Image processing device, distance measuring device, and image processing method

Abstract

To generate a distance image with an excellent S / N ratio at high resolution regardless of a beam diameter of an optical signal emitted from a light source.SOLUTION: An image processing device comprises: a light source which emits an optical signal at a prescribed time interval; a scan unit which can change at least one of a scanning range and a scanning timing of the optical signal for each frame; a light reception unit which receives a reflection light signal reflected after an object is irradiated with the optical signal; a distance image generation unit which generates a distance image for each frame on the basis of the reflection light signal received by the light reception unit; and an image composition unit which generates a high-resolution distance image by combining the distance images of the frames.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an image processing device, a distance measuring device, and an image processing method. [Background technology]

[0002] To improve the distance measurement accuracy of LiDAR (Light Detection and Ranging) devices used in autonomous driving, etc., it is desirable to narrow the beam diameter of the optical signal. However, narrowing the beam diameter requires increasing the lens aperture or the number of lenses, which increases the size of the LiDAR housing and the cost of parts.

[0003] It is possible to increase the resolution of the distance image by measuring distance by receiving optical signals with a pixel width smaller than the beam diameter; however, this reduces the amount of light received by each pixel, lowering the S / N ratio and decreasing the accuracy of distance measurement.

[0004] Furthermore, if distance measurement is performed in units of multiple pixels according to the size of the beam diameter, the S / N ratio increases, but the effective resolution decreases. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 110801 Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, one embodiment of the present invention provides an image processing device, a distance measuring device, and an image processing method that are capable of generating distance images with high resolution and excellent S / N ratio regardless of the beam diameter of the optical signal emitted from the light source. [Means for solving the problem]

[0007] In order to solve the above problems, according to one embodiment of the present invention, there is provided a light source that emits an optical signal at a predetermined time interval; a scanning unit capable of changing at least one of a scanning range and a scanning timing of the optical signal for each frame; a light receiving unit that receives a reflected light signal that is generated when the light signal is irradiated onto an object and reflected by the object; a distance image generating unit that generates a distance image for each frame based on a reflected light signal received by the light receiving unit; and an image synthesis unit that synthesizes distance images of the plurality of frames to generate a high-resolution distance image. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment. [Figure 2A] FIG. 13 is a diagram showing an example in which the beam width of the reflected optical signal is three times the size of the light receiving element. [Figure 2B] FIG. 13 is a diagram showing an example of receiving a reflected optical signal in units of three light receiving elements in accordance with the beam width of the reflected optical signal. [Figure 3A] FIG. 13 is a diagram showing an example in which the beam width of a reflected optical signal is the same as the width of a light receiving element. [Figure 3B] FIG. [Figure 3C] FIG. [Figure 4A] FIG. 13 is a diagram showing an example in which the beam width of a reflected light signal is a size of three pixels. [Figure 4B] FIG. [Figure 4C] FIG. [Diagram 5] 11A and 11B are diagrams for explaining why distance information includes error information; [Figure 6A] 4 is a diagram showing a light intensity distribution of an optical signal emitted from a light source. [Figure 6B] FIG. [Figure 6C] FIG. [Figure 6D]FIG. [Figure 7] FIG. 4 is a diagram showing a distance measurement sequence of a distance measuring device according to a comparative example. [Figure 8A] 8 is a diagram showing luminance information of a first frame detected by the distance measuring device of FIG. 7. [Figure 8B] 8 is a diagram showing luminance information of a second frame detected by the distance measuring device of FIG. 7. [Figure 9A] 8 is a diagram showing distance information of an object in the first frame detected by the distance measuring device of FIG. 7. [Figure 9B] 8 is a diagram showing distance information of an object in a second frame detected by the distance measuring device in FIG. 7. [Figure 10] 4 is a diagram showing a distance measurement sequence of the image processing device and the distance measuring device according to the embodiment. [Figure 11A] 2 is a diagram showing luminance information of a first frame detected by the distance measuring device of FIG. 1; [Figure 11B] 2 is a diagram showing luminance information of a second frame detected by the distance measuring device of FIG. 1; [Figure 11C] A diagram combining the luminance information of Figures 11A and 11B. [Figure 12A] 11 is a diagram showing distance information of an object in the first frame detected by the image processing device and distance measuring device of FIG. 10. [Figure 12B] 11 is a diagram showing distance information of an object in a second frame detected by the image processing device and distance measuring device of FIG. 10. [Figure 12C] FIG. 12C is a composite diagram of the distance information of FIGS. 12A and 12B. [Figure 13] FIG. 2 is a block diagram of an image processing apparatus according to a first specific example of FIG. 1; [Figure 14] FIG. 2 is a block diagram of an image processing device according to a second specific example of FIG. 1; [Figure 15] 10 is a diagram illustrating an example in which a scanning unit changes at least one of a scanning range and a scanning timing of an optical signal for each frame. [Figure 16] 4 is a flowchart showing a first example of the processing operation of the image processing device 1 and the distance measuring device 2 according to the first embodiment. [Figure 17A] FIG. 17 is a diagram showing a schematic diagram of a distance image generated in step S1 of FIG. 16. [Figure 17B] FIG. 17 is a diagram showing a schematic diagram of a distance image generated in step S2 of FIG. 16. [Figure 17C] FIG. 17 is a diagram showing a schematic diagram of a distance image generated in step S3 of FIG. 16. [Figure 18] 6 is a flowchart showing a second example of the processing operation of the image processing device and the distance measuring device according to the first embodiment. [Figure 19A] FIG. 20 is a diagram showing a schematic diagram of a distance image generated in step S11 of FIG. 18. [Figure 19B] FIG. 20 is a diagram showing a schematic diagram of a distance image generated in step S12 of FIG. 18. [Figure 19C] FIG. 20 is a diagram showing a schematic diagram of a distance image generated in step S13 of FIG. 18. [Figure 20] FIG. 11 is a block diagram showing a schematic configuration of an image processing apparatus according to a second embodiment. [Figure 21] FIG. 21 is a block diagram of an image processing device that further embodies the blur elimination processing unit in FIG. 20 . [Figure 22] 6A to 6C are diagrams for explaining the processing operation of a blur elimination processing unit. [Diagram 23] FIG. 11 is a block diagram showing a schematic configuration of an image processing apparatus according to a third embodiment. [Figure 24] FIG. 24 is a block diagram of an image processing device that further embodies the feature object recognition unit of FIG. 23 . [Diagram 25] 13 is a flowchart showing a first example of the processing operation of the image processing device and the distance measuring device according to the third embodiment. [Figure 26A] FIG. 26 is a diagram showing a schematic diagram of a distance image generated in step S21 of FIG. 25. [Figure 26B] FIG. 26 is a diagram showing a schematic diagram of a distance image generated in step S22 of FIG. 25. [Figure 26C] FIG. 26 is a diagram showing a schematic diagram of distance images of odd-numbered frames finally generated by the process of FIG. 25; [Figure 26D] FIG. 4 is a diagram illustrating distance images of even-numbered frames. [Figure 27] 13 is a flowchart showing a second example of the processing operation of the image processing device and the distance measuring device according to the third embodiment. [Figure 28A]FIG. 28 is a diagram showing a schematic diagram of a distance image generated in step S31 of FIG. 27. [Figure 28B] FIG. 28 is a diagram illustrating a feature object image generated in step S33 of FIG. 27. [Figure 28C] FIG. 28 is a diagram showing a schematic diagram of distance images of odd-numbered frames finally generated by the process of FIG. 27; [Figure 28D] FIG. 4 is a diagram illustrating distance images of even-numbered frames. [Figure 29] 13 is a flowchart showing a third example of the processing operation of the image processing device and the distance measuring device according to the third embodiment. [Figure 30A] FIG. 30 is a diagram showing a schematic diagram of a distance image generated in step S41 of FIG. 29. [Figure 30B] FIG. 30 is a diagram showing a schematic diagram of a distance image generated in step S42 of FIG. 29. [Figure 30C] FIG. 30 is a diagram showing a schematic diagram of a distance image generated in step S43 of FIG. 29. [Figure 30D] FIG. 30 is a diagram showing a schematic diagram of a distance image generated in step S44 of FIG. 29. [Figure 30E] FIG. 30 is a diagram showing a schematic diagram of a distance image finally generated by the image processing device and distance measuring device of FIG. 29. [Diagram 31] FIG. 13 is a diagram showing an example of a depression occurring on a road. [Diagram 32] FIG. 13 shows an example in which a depression on a road is recognized as a feature, and the optical signal is scanned with a 1 / 2 pixel shift in the second frame compared to the first frame. [Diagram 33] This figure shows an example in which a depression on a road is recognized as a feature, and the optical signal is scanned with a 1 / 3 pixel shift in the second frame compared to the first frame, and with a 2 / 3 pixel shift in the third frame compared to the first frame. [Diagram 34] This diagram shows an example in which a depression on the road is recognized as a feature, an optical signal is emitted with a 1 / 2 pixel shift for the area of ​​the depression on the road, the reflected optical signal from the depression is received to generate a feature image, and the feature images from the first and second frames are synthesized to generate a high-resolution feature image. [Diagram 35]FIG. 1 is a block diagram showing a hardware configuration of an image processing device and a distance measuring device according to the first to third embodiments. [Diagram 36] FIG. 2 is a block diagram showing a more specific hardware configuration of the image processing device and the distance measuring device according to the first to third embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of an image processing device, a distance measuring device, and an image processing method will be described with reference to the drawings. The following description will focus on the main components of the image processing device and the distance measuring device, but the image processing device and the distance measuring device may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0010] (First embodiment) Fig. 1 is a block diagram showing a schematic configuration of an image processing device 1 according to the first embodiment. The image processing device 1 in Fig. 1 is built into, for example, a distance measuring device 2. The hardware configurations of the image processing device 1 and the distance measuring device 2 will be described later.

[0011] Image processing device 1 in Fig. 1 includes a light source 3, a scanning unit 4, a light receiving unit 5, a distance image generating unit 6, and an image synthesis unit 7. The light source 3 and scanning unit 4 in Fig. 1 form a light projector 8, and the light receiving unit 5, the distance image generating unit 6, and the image synthesis unit 7 form a light receiver 9.

[0012] The light source 3 emits a plurality of optical signals at a predetermined time interval. More precisely, the optical signal emitted by the light source 3 is an optical pulse signal having a predetermined pulse width. The light source 3 emits an optical pulse signal consisting of, for example, a laser beam or an LED beam. The light source 3 may have a plurality of light-emitting elements, and each of the light-emitting elements may emit a plurality of optical signals at the same timing.

[0013] The scanning unit 4 can change at least one of the scanning range and scanning timing of the optical signal for each frame. The scanning unit 4 is configured using, for example, a polygon mirror or a MEMS (Micro Electro Mechanical System) mirror. The scanning unit 4 changes the traveling direction of the optical signal emitted from the light source 3 according to time, thereby scanning the optical signal in a one-dimensional direction or a two-dimensional direction. A frame is a still image acquired by scanning the FoV (Field of View) of the distance measuring device 2. The distance image generating unit 6, which will be described later, generates a distance image on a frame-by-frame basis.

[0014] The scanning unit 4 can change the scanning range of the optical signal in each frame. That is, at least a part of the scanning range of the optical signal may be different between each of a predetermined number of adjacent frames (two or more).

[0015] The scanning unit 4 can change the scanning timing of the optical signal in each frame. In this case, the scanning range of the optical signal in each frame may be the same or different. For example, the scanning unit 4 can change the scanning timing by changing the emission timing of the optical signal from the light source 3 for each frame.

[0016] The light receiving unit 5 receives a reflected light signal that is reflected when the light signal is irradiated on the object 13. The light receiving unit 5 has a plurality of light receiving elements arranged in a one-dimensional or two-dimensional direction. The light receiving signal received by each light receiving element is converted into, for example, a luminance signal, and finally a distance signal is generated. In this specification, the luminance signal or distance signal corresponding to the light receiving signal of each light receiving element may be called a pixel. In this manner, the distance image generated on a frame-by-frame basis contains a plurality of pixels, and each pixel corresponds to a different light receiving element. The light signal scanned by the scanning unit 4 is irradiated on the object 13, and the reflected light signal from the object 13 is received by one or more light receiving elements. The scanning unit 4 can change the scanning range of the light signal on a pixel-by-pixel basis for each frame.

[0017] The distance image generating unit 6 generates a distance image for each frame based on the reflected light signal received by the light receiving unit 5. The distance image generating unit 6 generates a distance image based on the time difference between the time when the light signal is emitted from the light source 3 and the time when the reflected light signal is received by the light receiving unit 5.

[0018] The image synthesis unit 7 synthesizes distance images of multiple frames to generate a high-resolution distance image. As described below, the distance image generation unit 6 can generate multiple distance images by varying at least one of the scanning range and scanning timing of the optical signal for each frame. Therefore, the image synthesis unit 7 can easily generate a high-resolution distance image by synthesizing these multiple distance images.

[0019] The beam width of the reflected light signal received by the light receiving unit 5 does not necessarily match the size of each light receiving element in the light receiving unit 5, and if the beam width is wider than the width of the light receiving element, this can result in a reduced S / N ratio, limited measurable distances, or reduced resolution of the distance image.

[0020] A distance measuring device 2 can be configured by adding a distance measuring unit 10 to the block configuration of the image processing device 1 in Fig. 1. The distance measuring unit 10 measures the distance to an object 13 based on the time when the reflected light signal is received by the light receiving unit 5 and the time when the light signal is emitted from the light source 3.

[0021] 2A and 2B are diagrams showing the relationship between the beam width of the reflected light signal received by the light receiving unit 5 and the size of the light receiving element 5a in the light receiving unit 5. The dashed frame 11 in Fig. 2A and 2B indicates the beam width, and the dashed-dotted frame 12 indicates the detection range of the object 13. Fig. 2A and 2B show an example in which the beam width has a size of three pixels, and the object 13 has a size of four pixels.

[0022] FIG. 2A shows an example in which the beam width of the reflected light signal is three times the size of the light receiving element 5a. In this case, the received light intensity S at each pixel of the distance image is 1 / 3 of the light intensity of the reflected light signal. If the ambient light noise N incident on each pixel is 1, the S / N ratio is 1 / 3. Therefore, the total signal intensity (S+N) of the received light intensity S and the ambient light noise N is 1.33. In the case of FIG. 2A, if the beam width is 3°, the pixel resolution is 1°.

[0023] Thus, in the case of FIG. 2A, the effective resolution of the distance image is the same as the pixel width, but the total signal strength is low, so the measurable distance is limited.

[0024] FIG. 2B shows an example in which a reflected light signal is received in units of three light receiving elements 5a in accordance with the beam width of the reflected light signal. In FIG. 2B, three light receiving elements 5a form one pixel, so the received light intensity S at each pixel is the same as the light intensity of the reflected light signal (S=1). The ambient light noise N incident on one pixel is N=3 because there are three light receiving elements 5a within one pixel. Therefore, the S / N ratio is 1 / 3. The total signal intensity (S+N) of the received light intensity S and the ambient light noise N is 4. In the case of FIG. 2B, if the beam width is 3°, the effective resolution is 3°.

[0025] In this way, in the case of FIG. 2B, the reflected light signal is received in units of three pixels in accordance with the beam width of the reflected light signal, so the measurable distance range can be expanded, but the effective resolution is degraded compared to the pixel resolution.

[0026] 2A and 2B, increasing the resolution of the distance image reduces the S / N ratio and narrows the measurable distance range. On the other hand, receiving reflected light signals from multiple light receiving elements 5a can widen the measurable distance range, but the resolution of the distance image decreases.

[0027] Furthermore, there is a problem that the larger the beam width of the reflected light signal, the more blurred the edges of the distance image become, resulting in a decrease in resolution.

[0028] Fig. 3A is a diagram showing an example in which the beam width of the reflected light signal is the same as the width of the light receiving element 5a. The dashed frame 11 in Fig. 3A indicates the beam width, and the dashed-dotted frame 12 indicates the detection range of the object 13. This shows an example in which the beam width has a size of one pixel, and the object 13 has a size of four pixels.

[0029] Fig. 3B is a diagram showing luminance information of object 13, and Fig. 3C is a diagram showing distance information of object 13. The horizontal axis of Fig. 3B and Fig. 3C is the pixel number corresponding to each light receiving element 5a, the vertical axis of Fig. 3B is the luminance value of object 13, and the vertical axis of Fig. 3C is the distance of object 13. In the case of Fig. 3A, since the beam width of the reflected light signal and the width of the light receiving element 5a match, both the luminance information and the distance information change sharply, making the edges clear.

[0030] Fig. 4A is a diagram showing an example in which the beam width of the reflected light signal is three pixels in size. A dashed frame 11 in Fig. 4A indicates the beam width, and a dashed-dotted frame 12 indicates the detection range of an object 13. This shows an example in which the beam width has a size of three pixels, and the object 13 has a size of four pixels.

[0031] Fig. 4B is a diagram showing luminance information of object 13, and Fig. 4C is a diagram showing distance information of object 13. The horizontal axis of Fig. 4B and Fig. 4C is the pixel number corresponding to each light receiving element 5a, the vertical axis of Fig. 4B is the luminance value of object 13, and the vertical axis of Fig. 4C is the distance of object 13. The area within the two dashed lines in Fig. 4B and Fig. 4C is the actual position of object 13, but information about object 13 is also detected outside these dashed lines. Object information outside the two dashed lines is error information.

[0032] In the case of Figure 4A, the beam width of the reflected light signal is three pixels, so the resolution is lower than the pixel resolution, and the edges of the brightness information become unclear. Therefore, if distance information is calculated based on this brightness information, it will contain error information, as shown in Figures 4B and 4C.

[0033] Fig. 5 is a diagram for explaining the reason why error information is included in distance measurement information. Like Fig. 4A, Fig. 5 shows an example in which the beam width of the reflected light signal has a size of three pixels and the object 13 has a size of four pixels.

[0034] 6A is a diagram showing the light intensity distribution of an optical signal emitted from light source 3, where the horizontal axis represents the pixel number in the horizontal (scanning) direction and the vertical axis represents the light intensity. The light intensity distribution in FIG. 6A is known.

[0035] 6B is a diagram showing the reflectance of object 13, where the horizontal axis is the pixel number in the horizontal (scanning) direction and the vertical axis is the reflectance. The reflectance of object 13 differs for each object 13. Also, the length of the horizontal axis differs depending on the size of object 13.

[0036] Fig. 6C is a diagram showing luminance information of object 13, and Fig. 6D is a diagram showing distance information of object 13. The horizontal axis of Fig. 6C and Fig. 6D is the pixel number corresponding to each light receiving element 5a, the vertical axis of Fig. 6C is the luminance value of object 13, and the vertical axis of Fig. 6D is the distance of object 13.

[0037] The luminance value of the object 13 is obtained by performing a convolution process of the light intensity g[n] of the optical signal and the reflectance f[n] of the object 13, as shown in the following equation (1). f[n]×g[n]=Σf[m]×g[nm] …(1)

[0038] If the beam width of the reflected light signal is wider than the width of the light receiving element 5a, the luminance distribution of the object 13 is convoluted in the scanning direction as shown in equation (1), so that even pixels outside the pixel corresponding to the actual detection position of the object 13 will detect a certain amount of luminance value, making the edge of the object 13 unclear.

[0039] Fig. 7 is a diagram showing a distance measurement sequence of a distance measuring device 2 according to a comparative example. Fig. 7 shows an example in which each frame has four pixels in the scanning direction. A dashed frame 11 in Fig. 7 indicates the beam width of a reflected light signal, and a dashed-dotted frame 12 indicates the detection range of an object 13.

[0040] Fig. 8A shows luminance information for the first frame detected by the distance measuring device 2 in Fig. 7, and Fig. 8B shows luminance information for the second frame detected by the distance measuring device 2 in Fig. 7. The horizontal axis of Fig. 8A and Fig. 8B is the pixel number, and the vertical axis is the luminance information of the reflected light signal. In each plot of Fig. 8A and Fig. 8B, the solid line indicates the light receiving position of the reflected light signal, and the dashed line indicates a case where an object is ideally detected.

[0041] As shown in Fig. 7, object 13 is not detected in the first pixel, object 13 is detected only in a part of the second pixel, object 13 is detected in the entire area of ​​the third pixel, and object 13 is not detected in the fourth pixel. Therefore, as shown in Fig. 8A and Fig. 8B, the luminance information is at an intermediate level in the second pixel, the luminance information is at the maximum level in the third pixel, and the luminance information is at the minimum level in the first pixel and the fourth pixel.

[0042] Fig. 9A is a diagram showing distance information for object 13 in the first frame detected by distance measuring device 2 in Fig. 7, and Fig. 9B is a diagram showing distance information for object 13 in the second frame detected by distance measuring device 2 in Fig. 7. The horizontal axis in Fig. 9A and Fig. 9B is the pixel number, and the vertical axis is the distance to object 13. In each plot in Fig. 9A and Fig. 9B, the solid line indicates the light receiving position of the reflected light signal, and the dashed line indicates a case where the object is ideally detected.

[0043] 8A and 8B and 9A and 9B, distance measurement is always performed with the same FoV, so the same distance and the same luminance information are always detected for pixel numbers 2 and 3. For this reason, the resolution is equivalent to 1°.

[0044] Fig. 10 is a diagram showing a distance measurement sequence of the image processing device 1 and distance measuring device 2 according to this embodiment. Like Fig. 7, Fig. 10 shows an example in which each frame has four pixels in the scanning direction. A dashed frame 11 in Fig. 10 indicates the beam width of the reflected light signal, and a dashed-dotted frame 12 indicates the detection range of an object 13.

[0045] Fig. 11A is a diagram showing luminance information of the first frame detected by the distance measuring device 2 of Fig. 1, Fig. 11B is a diagram showing luminance information of the second frame detected by the distance measuring device 2 of Fig. 1, and Fig. 11C is a diagram combining the luminance information of Fig. 11A and Fig. 11B. The horizontal axis of Fig. 11A to Fig. 11C is the pixel number, and the vertical axis is the luminance information of the reflected light signal. In each plot of Fig. 11A to Fig. 11C, the solid line indicates the light receiving position of the reflected light signal, and the dashed line indicates a case where an object is ideally detected.

[0046] The first frame in FIG. 10 is similar to the first frame in FIG. 7, and the luminance information of the reflected light signal in the first frame shown in FIG. 11A is the same as the luminance information in FIG. 8A.

[0047] In the second frame in Fig. 10, the light source 3 emits a light signal with a delay of 1 / 2 pixel from the first frame. Therefore, the luminance information of the reflected light signal in the second frame is detected with a delay of 1 / 2 pixel from Fig. 11A, as shown in Fig. 11B. When the scanning unit 4 scans with a light signal at a constant speed, if the light source 3 emits a light signal with a delay of 1 / 2 pixel, the light signal is emitted to a position shifted by 1 / 2 pixel on the object 13, and the reflected light signal is received at a position shifted by 1 / 2 pixel on the light receiving unit 5.

[0048] By combining the luminance information of the first and second frames of the reflected light signal, the luminance information shown in Fig. 11C is obtained. The luminance information in Fig. 11C has twice the resolution of the luminance information in Fig. 11A and Fig. 11B, and can be detected more accurately.

[0049] Fig. 12A is a diagram showing distance information of object 13 in the first frame detected by image processing device 1 and distance measuring device 2 in Fig. 10, Fig. 12B is a diagram showing distance information of object 13 in the second frame detected by image processing device 1 and distance measuring device 2 in Fig. 10, and Fig. 12C is a diagram combining the distance information of Fig. 12A and Fig. 12B. Each plot in Fig. 12A to Fig. 12C indicates the detection position of the distance information.

[0050] In the second frame, a light signal is emitted from the light source 3 with a delay of 1 / 2 pixel from the first frame to detect distance information of the object 13, so the distance information of the object 13 is detected with a delay of 1 / 2 pixel from the first frame.

[0051] By combining the distance information of the first and second frames, as shown in FIG. 12C, the distance near the edge of the object 13 can be detected accurately, and the distance resolution can be improved.

[0052] 1 is characterized in that at least one of the scanning range and the scanning timing of the optical signal can be changed for each frame. There are several possible specific examples of the method in which the scanning unit 4 scans the optical signal.

[0053] Fig. 13 is a block diagram of the image processing device 1 according to the first specific example of Fig. 1. In Fig. 13, components common to Fig. 1 are given the same reference numerals, and the following mainly describes the differences. The light source 3 and scanning unit 4 in Fig. 13 constitute a projector 8. The light receiving unit 5, distance measuring unit 10, distance image generating unit 6, and image synthesis unit 7 in Fig. 13 constitute a light receiving unit 9.

[0054] The image processing device 1 of FIG. 13 includes a scanning control unit 15 in addition to the configuration of FIG. 1. The scanning control unit 15 controls the scanning unit 4. For example, the scanning control unit 15 controls switching of the light emission direction of the scanning unit 4. As described later, the scanning unit 4 has a mirror member, such as a polygon mirror or a MEMS mirror, that switches the traveling direction of the optical signal emitted from the light source 3. The scanning control unit 15 not only controls the rotation of the mirror surface of the mirror member, but may also vary the direction of the optical signal emitted from the scanning unit 4 for each frame by rotating the mirror member itself within a predetermined angle range.

[0055] Alternatively, the scanning control unit 15 may control switching of the light emission direction of the housing unit incorporating the light source 3. This makes it possible to vary the traveling direction of the optical signal emitted from the light source 3, and to change the scanning range of the optical signal emitted from the scanning unit 4 for each frame.

[0056] Alternatively, the scan control unit 15 may variably control the speed at which the scanning unit 4 switches the traveling direction of the optical signal. That is, the scan control unit 15 can variably control the resolution of the distance image by controlling the switching of the scanning speed of the optical signal by the scanning unit 4.

[0057] Fig. 14 is a block diagram of the image processing device 1 according to the second specific example of Fig. 1. The image processing device 1 of Fig. 14 includes a timing control section 16 in addition to the configuration of Fig. 13.

[0058] The timing control unit 16 controls the emission start timing at which the light source 3 starts emitting an optical signal. The scanning unit 4 switches the scanning timing of the optical signal for each frame based on the emission start timing of the optical signal by the light source 3. The timing control unit 16 variably controls the emission start timing of the optical signal for each frame without changing the emission interval of the optical signal emitted from the light source 3.

[0059] More specifically, the light source 3 emits a plurality of optical signals at the same emission intervals, starting from different emission start times, in each of n consecutive frames (n is an integer equal to or greater than 2). The image synthesis unit 7 synthesizes the distance images generated in each of the n frames.

[0060] Fig. 15 is a diagram showing an example in which the scanning unit 4 changes at least one of the scanning range and scanning timing of the optical signal for each frame. In frame F1 of Fig. 15, the scanning unit 4 scans the optical signal in the horizontal direction, and when scanning of one horizontal line is completed, the scanning unit 4 scans the optical signal for the next horizontal line from left to right, repeating this operation for all horizontal lines. Each square in frame F1 indicates a pixel that is received by each light receiving element 5a.

[0061] After the distance image of frame F1 has been generated, in the next frame F2, an optical signal is emitted from the light source 3 with a delay of 1 / 3 pixel, and the reflected optical signal from the object 13 is received by the light-receiving unit 5 to generate a distance image. In the next frame F3, an optical signal is emitted from the light source 3 with a delay of 2 / 3 pixel, and the reflected optical signal from the object 13 is received by the light-receiving unit 5 to generate a distance image. In the next frame F4, an optical signal is emitted from the light source 3 with the same timing as frame F1, and a distance image is generated.

[0062] The image synthesis unit 7 synthesizes the three distance images of frames F1 to F3 to generate a high-resolution distance image. Although an example in which the three distance images of frames F1 to F3 are synthesized is shown in Fig. 15, the number of distance images to be synthesized is arbitrary.

[0063] Fig. 16 is a flowchart showing a first example of the processing operation of the image processing device 1 and distance measuring device 2 according to the first embodiment. First, in the first frame, a distance image G1 is generated by scanning with an optical signal without shifting the scanning range (step S1). Fig. 17A is a diagram showing a schematic diagram of the distance image G1 generated in step S1 of Fig. 16. Fig. 17A shows an example of a distance image G1 with six pixels in the horizontal direction and four pixels in the vertical direction.

[0064] Next, in the second frame, the scanning range is shifted (delayed) in the horizontal direction by 1 / 2 pixel to generate distance image G2 (step S2). Fig. 17B is a diagram showing a schematic diagram of distance image G2 generated in step S2 of Fig. 16.

[0065] Next, the distance image G1 of the first frame and the distance image G2 of the second frame are combined to generate a high-resolution distance image G3 (step S3). Figure 17C is a diagram showing a schematic of the distance image G3 generated in step S3 of Figure 16. As shown in Figure 17C, by combining the distance images G1 and G2 of two frames, a high-resolution distance image G3 with twice the horizontal resolution can be generated.

[0066] Fig. 18 is a flowchart showing a second example of the processing operation of the image processing device 1 and the distance measuring device 2 according to the first embodiment. First, in the first frame, a distance image is generated by scanning the optical signal in a reference scanning range (step S11). Fig. 19A is a diagram showing a schematic diagram of the distance image G1 generated in step S11 of Fig. 18.

[0067] Next, in the second frame, the scanning range is shifted (delayed) in the vertical direction by 1 / 2 pixel, and a distance image is generated (step S12). Fig. 19B is a diagram showing a schematic diagram of distance image G2 generated in step S12 of Fig. 18.

[0068] Next, the distance image of the first frame and the distance image of the second frame are synthesized to generate a high-resolution distance image G3 (step S13). Figure 19C is a schematic diagram showing the distance image generated in step S13 of Figure 18. As shown in Figure 19C, by synthesizing the distance images of two frames, the vertical resolution can be doubled.

[0069] In this way, in the first embodiment, a high-resolution distance image is generated by combining multiple distance images generated by changing at least one of the scanning range and scanning timing of the optical signal for each frame. This makes it possible to increase the resolution of the distance image with a simple configuration without providing a lens or the like for reducing the beam width of the optical signal emitted from the light source 3.

[0070] Second embodiment 20 is a block diagram showing a schematic configuration of an image processing device 1 according to the second embodiment. The image processing device 1 in FIG. 20 is obtained by adding a blur removal processing unit 17 to the configuration in FIG.

[0071] The blur removal processor 17 performs processing to make the high-resolution distance image generated by the image synthesis unit 7 clearer. Specifically, the blur removal processor 17 removes blur from the edge portions of the distance image and emphasizes the edges.

[0072] Fig. 21 is a block diagram of the image processing device 1 which more specifically embodies the blur elimination processing unit 17 of Fig. 20. The blur elimination processing unit 17 of Fig. 21 has a deconvolution processing unit 18 and an edge enhancement unit 19.

[0073] The deconvolution processing unit 18 performs deconvolution processing on the high-resolution range image generated by the image synthesis unit 7 using the light intensity distribution of the optical signal emitted from the light source 3 to generate the reflectance of the object 13.

[0074] The edge enhancement unit 19 enhances the edges of the high-resolution range image generated by the image synthesis unit 7 based on the reflectance of the object 13 generated by the deconvolution processing unit 18 .

[0075] 22 is a diagram illustrating the processing operation of blur elimination processor 17. Distance image generator 6 generates luminance information and distance information based on the light reception signal received by light receiver 5. The luminance information generated by distance image generator 6 is represented, for example, by waveform w1, and the distance information is represented, for example, by waveform w2. As can be seen from waveforms w1 and w2, when the beam width of the reflected light signal is wider than the width of light receiving element 5a, the luminance information and distance information will have values ​​that include errors.

[0076] The light intensity g[n] of the optical signal emitted from the light source 3 is known. Therefore, the deconvolution processing unit 18 can generate the reflectance of the object 13 by performing a deconvolution process of g[n] on the luminance information. The reflectance of the object 13 is represented, for example, by a waveform w3 in FIG. 22. The reflectance of the object 13 generated by the deconvolution processing unit 18 is information that does not include errors. The light intensity g[n] can be obtained by measuring the light intensity in advance. Alternatively, it can be obtained by acquiring the intensity distribution of the laser light each time distance measurement is performed using a light receiving element provided in the projector.

[0077] The edge enhancement unit 19 corrects the luminance information generated by the distance image generation unit 6 based on the reflectance of the object 13 generated by the deconvolution processing unit 18, and also corrects the distance information. This makes it possible to obtain luminance information and distance information with clear edges, as shown by waveforms w4 and w5.

[0078] In Fig. 22, deconvolution processing is performed on the luminance information contained in the distance image generated by the distance image generation unit 6, but it is also possible to perform deconvolution processing on the luminance information contained in the high-resolution distance image generated by the image synthesis unit 7 as in Fig. 21. In this case, the edge enhancement unit 19 sets a threshold value for the luminance signal corresponding to the reflected light signal received by the light receiving unit 5 based on the reflectance of the object 13 generated by the deconvolution processing unit 18, binarizes the luminance signal based on the threshold value, and corrects the high-resolution distance image generated by the image synthesis unit 7 based on the binarized luminance value. Alternatively, the edge enhancement unit 19 may estimate a luminance value corresponding to the reflected light signal received by the light receiving unit 5 based on the reflectance of the object 13 generated by the deconvolution processing unit 18, and correct the high-resolution distance image generated by the image synthesis unit 7 based on the luminance value.

[0079] As described above, in the second embodiment, the reflectance of the object 13 is generated by performing deconvolution processing of the light intensity of the optical signal on the luminance information based on the light reception signal received by the light receiving unit 5, and the luminance information and distance information are corrected based on the generated reflectance of the object 13. This makes it possible to eliminate blurring in the high-resolution distance image generated by the image synthesis unit 7 and emphasize edges.

[0080] (Third embodiment) 23 is a block diagram showing a schematic configuration of an image processing device 1 according to a third embodiment. The image processing device 1 in FIG. 23 includes a feature object recognition unit 20 in addition to the configuration in FIG.

[0081] The feature recognition unit 20 recognizes feature objects included in the distance image generated by the distance image generation unit 6. A feature object is, for example, a stationary object that satisfies a predetermined condition. The predetermined condition is, for example, a stationary object that did not exist before a predetermined period of time. Specific examples include a depression in the road and an obstacle placed on the road.

[0082] The image synthesis unit 7 generates a high-resolution distance image of the feature object at a first frame rate, and generates a distance image of the object other than the feature object at a second frame rate higher than the first frame rate without changing the resolution. As described later, the feature object recognition unit 20 may recognize a moving object. The image synthesis unit 7 generates a distance image of the moving object recognized by the feature object recognition unit 20 at the second frame rate without changing the resolution.

[0083] Fig. 24 is a block diagram of the image processing device 1 which further embodies the feature object recognition unit 20 of Fig. 23. The feature object recognition unit 20 of Fig. 24 has a frame memory 21 and an inter-frame difference detection unit 22.

[0084] The frame memory 21 stores at least one frame of distance images. The frame difference detection unit 22 detects differences between distance images of a plurality of frames and recognizes features.

[0085] More specifically, inter-frame difference detection unit 22 may have a first recognition unit and a second recognition unit. The first recognition unit recognizes stationary features that did not exist prior to a predetermined period of time by taking the difference between a distance image generated by distance image generation unit 6 and a distance image generated prior to a predetermined period of time. The second recognition unit recognizes moving object 13 by taking the difference between a plurality of distance images most recently generated by distance image generation unit 6.

[0086] For example, based on the recognition result by the first recognition unit, the image synthesis unit 7 generates a high-resolution distance image at a first frame rate in an area including a feature, and generates a distance image at a second frame rate lower than the first frame rate in an area not including the feature. Alternatively, the image synthesis unit 7 generates a high-resolution distance image at the first frame rate in an area including a feature recognized by the first recognition unit, and generates a distance image at the second frame rate in an area including a moving object recognized by the second recognition unit.

[0087] Fig. 25 is a flowchart showing a first example of the processing operation of the image processing device 1 and the distance measuring device 2 according to the third embodiment. First, in the first frame, a distance image is generated by scanning an optical signal within a predetermined reference scanning range (step S21). Fig. 26A is a diagram showing a schematic diagram of the distance image G1 generated in step S21 of Fig. 25.

[0088] Next, in the second frame, the scanning range is shifted (delayed) in the horizontal direction by half a pixel to generate distance image G2 (step S22). Fig. 26B is a diagram showing a schematic diagram of distance image G2 generated in step S22 of Fig. 25.

[0089] Next, the feature recognition unit 20 recognizes the feature and identifies the position of the feature contained in the distance image (step S23). The identified feature is, for example, a stationary object. The feature is recognized in each of the distance images G1 and G2. Next, the image synthesis unit 7 synthesizes the feature images PG of the first and second frames for the feature to generate a high-resolution feature image PG (step S24). Furthermore, for objects other than the feature, the image synthesis unit 7 generates a distance image at a normal frame rate without image synthesis (step S25).

[0090] Fig. 26C is a schematic diagram showing a distance image G3 of an odd-numbered frame finally generated by the processing of Fig. 25, and Fig. 26D is a schematic diagram showing a distance image G4 of an even-numbered frame. As shown in Figs. 26C and 26D, a feature image PG is generated at a high resolution and a low frame rate only in the region of the feature included in the distance image, and distance images G3 and G4 are generated at a low resolution and a high frame rate for the region other than the feature.

[0091] Fig. 27 is a flowchart showing a second example of the processing operation of the image processing device 1 and the distance measuring device 2 according to the third embodiment. First, in the first frame, a distance image is generated by scanning an optical signal within a predetermined reference scanning range (step S31). Fig. 28A is a diagram showing a schematic diagram of the distance image G1 generated in step S31 of Fig. 27.

[0092] Next, the position of a feature included in the distance image is identified (step S32). The identified feature is, for example, a stationary object. Next, for the feature, a light signal is emitted from the light source 3 with a 1 / 2 pixel shift (delay) in the second frame, and a reflected light signal from the object 13 is received to generate a feature image PG (step S33). FIG. 28B is a diagram showing a schematic diagram of the feature image PG generated in step S33 of FIG. 27. Next, the feature image PG of the first frame and the feature image PG of the second frame are synthesized to generate a high-resolution feature image PG (step S34).

[0093] On the other hand, for objects other than the features, in the second frame, a light signal is emitted from the light source 3 in the same scanning range or scanning timing as the first frame, and a reflected light signal from the object 13 is received, and a distance image is generated for each frame (step S35).

[0094] Fig. 28C is a schematic diagram showing a distance image G3 of an odd-numbered frame finally generated by the processing of Fig. 27, and Fig. 28D is a schematic diagram showing a distance image G4 of an even-numbered frame. As shown in these figures, a feature image PG with high resolution and a low frame rate is generated in the feature region, and distance images G3 and G4 with high frame rates are generated in the non-feature region.

[0095] Fig. 29 is a flowchart showing a third example of the processing operation of the image processing device 1 and the distance measuring device 2 according to the third embodiment. First, in the first frame, a distance image is generated by scanning an optical signal within a predetermined reference scanning range (step S41). Fig. 30A is a diagram showing a schematic diagram of the distance image G1 generated in step S41 of Fig. 29.

[0096] Next, in the second frame, the scanning range is shifted (delayed) in the horizontal direction by 1 / 2 pixel to generate distance image G2 (step S42). Fig. 30B is a schematic diagram showing distance image G2 generated in step S42 of Fig. 29.

[0097] Next, in the third frame, the scanning range is shifted (delayed) in the vertical direction by 1 / 2 pixel to generate distance image G3 (step S43). Figure 30C is a schematic diagram showing distance image G3 generated in step S43 of Figure 29.

[0098] Next, in the fourth frame, the scanning range is shifted (delayed) by 1 / 2 pixel in the horizontal and vertical directions to generate distance image G4 (step S44). Fig. 30D is a schematic diagram showing distance image G4 generated in step S44 of Fig. 29.

[0099] Next, the positions of the features included in the distance image are identified (step S45). For the features, the feature images PG of the first to fourth frames are synthesized to generate a high-resolution feature image PG. For the non-feature objects, a distance image G5 is generated for each frame (step S46).

[0100] Fig. 30E is a diagram showing a schematic diagram of a distance image G5 finally generated by the image processing device 1 and distance measuring device 2 in Fig. 29. A feature image PG with high resolution and a low frame rate is generated for the feature region, and a distance image G5 with a high frame rate is generated for the region other than the feature.

[0101] 31 to 34 are schematic diagrams showing how infrastructure is monitored using an image processing device 1 and a distance measuring device 2 according to the third embodiment. More specifically, FIGS. 31 to 34 show how it is monitored to see if a depression 34 has occurred on a road. The image processing device 1 and the distance measuring device 2 generate a distance image by scanning an optical signal emitted from a light source 3 with a scanning unit 4. Alternatively, a feature may be recognized first as in FIG. 27, and only the feature may be shifted by 1 / 2 pixel for each frame.

[0102] Fig. 31 shows an example in which a depression 34 has occurred on a road. Before the depression 34 on the road is recognized as a feature, as shown in Fig. 31, the scanning range and scanning timing of the optical signal by the scanning unit 4 are the same in the first and second frames.

[0103] Figure 32 shows an example in which a depression 34 on a road is recognized as a feature, and the optical signal is scanned with a 1 / 2 pixel shift in the second frame from the first frame. The image synthesis unit 7 generates a high-resolution distance image by synthesizing the distance images of the first and second frames. Figure 32 shows an example in which a high-resolution distance image is generated over the entire frame.

[0104] In Fig. 33, a depression 34 on a road is recognized as a feature, and in the second frame, the optical signal is scanned with a shift of 1 / 3 pixel from the first frame, and in the third frame, the optical signal is scanned with a shift of 2 / 3 pixel from the first frame. The image synthesis unit 7 synthesizes the feature images PG of the first to third frames to generate a feature image PG with higher resolution than that of Fig. 32. Fig. 33 also shows an example of generating a high-resolution distance image over the entire frame.

[0105] In Fig. 34, a depression 34 on the road is recognized as a feature, and a light signal is emitted with a half pixel shift for the area of ​​the depression 34 on the road, and a reflected light signal from the depression 34 is received to generate a feature image PG, and the feature images PG of the first and second frames are synthesized to generate a high-resolution feature image PG. Meanwhile, in areas other than the feature, moving vehicles exist, so distance images are generated at a high frame rate for each frame. The above-mentioned second recognition unit 24 may recognize moving objects and generate distance images at a high frame rate.

[0106] In this way, the feature recognition unit 20 according to the third embodiment can recognize a feature that did not exist before a predetermined period and is stationary by taking the difference between a newly generated distance image and a distance image generated before a predetermined period. Also, the feature recognition unit 20 can recognize a moving object 13 by taking the difference between a plurality of distance images generated most recently. The image synthesis unit 7 can generate a high-resolution feature image PG for a feature that did not exist before a predetermined period and is stationary by synthesizing feature images PG in a plurality of frames. Also, for a moving object, a distance image is generated for each frame, so that the moving object can be tracked with high accuracy.

[0107] (Fourth embodiment) At least a part of the image processing device 1 and the distance measuring device 2 according to the above-described first to third embodiments may be configured with hardware or software.

[0108] 35 is a block diagram showing an example in which the image processing device 1 and the distance measuring device 2 according to the first to third embodiments are configured as hardware. The image processing device 1 and the distance measuring device 2 in FIG. 35 include a light source 3, a scanning device 31, a light receiving device 32, and a signal processor 33.

[0109] The light source 3 may be a laser light source or an LED light source. The light source 3 may have a plurality of laser light sources or a plurality of LED light sources.

[0110] The scanning device 31 corresponds to the scanning unit 4 according to the first to third embodiments. The scanning device 31 is, for example, a polygon mirror or a MEMS mirror, and has a function of scanning an optical signal in a predetermined scanning range.

[0111] The light receiving device 32 corresponds to the light receiving section 5 according to the first to third embodiments. The light receiving device 32 may have a plurality of light receiving elements 5a.

[0112] The signal processor 33 may be a semiconductor chip such as a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or a DSP (Digital Signal Processor), or may be a computer device such as a PC (Personal Computer), a workstation, or a server. The signal processor 33 has the functions of the scan control unit 15 and the timing control unit 16 according to the first to third embodiments. The signal processor 33 may also have the function of the distance measurement unit 10.

[0113] 36 is a block diagram showing a more specific hardware configuration of the image processing device 1 and the distance measuring device 2 according to the first to third embodiments. In FIG. 36, a polygon mirror 31a is used as an example of the scanning device 31.

[0114] The image processing device 1 and the distance measuring device 2 in Fig. 36 employ a SIP (Silicon in Package) in which a light receiving device 32 and a signal processor 33 are built in the same package. In Fig. 36, a first die 42 and a second die 43 are provided on a support substrate 41.

[0115] A plurality of light receiving devices 32 are arranged in a two-dimensional direction on the first die 42. Each light receiving device 32 has a silicon photomultiplier (SiPM) 44 and an active quench circuit (AQs) 45. Each SiPM 44 has one or more avalanche photo diodes (APDs). Note that a passive quench circuit may be provided instead of the active quench circuit 45.

[0116] On the second die 43, there are provided a plurality of A / D converters (hereinafter, ADCs) 46 that convert light reception signals received by each SiPM 44 into digital pixel data, and a signal processor 33. A pad 47 on the first die 42 and a corresponding pad 48 on the second die 43 are connected by a bonding wire 49.

[0117] In the image processing device 1 and distance measuring device 2 in FIG. 36, components other than the light source 3 and the scanning device 31 can be configured on a single semiconductor chip, making it possible to achieve miniaturization and low power consumption.

[0118] The aspects of the present disclosure are not limited to the above-mentioned individual embodiments, but include various modifications that may be conceived by a person skilled in the art, and the effects of the present disclosure are not limited to the above-mentioned contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure derived from the contents defined in the claims and their equivalents. [Explanation of symbols]

[0119] 1 image processing device, 2 distance measuring device, 3 light source, 4 scanning unit, 5 light receiving unit, 5a light receiving element, 6 distance image generating unit, 7 image synthesis unit, 8 light projector, 9 light receiver, 10 distance measuring unit, 11 dashed frame, 12 dashed-dotted frame, 13 object, 15 scan control unit, 16 timing control unit, 17 blur elimination processing unit, 18 processing unit, 19 edge enhancement unit, 20 feature recognition unit, 21 frame memory, 22 frame difference detection unit, 24 second recognition unit, 31 scan device, 31a polygon mirror, 32 light receiving device, 33 signal processing processor, 34 depression, 41 support substrate, 42 first die, 43 second die, 45 active quench circuit (AQs), 46 converter (hereinafter, ADC), 47 pad, 48 pad, 49 bonding wire

Claims

1. a light source that emits an optical signal at a predetermined time interval; a scanning unit capable of changing at least one of a scanning range and a scanning timing of the optical signal for each frame; a light receiving unit that receives a reflected light signal that is generated when the light signal is irradiated onto an object and reflected by the object; a distance image generating unit that generates a distance image for each frame based on a reflected light signal received by the light receiving unit; an image synthesis unit that synthesizes distance images of a plurality of frames generated by shifting pixel positions of the light receiving unit that receives the reflected light signal by less than one pixel for each frame, to generate a distance image with a higher resolution than each of the distance images of the plurality of frames; a blur removal processing unit that performs processing to remove blur from the high-resolution distance image generated by the image synthesis unit; a measurement unit that measures a light intensity distribution of an optical signal emitted from the light source, The blur removal processing unit is a deconvolution processing unit that performs a deconvolution process on the luminance information of the high-resolution distance image generated by the image synthesis unit using the light intensity distribution of the optical signal emitted from the light source measured by the measurement unit to generate a reflectance of the object; and an edge enhancement unit that enhances edges of the high-resolution distance image generated by the image synthesis unit based on the reflectance of the object generated by the deconvolution processing unit.

2. The deconvolution processing unit performs a deconvolution process using a known light intensity distribution of the optical signal emitted from the light source to generate a reflectance of the object. The image processing device according to claim 1 .

3. the scanning unit makes at least one of a scanning range and a scanning timing of the optical signal different in each of the predetermined number of consecutive frames, which are two or more, and repeatedly and periodically switches at least one of the scanning range and the scanning timing of the optical signal in units of the predetermined number of frames; The image processing device according to claim 1 , wherein the image synthesis unit synthesizes distance images of the predetermined number of frames to generate the high-resolution distance image.

4. The image processing device according to claim 1 , wherein the scanning unit changes a scanning range of the optical signal emitted from the light source for each frame.

5. The image processing device according to claim 1 , further comprising a scanning control unit that switches and controls a light emission direction of the scanning unit for each frame.

6. a scanning control unit that switches and controls a light emission direction of a housing unit having the light source built therein for each frame; The image processing device according to claim 1 , wherein the scanning section changes a scanning range of the optical signal for each frame by controlling switching of a light emission direction of the housing section using the scanning control section.

7. The image processing device according to claim 1 , further comprising a scanning control unit that variably controls a speed at which the scanning unit switches the traveling direction of the optical signal for each frame.

8. a timing control unit that controls a timing at which the light source starts emitting the optical signal, The image processing device according to claim 1 , wherein the scanning unit switches a scanning timing of the optical signal for each frame based on a timing at which the light source starts emitting the optical signal.

9. The image processing device according to claim 8 , wherein the timing control unit variably controls an emission start timing of the optical signal for each frame without changing an emission interval of the optical signal emitted from the light source.

10. the light source emits a plurality of the optical signals at the same emission intervals, with different emission start times as starting points, in each of the consecutive n (n is an integer equal to or greater than 2) frames; The image processing device according to claim 9 , wherein the image synthesis unit synthesizes distance images generated in the n frames.

11. 11. An image processing device as claimed in any one of claims 1 to 10, wherein the edge enhancement unit sets a threshold value for a luminance signal corresponding to the reflected light signal received by the light receiving unit based on the reflectance of the object generated by the deconvolution processing unit, binarizes the luminance signal based on the threshold value, and corrects the high-resolution distance image generated by the image synthesis unit based on the binarized luminance value.

12. 11. The image processing device according to claim 1, wherein the edge enhancement unit estimates a luminance value corresponding to the reflected light signal received by the light receiving unit based on the reflectance of the object generated by the deconvolution processing unit, and corrects the high-resolution distance image generated by the image synthesis unit based on the luminance value.

13. a feature recognition unit that recognizes feature objects included in the distance image generated by the distance image generation unit, 13. The image processing device according to claim 1, wherein the image synthesis unit outputs the high-resolution distance image generated at a first frame rate for the feature, and outputs a distance image at a second frame rate higher than the first frame rate without changing the resolution for the object other than the feature.

14. The image processing device according to claim 13 , wherein the feature is a stationary object that satisfies a predetermined condition.

15. The image processing device described in claim 13 or 14, wherein the feature recognition unit has a first recognition unit that recognizes the feature that did not exist prior to the specified period and is stationary by taking the difference between the distance image generated by the distance image generation unit and a distance image generated prior to the specified period.

16. The image processing device according to claim 15 , wherein the feature object recognition unit has a second recognition unit that recognizes the moving object by taking a difference between a plurality of distance images most recently generated by the distance image generation unit.

17. 17. The image processing device according to claim 16, wherein the image synthesis unit outputs the high-resolution distance image generated at the first frame rate in a partial region including the feature, and outputs the distance image generated at the second frame rate in a partial region including the moving object, based on the recognition results by the first recognition unit and the second recognition unit.

18. An image processing device according to any one of claims 1 to 17; a distance measuring unit that measures a distance to the object based on the time when the reflected light signal is received by the light receiving unit and the time when the light signal is emitted from the light source.

19. At least one of a scanning range and a scanning timing of an optical signal emitted at a predetermined time interval from a light source that emits an optical signal at a predetermined time interval is varied for each frame; receiving a reflected optical signal that is reflected by an object after the optical signal is irradiated onto the object; generating a distance image for each of the frames based on the received reflected light signal; synthesizing distance images of the plurality of frames, the pixel positions of which are shifted by less than one pixel, to generate a distance image having a higher resolution than each of the distance images of the plurality of frames; performing a process of eliminating blurring of the generated high-resolution distance image; measuring a light intensity distribution of an optical signal emitted from the light source; In the process of eliminating blur, performing a deconvolution process on the luminance information of the generated high-resolution distance image using the measured light intensity distribution to generate a reflectance of the object; 20. A method of image processing comprising: enhancing edges of the generated high resolution range image based on the generated reflectance of the object.

Images