Electromagnetic wave detection device and control device

Abstract

To provide an electromagnetic wave detection device and a control device that can improve the accuracy of object recognition, self-position estimation, or the like.SOLUTION: An electromagnetic wave detection device (10) includes an irradiation unit (12) which outputs an electromagnetic wave, a changing unit (13) which outputs the electromagnetic wave to an irradiation area while changing an irradiation direction of the electromagnetic wave to a plurality of different directions, a detection unit (20) which detects a reflected wave of the electromagnetic wave reflected against an object existing in the irradiation area, an imaging unit (17) which acquires an image including the area, and a control unit (14) for controlling the irradiation unit and the changing unit to change the irradiation direction of the electromagnetic wave and perform irradiation multiple times at a predetermined period. The control unit interpolates distance information based on detection results of the detection unit at the predetermined period using an image acquired by the imaging unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an electromagnetic wave detection device and a control device. [Background technology]

[0002] In recent years, devices have been developed that detect electromagnetic waves and obtain information about the surroundings from the detection results. For example, Patent Document 1 discloses a technology that uses LiDAR (Light Detection and Ranging) to detect obstacles, such as fallen objects on the road, as targets. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-56163 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, it is desirable for a traveling vehicle to have higher accuracy in object recognition, self-position estimation, and the like.

[0005] In view of the above circumstances, an object of the present disclosure is to provide an electromagnetic wave detection device and a control device that can improve the accuracy of object recognition, self-position estimation, and the like. [Means for solving the problem]

[0006] (1) An electromagnetic wave detection device according to an embodiment of the present disclosure, an irradiation unit that outputs electromagnetic waves; a change unit that changes the irradiation direction of the electromagnetic wave to a plurality of different directions and outputs the electromagnetic wave to an irradiation area; a detection unit that detects a reflected wave of the electromagnetic wave reflected by an object present in the irradiation area; an imaging unit that acquires an image including the illumination area; a control unit that controls the irradiation unit and the change unit to change the irradiation direction of the electromagnetic wave and irradiate it multiple times in a predetermined period, The control unit interpolates distance information based on the detection result of the detection unit in the predetermined cycle using the image acquired by the imaging unit.

[0007] (2) As one embodiment of the present disclosure, in (1), The control unit projects three-dimensional information based on the distance information onto the image, and interpolates distance information for a portion of the image where the three-dimensional information is not projected.

[0008] (3) As an embodiment of the present disclosure, in (1) or (2), The control unit interpolates the distance information based on first distance information indicating the distance to a first region included in the distance information and characteristics of an image region of the image corresponding to the first region.

[0009] (4) As an embodiment of the present disclosure, in (3), When a peripheral area of ​​the image having the same characteristics exists around an image area of ​​the image corresponding to the first area, the control unit interpolates second distance information indicating the distance to the peripheral area using the first distance information.

[0010] (5) As an embodiment of the present disclosure, in (3) or (4), The feature may be brightness or color.

[0011] (6) As an embodiment of the present disclosure, An electromagnetic wave detection device according to (1) that is mounted on a moving body, the control unit corrects the distance information in accordance with a traveling state of the moving object, and interpolates the corrected distance information using the image acquired by the imaging unit. Electromagnetic wave detection device.

[0012] (7) As an embodiment of the present disclosure, in (6), The control unit identifies the traveling state based on a difference between the shape of an object based on the plurality of pieces of distance information acquired at the predetermined period and the shape of the object included in the image.

[0013] (8) A control device according to an embodiment of the present disclosure includes: A control device for controlling an electromagnetic wave detection unit including an irradiation unit that outputs electromagnetic waves, a change unit that changes the irradiation direction of the electromagnetic waves to a plurality of different directions and outputs the electromagnetic waves to an irradiation area, a detection unit that detects reflected waves of the electromagnetic waves reflected by an object present in the irradiation area, and an imaging unit that acquires an image including the irradiation area, a control unit that controls the irradiation unit and the change unit to change the irradiation direction of the electromagnetic wave and irradiate it multiple times in a predetermined period, The control unit interpolates distance information based on the detection result of the detection unit in the predetermined cycle using the image acquired by the imaging unit. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to provide an electromagnetic wave detection device and a control device that can improve the accuracy of object recognition, self-position estimation, and the like. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an electromagnetic wave detection device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram for explaining the traveling directions of electromagnetic waves in the first state and the second state of the electromagnetic wave detection device of FIG. [Figure 3] FIG. 3 is a diagram for explaining the detection of electromagnetic waves including reflected waves. [Figure 4] FIG. 4 is a timing chart for explaining the calculation of the distance. [Figure 5] FIG. 5 is a diagram for explaining the interpolation of distance information. [Figure 6]FIG. 6 is a flowchart showing the steps of an information processing method for generating dense distance information. [Figure 7] FIG. 7 is a flowchart showing the detailed interpolation process. [Figure 8] FIG. 8 is a diagram for explaining the influence of the traveling state of a moving object. [Figure 9] FIG. 9 is a flowchart showing a process for identifying the traveling state of a moving object. DETAILED DESCRIPTION OF THE INVENTION

[0016] FIG. 1 shows a schematic configuration of an electromagnetic wave detection device 10 according to an embodiment of the present disclosure. The electromagnetic wave detection device 10 includes an irradiation system 111, a light receiving system 110, and a control unit 14. In this embodiment, the electromagnetic wave detection device 10 functions as a distance measuring device. Furthermore, for example, the electromagnetic wave detection device 10 may function as a driving assistance device that detects the presence of an object ob, which is an obstacle on the road, and issues an alarm. The electromagnetic wave detection device 10 may be mounted on a mobile object, such as a vehicle traveling on a road. In this embodiment, the electromagnetic wave detection device 10 is described as being mounted on a mobile object.

[0017] The irradiation system 111 includes an irradiation unit 12 and a change unit 13. The light receiving system 110 includes an incident unit 15, a separation unit 16, a detection unit 20, an imaging unit 17, a switching unit 18, and a post-stage optical system 19. The control unit 14 controls the overall operation of the electromagnetic wave detection device 10 and also functions as a calculation unit that performs, for example, distance calculations.

[0018] In the drawings, dashed lines connecting each functional block indicate the flow of control signals or communicated information. The communication indicated by the dashed lines may be wired communication or wireless communication. Furthermore, solid arrows indicate beam-shaped electromagnetic waves. Furthermore, in the drawings, object ob is the subject of the electromagnetic wave detection device 10. The subject may include, for example, a road, a median strip, a sidewall, an object installed on the road, a vehicle, a fallen object, or an object such as a person. Furthermore, the number of objects ob is not limited to one.

[0019] The electromagnetic wave detection device 10 acquires an image including a subject and can identify the subject by detecting waves reflected by the subject. The control unit 14 of the electromagnetic wave detection device 10 measures the distance to the object ob, and functions as a distance measuring device as described above.

[0020] (irradiation system) The irradiation system 111 irradiates electromagnetic waves into a space where the object ob exists. In this embodiment, the irradiation system 111 outputs the electromagnetic waves irradiated by the irradiation unit 12 toward the space where the object ob exists, with the irradiation direction changed by the change unit 13.

[0021] The irradiation unit 12 irradiates at least one of infrared light, visible light, ultraviolet light, and radio waves. In this embodiment, the irradiation unit 12 irradiates electromagnetic waves in the form of a narrow beam. The irradiation unit 12 also irradiates electromagnetic waves in pulses. The irradiation unit 12 may be configured to include, for example, an LED (Light Emitting Diode) as an electromagnetic wave irradiating element. The irradiation unit 12 may also be configured to include, for example, an LD (Laser Diode) as an electromagnetic wave irradiating element. The irradiation unit 12 switches between irradiating and stopping the electromagnetic waves under the control of the control unit 14.

[0022] The change unit 13 outputs the electromagnetic waves irradiated by the irradiation unit 12 in a plurality of different directions. Output in a plurality of different directions may be performed by reflecting the electromagnetic waves from the irradiation unit 12 while changing the orientation of the change unit 13. For example, the change unit 13 scans the target ob in one-dimensional or two-dimensional directions with the electromagnetic waves irradiated by the irradiation unit 12.

[0023] The change unit 13 is configured so that at least a part of the irradiation area, which is the space from which the electromagnetic waves are output, is included in the detection range of the electromagnetic waves in the light-receiving system 110. Therefore, at least a part of the electromagnetic waves irradiated into the space in which the target ob exists via the change unit 13 is reflected by at least a part of the target ob and can be detected by the light-receiving system 110. Here, the electromagnetic waves that are irradiated by at least a part of the target ob (i.e., an object existing in the irradiation area) are referred to as reflected waves. The irradiated waves are electromagnetic waves irradiated from the irradiation system 111 in multiple directions in the space in which the target ob exists.

[0024] The change unit 13 includes, for example, a MEMS (Micro Electro Mechanical Systems) mirror, a polygon mirror, or a galvanometer mirror.

[0025] The change unit 13 changes the direction in which the electromagnetic waves are reflected based on the control of the control unit 14. The change unit 13 may also have an angle sensor such as an encoder, and may notify the control unit 14 of the angle detected by the angle sensor as directional information in which the electromagnetic waves are reflected. In such a configuration, the control unit 14 can calculate the irradiation position of the electromagnetic waves based on the directional information acquired from the change unit 13.

[0026] (Light receiving system) Hereinafter, "electromagnetic waves including reflected waves" refers to electromagnetic waves that include waves reflected by the object ob and are incident on the light-receiving system 110. That is, to distinguish them from irradiated waves, electromagnetic waves that are incident on the light-receiving system 110 are sometimes referred to as "electromagnetic waves including reflected waves." Electromagnetic waves including reflected waves include not only reflected waves that are electromagnetic waves irradiated from the irradiating system 111 and reflected by the object ob, but also external light such as sunlight, light that is external light reflected by the object ob, and the like.

[0027] The incident section 15 is an optical system having at least one optical member, and forms an image of the object ob, which is the subject. The optical member includes at least one of a lens, a mirror, a diaphragm, an optical filter, and the like.

[0028] The separator 16 is provided between the incident portion 15 and the primary imaging position, which is the imaging position of the image of the object ob. The separator 16 separates the electromagnetic waves including the reflected waves according to wavelength, and separates them so that they travel in a first direction d1 or a second direction d2. The separator 16 may separate the electromagnetic waves including the reflected waves into the reflected waves and electromagnetic waves excluding the reflected waves. The electromagnetic waves excluding the reflected waves may include light, such as visible light.

[0029] In this embodiment, the separator 16 reflects visible light, which is incident electromagnetic waves and is environmental light such as sunlight reflected by the object ob, in a first direction d1. The separator 16 also transmits, in a second direction d2, electromagnetic waves, which are incident electromagnetic waves and are irradiated by the irradiating unit 12 and reflected by the object ob. The separator 16 may be, for example, a half mirror, a beam splitter, a dichroic mirror, a cold mirror, a hot mirror, a metasurface, a polarization-changing element, or a prism.

[0030] The imaging unit 17 is provided on the path of the electromagnetic wave traveling in the first direction d1 from the separation unit 16. The imaging unit 17 is provided at or near the imaging position of the object ob in the first direction d1. The imaging unit 17 detects the electromagnetic wave traveling in the first direction d1 from the separation unit 16.

[0031] Furthermore, the imaging unit 17 may be disposed so that a first propagation axis of the electromagnetic wave propagating in the first direction d1 from the separation unit 16 is parallel to a first detection axis of the imaging unit 17. The first propagation axis is the central axis of the electromagnetic wave propagating while spreading radially from the separation unit 16 in the first direction d1. In this embodiment, the first propagation axis is an axis obtained by extending the optical axis of the incident unit 15 to the separation unit 16 and bending it at the separation unit 16 so as to be parallel to the first direction d1. The first detection axis is an axis that passes through the center of the detection surface of the imaging unit 17 and is perpendicular to the detection surface.

[0032] In this embodiment, the imaging unit 17 is disposed so that the first axis of travel and the first axis of detection coincide with each other. Also, in this embodiment, the imaging unit 17 is disposed so that the angle between the first axis of travel and the detection surface of the imaging unit 17 is 90°.

[0033] In this embodiment, the imaging unit 17 is a passive sensor, and more specifically, includes an element array. For example, the imaging unit 17 includes an imaging element such as an image sensor or an imaging array, and captures an image formed by electromagnetic waves on a detection surface to generate image information of a space including the imaged object ob. In other words, the imaging unit 17 generates an image including the illuminated area.

[0034] In this embodiment, more specifically, the imaging unit 17 captures an image using visible light. The imaging unit 17 transmits the generated image information to the control unit 14. Here, the imaging unit 17 may capture an image using light other than visible light, such as infrared light or ultraviolet light.

[0035] The switching unit 18 is provided on the path of the electromagnetic wave traveling in the second direction d2 from the separation unit 16. The switching unit 18 is provided at or near the primary imaging position of the object ob in the second direction d2.

[0036] In this embodiment, the switching unit 18 is provided at the primary imaging position. The switching unit 18 has an action surface as onto which the electromagnetic wave that has passed through the incident unit 15 and the separation unit 16 is incident. The action surface as is composed of a plurality of switching elements se arranged two-dimensionally. The action surface as is a surface that causes an action, such as reflection or transmission, on the electromagnetic wave in at least one of a first state and a second state described below.

[0037] The switching unit 18 can switch, for each switching element se, between a first state in which the electromagnetic wave incident on the action surface as travels in a third direction d3 and a second state in which the electromagnetic wave travels in a fourth direction d4. In this embodiment, the first state is a first reflection state in which the electromagnetic wave incident on the action surface as is reflected in the third direction d3. The second state is a second reflection state in which the electromagnetic wave incident on the action surface as is reflected in the fourth direction d4.

[0038] In this embodiment, the switching unit 18 includes a reflective surface that reflects electromagnetic waves for each switching element se. The switching unit 18 switches between the first reflective state and the second reflective state for each switching element se by arbitrarily changing the orientation of the reflective surface for each switching element se.

[0039] In this embodiment, the switching unit 18 includes, for example, a DMD (Digital Micromirror Device). The DMD drives tiny reflective surfaces that make up the acting surface as, thereby switching the reflective surface for each switching element se to an inclination state of either +12° or −12° with respect to the acting surface as. The acting surface as is parallel to the surface of a substrate on which the tiny reflective surfaces of the DMD are mounted.

[0040] The switching unit 18 switches between a first state and a second state for each switching element se under the control of the control unit 14. For example, as shown in FIG. 2, the switching unit 18 can cause the electromagnetic wave incident on the switching element se1 to travel in the third direction d3 by switching some switching elements se1 to the first state. The switching unit 18 can also cause the electromagnetic wave incident on the switching element se2 to travel in the fourth direction d4 by simultaneously switching some other switching elements se2 to the second state. More specifically, the control unit 14 detects the direction or position of the electromagnetic wave based on the direction information from the change unit 13. Then, the control unit 14 sets the switching element se1 to the first state and the other switching elements se2 to the second state according to the detected direction or position of the electromagnetic wave, thereby selectively causing the reflected wave from the target ob to travel in the third direction d3. Of the electromagnetic waves that pass through the separation unit 16, those other than the reflected wave from the target ob travel in the fourth direction d4 and are therefore not incident on the detection unit 20.

[0041] 1, the post-stage optical system 19 is provided in a third direction d3 from the switching unit 18. The post-stage optical system 19 includes, for example, at least one of a lens and a mirror.

[0042] The detection unit 20 detects the electromagnetic waves reflected by an object (object ob) present in the irradiation area. The detection unit 20 is disposed at a position where it can detect the electromagnetic waves traveling via the post-stage optical system 19. The detection unit 20 detects the electromagnetic waves that have traveled via the post-stage optical system 19, i.e., the electromagnetic waves that have traveled in the third direction d3, and outputs a detection signal.

[0043] Furthermore, the detection unit 20 may be disposed so that the second propagation axis of the electromagnetic wave, the propagation direction of which has been switched to the third direction d3 by the switching unit 18, is parallel to the second detection axis of the detection unit 20. The second propagation axis is the central axis of the electromagnetic wave that propagates while spreading radially from the switching unit 18 in the third direction d3. In this embodiment, the second propagation axis is an axis that extends the optical axis of the incident unit 15 to the switching unit 18 and is bent at the switching unit 18 so as to be parallel to the third direction d3. The second detection axis is an axis that passes through the center of the detection surface of the detection unit 20 and is perpendicular to the detection surface.

[0044] In this embodiment, the detector 20 is disposed so that the second axis of travel and the second axis of detection coincide with each other. Also, in this embodiment, the detector 20 is disposed so that the angle between the second axis of travel and the detection surface of the detector 20 is 90°.

[0045] In this embodiment, the detection unit 20 is an active sensor that detects reflected waves. The detection unit 20 may include a single element such as an APD (Avalanche PhotoDiode), a PD (PhotoDiode), or a ranging image sensor. Alternatively, the detection unit 20 may include an element array such as an APD array, a PD array, a ranging imaging array, or a ranging image sensor.

[0046] In this embodiment, the detection unit 20 transmits detection information indicating that a reflected wave from the subject has been detected as a signal to the control unit 14. Also, in this embodiment, the detection unit 20 is used as a detection element for measuring the distance to the object ob. In other words, the detection unit 20 is an element that constitutes a distance measurement sensor.

[0047] With the above-described configuration, the electromagnetic wave detection device 10 aligns a predetermined position on an image with the optical axis of a reflected wave for measuring the distance to that position.

[0048] FIG. 3 is a diagram illustrating the detection of electromagnetic waves including reflected waves. In FIG. 3, the space in which the object ob exists is divided into a grid pattern based on the number of times per frame that the irradiation system 111 irradiates electromagnetic waves. Generally, the time required to detect one frame of electromagnetic waves including reflected waves is longer than the time required to acquire one frame of image using an image sensor or the like. As an example, an image sensor can acquire 30 frames of 1920 × 1080 pixel images per second. On the other hand, the time required to measure the distance by receiving the reflected waves of the irradiated electromagnetic waves may take approximately 20 μs per point (each range in FIG. 3 where the space is divided into a grid pattern). Therefore, the number of points (number of points) at which reflected waves from the space are received to acquire distance information (distance information acquisition points) is smaller than 1920 × 1080 per frame.

[0049] In the example of FIG. 3, a beam-shaped electromagnetic wave emitted from the irradiation unit 12 is reflected by the change unit 13 and is incident on one region R in space as an irradiated wave. The irradiated wave is, for example, infrared light. Electromagnetic waves including reflected waves (e.g., infrared light) reflected from an object ob present in the region R are incident on the incidence unit 15. The electromagnetic waves including reflected waves include visible light resulting from external light being reflected from an object ob present in the region R. The separation unit 16 reflects visible light from the electromagnetic waves including reflected waves in a first direction d1. The reflected visible light is detected by the imaging unit 17. The separation unit 16 also transmits infrared light from the electromagnetic waves including reflected waves in a second direction d2. The infrared light transmitted through the separation unit 16 is reflected by the switching unit 18, and at least a portion of the infrared light travels in a third direction d3. The infrared light traveling in the third direction d3 passes through a post-stage optical system 19 and is detected by the detection unit 20.

[0050] (Control unit) As described above, the control unit 14 controls the overall operation of the electromagnetic wave detection device 10 and performs distance calculations, etc. The control unit 14 controls the irradiation system 111. For example, the control unit 14 causes the irradiation unit 12 to switch between emitting and stopping electromagnetic waves. The control unit 14 causes the change unit 13 to change the direction in which the electromagnetic waves are reflected. In this embodiment, the control unit 14 controls the irradiation unit 12 and the change unit 13 to change the irradiation direction of the electromagnetic waves and emit them multiple times in a predetermined cycle. The control unit 14 also controls the light-receiving system 110. For example, the control unit 14 causes the switching unit 18 to switch between a first state and a second state for each switching element se. The control unit 14 also calculates the distance to the target ob based on detection information from the detection unit 20. The control unit 14 can calculate the distance based on the acquired detection information using, for example, a ToF (Time-of-Flight) method.

[0051] As shown in FIG. 4, the control unit 14 inputs an electromagnetic wave emission signal to the irradiation unit 12, thereby causing the irradiation unit 12 to emit pulsed electromagnetic waves (see "Electromagnetic wave emission signal"). The irradiation unit 12 emits electromagnetic waves based on the input electromagnetic wave emission signal (see "Amount of irradiation unit radiation"). The electromagnetic waves emitted by the irradiation unit 12 and reflected by the change unit 13 and irradiated into the irradiation area, which is the space where the target ob exists, are reflected in the irradiation area. The detection unit 20 detects the electromagnetic waves reflected in the irradiation area (see "Amount of electromagnetic waves detected") and notifies the control unit 14 of the detection information.

[0052] The control unit 14 acquires information from the above-mentioned signals, including the detection information. The control unit 14 includes, for example, a time measurement LSI (Large Scale Integrated circuit), and measures the time ΔT from time T1, when the irradiating unit 12 irradiates the electromagnetic waves, to time T2, when the detection information is acquired (see "Acquisition of Detection Information"). The control unit 14 calculates the distance to the irradiation position by multiplying the time ΔT by the speed of light and dividing by 2. In this way, the control unit 14 calculates distance information indicating the distance to the reflection point of the electromagnetic waves, based on the detection information from the detection unit 20.

[0053] The control unit 14 may be configured to include one or more processors. The processor may include at least one of a general-purpose processor that loads a specific program to execute a specific function and a dedicated processor specialized for a specific process. The dedicated processor may include an application-specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 14 may include at least one of a system-on-a-chip (SoC) and a system in a package (SiP) in which one or more processors work together.

[0054] Here, particularly when the electromagnetic wave detection device 10 is mounted on a moving object such as a vehicle, movement of the detection unit 20 can distort the image (three-dimensional information of an object present in the irradiation area) based on distance information. As described with reference to FIG. 3 , the time required for the detection unit 20 to detect one frame of reflected waves is longer than the time required for the imaging unit 17 to acquire one frame of image. Therefore, if the detection unit 20 moves quickly, the distance between the detection unit 20 and the object (object ob) can change significantly between the acquisition of distance information at a certain point and the acquisition of the next distance information (distance information acquisition point), resulting in distortion of the three-dimensional information of the object. As a result, the three-dimensional information of the object may not match the shape of the object in the image acquired by the imaging unit 17. To address such distortion, a conventional method generates three-dimensional information of the object based on distance information, and then performs a process (scanning distortion correction process) to correct the three-dimensional information of the object so as to cancel out the movement of the moving object in three-dimensional space, for example, in the yaw angle and traveling direction of the moving object. Here, the scanning distortion correction process corrects distance information and corrects the three-dimensional positions (coordinates) of the three-dimensional point cloud that constitutes the three-dimensional information based on the corrected distance information. Information about the yaw angle and movement in the direction of travel of the moving object is obtained from a sensor (e.g., an inertial measurement unit (IMU)) equipped on the moving object. In the scanning distortion correction process, for each point included in the three-dimensional point cloud, linear interpolation of two pieces of movement information acquired at the time closest to the time the point was acquired and the next closest time may be used as the movement information at the time the point was acquired. However, when the three-dimensional information after scanning distortion correction is projected into two-dimensional space to generate an RGBD image, distance information corresponding to depth data is not obtained at equal intervals for the pixels that constitute the RGB image. Here, an RGBD image is an image that combines RGB data, which is color information, and depth data. In other words, because the three-dimensional positions of some points corresponding to distance information included in the three-dimensional point cloud are moved by scanning distortion correction, there are pixels in the RGBD image that do not contain distance information, resulting in areas with little information. Therefore, the three-dimensional information after scanning distortion correction contains areas with little information, resulting in sparse distance information.Similarly, the RGBD image generated from the 3D information after the scanning distortion correction also contains areas with little information, resulting in sparse distance information. To solve this problem, the electromagnetic wave detection device 10 according to this embodiment performs the interpolation described below.

[0055] FIG. 5 is a diagram illustrating the interpolation of distance information performed by the electromagnetic wave detection device 10 according to this embodiment. In the electromagnetic wave detection device 10 according to this embodiment, the optical axes of the image and the image of distance information are aligned by the above-described configuration. This allows the control unit 14 to easily associate the image with the distance information. The control unit 14 interpolates the distance information calculated based on the detection information from the detection unit 20 using the image generated by the imaging unit 17, thereby generating detailed distance information through the interpolation. As shown in FIG. 5, the control unit 14 performs interpolation processing after scanning distortion correction. In the scanning distortion correction, for example, distortion of an object that shifts in the vertical direction in three-dimensional space due to the movement of a mobile object carrying the electromagnetic wave detection device 10 (see "Before Correction" at the top of FIG. 5) is corrected to three-dimensional information with the distortion eliminated (see "After Correction" at the top of FIG. 5). In this embodiment, the movement of the mobile object considered in the scanning distortion correction is not limited to the yaw angle and movement in the traveling direction, but also considers, for example, the pitch angle, roll angle, and movement in the vertical direction. Then, for sparse portions of the distance information (see "Before Interpolation" at the bottom of FIG. 5), control unit 14 interpolates the distance information based on the features of the same region in the image (see "After Interpolation" at the bottom of FIG. 5). Here, the "features" of the image are, for example, brightness or color.

[0056] (Information processing method) The control unit 14 of the electromagnetic wave detection device 10 executes the information processing method according to this embodiment, for example, in accordance with the flowchart of Fig. 6. The information processing method solves the problem of the conventional technology in which sparse distance information is generated, and generates dense distance information by interpolation.

[0057] The control unit 14 performs scanning distortion correction so as to cancel out the influence of the movement of the mobile object on which the electromagnetic wave detection device 10 is mounted (step S1). The scanning distortion correction may be performed by a known method.

[0058] The control unit 14 projects the 3D information after the scanning distortion correction onto 3D coordinates (step S2). The control unit 14 then projects the 3D information after the scanning distortion correction onto an image generated by the imaging unit 17 (step S3). Here, the image generated by the imaging unit 17 has pixels with a higher density than the distance information (distance information acquisition points) acquired by the detection unit 20 in one frame. The control unit 14 also uses, as the projected image, an image generated by the imaging unit 17 in a time period (a frame in which the imaging unit 17 generates an image) closest to the one frame in which the distance information (3D information) before the scanning distortion correction was acquired. Here, the time period closest to the one frame refers to, for example, one or more frames in which the imaging unit 17 generated an image, which are included in the time period of the one frame in which the distance information before the scanning distortion correction was acquired. The control unit 14 can identify sparse portions of the distance information by step S3. Here, sparse parts in the distance information are, for example, pixels in an RGBD image that do not contain depth data (distance information after scanning distortion correction) when an RGBD image is generated by projecting three-dimensional information after scanning distortion correction onto three-dimensional coordinates.

[0059] The control unit 14 interpolates distance information based on image features for the sparse distance information including the sparse portion identified in step S3 (step S4). Then, the control unit 14 generates dense distance information after the interpolation (step S5). The control unit 14 may further generate three-dimensional information of the object based on the generated dense distance information.

[0060] Here, the interpolation process (step S4 in FIG. 6) performed by the control unit 14 is not limited to a specific method as long as it is performed based on the characteristics of the same region (region corresponding to the sparse portion) in the image. For example, the control unit 14 may interpolate the distance information based on first distance information indicating the distance to the first region included in the distance information and the characteristics of the image region corresponding to the first region. For example, the interpolation process described in the flowchart in FIG. 7 may be adopted.

[0061] The control unit 14 divides the image generated by the imaging unit 17 (step S11). The division is performed based on the characteristics of the image. For example, portions with similar brightness or similar color may be divided into one region. In this case, portions with brightness values ​​within a predetermined range may be divided into one region. Alternatively, portions with color information (e.g., RGB values) within a predetermined range may be divided into one region. A technique for generating superpixels may be used to divide the image into one region. In the example of FIG. 5, for example, the road surface, one sidewall, and the opposite sidewall may each be divided into one region (first region). Alternatively, the control unit 14 may recognize the type of object the portion with similar brightness or color is based on its shape, or may further recognize the shape of a part of the object, and divide the image into one region using any of the recognition results.

[0062] Control unit 14 estimates a plane for each divided region of the image (step S12). In other words, control unit 14 sets a virtual plane for each region. Then, control unit 14 calculates an intersection between a plane (second plane) of a region included in the set virtual plane and corresponding to the second distance information acquisition point (one region including the position of the distance information acquisition point to be interpolated) and a ray to this distance information acquisition point (step S13). Here, the ray to the distance information acquisition point is a virtual straight line from detection unit 20 to the distance information acquisition point. Control unit 14 uses the distance to the calculated intersection point as the distance information to be interpolated. For example, control unit 14 uses information about the first plane of a region included in the set virtual plane and corresponding to the first distance information acquisition point where the first distance information was acquired (one region included in the 3D information after scanning distortion correction and including the position of the first distance information acquisition point where the first distance information has been acquired). Control unit 14 calculates the intersection point between detection unit 20 and the second distance information acquisition point using (1) first distance information indicating the distance from detection unit 20 to the first distance information acquisition point, (2) the angle between the ray from detection unit 20 to the first plane and the ray from detection unit 20 to the second plane, and (3) the angle between the ray from detection unit 20 to the first plane and a virtual plane. Here, (2) can be obtained by understanding the intervals between each distance information acquisition point, because the distance information acquisition points are installed at regular intervals as shown in FIG. 3 .

[0063] Here, the control unit 14 may estimate a surface for each divided region, and then combine the regions based on the similarity of the normals of the surfaces to generate a larger region. Generating a larger region can further improve the accuracy of the interpolated distance information. For example, if the normals of the virtual surfaces of multiple regions point upward, the control unit 14 may determine that these multiple regions correspond to the same road surface and perform a combining process. Here, the similarity of the normals of the surfaces may be determined by whether the angle between the normals is equal to or smaller than a preset threshold angle.

[0064] Furthermore, image information may be used to accurately identify the movement of the moving object in the scanning distortion correction (step S1 in FIG. 6) performed by the control unit 14. Accurate identification of the movement of the moving object contributes to accurate estimation of its own position (particularly the position of the detection unit 20). The movement of the moving object (i.e., the movement of the detection unit 20) is obtained from an inertial measurement unit or the like as described above. However, the detection results of the inertial measurement unit may contain large errors, for example, when the moving object moves up or down due to bouncing. Using image information makes it possible to correct such errors. The image used may be an image generated by the imaging unit 17 in a frame in a time period closest to the frame in which distance information (three-dimensional information) before scanning distortion correction was acquired. Here, the time period closest to the frame is, for example, the time period in which the imaging unit 17 generated the image, which is included in the time period of the frame in which distance information before scanning distortion correction was acquired. For example, in the example of FIG. 8, the actual distance to an object having a rectangular parallelepiped shape (see “Actual Distance” on the right) may be distorted due to bouncing of the moving object (see “Input Distance Information” on the left). The control unit 14 can estimate distortion of distance information in a target region, which is a region containing an object of a known shape recognized from an image, calculate a correction value for adjusting the distorted distance information in the target region to the known shape, and correct the distance information based on the correction value. The control unit 14 can correct the input distance information (perform scanning distortion correction) using image information in this way. The movement state of the detection unit 20 can also be determined using the correction value for adjusting to the known shape. For example, the movement state of the detection unit 20 can be estimated from the difference between the detection time of a reflected wave at a distance information acquisition point included in the known shape and the detection time of a reflected wave at another distance information acquisition point from which distorted distance information was obtained, and the correction value for adjusting to the known shape. Here, the movement state of the detection unit 20 can be, for example, the speed or the frequency of vertical movement (vibration) due to bouncing, etc. Such correction can also utilize the results of the processing of FIG. 6 that has already been performed. For example, the control unit 14 can estimate the movement state of the detection unit 20 based on multiple interpolated distance information pieces obtained over a predetermined period (e.g., the immediately preceding few seconds). For example, the process of identifying the traveling state of the mobile object, which is explained in the flowchart of FIG. 9, may be executed.

[0065] The control unit 14 divides the area of ​​the distance information (step S21). When dividing the distance information, for example, distance information whose difference falls within a preset range may be divided into one area. The control unit 14 also divides the area of ​​the image so as to correspond to the division of the distance information (step S22). Then, the control unit 14 selects one of the divided areas as a target area (step S23).

[0066] The control unit 14 associates the distance information within the target region with the shape of the object in the image (step S24). Then, the control unit 14 identifies the traveling state of the moving object from the difference in the shape of the object (step S25). The identified traveling state is used in scanning distortion correction (step S1 in FIG. 6).

[0067] Here, when there is one or more target regions, the control unit 14 may perform steps S23 to S25 for each target region and increase the accuracy of identifying the traveling state of the mobile object by averaging or the like. Furthermore, the control unit 14 may simplify the processing when identifying only forward and backward as the traveling state of the mobile object. That is, the control unit 14 may omit step S21 and calculate the magnitude of distortion of the distance information for, for example, a rectangular parallelepiped object included in the target region of the image obtained by segmentation, based on the time interval between scanning and acquiring the distance information. Then, the control unit 14 may identify the forward or backward speed of the mobile object based on the magnitude of distortion of the distance information.

[0068] As described above, the electromagnetic wave detection device 10 according to this embodiment can improve the accuracy of object recognition, self-position estimation, and the like.

[0069] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and alterations based on the present disclosure, and therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure.

[0070] In this embodiment, the configuration of the electromagnetic wave detection device 10 has been described with reference to Fig. 1. However, the electromagnetic wave detection device 10 is not limited to the configuration shown in Fig. 1 and may be configured, for example, with a control device including a control unit 14 and an electromagnetic wave detection unit including components other than the control unit 14. In other words, the control device may be a device separate from the electromagnetic wave detection unit, controlling the electromagnetic wave detection unit including the irradiation unit 12, the change unit 13, the detection unit 20, and the imaging unit 17. A control device configured in this way can also improve the accuracy of object recognition, self-position estimation, and the like.

[0071] For example, in the interpolation process, when a peripheral area of ​​the image corresponding to a second area having the same characteristics as a first area exists around the first area in the image, the control unit 14 may interpolate second distance information indicating the distance to the second area using the first distance information. As a specific example, instead of or in addition to the process of Fig. 7, a bilateral filter that inserts similar depth information for parts of the image having similar characteristics (e.g., colors) may be used.

[0072] In the electromagnetic wave detection device 10, the optical axes of the image and the image of distance information are aligned in the light receiving system 110. However, if the correspondence between the image and the image of distance information is acquired in advance as information, the optical axes of the image and the image of distance information do not need to be aligned.

[0073] While the solution of the present disclosure has been described as an apparatus and a method, the present disclosure can also be realized as aspects including these. The present disclosure can also be realized as a program that substantially corresponds to these, or a storage medium on which a program is recorded, and it should be understood that these are also included within the scope of the present disclosure. [Explanation of symbols]

[0074] 10. Electromagnetic wave detection device 12 Irradiation unit 13 Changes 14 Control Unit 15 Input part 16 Separation part 17 Imaging unit 18 Switching section 19 Post-stage optical system 20 Detector 110 Light receiving system 111 Irradiation system as action surface d1, d2, d3, d4 First direction, second direction, third direction, fourth direction ob target

Claims

1. an irradiation unit that outputs electromagnetic waves; a change unit that changes the irradiation direction of the electromagnetic wave to a plurality of different directions and outputs the electromagnetic wave to an irradiation area; a detection unit that detects a reflected wave of the electromagnetic wave reflected by an object present in the irradiation area; an imaging unit that acquires an image including the illumination area; a control unit that controls the irradiation unit and the change unit to change the irradiation direction of the electromagnetic wave and irradiate it multiple times in a predetermined period, the control unit interpolates distance information based on the detection result of the detection unit in the predetermined cycle using the image acquired by the imaging unit. Electromagnetic wave detection device.

2. the control unit projects three-dimensional information based on the distance information onto the image, and interpolates distance information for a portion of the image where the three-dimensional information is not projected.

2. The electromagnetic wave detection device according to claim 1.

3. the control unit interpolates the distance information based on first distance information indicating a distance to a first region included in the distance information and a feature of an image region of the image corresponding to the first region.

3. The electromagnetic wave detection device according to claim 1 or 2.

4. 4. The electromagnetic wave detection device of claim 3, wherein when a peripheral area of ​​the image corresponding to the first area exists around the image area of ​​the image having the same characteristics, the control unit interpolates second distance information indicating the distance to the peripheral area using the first distance information.

5. the characteristic is brightness or color; 4. The electromagnetic wave detection device according to claim 3.

6. 2. The electromagnetic wave detection device according to claim 1, which is mounted on a moving body, the control unit corrects the distance information in accordance with a traveling state of the moving object, and interpolates the corrected distance information using the image acquired by the imaging unit. Electromagnetic wave detection device.

7. the control unit identifies the traveling state based on a difference between a shape of an object based on the plurality of pieces of distance information acquired at the predetermined period and a shape of the object included in the image.

7. The electromagnetic wave detection device according to claim 6.

8. A control device for controlling an electromagnetic wave detection unit including an irradiation unit that outputs electromagnetic waves, a change unit that changes the irradiation direction of the electromagnetic waves to a plurality of different directions and outputs the electromagnetic waves to an irradiation area, a detection unit that detects reflected waves of the electromagnetic waves reflected by an object present in the irradiation area, and an imaging unit that acquires an image including the irradiation area, a control unit that controls the irradiation unit and the change unit to change the irradiation direction of the electromagnetic wave and irradiate it multiple times in a predetermined period, the control unit interpolates distance information based on the detection result of the detection unit in the predetermined cycle using the image acquired by the imaging unit. Control device.

Images