Movement amount estimation device, movement amount estimation method, movement amount estimation program, and movement amount estimation system

Abstract

The image processing ECU (30) includes an image acquisition unit (41) and an image processing module (42), and functions as a movement amount estimation device. The image acquisition unit (41) acquires a reflected light image (ImR) containing distance information obtained by detecting reflected light reflected from an object by light irradiation through a light receiving element (12a), and a background light image (ImL) containing brightness information by detecting background light relative to the reflected light using the light receiving element (12a). In addition, the image acquisition unit (41) acquires a camera image (ImC) captured by a camera element (22a) different from the light receiving element (12a). The image processing module (42) estimates the movement amount of the estimated object using information of the subject detected in common in the reflected light image (ImR), the background light image (ImL), and the camera image (ImC).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Patent Application No. 2019-195548 filed in Japan on October 28, 2019, the entire contents of which are hereby incorporated by reference. Technical Field

[0003] The disclosure based on this specification relates to a technology for estimating a movement amount. Background Art

[0004] For example, Patent Document 1 discloses a self-position estimation device that uses LiDAR (Light Detection and Ranging) as a distance sensor. LiDAR is installed on a moving body that can move autonomously, and measures the distance to objects around the moving body. The self-position estimation device estimates the movement amount of the moving body using the distance information detected by the LiDAR, and further estimates the self-position of the moving body.

[0005] Patent document 1: Japanese Patent Application Publication No. 2019-128705.

[0006] The LiDAR of Patent Document 1 repeatedly outputs the detected distance measurement information. However, the number of distance measurement information output per unit time is limited. Therefore, if only the distance measurement information is used to estimate the movement amount of the moving object, it may be difficult to ensure the estimation accuracy of the movement amount due to the insufficient number of frames of distance measurement information per unit time, that is, the frame rate. Summary of the invention

[0007] An object of the present disclosure is to provide a movement amount estimation device and the like that can easily ensure the estimation accuracy of the movement amount.

[0008] In order to achieve the above-mentioned purpose, a disclosed method is a movement amount estimation device, comprising: an image acquisition unit, which acquires a reflected light image containing distance information obtained by detecting reflected light reflected from an object by light irradiation by a light receiving element, a background light image containing brightness information obtained by detecting background light relative to the reflected light by the light receiving element, and a camera image captured by a camera element different from the light receiving element; and an estimation unit, which estimates the movement amount of the estimated object using information of the subject detected jointly in the reflected light image, the background light image and the camera image.

[0009] Another disclosed method is a method for estimating the amount of movement, in which processing executed by at least one processor includes: obtaining a reflected light image containing distance information obtained by detecting reflected light reflected from an object by light irradiation through a light receiving element, a background light image containing brightness information of background light relative to the reflected light detected by the light receiving element, and a camera image captured by a camera element different from the light receiving element; and estimating the amount of movement of the estimated object using information of the subject detected jointly in the reflected light image, the background light image, and the camera image.

[0010] Another disclosed method is a movement amount estimation program, which causes at least one processor to execute the following steps: obtaining a reflected light image containing distance information obtained by detecting reflected light reflected from an object by light irradiation through a light receiving element, a background light image containing brightness information of background light relative to the reflected light detected by the light receiving element, and a camera image captured by a camera element different from the light receiving element; and estimating the movement amount of the estimated object using information of the photographed object detected jointly in the reflected light image, the background light image, and the camera image.

[0011] Another disclosed method is a movement amount estimation system, comprising: a distance measuring device, which generates a reflected light image containing distance information obtained by detecting reflected light reflected from an object by light irradiation by a light receiving element, and a background light image containing brightness information by detecting background light relative to the reflected light by the light receiving element; a camera device, which generates a camera image by photographing with a camera element different from the light receiving element; and a computing device, which estimates the movement amount of the estimated object using information of the subject detected in the reflected light image, the background light image, and the camera image.

[0012] In these methods, the estimation of the movement amount of the object is not only based on the reflected light image, but also based on the background light image and the camera image. As described above, the estimation of the movement amount can be implemented at a higher frame rate than when the movement amount is estimated based on the reflected light image alone. Therefore, it is easy to ensure the estimation accuracy of the movement amount.

[0013] In addition, the reference numerals in parentheses in the claims and the like merely show an example of the correspondence relationship with the specific configurations in the embodiments described later, and do not limit the technical scope at all. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a diagram showing the overall situation of the sensor package and the autonomous driving system according to the first embodiment of the present disclosure.

[0015] Figure 2 It is a diagram showing the state in which the distance measuring sensor and the external camera are mounted on the vehicle.

[0016] Figure 3 This is a block diagram for explaining the process of image processing in the image processing ECU.

[0017] Figure 4 This is a diagram showing an example of the capturing timing of the sensor image and the camera image.

[0018] Figure 5 FIG. 1 is a diagram schematically showing details of processing related to movement amount estimation.

[0019] Figure 6 This is a flowchart of the movement amount estimation process performed in the image processing ECU.

[0020] Figure 7 It is a diagram showing the imaging timing of the sensor image and the camera image in the second embodiment. DETAILED DESCRIPTION

[0021] Hereinafter, multiple embodiments of the present disclosure are described based on the accompanying drawings. In addition, sometimes, by adding the same figure marks to the corresponding components in each embodiment, repeated descriptions are omitted. In the case where only a part of the composition is described in each embodiment, the other parts of the composition can apply the composition of other embodiments described in advance. In addition, not only the combination of the composition explicitly stated in the description of each embodiment, as long as the combination does not particularly cause hindrance, the composition of multiple embodiments can be partially combined with each other even if it is not explicitly stated. Moreover, the unstated combination of the composition described in multiple embodiments and modified examples is also disclosed through the following description.

[0022] <First embodiment>

[0023] Used in vehicle A as a moving object Figure 1 The image processing ECU (Electronic Control Unit: Electronic Control Unit) 30 of the first embodiment of the present disclosure is shown. Figure 1 as well as Figure 2 As shown, the image processing ECU 30, the distance measuring sensor 10, the external camera 20, etc., together constitute a sensor package 100. Such a sensor package 100 can constitute an automatic driving system 110 together with the automatic driving ECU 50 and the power supply device 60, etc. In the automatic driving system 110, the image processing ECU 30 and the sensor package 100 integrate the measurement results of the distance measuring sensor 10 and the external camera 20 to function as a movement amount estimation device and a movement amount estimation system that estimates the movement amount of the vehicle A.

[0024] Figure 1 as well as Figure 3The autonomous driving system 110 shown is, for example, a system package that enables autonomous driving of a vehicle A without a driver. The autonomous driving ECU 50 that constitutes the core of the autonomous driving system 110 is a high-speed computing device that includes a computer as a main body, which includes a processor, RAM (Random Access Memory), a storage unit, an input / output interface, and a bus connecting them. The autonomous driving ECU 50 includes an identification and judgment unit 51 and a path generation unit 52 as functional units for realizing autonomous driving. The identification and judgment unit 51 identifies the driving environment around the vehicle A based on the information provided from the sensor package 100 and determines the next action of the vehicle A. The path generation unit 52 generates a driving path for the vehicle A to travel based on the decision of the identification and judgment unit 51. In addition, the autonomous driving system 110 may not be a system that can provide a complete autonomous driving function that can perform unmanned driving, but a system that can provide a high-level driving support function that can provide driving support for the driver.

[0025] Figure 1 to Figure 3 The sensor package 100 shown includes the distance measuring sensor 10 , the external camera 20 , and the image processing ECU 30 as described above.

[0026] As an example, the distance measuring sensor 10 is a SPAD (Single Photon Avalanche Diode) LiDAR. The distance measuring sensor 10 is disposed at the front end of the vehicle A or the roof of the vehicle A. The measurement range MA1 of the distance measuring sensor 10 is set to include at least the front range of the periphery of the vehicle A.

[0027] The distance measuring sensor 10 is composed of a light emitting unit 11, a light receiving unit 12, and a control unit 13. The light emitting unit 11 irradiates a light beam emitted from a light source toward the measurement range MA1 by scanning using a movable optical component (e.g., a polygon mirror). The light source is, for example, a semiconductor laser (Laser diode), and according to an electrical signal from the control unit 13, a light beam in a non-visible area (e.g., a near-infrared area) that cannot be visually recognized by passengers and outsiders is irradiated in a pulsed manner.

[0028] The light receiving unit 12 has a focusing lens and a light receiving element 12a. The focusing lens focuses the reflected light of the light beam reflected by the object within the measurement range MA1 and the background light relative to the reflected light, and makes it incident on the light receiving element 12a. The light receiving element 12a is an element that converts light into an electrical signal by photoelectric conversion. In order to efficiently detect the reflected light of the light beam, the light receiving element 12a is a CMOS sensor that sets the sensitivity of the near-infrared domain higher than the visible domain. The sensitivity of the light receiving element 12a to each wavelength domain can be adjusted by an optical filter. The light receiving element 12a has a plurality of light receiving pixels arranged in an array in a one-dimensional direction or a two-dimensional direction. Each light receiving pixel is constructed using a SPAD, and the electrons generated by the injection of photons are amplified by avalanche multiplication, so that high-sensitivity light detection can be performed.

[0029] The control unit 13 controls the light emitting unit 11 and the light receiving unit 12. The control unit 13 is, for example, arranged on the same substrate as the light receiving element 12a. The control unit 13 is, for example, mainly composed of a processor in a broad sense such as a microcontroller or FPGA (Field-Programmable Gate Array). The control unit 13 realizes a scanning control function, a reflected light measurement function, and a background light measurement function.

[0030] The scanning control function is a function of controlling the scanning of the light beam by the light emitting unit 11. The control unit 13 causes the light beam to oscillate multiple times in a pulsed manner from the light source at the timing based on the operation clock of the clock oscillator provided in the distance measuring sensor 10. In addition, the control unit 13 causes the movable optical component to operate synchronously with the irradiation of the light beam.

[0031] The reflected light measurement function is a function of reading the voltage value based on the reflected light received by each light-receiving pixel and measuring the intensity of the reflected light according to the timing of the scanning of the light beam. The control unit 13 senses the arrival time of the reflected light according to the generation timing of the peak in the output pulse of the light-receiving element 12a. The control unit 13 measures the time of flight of the light by measuring the time difference between the emission time of the light beam from the light source and the arrival time of the reflected light.

[0032] Through the cooperation of the above scanning control function and the reflected light measurement function, image-like data, i.e., the reflected light image ImR, is generated. The control unit 13 measures the reflected light using a rolling shutter method to generate the reflected light image ImR. In detail, the control unit 13 generates information of pixel groups arranged horizontally on an image plane corresponding to the measurement range MA1, row by row or multiple rows at a time, in coordination with the scanning of the light beam in the horizontal direction, for example. The control unit 13 synthesizes the pixel information generated sequentially in each row in the vertical direction to generate a reflected light image ImR.

[0033] The reflected light image ImR is image data containing distance information obtained by detecting (sensing) reflected light corresponding to light irradiation from the light-emitting unit 11 by the light-receiving element 12a. Each pixel of the reflected light image ImR contains a value indicating the flight time of light. In other words, the value indicating the flight time of light is a distance value indicating the distance from the distance measuring sensor 10 to the reflection point of the object located in the measurement range MA1. In addition, each pixel of the reflected light image ImR also contains a value indicating the intensity of the reflected light. That is, the reflected light image ImR becomes image data indicating the brightness distribution of the reflected light.

[0034] The background light measurement function is a function of measuring the intensity of the background light by reading out the voltage value based on the background light received by each light-receiving pixel at a timing immediately before measuring the reflected light. Here, the background light refers to the incident light that enters the light-receiving element 12a from the external measurement range MA1 and does not actually include the reflected light. The incident light includes natural light, display light that enters from external displays, etc. The control unit 13 measures the background light in a rolling shutter manner in the same manner as the reflected light image ImR, and generates a background light image ImL. The background light image ImL is image data representing the brightness distribution of the background light before light irradiation, and includes the brightness information of the background light detected by the same light-receiving element 12a as the reflected light image ImR. That is, the value of each pixel arranged two-dimensionally in the background light image ImL is a brightness value representing the intensity of the background light at the corresponding position of the measurement range MA1.

[0035] The reflected light image ImR and the background light image ImL are sensed by the common light receiving element 12a, and are obtained from the common optical system including the light receiving element 12a. Therefore, the coordinate system of the reflected light image ImR and the coordinate system of the background light image ImL can be regarded as mutually consistent same coordinate systems. In addition, there is almost no offset in the measurement timing (for example, less than 1ns) between the reflected light image ImR and the background light image ImL. Therefore, it can also be regarded that a group of reflected light images ImR and background light images ImL acquired continuously are also synchronized. In addition, the reflected light image ImR and the background light image ImL can uniquely obtain the correspondence between each pixel. The reflected light image ImR and the background light image ImL correspond to each pixel, and are output to the image processing ECU 30 in sequence as an integrated image data including the intensity of the reflected light, the distance to the object, and the intensity of the background light in three channels. In addition, in the following description, a group of reflected light images ImR and background light images ImL are sometimes recorded as "sensor image ImS".

[0036] The outside camera 20 is disposed, for example, inside the front windshield of the vehicle A. The outside camera 20 can measure at least a measurement range MA2 in front of the outside of the vehicle A. The measurement range MA2 of the outside camera 20 is defined to overlap at least a portion with the measurement range MA1 of the distance measuring sensor 10 .

[0037] The external camera 20 includes a light receiving unit 22 and a control unit 23. The light receiving unit 22 collects incident light (background light) incident from a measurement range MA2 outside the camera using, for example, a light receiving lens, and causes the light to enter a camera element 22a.

[0038] The camera element 22a is an element that converts light into an electrical signal by photoelectric conversion, and has a different structure from the light receiving element 12a of the distance measuring sensor 10. The camera element 22a can be a CCD sensor or a CMOS sensor, for example. In the camera element 22a, in order to efficiently receive natural light in the visible range, the sensitivity of the visible region is set higher than that of the near-infrared region. The camera element 22a has a plurality of light receiving pixels arranged in an array in two dimensions. The light receiving pixels of the camera element 22a are equivalent to so-called sub-pixels. For example, red, green, and blue color filters are arranged in adjacent light receiving pixels. Each light receiving pixel receives visible light of a color corresponding to the configured color filter. The camera element 22a uses each light receiving pixel to measure the intensity of red, green, and blue, respectively, and captures a camera image ImC. The camera image ImC is a color image of the visible range, and becomes a high-resolution image compared with the reflected light image ImR and the background light image ImL.

[0039] The control unit 23 controls the light receiving unit 22. The control unit 23 is disposed on a common substrate with the camera element 22a, for example, and is mainly composed of a processor in a broad sense such as a microcontroller or FPGA. The control unit 23 realizes a photographing function.

[0040] The photographing function is a function of photographing the color image described above. The control unit 23 reads the voltage value of each light-receiving pixel corresponding to the amount of incident light at the timing based on the operation clock of the clock oscillator provided in the external camera 20. The clock oscillator is provided independently of the clock oscillator of the distance measuring sensor 10. The control unit 23 measures the background light based on the camera element 22a at all light-receiving pixels substantially at the same time using a global shutter method, and generates image-like data, namely, a camera image ImC.

[0041] The camera image ImC is an image showing the brightness distribution of the background light in the visible area sensed by the camera element 22a. In each of the images described so far, the brightness distribution is a distribution obtained by converting the intensity distribution of the corresponding light (reflected light or background light, or light in the non-visible area or light in the visible area) into grayscale values. That is, the value of each pixel arranged two-dimensionally in the camera image ImC becomes a brightness value showing the intensity of the incident light of each color incident on the camera element 22a from the measurement range MA2. Such camera images ImC are sequentially output to the image processing ECU 30.

[0042] The image processing ECU 30 is an electronic control device that includes a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, and a bus connecting them as a main body. The processing unit 31 is hardware for calculation processing combined with the RAM 32. The processing unit 31 includes at least one CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an FPGA and other calculation cores. The processing unit 31 can be configured as an image processing chip that further includes an IP core with other dedicated functions. Such an image processing chip can be an ASIC (Application Specific Integrated Circuit) designed for autonomous driving purposes. The processing unit 31 performs various processes for realizing the functions of each functional unit described later by accessing the RAM 32. The storage unit 33 is a structure including a non-volatile storage medium. Various programs (such as movement estimation programs) executed by the processing unit 31 are stored in the storage unit 33.

[0043] The image processing ECU 30 processes the image information of the distance measuring sensor 10 and the image information of the external camera 20, and generates the provided information to be provided to the automatic driving ECU 50. The provided information generated by the image processing ECU 30 includes the estimated result of the movement amount (hereinafter referred to as "self-movement amount") of the vehicle A equipped with the sensor package 100. In order to estimate the self-movement amount, the image processing ECU 30 executes the movement amount estimation program stored in the storage unit 33 through the processing unit 31, and has functional units such as an image acquisition unit 41, a calibration unit 43, a distance value acquisition unit 44, a movement amount estimation unit 45, and a self-movement amount calculation unit 46. Among these functional units, the calibration unit 43, the distance value acquisition unit 44, the movement amount estimation unit 45, and the self-movement amount calculation unit 46 are image processing modules 42 that perform image processing.

[0044] The image acquisition unit 41 acquires the reflected light image ImR and the background light image ImL from the distance measuring sensor 10. In addition, the image acquisition unit 41 acquires the camera image ImC from the external camera 20. In the first embodiment, the frame rate of the reflected light image ImR and the background light image ImL is set to be substantially the same as the frame rate of the camera image ImC. On the other hand, the shooting timing STs of the reflected light image ImR and the background light image ImL in the distance measuring sensor 10 is different from the shooting timing STc of the camera image ImC in the external camera 20. Therefore, the shooting timing STs of the reflected light image ImR and the background light image ImL and the shooting timing STc of the camera image ImC come alternately (refer to Figure 4 As a result, the image acquisition unit 41 alternately acquires the reflected light image ImR and the background light image ImL, and acquires the camera image ImC.

[0045] For convenience, the detection period during which the light receiving element 12a performs detection for generating the reflected light image ImR and the background light image ImL is set as a frame period, and the start time of the frame period is set as the shutter start time Tf of the reflected light image ImR and the background light image ImL. On the other hand, the start time of the exposure period during which the camera element 22a performs exposure for capturing the camera image ImC is set as the shutter start time Te of the camera image ImC.

[0046] As described above, the background light image ImL and the reflected light image ImR are formed at substantially the same shutter start time Tf (see Figure 4 On the other hand, the shutter start time Te of the camera image ImC (refer to Figure 4 The time T2) is offset relative to the shutter start time Tf of the background light image ImL and the reflected light image ImR.

[0047] In addition, the entire frame period in the light receiving element 12a is set to be offset relative to the entire exposure period in the camera element 22a. That is, after the frame period of the light receiving element 12a ends, the exposure period of the camera element 22a starts. And after the exposure period ends, the next frame period starts in the light receiving element 12a. Based on the above, the image acquisition unit 41 alternately acquires image data from the distance measuring sensor 10 and the external camera 20 in the order of the background light image ImL and the reflected light image ImR at time T1, the camera image ImC at time T2, the background light image ImL and the reflected light image ImR at time T3, etc.

[0048] like Figure 4As shown, the state of vehicle A is defined by six-axis coordinate information, namely, the coordinate information R (roll, pitch, yaw) of the rotation system representing the posture of the moving body and the coordinate information t (x, y, z) of the translation system representing the position of the moving body. During the period from time T1 to time T2, a posture change of ΔR1 and a position change of Δt1 occur in vehicle A. Similarly, during the period from time T2 to time T3, a posture change of ΔR2 and a position change of Δt2 occur in vehicle A. The differences between these times (ΔR1, Δt1) and (ΔR2, Δt2) are equivalent to the difference between the two times. Figure 1 as well as Figure 3 The image processing ECU 30 shown estimates the movement amount of the vehicle A.

[0049] The calibration unit 43 sets the internal parameters of the distance measuring sensor 10, the internal parameters of the external camera 20, and the external parameters between the distance measuring sensor 10 and the external camera 20. The calibration unit 43 reads the values ​​of the preset parameters from the storage unit 33 and sets them in a specific storage area of ​​the RAM 32. The values ​​of the internal parameters prepared in the RAM 32 are referred to by the movement amount estimation unit 45. On the other hand, the values ​​of the external parameters are referred to by the own movement amount calculation unit 46.

[0050] Figure 3 as well as Figure 5 The distance value acquisition unit 44 shown refers to the reflected light image ImR and the background light image ImL constituting a sensor image ImS, and uses these images ImR and ImL to prepare sensor element information. In detail, the distance value acquisition unit 44 acquires the distance value of the reflected light image ImR corresponding to each pixel position of the background light image ImL. The distance value acquisition unit 44 determines the subject PO reflected in both the reflected light image ImR and the background light image ImL based on the degree of change (gradient) of the brightness value and the distance value between adjacent pixels on the image. For example, the distance value acquisition unit 44 sets the pixel on the background light image ImL corresponding to the position of the pixel where the intensity value of the reflected light contained in the reflected light image ImR is constant or above as the feature point Fpa. The distance value acquisition unit 44 uses the distance value and brightness value of the feature point Fpa related to the subject PO and the pixels in its vicinity as sensor element information.

[0051] The movement amount estimation unit 45 estimates the movement amount of the estimated object ET using the information of the object PO detected in the reflected light image ImR, the background light image ImL, and the camera image ImC. Specifically, the movement amount estimation unit 45 tracks a plurality of feature points Fpa related to the object PO and their vicinities between the sensor image ImS and the camera image ImC that are continuous in time, and calculates the movement vector of the object PO. The object PO is preferably a stationary three-dimensional structure. If the object PO is a stationary object, the movement amount of the vehicle A can be easily estimated based on the movement amount of the object PO on the image.

[0052] Specifically, the movement amount estimation unit 45 also determines the subject PO shared with the sensor image ImS based on the camera image ImC, and prepares the brightness value of the pixel related to the subject PO as the camera element information. In addition, in order to implement the processing of treating the sensor image ImS and the camera image ImC as continuous frame images, the movement amount estimation unit 45 converts the sensor element information prepared by the distance value acquisition unit 44 into information that can be directly compared with the camera element information. The movement amount estimation unit 45 also appropriately adjusts the difference in resolution between the sensor image ImS and the camera image ImC, the difference in background light intensity (brightness difference) in the visible light domain and the near infrared domain, etc. The movement amount estimation unit 45 uses the camera element information as a reference and compares the brightness value of the camera element information with the brightness value of the sensor element information. As an example, the movement amount estimation unit 45 uses the Lucas-Kanade gradient method or the like to obtain ΔR and Δt that minimize the left side of the following formula (1).

[0053] [Number 1]

[0054] C(ΔR, Δt)=||W(I_spad, d_spad)-I_image||^2...Equation (1)

[0055] In the above formula (1), W is a conversion function determined by ΔR and Δt and the internal parameters of the external camera 20. The conversion function W is a function for performing general rotation, translation, and perspective projection (deformation) on an image. In addition, I_spad is the brightness value of the background light image ImL, d_spad is the distance value of the reflected light image ImR, and I_image is the brightness value of the camera image ImC. The movement amount estimation unit 45 estimates the rotational movement amount ΔR and the translational movement amount Δt of the subject PO in a small time based on the above formula (1).

[0056] The own movement amount calculation unit 46 performs a calculation process of removing the external parameters from the movement amounts ΔR and Δt estimated by the movement amount estimation unit 45, and calculates the rotational movement amount ΔR_ego and the translational movement amount Δt_ego of the vehicle A as the estimation object ET. Specifically, the own movement amount calculation unit 46 obtains the differences in posture and position between the distance measuring sensor 10 and the external camera 20 as external parameters (see Figure 2 The own movement amount calculation unit 46 applies the movement amounts ΔR, Δt and the external parameters R_cam, t_cam to the following formula (2) to calculate the own movement amounts ΔR_ego, Δt_ego of the vehicle A. The own movement amount calculation unit 46 outputs the calculated own movement amounts ΔR_ego, Δt_ego to the automatic driving ECU 50 in sequence.

[0057] [Number 2]

[0058] ΔR_ego=R_cam·ΔR

[0059] Δt_ego=R_cam·Δt+t_cam...Equation (2)

[0060] The following is based on Figure 6 The flowchart shown in Figure 1 , Figure 3 as well as Figure 5 The above method of estimating the amount of movement for realizing the estimation of the amount of movement itself is described in detail. The power supply of the power supply device 60 to each component of the automatic driving system 110 is implemented, and the image processing ECU 30 starts after the initial startup process etc. is completed. Figure 6 The movement amount estimation process shown.

[0061] In S101, the internal parameters and external parameters related to the distance measuring sensor 10 and the external camera 20 are read from the storage unit 33, and the values ​​of each parameter are set in the referenceable area of ​​the RAM 32, and then the process proceeds to S102. In S102, the process of acquiring image data from one of the distance measuring sensor 10 and the external camera 20 is implemented, and then the process proceeds to S103. In S102, the reflected light image ImR and the background light image ImL captured at substantially the same time are acquired from the distance measuring sensor 10, or the camera image ImC is acquired from the external camera 20.

[0062] In S103, it is determined whether the image acquired in S102 is the sensor image ImS of the distance measuring sensor 10. If it is determined in S103 that the camera image ImC is acquired from the external camera 20, the process proceeds to S105. On the other hand, if it is determined in S103 that the sensor image ImS is acquired from the distance measuring sensor 10, the process proceeds to S104.

[0063] In S104, matching processing is performed on the reflected light image ImR and the background light image ImL generated in the same coordinate system to obtain the distance value of the reflected light image ImR corresponding to each pixel position of the background light image ImL. In S104, sensor element information including the distance value and brightness value of the feature point Fpa related to the subject PO is prepared, and the process proceeds to S105.

[0064] In S105, the sensor element information prepared using the background light image ImL and the reflected light image ImR and the camera pixel information prepared using the camera image ImC are compared. When the camera image ImC is acquired in the immediately preceding S102, the sensor element information prepared in the previous S104 is read out and compared with the camera element information prepared based on the current camera image ImC. On the other hand, when the sensor image ImS is acquired in the immediately preceding S102, the sensor element information prepared in the immediately preceding S104 is compared with the camera element information based on the latest camera image ImC acquired in the previous S102. In S105, the brightness values ​​of each element information based on the above-mentioned formula (1) are compared by using the camera element information as a reference, and the movement amounts ΔR and Δt are estimated, and the process proceeds to S106.

[0065] In S106, based on the above formula (2), the external parameter is removed from the movement amount estimated in S105 to calculate the own movement amount ΔR_ego and Δt_ego of vehicle A. The latest own movement amount estimated in S107 is output to the external automatic driving ECU 50, etc., and the process proceeds to S107.

[0066] In S107, it is determined whether the automatic driving system 110 has been turned off. In S107, if it is determined that the system is continuing to operate, the process returns to S102 to continue the estimation of the self-movement. On the other hand, if it is determined that the system has been turned off in S107, the process proceeds to S108. Then, in S108, the sensor package 100 is terminated, and a series of movement estimation processes are terminated.

[0067] In the first embodiment described so far, the estimation of the movement amount of the estimated object ET uses not only the reflected light image ImR but also the background light image ImL and the camera image ImC. Based on the above, the estimation of the movement amount can be implemented at a higher frame rate compared to the case where the movement amount is estimated only based on the reflected light image ImR. Therefore, it is easy to ensure the estimation accuracy of the movement amount.

[0068] In addition, in the first embodiment, not only the shape information of the subject PO based on the distance value of the reflected light image ImR, but also the texture information of the subject PO based on the brightness value of the background light image ImL and the camera image ImC is used for the movement estimation. Based on the above, even in a scene where there are few stationary three-dimensional structures in each measurement range MA1, MA2, the image processing ECU 30 can estimate the movement amount of the estimated object ET with good accuracy.

[0069] In the first embodiment, the vehicle A equipped with the light receiving element 12a and the camera element 22a is the estimation target ET, and the own movement amount calculation unit 46 estimates the own movement amount of the vehicle A. As described above, due to the improvement of the frame rate by the combination of the sensor image ImS and the camera image ImC, even if the estimation target ET is the vehicle A, it is easy to ensure high estimation accuracy of its own movement amount.

[0070] In the first embodiment, the shutter start time Tf for generating the reflected light image ImR and the background light image ImL is offset from the shutter start time Te of the camera image ImC. Therefore, the sensor image ImS and the camera image ImC, which are continuous in time, become images acquired in a state where the positions of the estimated object ET are different from each other. Based on the above, the effect of improving the estimation accuracy of the movement amount brought about by the increase in the frame rate can be more reliably exerted.

[0071] In addition, in the first embodiment, the entire frame period in the light receiving element 12a is offset relative to the entire exposure period in the camera element 22a. As described above, if the frame period and the exposure period are offset greatly so that these periods do not overlap, the image acquisition interval of the image processing ECU 30 becomes stable. As a result, the accuracy of the estimated movement amount is also suppressed from changing and can be easily and stably maintained at a high state.

[0072] In the first embodiment, the sensor element information related to the photographed object PO prepared using the sensor image ImS is compared with the camera element information related to the photographed object PO prepared using the camera image ImC. As described above, by not comparing the values ​​of the pixels of the entire image data, but limiting the comparison to the pixels related to the photographed object PO, the image processing module 42 can estimate the movement amount at a high speed.

[0073] In the first embodiment, the sensor element information is converted into information that can be compared with the camera element information by applying the conversion function W. Then, the movement amount estimation unit 45 uses the camera element information as a reference, compares the brightness value included in the sensor element information and the brightness value included in the camera element information, and estimates the movement amount. As described above, the difference calculation between images for estimating the movement amount mainly uses the brightness value commonly included in each image ImS, ImC. In other words, the process of generating a value for comparison with the distance value of the sensor element information based on the camera element information is not performed. As a result, the calculation load required for the movement amount estimation process can be reduced.

[0074] In addition, in the first embodiment, the determination of the subject PO and the feature point Fpa in the sensor image ImS uses not only the brightness value of the background light image ImL but also the distance value of the reflected light image ImR. Based on the above, even if the sensor image ImS has a relatively low resolution relative to the camera image ImC, the position of the pixel in the image as the comparison object of the brightness value can be set with good accuracy in the sensor image ImS. As a result, it is also easy to ensure the accuracy of the movement amount ΔR and Δt estimated by the gradient method.

[0075] In the first embodiment, the distance measuring sensor 10 corresponds to the "distance measuring device", the external camera 20 corresponds to the "camera device", and the image processing ECU 30 corresponds to the "calculation device" and the "movement amount estimation device". In addition, the processing unit 31 corresponds to the "processor", the image processing module 42 corresponds to the "estimation unit", the vehicle A corresponds to the "moving body", and the sensor package 100 corresponds to the "movement amount estimation system".

[0076] <Second embodiment>

[0077] exist Figure 7 In the second embodiment shown, the shooting period of the camera image ImC in the external camera 20 is different from the shooting period of the sensor image ImS in the distance measuring sensor 10. Specifically, the frame rate of the camera image ImC is set to be higher than the frame rate of the sensor image ImS. As an example, in the second embodiment, the frame rate of the camera image ImC is about three times that of the sensor image ImS. Moreover, during the shooting timing STs of the sensor image ImS, that is, the time T1 and T5, three shooting timings STc of the camera image ImC are generated (refer to the time T2, T3, and T4). Figure 3 as well as Figure 7 As shown, the image acquisition unit 41 sequentially acquires a sensor image ImS at time T1 , camera images ImC at times T2 to T4 , and a sensor image ImS at time T5 from the distance measuring sensor 10 and the external camera 20 .

[0078] In addition to the movement amount estimation by combining the sensor image ImS and the camera image ImC, the image processing module 42 also performs the movement amount estimation using a plurality of camera images ImC that are continuous in time. Specifically, when the image processing module 42 acquires the camera image ImC at time T2 to T4, etc., it combines the camera image ImC at each time with the latest (time T1) sensor image ImS to perform the movement amount estimation. In addition, at time T3 and T4, the image processing module 42 combines the latest camera image ImC at each time with the camera image ImC of the previous frame (time T2 and T3) to perform the movement amount estimation.

[0079] When the image processing module 42 acquires the sensor image ImS, it can independently compare the sensor image ImS with the plurality of camera images ImC to estimate the amount of movement. Specifically, when the image processing module 42 acquires the sensor image ImS at time T5, it compares the sensor image ImS with the camera image ImC at time T3 in addition to the camera image ImC at time T4. The image processing module 42 integrates the plurality of comparison results to estimate the amount of movement.

[0080] In the second embodiment described so far, the same effect as the first embodiment is achieved, and estimation accuracy can be ensured by estimating the amount of movement at a higher frame rate. In addition, in the second embodiment, when it is considered that the camera image ImC and the sensor image ImS constitute a plurality of frames, not only brightness value comparison between temporally adjacent frames but also brightness value comparison between skipped frames is performed. By adopting such processing, it is easier to ensure the estimation accuracy of the amount of movement.

[0081] <Other embodiments>

[0082] As mentioned above, although several embodiments of the present disclosure have been described, the present disclosure is not limited to the above-mentioned embodiments and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0083] In the first modification of the above embodiment, the frame rate of the distance measuring sensor 10 is set higher than the frame rate of the external camera 20. In the second modification, the frame rates of the distance measuring sensor 10 and the external camera 20 are made variable. As in the first and second modifications, the frame rates of the distance measuring sensor 10 and the external camera 20 can be changed as appropriate.

[0084] In addition, the shutter start time Tf of the sensor image ImS and the shutter start time Te of the camera image ImC may not be completely offset at all shooting timings STs and STc. In other words, when the frame rates of the ranging sensor 10 and the external camera 20 are different from each other, the shutter start times Tf and Te may coincide with each other at a specific timing. The image processing module 42 of the modified example 3 has a function of grasping the shutter start time Tf and Te of each image ImS and ImC, and terminating the comparison of the brightness values ​​between these images when the shutter start times Tf and Te are substantially the same.

[0085] Here, the entire frame period and the entire exposure period are shifted in front and back, and the sensor image ImS and the camera image ImC whose end time of one of the frame period and the exposure period is close to the start time of the other are the image groups suitable for motion estimation. When the frame rates are different from each other, the image processing module 42 can select the image group that meets the above conditions and perform motion estimation.

[0086] In the fourth variation of the above embodiment, the distance measuring sensor 10 and the external camera 20 are fixed to a stationary structure. The image processing module 42 uses the subject PO detected in both the sensor image ImS and the camera image ImC as the estimated object ET, and estimates the movement amount of the subject PO. As in the fourth variation above, the estimated object ET is not limited to the moving body such as the vehicle A itself. In addition, the distance measuring sensor 10 and the external camera 20 may also be set on a non-moving structure.

[0087] The image processing module 42 of the modified example 5 correctly grasps the shutter start time Tf and frame period of the distance measuring sensor 10, and the shutter start time Te and exposure period of the external camera 20. After estimating its own translational movement amount Δt_ego, the image processing module 42 uses the difference time between the shutter start times Tf and Te to calculate the moving speed of the vehicle A as a moving body. That is, the image processing module 42 may also have a moving speed estimation function in addition to the moving amount estimation function. In addition, in the case of not having a moving speed calculation function, the image processing module 42 may also incorrectly grasp the corresponding relationship between each shutter start time Tf and Te.

[0088] The image processing module 42 of the sixth modification performs a calculation for accumulating the amount of movement from a specific reference point (eg, the movement start point) to estimate the current position of the vehicle A. That is, the image processing module 42 may have a position estimation function in addition to the movement amount estimation function.

[0089] In the above embodiment, the pixel information of the reflected light image ImR and the background light image ImL captured at substantially the same time is combined to generate sensor element information including distance value and brightness value. Then, the movement amount is estimated by converting such sensor element information using the conversion function W and comparing it with the camera pixel information.

[0090] In contrast, in the image processing module 42 of the modified example 7 of the above-mentioned embodiment, the comparison between the reflected light image ImR and the camera image ImC, and the comparison between the background light image ImL and the camera image ImC are independently implemented. Then, the image processing module 42 estimates the movement amount by integrating the various comparison results. In this way, the specific processing steps for estimating the movement amount using the reflected light image ImR, the background light image ImL, and the camera image ImC can be appropriately changed. In addition, the reflected light image ImR and the background light image ImL may not be images actually captured at the same time. In addition, data processing such as applying a conversion function may be implemented on the brightness value of the camera image ImC so that it can be compared with the distance value of the reflected light image ImR. That is, a movement amount estimation process based on the sensor image ImS may also be implemented.

[0091] In variant 8 of the above-mentioned embodiment, the distance measuring sensor 10 and the external camera 20 constitute an integrated sensor unit. In variant 9, the sensor unit is integrated with the image processing ECU 30 to constitute a sensor package unit. In variant 10, the functions of the image processing ECU 30 and the automatic driving ECU 50 are centrally installed in an integrated control ECU. In variant 11, at least a part of the processing related to the movement amount estimation of the image processing ECU 30 is executed by a computer set on the network. In such variant 11, the result of the movement amount estimation is provided to the automatic driving ECU 50 of vehicle A via the network.

[0092] In the modification 12 of the above-mentioned embodiment, a plurality of distance measuring sensors 10 or a plurality of external cameras 20 sequentially output image data to the image processing ECU 30. The image processing ECU 30 can appropriately combine images containing a common subject PO from the acquired image data and perform movement estimation. If the same subject PO can be detected by movement as in such a modification 12, the respective measurement ranges may not necessarily overlap. In addition, the respective measurement ranges of the distance measuring sensor 10 and the external camera 20 are not limited to the front range of the vehicle A, but may also be the side range or the rear range.

[0093] The distance measuring sensor 10 of modification example 13 is a lidar device on the non-scanning side without movable optical components such as a polygonal mirror. In addition, the reflected light image ImR and the background light image ImL can also be captured using a global shutter method. In addition, the camera image ImC output by the external camera 20 of modification example 14 is not a color image, but a grayscale image. Such a camera image ImC can also be captured using a rolling shutter method. In the image processing ECU 30 of modification example 15, in addition to the distance measuring sensor 10 and the external camera 20, autonomous sensors such as a radar device and a sonar device are also electrically connected. The image processing ECU 30 can further combine the detection information of each device to estimate its own movement amount, etc.

[0094] In the above-mentioned embodiment, each function provided by the image processing ECU 30 can be provided by software and hardware that executes the software, only software, only hardware, or a combination of these. Furthermore, when such functions are provided by an electronic circuit as hardware, each function can also be provided by a digital circuit including a large number of logic circuits, or an analog circuit.

[0095] Furthermore, the form of the storage medium (non-transitory tangible storage medium) storing the movement amount estimation program that can realize the above-mentioned movement amount estimation method can be appropriately changed. For example, the storage medium is not limited to the configuration provided on the circuit substrate, but can also be provided in the form of a memory card, etc., inserted into the socket, and electrically connected to the control circuit of the image processing ECU 30. Furthermore, the storage medium can also be an optical disk or a hard disk that is a copy source of the program of the image processing ECU 30.

[0096] The control unit and method described in the present disclosure may also be implemented by a dedicated computer that is programmed to execute one or more functions embodied by a computer program. Alternatively, the device and method described in the present disclosure may also be implemented by a dedicated hardware logic circuit. Alternatively, the device and method described in the present disclosure may also be implemented by one or more dedicated computers that are composed of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may also be stored as an instruction executed by a computer in a non-migratable tangible recording medium that can be read by a computer.

Claims

1. A movement amount estimation device, wherein: have: an image acquisition unit that acquires a reflected light image including distance information obtained by detecting reflected light reflected from an object by light irradiation by a light receiving element, a background light image including brightness information obtained by detecting background light relative to the reflected light by the light receiving element, and a camera image captured by a camera element different from the light receiving element; as well as an estimating unit, which estimates a movement amount of an estimated object using information of the object detected in common in the reflected light image, the background light image, and the camera image, The estimation unit estimates the movement amount of the subject by comparing the brightness value included in the sensor element information related to the subject prepared using the reflected light image and the background light image with the brightness value included in the camera element information related to the subject prepared using the camera image.

2. The movement amount estimation device according to claim 1, wherein: The estimation unit estimates a movement amount of a moving object provided with the light receiving element and the camera element as the estimation object.

3. The movement amount estimation device according to claim 1 or 2, wherein: A start time of a detection period in which the light receiving element performs detection for generating the reflected light image and the background light image is offset from a start time of an exposure period in which the camera element performs exposure for capturing the camera image.

4. The movement amount estimation device according to claim 3, wherein: The entire detection period in the light receiving element is shifted relative to the entire exposure period in the camera element.

5. The movement amount estimation device according to claim 1 or 2, wherein: The information of the object includes information of a plurality of feature points related to the object.

6. The movement amount estimation device according to claim 5, wherein: The estimation unit calculates a motion vector of the object by tracking the plurality of feature points, and estimates a motion amount of the estimation object based on a motion amount of the object on the image.

7. A method for estimating movement amount, wherein: The processing performed by at least one processor includes: Acquire a reflected light image including distance information obtained by detecting reflected light reflected from an object by light irradiation by a light receiving element, a background light image including brightness information obtained by detecting background light relative to the reflected light by the light receiving element, and a camera image captured by a camera element different from the light receiving element; as well as using information of the subject detected in common in the reflected light image, the background light image, and the camera image, estimating the movement amount of the estimated object; In the estimation process, the movement amount of the above-mentioned subject is estimated by comparing the brightness value contained in the sensor element information related to the above-mentioned subject prepared using the above-mentioned reflected light image and the above-mentioned background light image with the brightness value contained in the camera element information related to the above-mentioned subject prepared using the above-mentioned camera image.

8. A storage medium storing a movement amount estimation program, wherein: The movement amount estimation program causes at least one processor to execute the following processing: Acquire a reflected light image including distance information obtained by detecting reflected light reflected from an object by light irradiation by a light receiving element, a background light image including brightness information obtained by detecting background light relative to the reflected light by the light receiving element, and a camera image captured by a camera element different from the light receiving element; as well as using information of the subject detected in common in the reflected light image, the background light image, and the camera image, estimating the movement amount of the estimated object; In the estimation process, the movement amount of the above-mentioned subject is estimated by comparing the brightness value contained in the sensor element information related to the above-mentioned subject prepared using the above-mentioned reflected light image and the above-mentioned background light image with the brightness value contained in the camera element information related to the above-mentioned subject prepared using the above-mentioned camera image.

9. A movement amount estimation system, wherein: have: a distance measuring device for generating a reflected light image including distance information obtained by detecting reflected light reflected from an object by light irradiation through a light receiving element, and a background light image including brightness information obtained by detecting background light relative to the reflected light through the light receiving element; A camera device that generates a camera image by photographing with a camera element different from the above-mentioned light receiving element; as well as The computing device estimates the movement amount of the estimated object using the information of the subject detected in the reflected light image, the background light image and the camera image. In the computing device, the amount of movement of the subject is estimated by comparing the brightness value contained in the sensor element information related to the subject prepared using the reflected light image and the background light image with the brightness value contained in the camera element information related to the subject prepared using the camera image.

Images