Synchronization device, synchronization method, and storage medium

Abstract

The synchronization device (1) comprises: a synchronization prediction unit (180) for predicting a synchronization offset time (T) between a radar sensor (22) and an external camera (24); a variation calculation unit (120) for calculating a time variation (ΔR) of a reflection wave image (Ir) between shutter frames (Fo) of the external camera (24); an image estimation unit (130) for estimating a reflection wave image (Ir) at a synchronization timing (ts) after the start timing of the shutter frame (Fo) is offset by the synchronization offset time (T) based on the time variation (ΔR); a difference extraction unit (140) for extracting a difference (E) by comparing the reflection wave image (Ir) at the synchronization timing (ts) with the external light image (Io); and a processing determination unit (160) for determining whether to return the synchronization processing to the prediction of the synchronization offset time (T) based on the difference (E).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on patent application No. 2019-185388 filed in Japan on October 8, 2019, and the contents of the basic application are cited by reference in their entirety. Technical Field

[0003] The present disclosure relates to synchronization processing technology. Background Art

[0004] In recent years, vehicles are equipped with a radar sensor that obtains a reflection wave image corresponding to a reflection wave obtained by irradiation with electromagnetic waves, and an external camera that obtains an external light image corresponding to external light sensed from the outside world. In order to synchronize the radar sensor and the external camera in such a vehicle, Patent Document 1 discloses a technology for adding time information to each obtained image.

[0005] Patent Document 1: Japanese Patent No. 3920769

[0006] However, in the disclosed technology of Patent Document 1, a delay of the image signal occurs on the path between the radar sensor and the external camera and the processing circuit that performs synchronization processing. As a result, there is a concern that the time information added by the processing circuit receiving the image signal includes an error with the regular time when the radar sensor and the external camera actually acquired the image. Summary of the invention

[0007] The subject of the present disclosure is to provide a synchronization device that synchronizes a radar sensor with an external camera with high precision. Another subject of the present disclosure is to provide a synchronization method that synchronizes a radar sensor with an external camera with high precision. Another subject of the present disclosure is to provide a synchronization program that synchronizes a radar sensor with an external camera with high precision.

[0008] Hereinafter, technical means of the present disclosure for solving the problems will be described.

[0009] A first aspect of the present disclosure is a synchronization device for executing synchronization processing of the radar sensor and the external camera in a vehicle equipped with a radar sensor for acquiring a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave and an external camera for acquiring an external light image corresponding to external light sensed from the outside, the synchronization device comprising:

[0010] A synchronization prediction unit predicts the synchronization offset time between the radar sensor and the external camera;

[0011] A variation calculation unit calculates a time variation of a reflected wave image between shutter frames of an external camera;

[0012] an image estimating unit that estimates a reflected wave image at a synchronization timing that is shifted from a start timing of a shutter frame by a synchronization shift time predicted by a synchronization predicting unit, based on the time variation calculated by the variation calculating unit;

[0013] a difference extracting section that extracts a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the image estimating section; and

[0014] The process determination unit determines whether to return the synchronization process to the prediction based on the synchronization offset time by the synchronization prediction unit based on the difference calculated by the difference extraction unit.

[0015] A second aspect of the present disclosure is a synchronization method for executing synchronization processing of a radar sensor and an external camera in a vehicle equipped with a radar sensor for acquiring a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave and an external camera for acquiring an external light image corresponding to external light sensed from the outside, including:

[0016] Prediction process, predicting the synchronization offset time between the radar sensor and the external camera;

[0017] A calculation step, calculating a time variation of a reflection wave image between shutter frames of an external camera;

[0018] an estimating step of estimating a reflected wave image at a synchronization timing shifted from a start timing of a shutter frame by a synchronization shift time predicted by the predicting step based on the time variation calculated by the calculating step;

[0019] an extraction step of extracting a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the estimation step; and

[0020] The determination step determines, based on the difference extracted in the extraction step, whether to return the synchronization process to the prediction of the synchronization deviation time in the prediction step.

[0021] A third aspect of the present disclosure is a synchronization program stored in a storage medium and including commands for causing a processor to execute, the synchronization program being used to execute synchronization processing of the radar sensor and the external camera in a vehicle equipped with a radar sensor for acquiring a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave and an external camera for acquiring an external light image corresponding to external light sensed from the outside world,

[0022] The commands include:

[0023] Prediction process, predicting the synchronization offset time between the radar sensor and the external camera;

[0024] A calculation step, calculating a time variation of a reflection wave image between shutter frames of an external camera;

[0025] an estimating step of estimating a reflected wave image at a synchronization timing shifted from a start timing of a shutter frame by a synchronization shift time predicted by the predicting step based on the time variation calculated by the calculating step;

[0026] an extraction step of extracting a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the estimation step; and

[0027] The determination step determines, based on the difference extracted in the extraction step, whether to return the synchronization process to the prediction of the synchronization deviation time in the prediction step.

[0028] According to the synchronization processing of these first to third forms, the reflected wave image at the synchronization timing after the synchronization offset time between the radar sensor and the external camera is offset from the start timing of the frame is estimated based on the time change amount of the reflected wave image between the shutter frames in the external camera. The more accurate the prediction of the synchronization offset time is, the smaller the difference extracted by comparing the reflected wave image at the synchronization timing with the external light image is. Therefore, by determining whether to return the synchronization processing to the prediction of the synchronization offset time based on the difference between the reflected wave image and the external light image, the synchronization processing can be repeated until the synchronization offset time can be correctly predicted. Therefore, the radar sensor and the external camera can be synchronized with high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a block diagram showing the overall configuration of the synchronizing device according to the first embodiment.

[0030] Figure 2 This is a block diagram showing the detailed configuration of the synchronization device according to the first embodiment.

[0031] Figure 3 It is a schematic diagram for explaining the outside camera (a) and the radar sensor (b) of the first embodiment.

[0032] Figure 4 It is a schematic diagram for explaining the outside world image (a) and the reflected wave image (b) according to the first embodiment.

[0033] Figure 5 It is a schematic diagram for explaining the motion compensation block according to the first embodiment.

[0034] Figure 6 It is a schematic diagram for explaining the motion compensation block according to the first embodiment.

[0035] Figure 7It is a schematic diagram for explaining the variation calculation block and the image estimation block according to the first embodiment.

[0036] Figure 8 It is a schematic diagram for explaining the difference extraction block of the first embodiment.

[0037] Fig. 9 It is a schematic diagram for explaining the difference extraction block of the first embodiment.

[0038] Fig.10 This is a flowchart showing the synchronization method according to the first embodiment.

[0039] Fig.11 It is a block diagram showing a detailed configuration of a synchronizing device according to the second embodiment.

[0040] Fig.12 : is a flowchart showing the synchronization method according to the second embodiment. DETAILED DESCRIPTION

[0041] Hereinafter, a plurality of embodiments are described based on the accompanying drawings. In addition, sometimes, by attaching the same reference numerals to the corresponding constituent elements in each embodiment, repeated descriptions are omitted. In addition, when only a part of the composition is described in each embodiment, the other parts of the composition can apply the composition of other embodiments described previously. Moreover, not only the combination of the composition explicitly stated in the description of each embodiment, but also the composition of a plurality of embodiments can be partially combined with each other even if it is not explicitly stated, if the combination does not particularly cause an obstacle.

[0042] (First Embodiment)

[0043] like Figure 1 As shown, the synchronization device 1 of the first embodiment is mounted on a vehicle 3. The vehicle 3 is a highly-assisted vehicle or an automatic driving vehicle that travels based on the result of motion estimation such as self-position estimation. In addition, in the following description, the lateral direction and the front-back direction of the vehicle 3 on the horizontal plane are simply referred to as the lateral direction and the front-back direction, respectively. In addition, in the following description, the vertical direction of the vehicle 3 on the horizontal plane is simply referred to as the vertical direction.

[0044] The vehicle 3 is equipped with a sensor system 2 together with the synchronization device 1. Figure 1 , 2 As shown, the sensor system 2 includes at least an inertial sensor 20 , a radar sensor 22 , and an external camera 24 .

[0045] The inertial sensor 20 is, for example, a so-called IMU (Inertial Measurement Unit) that acquires inertial information Ib that can be used for motion estimation of the vehicle 3. The inertial sensor 20 acquires at least one of angles, angular velocities, and angular accelerations about three axes of the vehicle 3, namely, the front-rear direction, the lateral direction, and the vertical direction, as the inertial information Ib. Therefore, the inertial sensor 20 is configured to include, for example, at least one of a gyroscope and an acceleration sensor.

[0046] The radar sensor 22 of the first embodiment is, for example, a so-called LIDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging) that acquires a reflected wave image Ir that can be used for motion estimation of the vehicle 3, etc. The radar sensor 22 acquires a reflected wave image Ir corresponding to a reflected wave obtained by irradiating an electromagnetic wave to the outside of the vehicle 3. Figure 2 As shown, the radar sensor 22 includes a laser element 220 , an imaging element 221 , and an imaging circuit 222 .

[0047] The imaging circuit 222 controls the infrared laser (hereinafter, simply referred to as laser) as an electromagnetic wave from the laser element 220 toward the outside of the vehicle 3, so that it becomes a pulse beam shape that is intermittent according to the change of the beam steering angle in the lateral direction. Accompanying the laser irradiation, the imaging circuit 222 implements a rolling shutter, which sequentially exposes and sequentially scans the longitudinal pixel columns in the lateral direction of a single column or multiple columns in the constituent pixels of the imaging element 221 according to each scanning line corresponding to the beam steering angle driven in the lateral direction. The imaging circuit 222 acquires the reflected wave image Ir by associating the distance value to the reflection point of the reflected light corresponding to the arrival time from the moment of laser irradiation of the reflected light sensed as a reflected wave according to each exposed pixel of the imaging element 221 with the beam steering angle corresponding to each pixel and digitizing it.

[0048] Here, the brightness value corresponding to the intensity of the reflected light as the reflected wave sensed by each exposed pixel of the imaging element 221 may be included in the reflected wave image Ir by associating it with the beam steering angle corresponding to each pixel and digitizing it by the imaging circuit 222. In addition, the imaging element 221 may also have a function of taking pictures according to the external light sensed from the outside of the vehicle 3 during the intervals of intermittent laser irradiation. In this case, the brightness value corresponding to the intensity of the external light sensed by each exposed pixel of the imaging element 221 may be included in the reflected wave image Ir by associating it with the beam steering angle corresponding to each pixel and digitizing it by the imaging circuit 242.

[0049] The external camera 24 is, for example, a so-called vehicle-mounted camera that acquires an external light image Io that can be used for motion estimation of the vehicle 3. The external camera 24 acquires an external light image Io corresponding to external light perceived from outside the vehicle 3. The external camera 24 includes an imaging element 241 and an imaging circuit 242.

[0050] The imaging circuit 242 implements a global shutter that collectively exposes and collectively scans a plurality of vertical pixel columns in the horizontal direction among the constituent pixels of the imaging element 241 using the corresponding scanning lines. The imaging circuit 242 acquires the external light image Io by associating and digitizing a brightness value corresponding to the intensity of external light sensed by each exposed pixel of the imaging element 241 with a pixel angle corresponding to each pixel.

[0051] In the global shutter frame Fo for scanning all the scanning lines in the external camera 24, a shorter frame period and a closer start timing are set than in the rolling shutter frame Fr for scanning all the scanning lines in the radar sensor 22. Figure 3 As shown, when the start timing of frames Fo and Fr are correctly synchronized, Figure 4 As shown, in appearance, compared with the external light image Io, the reflected wave image Ir is compressed on the side opposite to the traveling side of the vehicle 3 and the outside world corresponding to the compression amount is additionally observed and acquired. Figure 4 The images Ir and Io themselves are not shown, but the external environment captured by the images Ir and Io is schematically shown.

[0052] like Figure 1As shown, the synchronization device 1 of the first embodiment is connected to the components 20, 22, and 24 of the sensor system 2 via at least one of a LAN (Local Area Network), a wiring harness, and an internal bus. The synchronization device 1 is configured to include at least one dedicated computer. The dedicated computer constituting the synchronization device 1 may be an ECU (Electronic Control Unit) of a locator that estimates the own position of the vehicle 3. The dedicated computer constituting the synchronization device 1 may also be an ECU that controls the advanced driving assistance or automatic driving of the vehicle 3. The dedicated computer constituting the synchronization device 1 may also be an ECU that controls the communication between the vehicle 3 and the outside world.

[0053] The synchronization device 1 is constructed to include such a dedicated computer, each having at least one memory 10 and a processor 12. The memory 10 is a non-transitorytangible storage medium (non-transitorytangible storage medium) that non-temporarily stores programs and data that can be read by the computer, such as at least one of a semiconductor memory, a magnetic medium, and an optical medium. The processor 12 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a RISC (Reduced Instruction Set Computer)-CPU as a core. The processor 12 executes a plurality of commands included in the synchronization program stored in the memory 10. Thus, the synchronization device 1 is as follows. Figure 2 As shown, a plurality of functional blocks are constructed for executing synchronization processing of the radar sensor 22 and the outside camera 24. In the following description, the synchronization processing of the radar sensor 22 and the outside camera 24 is simply referred to as synchronization processing.

[0054] The plurality of functional blocks constructed by the synchronization device 1 include an error correction block 100 , a motion compensation block 110 , a variation calculation block 120 , an image estimation block 130 , a difference extraction block 140 , a processing decision block 160 , and a synchronization prediction block 180 .

[0055] The error correction block 100 receives the latest inertial information Ib acquired by the inertial sensor 20 at time k. At the same time, the error correction block 100 receives the latest bias error B estimated at time k by at least one of various navigation operations. Here, time k represents the start timing of the global shutter frame Fo (refer to Figure 3 ).

[0056] In response to such input, the error correction block 100 corrects the latest inertial information Ib by the latest bias error B. This correction is achieved by subtracting the bias error B from the inertial information Ib.

[0057] exist Figure 2 The motion compensation block 110 shown in the figure is input with the inertial information Ib at time k after the bias error B is corrected by the error correction block 100. At the same time, the motion compensation block 110 is input with the latest reflected wave image Ir acquired by the radar sensor 22 in the global shutter frame Fo with time k as the start timing. Figure 5 As shown, the latest reflected wave image Ir includes, for each pixel of each scanning line, a distance value Lr[k, i, j], a beam steering angle θr[k, i, j], and ψr[k, i, j] as the latest data values.

[0058] Here, [k, i, j] is an index for determining the latest data value Lr, θr, ψr based on the time k, the scan number i indicating the arrangement order and scan order of each scan line, and the pixel number j indicating each pixel of each scan line. That is, Lr[k, i, j], θr[k, i, j], and ψr[k, i, j] are the time h (refer to) of the following formula 1. Figure 3 ) is input to the motion compensation block 110 as the latest data value of each scan line of the scan timing.

[0059] [Formula 1]

[0060]

[0061] Here, the above formula 1 uses the time k, the scan number i, and the time interval Δh between scan lines (refer to Figure 5 ) approximately defines the time h after the synchronization offset time T further considering the start time of the frames Fo and Fr. Therefore, Figure 6 As shown, θr[k, i, j] represents the beam steering angle in the vertical direction for the laser beam sensed at time h by specific pixels numbered i and j in the imaging element 221 after being irradiated by the laser element 220. On the other hand, ψr[k, i, j] represents the beam steering angle in the lateral direction for the laser beam sensed at time h by specific pixels numbered i and j in the imaging element 221 after being irradiated by the laser element 220.

[0062] In response to such input, the motion compensation block 110 compensates the latest data value of the reflected wave image Ir based on the inertial information Ib corrected for the bias error B, based on the motion of the vehicle 3 in the global shutter frame Fo (see Figure 3Specifically, the motion compensation block 110 calculates the own position P[k, i] and the direction cosine matrix C[k, i] of the vehicle 3 at time k by at least one of the various navigation operations based on the inertial information Ib with the bias error B corrected.

[0063] Here, [k, i] is an index that determines the calculation results P and C based on the time k and the scan number i. That is, P[k, i] and C[k, i] are calculated in common for the pixels of the same scan line. Therefore, the motion compensation block 110 compensates for the movement amount of the vehicle 3 that changes for each scan number i in the global shutter frame Fo by the following equation 2 using the calculation results P[k, i] and C[k, i] for the latest data values ​​Lr[k, i, j], θr[k, i, j], and ψr[k, i, j]. As a result, the motion compensation block 110 calculates the three-dimensional polar coordinates R[k, i, j] of the following equation 2 for each pixel of each scan line. At this time, the calculated three-dimensional polar coordinates R[k, i, j] are associated with the time k as the latest data value of the reflected wave image Ir compensated for the movement amount of the vehicle 3 and stored in the memory 10. In addition, in the following description, the movement amount of the vehicle 3 in the global shutter frame Fo is simply referred to as the vehicle movement amount.

[0064] [Formula 2]

[0065]

[0066] exist Figure 2 The variation calculation block 120 shown is input (i.e., reads) the latest and last reflected wave images Ir that are compensated for the vehicle movement amount by the motion compensation block 110 and stored in the memory 10. Receiving this input, the variation calculation block 120 calculates the time variation ΔR[k, i, j] of the reflected wave image Ir between the global shutter frames Fo. Specifically, the variation calculation block 120 compares the latest and last reflected wave images Ir obtained in the global shutter frames Fo before and after the start timing, respectively, in the three-dimensional polar coordinates R shared by the index other than the time, using the following formula 3. As a result, the variation calculation block 120 calculates the time variation ΔR[k, i, j] that becomes the latest differential value at the time k for each pixel.

[0067] [Formula 3]

[0068] ΔR[k,i,j]=R[k,i,j]-R[k-1,i,j]

[0069] Here, the above formula 3 uses the coordinates R[k, i, j] in the global shutter frame Fo starting at the current time k and the coordinates R[k-1, i, j] in the global shutter frame Fo starting at the previous time k-1, and defines the change amount ΔR[k, i, j] according to the difference between these coordinates. Therefore, the change amount calculation block 120 uses the three-dimensional polar coordinates R[k, i, j] and R[k-1, i, j] calculated by the correction and compensation of the previous modules 100 and 110 in the current and previous synchronization processes. However, only in the initial synchronization process from the start of the driving of the vehicle 3, the change amount calculation block 120 uses the three-dimensional polar coordinates R[k-1, i, j] calculated by the correction and compensation of the modules 100 and 110 before the initial synchronization process together with the initial three-dimensional polar coordinates R[k, i, j].

[0070] exist Figure 2 The image estimation block 130 shown is input with the latest time variation ΔR[k, i, j] calculated by the variation calculation block 120. At the same time, the image estimation block 130 is input (i.e., read out) with the latest reflected wave image Ir compensated for the vehicle movement by the motion compensation block 110 and stored in the memory 10. Furthermore, the variation calculation block 120 is input (i.e., read out) with the latest synchronization offset time T predicted by the synchronization prediction block 180 and stored in the memory 10 as described in detail later.

[0071] Receiving such input, the image estimation block 130 sets the start timing of the global shutter frame Fo to time k, and sets the timing that deviates from time k by the latest synchronization offset time T in the direction of its sign (i.e., time k+T) to Figure 7 The synchronization timing ts shown. In addition, Figure 7 The reflected wave image Ir itself is not shown when the sign of the synchronization offset time T is positive (+), but the external range captured by the image Ir is schematically shown. Therefore, the image estimation block 130 estimates the reflected wave image Ir at the synchronization timing ts based on the time change ΔR[k, i, j] at the time k. Specifically, the image estimation block 130 converts the three-dimensional polar coordinates R[k, i, j] in the latest reflected wave image Ir using the following formula 4 using the time change ΔR[k, i, j] and the synchronization offset time T. As a result, the image estimation block 130 calculates the estimated polar coordinates Rp[k, i, j] of the reflected wave image Ir at the synchronization timing ts for each pixel.

[0072] [Formula 4]

[0073]

[0074] Here, the above formula 4 estimates the coordinates Rp[k,i,j] at the synchronization timing ts using the coordinates R[k,i,j] in the global shutter frame Fo starting at this time k, the change amount ΔR[k,i,j] at the time k, the synchronization offset time T, and the frame period Δk of the frame Fo. Therefore, the image estimation block 130 generates the reflected wave image Ir including the estimated polar coordinates Rp[k,i,j] as the estimated data value at the synchronization timing ts.

[0075] like Figure 2 As shown, the difference extraction block 140 includes sub-blocks 142 and 144 for pre-processing the reflected wave image Ir and the external light image Io, respectively, and a sub-block 146 for performing a comparison process on the pre-processed images Ir and Io.

[0076] The sub-block 142 in the difference extraction block 140 is input with the own position P[k, i] and the direction cosine matrix C[k, 1] corresponding to the time k in the direction cosine matrix C[k, i] calculated by the motion compensation block 110. At the same time, the reflected wave image Ir at the synchronization timing ts estimated by the image estimation block 130 is input to the sub-block 142.

[0077] Receiving such input, the sub-block 142 compensates the reflected wave image Ir for the offset from the start timing of the global shutter frame Fo, that is, time k. Specifically, the sub-block 142 converts the estimated polar coordinates Rp[k, i, j] of each pixel calculated by the image estimation block 130 as the estimated data value of the reflected wave image Ir at the synchronization timing ts into the three-dimensional orthogonal coordinates Rr[k, i, j] of the following formula 5.

[0078] [Formula 5]

[0079] Rr[k,i,j]=C T [k, 1] · Rp[k, i, j]

[0080] Here, the above equation 5 defines the three-dimensional orthogonal coordinates Rr[k,i,j] using the transposed matrix CT[k,1] of the direction cosine matrix C[k,1] at the start timing of the global shutter frame Fo, that is, at time k, and the estimated polar coordinates Rp[k,i,j] within the frame Fo. Therefore, the sub-block 142 further compensates the three-dimensional orthogonal coordinates Rr[k,i,j] of each pixel that becomes the latest data value of the reflected wave image Ir after the coordinate conversion by the offset from time k at time h using the following equation 6 using the own position P[k,1] at time k. As a result, the sub-block 142 calculates the relative orthogonal coordinates x, y, z of the following equation 6 as the relative position information when the laser reflection point corresponding to each pixel is observed from the vehicle 3 at time k.

[0081] [Formula 6]

[0082]

[0083] The sub-block 142 also converts the calculated relative orthogonal coordinates x, y, and z of each pixel using the following equations 7 to 9. As a result, the sub-block 142 calculates the distance value Lr[k, i, j], the beam steering angle θr[k, i, j], and ψr[k, i, j] as the latest data value of the reflected wave image Ir compensated for the offset in accordance with the time k.

[0084] [Formula 7]

[0085]

[0086] [Formula 8]

[0087]

[0088] [Formula 9]

[0089]

[0090] exist Figure 2 In the difference extraction block 140 shown in FIG. 1 , the beam steering angle θr[k, i, j] and ψr[k, i, j] in the latest data value of the reflected wave image Ir calculated by the sub-block 142 are input to the sub-block 144. At the same time, the latest external light image Io acquired by the external camera 24 in the global shutter frame Fo with the time k as the start timing is input to the sub-block 144. Figure 8 As shown, the latest external light image Io includes the brightness value So[k, m, n], the pixel angle θo[k, m, n] and ψo[k, m, n] as the latest data values ​​for each pixel of each scanning line.

[0091] Here, [k, m, n] is an index that determines the latest data value So, θo, ψo based on the time k, the scan number m indicating the arrangement order of each scan line, and the pixel number n indicating each pixel of each scan line. Here, the maximum value of the scan number m related to the external light image Io is greater than the maximum value of the scan number i related to the reflected wave image Ir, and the maximum value of the pixel number n related to the external light image Io is greater than the maximum value of the pixel number j related to the reflected wave image Ir. That is, the external light image Io is generated with a larger number of pixels than the reflected wave image Ir with high precision.

[0092] In addition, if Fig. 9As shown, θo[k, m, n] represents the pixel angle in the vertical direction for the external light perceived at time k by the specific pixels numbered m and n in the imaging element 241. On the other hand, ψo[k, i, j] represents the pixel angle in the horizontal direction for the external light perceived at time k by the specific pixels numbered m and n in the imaging element 241.

[0093] In response to such input, the sub-block 144 interpolates the high-definition side in coordination with the low-definition side in the reflected wave image Ir and the external light image Io. That is, the sub-block 144 makes the external light image Io on the high-definition side match the reflected wave image Ir on the low-definition side by interpolation. Specifically, the sub-block 144 performs the following steps: Figure 8 In this way, the beam steering angles θr[k, i, j] and ψr[k, i, j] with respect to each pixel are searched for pixel angles θo and ψo whose indices satisfy the following equations 10 and 11, respectively.

[0094] [Formula 10]

[0095] θo[k,m,n]<θr[k,i,j]≤θo[k,m+1,n]

[0096] [Formula 11]

[0097] Ψo[k,m,n]<Ψr[k,i,j]≤Ψo[k,m,n+1]

[0098] Here, the above equation 10 expresses the indices of pixels adjacent to each other in the horizontal direction as [k, m, n] and [k, m+1, n]. Under this expression, the above equation 10 uses θo[k, m, n] and θo[k, m+1, n] to define the search condition for the beam steering angle θr[k, i, j]. On the other hand, the above equation 11 expresses the indices of pixels adjacent to each other in the vertical direction as [k, m, n] and [k, m, n+1]. Under this expression, the above equation 11 uses the pixel angles θo[k, m, n] and θo[k, m, n+1] to define the search condition for the beam steering angle ψr[k, i, j]. Thus, the sub-block 144 is interpolated by the following equations 12 to 14 (refer to Figure 8 ), calculates the brightness value So[k, i, j] of the coordinates corresponding to the beam steering angles θr[k, i, j] and ψr[k, i, j] of each pixel on the external light image Io.

[0099] [Formula 12]

[0100]

[0101] [Formula 13]

[0102]

[0103] [Formula 14]

[0104]

[0105] Here, the above formula 12 expresses the indices of pixels adjacent to each other in the horizontal direction as [k, m, n] and [k, m+1, n]. Under this expression, the above formula 12 defines the variable S1 using the pixel angles θo[k, m, n] and θo[k, m+1, n], the corresponding brightness values ​​So[k, m, n] and So[k, m+1, n], and the beam steering angle θr[k, i, j].

[0106] On the other hand, the above formula 13 expresses the indices of pixels adjacent to each other in the vertical direction as [k, m, n] and [k, m, n+1]. In addition, when the above formula 13 expresses the indices of pixels adjacent to each other in the horizontal direction as [k, m, n] and [k, m+1, n], the indices of pixels adjacent to the latter [k, m+1, n] in the vertical direction are expressed as [k, m+1, n+1] (refer to Figure 8 ). In other words, in the above formula 13, when the indices of the pixels adjacent to each other in the vertical direction are expressed as [k, m, n] and [k, m, n+1], it can also be said that the indices of the pixels adjacent to the latter pixel [k, 1, n+1] in the horizontal direction are expressed as [k, m+1, n+1] (refer to Figure 8 ). Under these expressions, the above equation 13 uses the pixel angles θo[k, m, n+1] and θo[k, m+1, n+1], the corresponding brightness values ​​So[k, m, n+1] and So[k, m+1, n+1], and the beam steering angle θr[k, i, j] to define the variable S2.

[0107] Furthermore, the above formula 14 expresses the indices of pixels adjacent to each other in the vertical direction as [k, m, n] and [k, m, n+1]. Under this expression, the above formula 14 defines the brightness value So[k, i, j] of the interpolated external light image Io using the pixel angles ψo[k, m, n] and ψo[k, m, n+1], the variables S1 and S2 of the above formulas 12 and 13, and the beam steering angle ψr[k, i, j].

[0108] exist Figure 2In the difference extraction block 140 shown, the distance value Lr[k, i, j] is input to the sub-block 146 as the latest data value of the reflected wave image Ir calculated by the sub-block 142. Receiving this input, the sub-block 146 extracts the edge in the latest reflected wave image Ir by performing at least one of various filtering processes on the distance value Lr[k, i, j]. As a result, the sub-block 146 calculates the difference between the distance values ​​Lr[k, i, j] of the pixels adjacent to each other in the horizontal or vertical direction as the edge image value Lre[k, i, j] for each pixel corresponding to the extracted edge.

[0109] The sub-block 146 receives an input of the brightness value So[k, i, j] as the latest data value of the external light image Io interpolated by the sub-block 144. Receiving the input, the sub-block 146 extracts the edge in the latest external light image Io by performing at least one of various filtering processes on the brightness value So[k, i, j]. As a result, the sub-block 146 calculates the difference between the brightness values ​​So[k, i, j] of the pixels adjacent to each other in the horizontal or vertical direction as the edge image value Soe[k, i, j] for each pixel corresponding to the extracted edge.

[0110] The sub-block 146 extracts the difference E[k, i, j] between the images Io and Ir by comparing the reflected wave image Ir and the external light image Io for each corresponding pixel with respect to the edge image values ​​Lre[k, i, j] and Soe[k, i, j] calculated in this way. Specifically, the sub-block 146 calculates the difference E[k, i, j] between the image values ​​Lre[k, i, j] and Soe[k, i, j] having the same index with each other after normalizing them according to their respective deviation ranges. That is, the sub-block 146 compares the image values ​​Lre[k, i, j] and Soe[k, i, j] related to the edges of the reflected wave image Ir and the external light image Io with each other after normalizing them according to their respective deviation ranges.

[0111] [Formula 15]

[0112]

[0113] Here, the first term on the right side of the above formula 15 expresses the maximum value and the minimum value of the edge image value Lre[k, i, j] in the reflected wave image Ir as Lremax[k, i, j] and Lremin[k, i, j], respectively. Under this expression, the first term on the right side of the above formula 15 normalizes (i.e., de-dimensionalizes) the edge image value Lre[k, i, j] according to the deviation range after subtracting the minimum value Lremin[k, i, j] from the maximum value Lremax[k, i, j].

[0114] On the other hand, the second term on the right side of the above formula 15 expresses the maximum value and the minimum value of the edge image value Soe[k, i, j] in the external light image Io as Soemax[k, i, j] and Soemin[k, i, j], respectively. Under this expression, the first term on the right side of the above formula 15 normalizes (i.e., de-dimensionalizes) the edge image value Soe[k, i, j] according to the deviation range after subtracting the minimum value Soemin[k, i, j] from the maximum value Soemax[k, i, j].

[0115] exist Figure 2 The processing decision block 160 shown is input with the difference E[k, i, j] between the latest images Io and Ir extracted by the sub-block 146 in the difference extraction block 140. Receiving this input, the processing decision block 160 determines whether to return the synchronization processing to the prediction of the synchronization offset time T based on the synchronization prediction block 180 based on the difference E[k, i, j].

[0116] At this time, when the value obtained by summing the squares of the differences E[k, i, j] for each pixel of the corresponding edges is within the allowable range, the prediction of the synchronization offset time T is considered to be correct, and the processing decision block 160 makes a decision that it is not necessary to return. The synchronization offset time T when it is determined that it is not necessary to return is used, for example, to synchronize the radar sensor 22 with the external camera 24 when it is used for motion estimation of the vehicle 3. In the following description, the value obtained by summing the squares of the differences E[k, i, j] for each pixel is simply referred to as the sum of the squares of the differences E[k, i, j].

[0117] On the other hand, when the sum of the squares of the differences E[k, i, j] is outside the allowable range, the prediction of the synchronization offset time T is deemed to be incorrect, and a decision to return is made in the processing decision block 160. The synchronization offset time T when the decision to return is made is re-predicted to a value different from the latest value stored in the memory 10 by the synchronization prediction block 180 described in detail later.

[0118] Here, the upper limit of the value that can be judged as correct as the difference E[k, i, j] may be used as a threshold, and the allowable range that becomes the determination criterion of the synchronization prediction block 180 may be set to a numerical range below the threshold. The lower limit of the value that should be judged as incorrect as the difference E[k, i, j] may be used as a threshold, and the allowable range may be set to a numerical range smaller than the threshold.

[0119] exist Figure 2The synchronization prediction block 180 shown in the figure inputs (i.e., reads) the initial value of the synchronization offset time T stored in the memory 10 when judging that the initial synchronization process is required from the start of the running of the vehicle 3. Receiving the input, the synchronization prediction block 180 temporarily predicts the latest synchronization offset time T at the time k as the initial value and stores it in the memory 10. Here, the initial value of the synchronization offset time T may be a fixed value set in advance. The initial value of the synchronization offset time T may also be a variable value predicted in the past synchronization process.

[0120] In the synchronization prediction block 180, when a decision is made that a return is required by the processing decision block 160, the incorrectly predicted synchronization offset time T is input (i.e., read) from the memory 10. Receiving this input, the synchronization prediction block 180 re-predicts the latest synchronization offset time T at the time k by adding the adjustment value to the incorrectly predicted synchronization offset time T, and stores it in the memory 10. At this time, the synchronization prediction block 180 adjusts the sign of the offset adjustment value and the variable amount from the initial value according to the increase or decrease result between the sum of squares of the difference E[k, i, j] extracted in this and the previous synchronization processing. Therefore, when a decision is made that a return is required, the initial value of the offset adjustment value stored in the memory 10 is also input (i.e., read) in the synchronization prediction block 180. Here, the initial value of the offset adjustment value may be a fixed value set in advance. The initial value of the offset adjustment value may also be a variable value predicted in the past synchronization processing.

[0121] According to the above, in the first embodiment, the motion compensation block 110 corresponds to the "motion compensation unit", the variation calculation block 120 corresponds to the "variation calculation unit", and the image estimation block 130 corresponds to the "image estimation unit". In addition, in the first embodiment, the difference extraction block 140 corresponds to the "difference extraction unit", the processing determination block 160 corresponds to the "processing determination unit", and the synchronization prediction block 180 corresponds to the "synchronization prediction unit".

[0122] Below, according to Fig.10 The flow of the synchronization method implemented by the synchronization device 1 performing the synchronization processing through the cooperation of the functional blocks 100, 110, 120, 130, 140, 160, and 180 described so far is described. In addition, this flow is performed for each global shutter frame Fo repeated after the start of the driving of the vehicle 3. In addition, in this flow, each "S" refers to a plurality of steps executed according to a plurality of commands included in the synchronization program. In the following description, the expression of the index is omitted.

[0123] In S101 , the synchronization prediction block 180 temporarily predicts the synchronization offset time T generated between the radar sensor 22 and the external camera 24 at the start time k of the global shutter frame Fo as an initial value stored in the memory 10 , and stores it in the memory 10 .

[0124] In S102, the error correction block 100 corrects the latest bias error B for the latest inertial information Ib acquired at time k by the inertial sensor 20. In S103, the motion compensation block 110 compensates the vehicle movement amount in the global shutter frame Fo starting at time k for the latest reflected wave image Ir acquired by the radar sensor 22 in the frame Fo. The compensation at this time is based on the inertial information Ib after the bias error B is corrected in S102.

[0125] In S104, the variation calculation block 120 calculates the variation ΔR between the latest and previous images Ir compensated for the vehicle movement as the time variation ΔR of the reflected wave image Ir between the global shutter frames Fo. In addition, by executing the same correction as S101 and the same compensation as S102 once before executing the present flow of the initial synchronization process from the start of the vehicle 3, the three-dimensional polar coordinates R required for the initial time variation ΔR are calculated.

[0126] In S105, the image estimation block 130 estimates the reflected wave image Ir at the synchronization timing ts shifted from the start timing of the global shutter frame Fo at time k by the latest synchronization shift time T predicted most recently in S101 or S110 described later. The image estimation at this time is based on the time variation ΔR calculated in S104.

[0127] In S106 , the sub-block 142 of the difference extraction block 140 compensates the reflected wave image Ir at the synchronization timing ts estimated in S105 for the offset from the start timing of the global shutter frame Fo, that is, the time k.

[0128] In S107, the sub-block 144 of the difference extraction block 140 interpolates the latest external light image Io acquired by the external camera 24 in the global shutter frame Fo starting at time k based on the reflected wave image Ir that has been offset-compensated in S106. The interpolation at this time makes the external light image Io on the high-definition side match the reflected wave image Ir on the low-definition side.

[0129] In S108, the sub-block 146 of the difference extraction block 140 extracts the difference E by comparing the reflected wave image Ir that has been offset-compensated in S106 with the external light image Io that has been interpolated in S107. After the image values ​​Lre and Soe related to the edges of the reflected wave image Ir and the external light image Io are normalized according to their respective deviation ranges, the comparison and difference extraction are performed on these edges.

[0130] In S109, the process determination block 160 determines whether to return the synchronization process to the prediction of the synchronization offset time T based on the difference E extracted in S108. If it is determined that the process does not need to be returned, the flow ends.

[0131] On the other hand, if it is determined that a return is necessary, in S110, the synchronization prediction block 180 re-predicts the synchronization offset time T at the time k and stores it in the memory 10, and then executes S102 again. The re-prediction is achieved by adjusting the variable amount of the offset adjustment value added to the incorrectly predicted synchronization offset time T based on the increase or decrease result between the difference E extracted in the current and previous synchronization processes.

[0132] According to the above, in the first embodiment, S101 and S110 correspond to the "prediction process", S103 corresponds to the "compensation process", and S104 corresponds to the "calculation process". In addition, in the first embodiment, S105 corresponds to the "estimation process", S106, S107, and S108 correspond to the "extraction process", and S109 corresponds to the "determination process".

[0133] (Effect)

[0134] Next, the effects of the first embodiment described above will be described.

[0135] According to the synchronization process of the first embodiment, the reflected wave image Ir at the synchronization timing ts after the start timing of the frame Fo is shifted by the synchronization shift time T between the radar sensor 22 and the outside camera 24 based on the time change amount ΔR of the reflected wave image Ir between the global shutter frames Fo in the outside camera 24 is estimated. The more accurate the prediction of the synchronization shift time T is, the smaller the difference E extracted by comparing the reflected wave image Ir at the synchronization timing ts with the outside light image Io is. Therefore, by determining whether to return the synchronization process to the prediction of the synchronization shift time T based on the difference E between the reflected wave image Ir and the outside light image Io, the synchronization process can be repeated until the synchronization shift time T is correctly predicted. Therefore, the radar sensor 22 and the outside camera 24 can be synchronized with high accuracy.

[0136] The reflected wave image Ir of the first embodiment is compared with the external light image Io after compensating for the offset from the start timing of the global shutter frame Fo. Accordingly, the reflected wave image Ir acquired in the global shutter frame Fo can be compensated to become an image acquired together with the external light image Io at the start timing of the frame Fo. That is, the compensated reflected wave image Ir can be simulated to be acquired substantially at the same time as the external light image Io, so by comparing these images Ir and Io, the difference E can be correctly extracted, and the radar sensor 22 and the external camera 24 can be synchronized with high precision.

[0137] According to the first embodiment, before calculating the amount of change in the reflected wave image Ir between the frames Fo of the global shutter, the reflected wave image Ir is compensated for the amount of vehicle movement within the frame Fo. Accordingly, the time change ΔR after compensating for the amount of vehicle movement that changes within the frame Fo of the global shutter (specifically, for each frame Fr of the rolling shutter) is calculated. Therefore, the difference E with the external light image Io can be correctly extracted from the reflected wave image Ir at the synchronization timing ts predicted based on such a time change ΔR, so that the radar sensor 22 and the external camera 24 can be synchronized with high precision.

[0138] The reflected wave image Ir and the external light image Io of the first embodiment are compared after the high-definition side is interpolated in coordination with the low-definition side. Accordingly, by comparing the images Ir and Io after eliminating the mismatch between each other caused by the difference in fineness, the difference E can be correctly extracted, so that the radar sensor 22 and the external camera 24 can be synchronized with high precision.

[0139] The reflected wave image Ir and the external light image Io of the first embodiment compare their edges, which makes it easy to accurately extract the difference E from the edges that are easy to compare, so the radar sensor 22 and the external camera 24 can be synchronized with high accuracy.

[0140] The reflected wave image Ir and the external light image Io of the first embodiment are compared after the image values ​​Lre and Soe are normalized according to their respective deviation ranges. Accordingly, by comparing the image values ​​Lre and Soe scaled as a result of normalization, the difference E can be correctly extracted, so that the radar sensor 22 and the external camera 24 can be synchronized with high precision.

[0141] (Second Embodiment)

[0142] Fig.11 The second embodiment shown is a modification of the first embodiment.

[0143] The radar sensor 2022 of the second embodiment is, for example, a so-called imaging radar that acquires a reflected wave image Ir that can be used for motion estimation of the vehicle 3. The radar sensor 2022 acquires a reflected wave image Ir corresponding to a reflected wave obtained by irradiating a millimeter wave (radio wave) as an electromagnetic wave from the outside of the vehicle 3. The radar sensor 2022 includes a transmitting antenna 2220, a receiving antenna 2221, and an imaging circuit 2222.

[0144] The imaging circuit 2222 controls the millimeter wave irradiation to be a pulse beam shape as an electromagnetic wave from the plurality of transmitting antennas 2220 arranged in an array toward the outside of the vehicle 3. With the millimeter wave irradiation, the imaging circuit 2222 scans the plurality of vertical pixel columns in the horizontal direction in the constituent pixels corresponding to the plurality of receiving antennas 2221 arranged in an array, respectively, according to the corresponding scanning lines. The imaging circuit 2222 obtains the reflected wave image Ir by associating and digitizing the distance value to the reflection point of the reflected wave corresponding to the arrival time of the perceived reflected wave from the moment of millimeter wave irradiation with the beam steering angle corresponding to each pixel.

[0145] The difference extraction block 2140 of the second embodiment is supplemented with a sub-block 2145 for pre-processing the external light image Io. The external light image Io interpolated by the sub-block 144 is input to the sub-block 2145. Receiving the input, the sub-block 2145 extracts a specific target object with a high object capture rate of the radar sensor 2220 from the interpolated external light image Io.

[0146] Here, the specific object with a high object capture rate of the radar sensor 2220 is ideally a corner portion of the external object portion that is recessed in the inner direction when viewed from the vehicle 3. However, it is difficult to distinguish between the corner portion of the external object portion that is recessed in the inner direction when viewed from the vehicle 3 and the corner portion that is protruded in the near direction when viewed from the vehicle 3 based on the external light image Io. Therefore, the sub-block 2145 extracts both the recessed corner portion and the protruding corner portion of the external object portion from the external light image Io.

[0147] The difference extraction block 2140 of the second embodiment has a sub-block 2146 for performing a comparison process on the pre-processed images Ir and Io, instead of the sub-block 146 of the first embodiment. The sub-block 2146 receives the luminance value So[k, i, j] as the feature value of the specific object extracted from the external light image Io by the sub-block 2145, i.e., the latest data value.

[0148] The sub-block 2146 realizes the same functions as the sub-block 146 of the first embodiment except for the input. As a result, the reflected wave image Ir and the ambient light image Io are compared with respect to the corner portion which is a specific object captured in these images Ir and Io.

[0149] In the process of the synchronization device 1 of the second embodiment, as Fig.12 As shown, instead of S108 of the first embodiment, S2110 and S2111 are sequentially executed. In S2110, the sub-block 2146 of the difference extraction block 2140 extracts a specific object target with a high object capture rate of the radar sensor 2220 from the external light image Io interpolated in S107.

[0150] In S2111, the sub-block 2146 of the difference extraction block 2140 extracts the difference E by comparing the reflected wave image Ir offset-compensated in S106 with the specific object target of the external light image Io extracted in S2110. After the image values ​​Lre and Soe related to the edges of the specific object target of the reflected wave image Ir and the external light image Io are normalized according to their respective deviation ranges, the comparison and difference extraction are performed on the respective edges.

[0151] Based on the above, in the second embodiment, the difference extraction blocks 140 and 2140 correspond to the "difference extraction unit", and S106, S107, S2110, and S2111 correspond to the "extraction step".

[0152] According to the second embodiment described so far, even if the radar sensor 2022 of the first embodiment in which the electromagnetic wave irradiated to the outside of the vehicle 3 is converted into an infrared laser is used instead of the radar sensor 22 of the first embodiment in which the electromagnetic wave is converted into a millimeter wave, the same effect as the first embodiment can be achieved.

[0153] (Other Embodiments)

[0154] Although a plurality of embodiments have been described above, the present disclosure is not limited to these embodiments, and various embodiments and combinations can be applied within a scope that does not depart from the gist of the present disclosure.

[0155] The synchronization device 1 in the modified example may also be a dedicated computer including at least one of a digital circuit and an analog circuit as a processor. In particular, the digital circuit is at least one of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a SOC (System on a Chip), a PGA (Programmable Gate Array), and a CPLD (Complex Programmable Logic Device). In addition, such a digital circuit may also have a memory storing a program.

[0156] The error correction block 100 and S102 in the modified example may be omitted. In this case, the motion compensation block 110 and S103 may compensate for the vehicle movement amount based on the latest inertial information Ib acquired by the inertial sensor 20 at time k.

[0157] In the sub-block 142 and S106 of the difference extraction block 140 and 2140 in the modified example, the offset from the time k in the reflected wave image Ir may be compensated in the external light image Io. The reflected wave image Ir in the modified example may also be higher resolution than the external light image Io. In the sub-block 144 and S107 of the difference extraction block 140 and 2140 in this case, the reflected wave image Ir on the high-resolution side may be interpolated in coordination with the external light image Io on the low-resolution side.

[0158] In the sub-blocks 142, 146, 2146 and S106, S108, S2111 of the difference extraction blocks 140 and 2140 in the modified examples, a brightness value or an intensity value included in the image Ir and associated with the distance value Lr may be used instead of the distance value Lr of the reflected wave image Ir. In the sub-blocks 146, 2146 and S108, S2111 of the difference extraction blocks 140 and 2140 in the modified examples, the distance value Lr of the reflected wave image Ir and the brightness value So of the external light image Io may be directly used as image values ​​and provided for comparison and difference extraction. In the sub-blocks 146, 2146 and S108, S2111 of the difference extraction blocks 140 and 2140 in the modified examples, one of the reflected wave image Ir and the external light image Io may be normalized in coordination with the other, and then provided for comparison and difference extraction.

[0159] The radar sensor 22 in the modified example may also be configured to obtain the reflected wave image Ir through a global shutter. In other modified examples in this case, modules 100, 110, 142 and S102, S103, and S106 may also be omitted. Here, in sub-blocks 144 and S107 of other modified examples, the high-precision side may be interpolated in coordination with the low-precision side in the latest images Ir and Io obtained in the global shutter frame Fo. In addition, in sub-blocks 146 and S108 of other modified examples, the latest images Ir and Io obtained in the global shutter frame Fo may be compared with each other.

Claims

1. A synchronization device, which is a synchronization device for performing synchronization processing on a radar sensor and an external camera in a vehicle equipped with the radar sensor and the external camera, wherein the radar sensor obtains a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave, and the external camera obtains an external light image corresponding to external light perceived from the outside world, wherein: The above-mentioned synchronization device has: A synchronization prediction unit, which predicts a synchronization offset time between the radar sensor and the external camera; a variation calculation unit, for calculating a time variation of the reflected wave image between shutter frames of the external camera; an image estimating unit that estimates the reflected wave image at a synchronization timing that is shifted from the start timing of the shutter frame by the synchronization shift time predicted by the synchronization predicting unit, based on the time variation calculated by the variation calculating unit; a difference extracting section that extracts a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the image estimating section; and a processing determination unit that determines whether to return the synchronization processing to the prediction of the synchronization offset time based on the synchronization prediction unit based on the difference calculated by the difference extraction unit, The difference extracting section compares the reflected wave image with the external light image after compensating the reflected wave image for an offset from the start timing of the shutter frame. The synchronization device further includes a motion compensating unit that compensates the reflected wave image for a movement amount caused by a movement of the vehicle within the shutter frame before the change amount calculating unit calculates the time change amount.

2. The synchronization device according to claim 1, wherein: The difference extracting section compares the reflected wave image with the external light image after interpolating the high-definition side of the reflected wave image and the external light image in coordination with the low-definition side of the reflected wave image and the external light image.

3. The synchronization device according to claim 1, wherein: The difference extracting unit compares edges of the reflected wave image and the external light image.

4. The synchronization device according to any one of claims 1 to 3, wherein: The difference extracting unit compares the image values ​​of the reflected wave image and the ambient light image after normalizing the image values ​​according to respective deviation ranges.

5. A synchronization method, which is a synchronization method for performing synchronization processing on the radar sensor and the outside camera in a vehicle equipped with the radar sensor and the outside camera, wherein the radar sensor obtains a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave, and the outside camera obtains an outside light image corresponding to external light perceived from the outside, wherein: The above synchronization methods include: A prediction step of predicting a synchronization offset time between the radar sensor and the external camera; A calculation step of calculating a time variation of the reflected wave image between shutter frames of the external camera; an estimating step of estimating the reflected wave image at a synchronization timing shifted from the start timing of the shutter frame by the synchronization shift time predicted by the predicting step based on the time variation calculated by the calculating step; an extraction step of extracting a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the estimation step; and a determination step of determining, based on the difference extracted by the extraction step, whether to return the synchronization process to the prediction of the synchronization offset time based on the prediction step, The extraction step is to compare the reflected wave image with the external light image after compensating the reflected wave image for an offset from the start timing of the shutter frame. The synchronization method further includes a compensation step in which, before the calculation of the time variation in the calculation step, the reflected wave image is compensated for a movement amount caused by the motion of the vehicle within the shutter frame.

6. The synchronization method according to claim 5, wherein: The extraction step compares the reflected wave image with the external light image after interpolating the high-definition side of the reflected wave image and the external light image in coordination with the low-definition side of the reflected wave image and the external light image.

7. The synchronization method according to claim 5, wherein: The extraction step compares edges of the reflected wave image and the external light image with each other.

8. The synchronization method according to any one of claims 5 to 7, wherein: The extraction step includes comparing the image values ​​of the reflected wave image and the ambient light image after normalizing them according to respective deviation ranges.

9. A storage medium storing a synchronization program including commands for causing a processor to execute, the synchronization program being used to perform synchronization processing on the radar sensor and the external camera in a vehicle equipped with the radar sensor and the external camera, the radar sensor acquiring a reflected wave image corresponding to a reflected wave obtained by irradiating an electromagnetic wave, and the external camera acquiring an external light image corresponding to external light perceived from the outside, wherein: The above command includes: A prediction step of predicting a synchronization offset time between the radar sensor and the external camera; A calculation step of calculating a time variation of the reflected wave image between shutter frames of the external camera; an estimating step of estimating the reflected wave image at a synchronization timing shifted from the start timing of the shutter frame by the synchronization shift time predicted by the predicting step based on the time variation calculated by the calculating step; an extraction step of extracting a difference between the external light image and the reflected wave image by comparing the external light image with the reflected wave image at the synchronization timing estimated by the estimation step; and a determination step of determining, based on the difference extracted by the extraction step, whether to return the synchronization process to the prediction of the synchronization offset time based on the prediction step, The extraction step is to compare the reflected wave image with the external light image after compensating the reflected wave image for an offset from the start timing of the shutter frame. The command further includes a compensation step of compensating the reflected wave image for a movement amount caused by the motion of the vehicle within the shutter frame before the calculation of the time variation amount in the calculation step.

10. The storage medium according to claim 9, wherein: The extraction step compares the reflected wave image with the external light image after interpolating the high-definition side of the reflected wave image and the external light image in coordination with the low-definition side of the reflected wave image and the external light image.

11. The storage medium according to claim 9, wherein: The extraction step compares edges of the reflected wave image and the external light image with each other.

12. The storage medium according to any one of claims 9 to 11, wherein: The extraction step includes comparing the image values ​​of the reflected wave image and the ambient light image after normalizing them according to respective deviation ranges.

Images