Active LiDAR blind spot detection system and method for autonomous driving

Abstract

The present invention belongs to the technical field of autonomous driving, and specifically relates to an active laser radar blind spot detection system and method for the direction of autonomous driving, wherein the active laser radar blind spot detection system for the direction of autonomous driving comprises: a light source, wherein the light source is suitable for emitting laser to illuminate the environment in the direction of vehicle travel; a bionic scallop eye optical imaging subsystem, wherein the bionic scallop eye optical imaging subsystem is suitable for receiving scattered light reflected by the environment after being illuminated, and identifying and generating an image of an obscured object based on the scattered light, thereby enabling the autonomous driving vehicle to detect potential dangers in the front and surrounding areas in advance when driving, thereby improving the safety of the vehicle and protecting road users and public facilities without protective measures.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and in particular relates to an active laser radar blind spot detection system and method for autonomous driving direction. Background Art

[0002] With the goal of image imaging and motion trend prediction of non-line-of-sight hidden objects, the research is carried out by combining theoretical analysis, computer simulation and experiments to closely focus on improving the performance of long-distance and real-time potential danger detection. The research on the subject is carried out according to the three stages of bionics research: biological prototype research stage, theoretical model research stage and hardware model research stage. First, the special physiological structure of the bionic object scallop eye is quantitatively measured using instruments and equipment to obtain experimental data on structural parameters such as the internal geometric size of the eye, silver film thickness, and retinal photoreceptor cell distribution, and to carry out research on the mechanism of biological visual imaging characteristics. Secondly, the scallop eye is reconstructed and simulated based on relevant data, and a multi-layer micro-nano reflective bionic structure and MEMS manufacturing process are designed. The variable curvature control and dual-field off-axis variable focal length reflective imaging control method are established, and the hidden object image reconstruction and motion trend prediction algorithm based on the speckle correlation method are applied. Finally, the hardware and software system integration is carried out to develop a new non-line-of-sight detection device for anti-scallop eyes, and a performance test platform is established. The device structure is optimized and the performance is improved through continuous experiments.

[0003] It is very important for autonomous vehicles to detect potential dangers in front and surrounding areas in advance while driving. This not only improves the safety of the vehicle, but also protects unprotected road users and public facilities.

[0004] Therefore, based on the above technical problems, it is necessary to design a new active lidar blind spot detection system and method for autonomous driving. Summary of the invention

[0005] The object of the present invention is to provide an active laser radar blind spot detection system and method for autonomous driving.

[0006] In order to solve the above technical problems, the present invention provides an active laser radar blind spot detection system for autonomous driving, comprising:

[0007] a light source, the light source being adapted to emit laser light to illuminate an environment in a direction of travel of the vehicle;

[0008] A bionic scallop eye optical imaging subsystem, wherein the bionic scallop eye optical imaging subsystem is suitable for receiving scattered light reflected by the environment after being illuminated, and identifying and generating an image of an obscured object based on the scattered light.

[0009] Furthermore, the bionic scallop eye optical imaging subsystem comprises: a dual-field-of-view off-axis imaging optical path;

[0010] The dual-field-of-view off-axis imaging optical path is suitable for receiving scattered light reflected from the environment after being illuminated.

[0011] Furthermore, the dual-field off-axis imaging optical path includes: a wide-angle objective lens, a first plane reflector, a first bionic MEMS deformable reflector and a first CCD;

[0012] A central field of view light beam in the environment in the direction of vehicle travel is converged by a wide-angle objective lens and reaches a first plane reflector, and is then converged to a first bionic MEMS deformable reflector by the first plane reflector. A speckle pattern is formed on a first CCD after the wavelength of the incident light is filtered by the first bionic MEMS deformable reflector.

[0013] Furthermore, the dual-field off-axis imaging optical path further comprises: a collimating lens, a second plane reflector, a convex mirror, a second bionic MEMS deformable reflector and a second CCD;

[0014] The peripheral vision light beam in the environment in the direction of vehicle travel is converged by a wide-angle objective lens and then collimated by a collimating lens. The parallel light beam passes through an off-axis three-reflection imaging system composed of a second plane mirror, a convex mirror, and a second plane mirror. The second bionic MEMS deformable mirror completes the incident light wavelength filtering and controllable variable curvature zooming functions. Finally, the light beam converges onto the second CCD to form a ring-shaped speckle pattern.

[0015] Further, the bionic MEMS deformable reflector includes: a microcontroller, a plurality of micromirror units and a plurality of parallel plate drivers;

[0016] The micromirror units are independently controlled by at least one parallel plate driver;

[0017] The microcontroller is suitable for controlling the parallel plate driver to drive the corresponding micromirror unit to deform, so that the bionic MEMS deformable reflector as a whole produces a local controlled deformation to achieve the desired mirror surface shape;

[0018] The microcontroller is suitable for receiving a control signal sent by a host computer. The digital voltage signal is converted into an analog voltage signal after passing through a D / A converter. The microcontroller converts the control signal into a channel selection signal through a decoder, controls the corresponding amplifier channel to be turned on, and amplifies the analog voltage signal and applies it to the corresponding micromirror array.

[0019] Furthermore, the parallel plate driver is a parallel plate capacitor, and the relationship between the voltage and displacement of the parallel plate driver is:

[0020] E is the electric field strength between the two plates; V is the potential difference between the two plates; d is the distance between the two plates; Q is the total charge on the plates; C is the capacitance of the parallel plate driver; A is the area of ​​the plates; k is the equivalent elastic coefficient of the elastic structure;

[0021] When the upper plate is not fixed, the plate has a small displacement △z in the vertical direction under the action of electrostatic force. The work done by the electrostatic force Fe on the plate is Fe△z. According to the law of conservation of energy, the change in the energy of the parallel plate is equal to the sum of the work done by the electrostatic force and the energy provided by the power supply:

[0022]

[0023]

[0024] When a voltage is applied to the parallel plate driver, the upper plate moves a certain distance under the interaction of the electrostatic force Fe and the elastic force Fk to reach the equilibrium position. According to Hooke's law,

[0025]

[0026] The actual working voltage and displacement relationship of the parallel plate driver is:

[0027]

[0028] The voltage and displacement curve of the parallel plate driver is obtained, so as to adjust the displacement of the parallel plate driver by adjusting the voltage and accurately obtain the required mirror surface shape.

[0029] Furthermore, the active laser radar blind spot detection system further includes: a host computer;

[0030] The host computer is adapted to receive the speckle pattern sent by the bionic scallop eye optical imaging subsystem to obtain an image of the obscured object according to the speckle pattern, that is,

[0031] CNN data preparation:

[0032] Constructing an image training dataset by using shot noise in speckle patterns Statistical distribution law, forming the corresponding speckle pattern through the open source image training data set, and adding random shot noise to synthesize the required training data set;

[0033] Construct the loss function, use the SoftMax layer to generate a pair of complementary channels to represent the object and the background respectively, and use the average cross entropy as the loss function to promote sparsity. The loss function is:

[0034]

[0035] Where g is the segmentation gold standard pixel value; p is the predicted value; the average value is over all N pixels x on two channels c; g and p are both binary or continuous values;

[0036] Data preprocessing, cropping part of the output speckle image, setting the resolution of all input and output images to the preset pixels, adding simulated random noise to the speckle image according to the model, and normalizing the speckle image between 0 and 1;

[0037] CNN architecture implementation:

[0038] According to the overall structure of U-net, it includes a left encoder compression path and a right decoder expansion path;

[0039] The encoder compression path consists of 4 blocks, each of which uses a 3×3 convolution, a dense concatenation operation, and a 2×2 max pooling layer downsampling with a stride of 2;

[0040] The initial image is downsampled to reduce the image size. The number of feature channels is doubled after each downsampling. Finally, the encoder outputs a high-dimensional encoded feature map with a smaller size of 16×16 pixels.

[0041] The decoder expansion path consists of 4 blocks, each of which uses a 3×3 convolution, a dense connection operation, and a 2×2 deconvolution with a stride of 2 for upsampling;

[0042] The low-resolution high-dimensional feature map is upsampled through 4 2×2 deconvolution operations with a step size of 2. Each deconvolution operation doubles the size of the input feature map and halves the number of feature maps. The cropped feature maps from the corresponding compression path are concatenated through jump connections until the size of the input image is finally restored.

[0043] A 1×1 additional convolutional layer is used to produce the final network output, completing the restoration of the original image of the dual-channel speckle pattern and generating an image of the occluded object.

[0044] On the other hand, the present invention also provides an active laser radar blind spot detection method for autonomous driving direction, comprising:

[0045] Illuminating the environment in the direction of travel of the vehicle; and

[0046] Receive the scattered light reflected by the environment after being illuminated, and identify and generate an image of the obscured object based on the scattered light.

[0047] Furthermore, the active lidar blind spot detection method for the direction of autonomous driving is suitable for using the above-mentioned active lidar blind spot detection system for the direction of autonomous driving to generate an image of the obscured object.

[0048] The beneficial effect of the present invention is that, through a light source, the light source is suitable for emitting laser to illuminate the environment in the direction of vehicle travel; a bionic scallop eye optical imaging subsystem, the bionic scallop eye optical imaging subsystem is suitable for receiving scattered light reflected by the environment after illumination, and identifying and generating an image of the obscured object based on the scattered light, thereby enabling an autonomous driving vehicle to detect potential dangers in the front and surrounding areas in advance while driving, thereby improving the safety of the vehicle and protecting road users and public facilities without protection measures.

[0049] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 It is a principle block diagram of the active laser radar blind spot detection system for autonomous driving involved in the present invention;

[0053] Figure 2 This is a schematic diagram of the optical path of the bionic dual-field off-axis reflection zoom imaging system of the present invention;

[0054] Figure 3 It is a schematic diagram of improving the reflection efficiency of the system by using a multi-layer square close-paved structure of silver film crystals and thin film superposition in the present invention;

[0055] Figure 4 It is a schematic diagram of the arrangement of the bionic silver film square mirror unit and the square array in the present invention;

[0056] Figure 5 It is a schematic diagram of the structural design of the bionic silver film discrete mirror unit in the present invention;

[0057] Figure 6 This is the process flow of MEMS surface silicon processing in the present invention;

[0058] Figure 7 It is a schematic diagram of a multi-layer micro-nano discrete square array bionic structure in the present invention;

[0059] Figure 8 It is a virtual prototype and system simulation schematic diagram of the bionic silver film micromirror array in the present invention;

[0060] Fig. 9 Schematic diagram of the control system of the bionic silver film imaging system of the present invention;

[0061] Fig.10 It is a schematic diagram of the working principle of the electrostatic actuator in the present invention and the changes of electrostatic force and displacement under different voltages;

[0062] Fig.11 It is a line graph showing the working principle of the electrostatic actuator in the present invention and the relationship between the electrostatic force and the displacement change under different voltages;

[0063] Fig.12 It is a flow chart of speckle image reconstruction based on the improved U-net architecture deep convolutional network in the present invention;

[0064] Fig.13 It is a schematic diagram of the human body posture segmentation data in the BSDS500 data set and its simulated synthetic noise speckle test set in the present invention;

[0065] Fig.14 It is a schematic diagram of the CNN algorithm structure used in the present invention to learn the statistical relationship between the speckle pattern and the hidden object;

[0066] Fig.15 It is a schematic diagram of a special test platform for non-line-of-sight hidden object detection and imaging in the present invention;

[0067] Fig.16 It is a flow chart of the active laser radar blind spot detection method for autonomous driving direction involved in the present invention.

[0068] In the figure:

[0069] 1 is a wide-angle objective lens, 2 is a first plane reflector, 3 is a first bionic MEMS deformable reflector, 31 is a discrete mirror, 32 is an actuator column, 33 is a driver surface, 34 is a driver support, 35 is a drive electrode, 36 is a silicon-based material substrate, 4 is a first CCD, 5 is a collimating lens, 6 is a second plane reflector, 7 is a convex mirror, 8 is a second bionic MEMS deformable reflector, and 9 is a second CCD. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment 1 provides an active lidar blind spot detection system for the direction of autonomous driving, including: a light source (active laser lighting), wherein the light source is suitable for emitting laser to illuminate the environment in the direction of vehicle travel; a bionic scallop eye optical imaging subsystem, wherein the bionic scallop eye optical imaging subsystem is suitable for receiving scattered light reflected by the environment after illumination, and identifying and generating an image of the obscured object based on the scattered light, so as to effectively detect the image and movement trend of hidden objects hidden in front of the field of vision and in the surrounding corners, so that the autonomous driving vehicle can detect potential dangers in the front and surrounding areas in advance when driving, thereby improving the safety of the vehicle and protecting road users and public facilities without protection measures.

[0073] Specifically, the principle of the active LiDAR blind spot detection system for autonomous driving is as follows: the active laser transmitter (i.e., the light source, vertical cavity surface emitting laser VCSEL is proposed) emits a long-distance array laser (considering the safety and non-invasiveness of human eyes, a short-wave near-infrared laser with a wavelength of 905nm is selected), and the emitted light passes through various mirror surfaces (reflectors, metals, etc.) or irregular rough surfaces (walls, floors, etc.) in front of the field of view and in the surrounding corners as a secondary lighting source, and is reflected to the hidden object (obstructed object) behind the obstruction and produces diffuse reflection; secondly, the receiving end of the confocal bionic scallop eye optical system (bionic scallop eye optical imaging subsystem) can not only efficiently receive the scattered light reflected back from various indirect paths, but also use the thin film interference reflection enhancement wavelength filtering mechanism unique to the bionic structure to screen the effective scattered light (pre-processing and denoising, that is, only allowing 905nm light), and then use its special dual-field CCD structure to realize off-axis reflection zoom imaging (speckle pattern); finally, the AI ​​image algorithm is used to reconstruct the speckle image and predict the motion trend.

[0074] In this embodiment, the bionic scallop eye optical imaging subsystem includes: a dual-field-of-view off-axis imaging optical path; the dual-field-of-view off-axis imaging optical path is suitable for receiving scattered light reflected by the environment after being illuminated.

[0075] In this embodiment, the dual-field off-axis imaging optical path (dual-field off-axis reflective zoom imaging system) includes: a wide-angle objective lens 1, a first plane reflector 2, a first bionic MEMS deformable reflector 3 and a first CCD 4;

[0076] The central field of view light beam in the environment in the direction of vehicle travel is converged by the wide-angle objective lens 1 and reaches the first plane reflector 2, and is converged to the first bionic MEMS deformable reflector 3 by the first plane reflector 2. After the wavelength of the incident light is filtered by the first bionic MEMS deformable reflector 3, a speckle pattern is formed on the first CCD 4.

[0077] In this embodiment, the dual-field off-axis imaging optical path further includes: a collimating lens 5, a second plane reflector 6, a convex mirror 7, a second bionic MEMS deformable reflector 8 and a second CCD 9;

[0078] The peripheral vision light beam in the environment in the direction of vehicle travel is converged by the wide-angle objective lens 1 and then collimated by the collimating lens 5. The parallel light beam passes through the off-axis three-reflection imaging system composed of the second plane mirror 6, the convex mirror 7, and the second plane mirror 6. The second bionic MEMS deformable mirror 8 completes the incident light wavelength filtering and controllable variable curvature zoom function. Finally, the light beam converges onto the second CCD 9 to form a ring-shaped speckle pattern.

[0079] Drawing on the dual retina off-axis reflection zoom imaging mode of scallop eyes, synchronous imaging and zoom viewing distance switching of the central and peripheral fields of view are realized. Due to the need to consider the limitations of limited installation position and size on autonomous driving vehicles, and to cover the wide-angle field of view in front of the field of view and the surrounding area as much as possible, the dual-field off-axis reflection zoom optical imaging system designed in this embodiment introduces a wide-angle objective lens 1, two bionic MEMS deformable mirrors and a three-reflection off-axis zoom structure to design a compact, compact and efficient dual-field off-axis imaging optical path system. In addition, since the dual fields of view require synchronous reconnaissance and parallel prejudgment of various potential dangers, a wide-angle fisheye objective lens is used to achieve a maximum range of 180-degree field of view to receive the returned scattered light. The overall imaging optical path system can be decomposed into two sub-optical path systems, namely: a central field of view sub-optical path and a peripheral field of view sub-optical path. The central field of view (central visual field) simulates the imaging principle of the distal retina of the scallop, zooms and scans objects at different viewing distances in front of the field of view, and the bionic MEMS deformable mirror stacked interference structure filters the wavelength of the incident light, and controls the surface shape change of the micromirror through the actuator to achieve zoom reflection imaging, thereby ensuring that the imaging quality in any ROI at each focal length of the central field of view can reach the diffraction limit. The peripheral field of view (peripheral visual field) sub-optical path simulates the imaging principle of the proximal retina of the scallop. Since it is necessary to collect scattered light in the peripheral area as much as possible without blocking the central field of view sub-optical path, a three-mirror off-axis zoom structure is considered. The off-axis three-mirror optical system has no central obstruction and a high modulation transfer function (MTF). When the same energy utilization rate is obtained, the off-axis structure can achieve a smaller primary mirror aperture and system structure size than the coaxial structure. The concave mirror in the three-mirror off-axis structure is replaced by a programmable bionic MEMS deformable mirror, which improves the light energy utilization rate and imaging quality of the optical system.

[0080] Optical system (dual-field-of-view off-axis imaging optical path, bionic dual-field-of-view off-axis reflection zoom imaging system optical path) such as Figure 2As shown, the wide-angle objective lens 1 acts as the objective lens of the entire system (dual-field off-axis imaging optical path). Since it is necessary to collect all scattered light returned from non-line-of-sight hidden objects within the 180° viewing angle as much as possible, the wide-angle objective lens 1 is considered to use a fisheye lens. After the scattered light from the central and peripheral fields of view enter from the wide-angle objective lens 1, it is decomposed into two independent sub-optical path systems for parallel processing according to the different directions of the scattered light, that is, the central field of view light beam converges through the wide-angle objective lens 1 and reaches the first plane reflector 2, and is converged to the programmable first bionic MEMS deformable reflector 3 through mirror reflection. Due to the unique thin film interference reflection enhancement effect of the first bionic MEMS deformable reflector 3, the wavelength of the incident light is filtered, and the low-flux and effective scattered light of the obstructed object returned after indirect laser illumination is converged to the greatest extent, and then a high-resolution speckle pattern is formed on the first CCD 4, which provides a guarantee for the subsequent restoration of the obstructed object image based on speckle correlation in the central area of ​​the field of view. In addition, due to the controllable variable curvature zoom function of the first bionic MEMS deformable reflector 3, continuous scanning of the central field of view far-field and near-field incident light imaging can be achieved by applying a control signal to the first bionic MEMS deformable reflector 3. After the peripheral field of view light beam passes through the wide-angle objective lens 1, it converges to the collimating lens 5 for collimation, and the parallel light beam passes through the off-axis three-reflection imaging system composed of the second plane reflector 6, the convex mirror 7, and the programmable controllable second bionic MEMS deformable reflector 8. Finally, the second bionic MEMS deformable reflector 8 completes the incident light wavelength filtering and controllable variable curvature zoom function. Finally, the light beam converges to the second CCD 9 to form an annular speckle pattern, which provides a guarantee for the subsequent restoration of the image of the obstructed object in the peripheral area of ​​the field of view based on the speckle correlation. Therefore, the overall system (dual-field of view off-axis imaging optical path) can complete the dual-field of view off-axis reflection zoom imaging function of scattered light in the central and peripheral areas of the field of view in real time.

[0081] In this embodiment, the bionic MEMS deformable reflector (the first bionic MEMS deformable reflector 3, the second bionic MEMS deformable reflector 8) includes: a microcontroller, a plurality of micromirror units (discrete mirrors 31) and a plurality of parallel plate drivers (actuator columns 32, driver surfaces 33, driver supports 34, drive electrodes 35, silicon-based material substrates 36);

[0082] The driver surface 33 is arranged on a silicon-based material substrate 36 through a driver support 34, the driving electrode 35 is arranged on the silicon-based material substrate 36 and is located below the driver surface 33, and the discrete mirror 31 is arranged on the driver surface 33 through an actuator column 32;

[0083] The micromirror units are independently controlled by at least one parallel plate driver;

[0084] The microcontroller is suitable for controlling the parallel plate driver to drive the corresponding micromirror unit to deform, so that the bionic MEMS deformable reflector as a whole produces a local controlled deformation to achieve the desired mirror surface shape;

[0085] The microcontroller is suitable for receiving a control signal sent by a host computer. The digital voltage signal is converted into an analog voltage signal after passing through a D / A converter. The microcontroller converts the control signal into a channel selection signal through a decoder, controls the corresponding amplifier channel to be turned on, and amplifies the analog voltage signal and applies it to the corresponding micromirror array (composed of a plurality of micromirror units and a plurality of parallel plate drivers).

[0086] Scallops' spatial vision is achieved by focusing light onto the retina through a layered structure of mirrors located in each eye. By finely adjusting the layered structure, scallops can efficiently collect light of different wavelengths that penetrate their habitat. Figure 3 As shown in the figure, the scallop reflective silver film is made of 20-30 layers of square guanine crystals, which are separated by cytoplasm. The interface between the crystal and cytoplasm forms a reflector. The low-refractive-index cytoplasm and the high-refractive-index crystal sheet form a thin-film interference system. The scallop precisely controls the size of the square crystal and the liquid thin layer. When facing specific light, thin-film interference will occur, effectively increasing the reflection efficiency of light of a specific wavelength. The superposition of multiple films will significantly improve the reflection efficiency.

[0087] In actual application scenarios, there is not only scattered light indirectly reflected by hidden objects, but also scattered light from other objects captured from walls, floors and clutter in the scene, which can form diffuse background and speckle noise that is unrelated to speckle measurement, introduce unpredictable random noise, and increase the difficulty of image reconstruction algorithm design. Inspired by the special physiological structure of scallop eyes for wavelength filtering of light and dual-field off-axis reflection zoom imaging, this embodiment will structurally imitate the special reflective silver film of scallop eyes, design a multi-layer discrete unit concave reflector structure to achieve effective light filtering and off-axis reflection focusing imaging, and use MEMS technology to manufacture bionic micro-optical electromechanical systems, thereby effectively removing random noise, providing high-resolution pre-processed images, reducing the difficulty of subsequent image reconstruction algorithm design, and improving detection real-time performance.

[0088] First, the shape and arrangement of the micromirror unit are designed. Drawing on the structural characteristics of the scallop silver film ultra-micro unit array, this embodiment uses layered stacked square micromirror reflection units and a square array arrangement, such as Figure 4As shown. Since a multi-layer stacked "interference film" structure is required to achieve the screening and convergence reflection of light of a specific wavelength, the mirror reflection unit material needs to select a high-refractive index semi-transparent material, and the interlayer filling layer needs to select a low-refractive index material to form a thin film interference reflection enhancement effect. The surface mirror unit selects a material with good optical properties (polycrystalline silicon is proposed as the substrate). Since the reflectivity of silicon in the visible and near-infrared spectral range is very low, generally a layer of optical coating in the visible and near-infrared spectral range is deposited on the structural layer to achieve good optical performance.

[0089] Secondly, design the drive structure of the micromirror unit (parallel plate drive). Since the "concave reflective mirror surface" of the scallop eye is not a hemispherical or elliptical surface with a fixed curvature, its curvature will vary with different positions. This embodiment uses a discrete micromirror drive structure to achieve the variable curvature surface adjustment function. The discrete mirror surface 31 helps to eliminate the mutual coupling between unit drivers. Figure 5 As shown, the discrete mirror body is composed of a large number of closely arranged micromirror units, each of which is independently controlled by one or more micro-drivers (parallel plate drivers). By precisely controlling the spatial position of each micromirror, the deformable mirror as a whole is caused to produce local controlled deformation to achieve the desired mirror surface shape. Since each micromirror unit is independently controlled, there is no mutual influence between adjacent unit drivers.

[0090] Finally, choose a suitable manufacturing process. MEMS processing technology can better ensure the consistency of each unit driver, and can process a bionic deformable mirror with a large number of driver units at one time. Figure 6 As shown, surface silicon micromachining is a process for carving and shaping material layers sequentially deposited on a silicon substrate, using selective etching of sacrificial thin films to form independent vertical or even completely independent thin film microstructures. Since the substrate only provides mechanical support, various standard methods similar to IC manufacturing can be used to form complex multi-layer stacked microstructures through a series of thin film deposition and forming steps. This embodiment uses the MEMS surface silicon micromachining process, that is, the bionic silver film structure is manufactured using basic processes such as deposition and etching of multi-layer silicon. The MEMS micro-deformable mirror manufactured by this technology has high integration, small size, and flexible structural design, and can be manufactured using standard process equipment processing units compatible with ICs, such as Figure 7 shown.

[0091] This embodiment uses virtual prototype related tools (such as UG, ADAMS, ANSYS, Matlab, etc.) and design methods to accurately and effectively model and virtually design the bionic silver film MEMS micromirror array at the system level, and then conducts static analysis, dynamics, kinematics and processing technology simulation on the virtual prototype model micromirror array, and conducts virtual analysis and testing on the performance indicators such as the surface morphology, off-plane deformation, static voltage-displacement curve and resonant frequency of the bionic structure, so as to quickly obtain reliable design results, reduce design costs, and shorten development cycles, such as Figure 8 shown.

[0092] The second bionic MEMS deformable reflector 8 of this embodiment is driven by parallel plate electrostatic drive, and the surface shape is fitted and corrected by real-time control of each micro-mirror unit. Different control signals are used to control the deflection sequence, angle and direction of each unit structure in parallel, so as to achieve effective control of the mirror surface. Fig. 9 As shown, this embodiment uses a multi-channel D / A module and a multi-channel high-voltage amplifier module to form a control circuit. The microcontroller function is implemented by a field programmable gate array device (FPGA) provided by NI (National Instruments). The computer transmits data to the microcontroller through a CAN bus interface. The digital voltage signal is converted into an analog voltage signal after passing through a D / A converter. The microcontroller converts the control signal into a channel selection signal through a decoder, controls the corresponding amplifier channel to be turned on, and applies the analog voltage signal to the corresponding bionic micromirror electrode after being amplified.

[0093] In this embodiment, the driving control method of the mirror unit is as follows: due to the film reflection enhancement effect formed between the multi-layer reflection, the scattered light other than the non-laser illumination reflection wavelength of 905nm can be smoothly and effectively filtered out. Therefore, when the film thickness is 1 / 4 of the incident light wavelength λ, its interference constructive phase reaches the best reflected light and the reflection efficiency of the film can reach four times that of the same single interface. From this, the basic thickness of the film (i.e., the spacing between the optimal parallel plates) can be determined. In the bionic structure of this embodiment, a low refractive index material needs to be filled between the upper mirror unit and the lower mirror unit (air is more suitable), so parallel plate electrostatic drive is a more ideal choice;

[0094] The parallel plate driver is a parallel plate capacitor, and the relationship between the voltage and displacement of the parallel plate driver is: the parallel plate driver is essentially a parallel plate capacitor, which uses the electrostatic attraction of heterogeneous charges generated between the two plates of the capacitor to achieve electrostatic drive. When the potentials on the two plates of the parallel plate capacitor are different, equal amounts of heterogeneous charges will be generated on the two plates, thereby generating a uniform electrostatic field in the space between the two plates; the upper plate is the driver surface 33, and the lower plate is the driving electrode 35;

[0095] E is the electric field strength between the two plates; V is the potential difference between the two plates; d is the distance between the two plates; Q is the total charge on the plates; C is the capacitance of the parallel plate driver; A is the area of ​​the plates; k is the equivalent elastic coefficient of the elastic structure;

[0096] When the upper plate (driver surface 33) is not fixed, the plate has a small displacement △z in the vertical direction under the action of electrostatic force. The work done by the electrostatic force Fe on the plate is Fe△z. According to the law of conservation of energy, the change in the energy of the parallel plate is equal to the sum of the work done by the electrostatic force and the energy provided by the power supply:

[0097]

[0098]

[0099] If the upper plate is connected to the elastic structure, such as Fig.10 As shown in the figure, when a voltage is applied to the parallel plate driver, the upper plate moves a distance under the interaction of the electrostatic force Fe and the elastic force Fk to reach the equilibrium position. According to Hooke's law:

[0100]

[0101] Under known voltage conditions, the plate displacement solution is obtained by numerical solution. Fig.11 The figure shows the relationship between the elastic restoring force and the electrostatic force and displacement generated when different voltages are applied. As the applied voltage increases, the positions of the two equilibrium points A and B with different properties gradually approach each other. When the V2 value is reached, the two points coincide with point C, so the voltage value corresponding to V2 is the upper limit of the voltage that can be applied.

[0102] Therefore, the actual working voltage and displacement relationship of the parallel plate driver can be obtained as follows:

[0103]

[0104] The voltage and displacement curve of the parallel plate driver can be obtained according to the formula, so that the displacement of the parallel plate driver can be adjusted by adjusting the voltage to accurately obtain the required mirror surface shape.

[0105] In this embodiment, the active laser radar blind spot detection system further includes: a host computer (which may be a microcomputer, a controller in a car, etc.);

[0106] The host computer is adapted to receive the speckle pattern sent by the bionic scallop eye optical imaging subsystem to obtain an image of the obscured object according to the speckle pattern, that is,

[0107] Real-time reconstruction algorithm of the obstructed object image based on the improved U-net network and low signal-to-noise ratio speckle correlation map; In the case of non-line-of-sight, when the detector transmitter emits laser (the light source emits laser), the light forms a "secondary light source" through various mirror surfaces (reflectors, metals, etc.) or irregular rough surfaces (walls, floors, etc.) in front of the field of view and in the surrounding corners. After indirectly illuminating the obstructed object, the generated scattered light forms a speckle pattern through the "relay" rough surface. Since there are many possible light paths in the middle, it is very difficult to fully and deterministically describe the scattering process. The optical memory effect theory proves that the speckle correlation method is an efficient method to reconstruct the original image using the inherent correlation of the speckle image. The autocorrelation result of the speckle image is approximately equal to the autocorrelation of the original image signal. In the subsequent calculation, after removing the background term constant, the autocorrelation operation and phase recovery (PR, Phase Retrieval) algorithm can be used to effectively and quickly restore the original image of the obstructed object. Usually, the phase information can be recovered from the speckle autocorrelation by using the classic iterative phase recovery algorithm (i.e., HIO hybrid input-output algorithm, ER error reduction algorithm, or DM differential mapping algorithm, etc.), and combined with the image amplitude information obtained by the speckle image autocorrelation operation, the original image of the hidden object can be reconstructed. These methods are generally suitable for high-throughput laser illumination imaging mode and require a long sampling time and a short detection distance.

[0108] Autonomous vehicles need to actively detect and predict potential dangers in real time at long distances while driving fast. The scattered light emitted by the obstructed object will travel along multiple indirect paths and scatter multiple times before reaching the receiving end. Therefore, the indirect signal actually returned to the receiving end is very weak, and the intensity of the scattered light will be greatly attenuated with increasing distance, thereby greatly reducing the signal-to-noise ratio (SNR). Therefore, it is necessary to restore the original image of the obstructed object in real time and with high resolution from the photon-deficient long-distance low-flux indirect scattered signal;

[0109] In order to better model, characterize and utilize low signal-to-noise ratio speckle information, so that the detector (active lidar blind spot detection system for autonomous driving) can achieve the detection performance of long distance, weak scattered light and millisecond response when traveling quickly, and use the speckle pattern of indirect scattering imaging to achieve high resolution and real-time recovery of the original image of the obscured object, this embodiment designs an algorithm for real-time reconstruction and motion prediction of the obscured object image based on an improved U-net deep convolutional network and a low signal-to-noise ratio speckle correlation map, such as Fig.12As shown in the figure. U-Net adopts a typical Encoder-Decoder structure. The encoder gradually reduces the spatial dimension, and the decoder gradually repairs the details and spatial dimension of the object. There is usually a jump connection between the two, which can help the decoder better repair the details of the target. The CNN network only needs a small amount of image data to train a high-precision end-to-end image segmentation network. The execution efficiency is very high and it is very suitable for non-line-of-sight speckle image reconstruction. In addition, in order to further improve the training efficiency, it is planned to introduce a dense connection layer (DenseNet), that is, each layer obtains the feature maps of all previous layers as input and uses its own feature maps as input for all subsequent layers, thereby effectively eliminating the network gradient disappearance, strengthening feature reuse, and greatly reducing the number of parameters.

[0110] CNN data preparation:

[0111] Constructing image training datasets. Deep learning is a powerful tool for solving computational imaging problems, but it requires a large number of training datasets that meet the requirements to be successful. In the context of non-line-of-sight imaging, it is obviously very difficult to establish a training dataset by collecting a large amount of experimental data that meets the requirements on site. This is because the data collection of speckle image samples under real road conditions is affected by various factors such as environmental lighting, scattering characteristics of irregular rough surfaces, diversity of obstructions, and morphological characteristics of hidden objects themselves. A comprehensive deterministic description of the scattering process requires large-scale data measurement.

[0112] Shot noise in the speckle pattern According to the statistical distribution law, the corresponding speckle pattern is formed through the open source image training data set, and random shot noise is added to synthesize the required training data set; this embodiment uses the Berkeley Segmentation Dataset and Benchmark500 open source image segmentation and boundary detection data set (BSDS500), which is completed by 30 human subjects and contains a training set of 200 images and a test set of 100 images. The BSDS500 is cropped by the Canny edge detector to form a training data set of about 2000064×64 sparse, unstructured, connected edge images, such as Fig.13 shown.

[0113] Constructing the loss function, since the imaging of the occluded object in this embodiment is sparse, the widely used loss functions include mean square error (MSE) and mean absolute error (MAE), which cannot promote image sparsity because their basic signals follow Gaussian and Laplace statistics respectively. Using average cross entropy as the loss function can effectively promote sparsity. Therefore, a pair of complementary channels is generated through the SoftMax layer to represent the object and the background respectively, and the average cross entropy is used as the loss function to promote sparsity. The loss function is:

[0114]

[0115] Where g is the segmentation gold standard pixel value; p is the predicted value; the average value is over all N pixels x on two channels c; g and p are both binary or continuous values;

[0116] Data preprocessing,Based on the real-time computing and millisecond response requirements,,speckle pattern data from simulations and experiments need to be,preprocessed before using CNN for training.

[0117] Crop a small part of the output speckle image, and set the resolution of all input and output images to the preset pixel (256×256). Since each part of the output speckle image contains the information of the entire target, even a small part of the speckle image can be used for target recognition. This process reduces the number of network parameters and the amount of data required for training without overfitting. Image cropping will not affect the final redefinition result. Artificially add simulated random noise to the speckle image according to the model (noise model), and the speckle image is normalized between 0 and 1.

[0118] CNN architecture implementation:

[0119] In order to effectively improve the training efficiency of CNN, this embodiment improves the two repeated convolution layers in the original CNN network of U-net, and modifies one of them into a dense connection layer. Each dense connection includes multiple layers consisting of batch normalization, rectified linear unit (ReLU) nonlinear activation function and convolution (conv) with 16 filters, such as Fig.14 As shown: According to the overall structure of U-net, it includes a left encoder compression path and a right decoder expansion path;

[0120] The encoder compression path consists of 4 blocks, each of which uses a 3×3 convolution, a dense connection operation, and a 2×2 max pooling layer (Max Pooling) downsampling with a stride of 2;

[0121] The initial image is downsampled to reduce the image size, and the number of feature channels is doubled after each downsampling. Finally, the encoder outputs a high-dimensional encoded feature map with a smaller size of 16×16 pixels (with 1088 activation maps);

[0122] The decoder expansion path consists of 4 blocks, each of which uses a 3×3 convolution, a dense connection operation, and a 2×2 deconvolution (up conv) with a stride of 2 for upsampling;

[0123] The low-resolution high-dimensional feature map is upsampled through 4 2×2 deconvolution operations with a step size of 2. Each deconvolution operation doubles the size of the input feature map and halves the number of feature maps. The cropped feature maps from the corresponding compression path are concatenated through jump connections until the size of the input image is finally restored.

[0124] A 1×1 additional convolutional layer is used to generate the final network output, complete the dual-channel speckle pattern original image restoration imaging, and generate the image of the occluded object. The algorithm for reconstructing the original image of the occluded object from the photon-deficient low-flux indirect speckle information is deeply studied and optimized, so that the detector can adapt to the speckle imaging of long distance, weak scattered light, and millisecond-level indirect scattered light imaging in a fast-moving state to achieve high resolution and quickly restore the original image of the occluded object.

[0125] Through the virtual prototype design and simulation analysis of the bionic scallop eye, the multi-layer micro-nano bionic silver film structure and MEMS manufacturing process optimization are realized, and the various components of the bionic MEMS detector are assembled and trimmed in an orderly manner. At the same time, the microcontroller, FPGA, CAN bus controller and various electronic components are assembled and integrated according to the predetermined control circuit diagram, and the compiled device control program and image reconstruction software are burned into the dedicated embedded controller or main control computer to complete the hardware and software system integration of the principle prototype. Fig.15 As shown, a dedicated test platform for non-line-of-sight hidden object detection and imaging was built under laboratory and real road conditions, and relevant tests on the equipment effectiveness, stability and safety were carried out. Various experimental data were collected and analyzed, and factors for improving efficiency were discovered through experiments, and the device was further improved and optimized.

[0126] Example 2

[0127] like Fig.16 As shown, based on Example 1, this Example 2 further provides an active laser radar blind spot detection method for the direction of autonomous driving, including: illuminating the environment in the direction of vehicle travel; and

[0128] Receive the scattered light reflected by the environment after being illuminated, and identify and generate an image of the obscured object based on the scattered light.

[0129] In this embodiment, the active laser radar blind spot detection method for the direction of autonomous driving is suitable for using the above-mentioned active laser radar blind spot detection system for the direction of autonomous driving to generate an image of the obscured object.

[0130] In summary, the present invention uses a light source, which is suitable for emitting laser to illuminate the environment in the direction of vehicle travel; a bionic scallop eye optical imaging subsystem, which is suitable for receiving scattered light reflected by the environment after illumination, and identifying and generating an image of the obscured object based on the scattered light, so that the autonomous driving vehicle can detect potential dangers in the front and surrounding areas in advance when driving, thereby improving the safety of the vehicle and protecting road users and public facilities without protection measures.

[0131] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0133] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. An active laser radar blind spot detection system for autonomous driving, characterized in that: include: a light source, the light source being adapted to emit laser light to illuminate an environment in a direction of travel of the vehicle; A bionic scallop eye optical imaging subsystem, wherein the bionic scallop eye optical imaging subsystem is adapted to receive scattered light reflected by the environment after being illuminated, and to identify and generate an image of an obscured object based on the scattered light; The bionic scallop eye optical imaging subsystem comprises: a dual-field off-axis imaging optical path; The dual-field-of-view off-axis imaging optical path is suitable for receiving scattered light reflected by the environment after being illuminated; The dual-field off-axis imaging optical path comprises: a wide-angle objective lens, a first plane reflector, a first bionic MEMS deformable reflector and a first CCD; A central field of view light beam in the environment in the direction of vehicle travel is converged by a wide-angle objective lens and reaches a first plane reflector, and is then converged to a first bionic MEMS deformable reflector by the first plane reflector. A speckle pattern is formed on a first CCD after the wavelength of the incident light is filtered by the first bionic MEMS deformable reflector.

2. The active laser radar blind spot detection system for automatic driving direction according to claim 1, characterized in that: The dual-field off-axis imaging optical path also includes: a collimating lens, a second plane reflector, a convex mirror, a second bionic MEMS deformable reflector and a second CCD; The peripheral vision light beam in the environment in the direction of vehicle travel is converged by a wide-angle objective lens and then collimated by a collimating lens. The parallel light beam passes through an off-axis three-reflection imaging system composed of a second plane mirror, a convex mirror, and a second plane mirror. The second bionic MEMS deformable mirror completes the incident light wavelength filtering and controllable variable curvature zooming functions. Finally, the light beam converges onto the second CCD to form a ring-shaped speckle pattern.

3. The active laser radar blind spot detection system for automatic driving direction according to claim 2, characterized in that: The bionic MEMS deformable reflector comprises: a microcontroller, a plurality of micromirror units and a plurality of parallel plate drivers; The micromirror units are independently controlled by at least one parallel plate driver; The microcontroller is suitable for controlling the parallel plate driver to drive the corresponding micromirror unit to deform, so that the bionic MEMS deformable reflector as a whole produces a local controlled deformation to achieve the desired mirror surface shape; The microcontroller is suitable for receiving a control signal sent by a host computer. The digital voltage signal is converted into an analog voltage signal after passing through a D / A converter. The microcontroller converts the control signal into a channel selection signal through a decoder, controls the corresponding amplifier channel to be turned on, and amplifies the analog voltage signal and applies it to the corresponding micromirror array.

4. The active laser radar blind spot detection system for automatic driving direction as claimed in claim 3, characterized in that: When the upper plate is not fixed, the plate has a small displacement △z in the vertical direction under the action of electrostatic force. The electrostatic force F e The work done on the plate is F e △z, according to the law of conservation of energy, the change in energy of the parallel plates is equal to the sum of the work done by the electrostatic force and the energy provided by the power supply: V is the potential difference between the two plates; d is the distance between the two plates; C is the capacitance of the parallel plate driver; When a voltage is applied to the parallel plate driver, the upper plate is subjected to an electrostatic force F e and elastic force F k After the interaction reaches the equilibrium position after moving a certain distance, according to Hooke's law: A is the area of ​​the plate; k is the equivalent elastic coefficient of the elastic structure; The actual working voltage and displacement relationship of the parallel plate driver is: The voltage and displacement curve of the parallel plate driver is obtained, so as to adjust the displacement of the parallel plate driver by adjusting the voltage and accurately obtain the required mirror surface shape.

5. The active laser radar blind spot detection system for automatic driving direction according to claim 4, characterized in that: The active laser radar blind spot detection system also includes: a host computer; The host computer is adapted to receive the speckle pattern sent by the bionic scallop eye optical imaging subsystem to obtain an image of the obscured object according to the speckle pattern, that is, CNN data preparation: Constructing an image training dataset by using shot noise in speckle patterns Statistical distribution law, forming the corresponding speckle pattern through the open source image training data set, and adding random shot noise to synthesize the required training data set; Construct the loss function, use the SoftMax layer to generate a pair of complementary channels to represent the object and the background respectively, and use the average cross entropy as the loss function to promote sparsity. The loss function is: Where g is the segmentation gold standard pixel value; p is the predicted value; the average value is obtained based on all N pixels x on two channels c; g and p are both binary or continuous values; Data preprocessing, cropping part of the output speckle image, setting the resolution of all input and output images to the preset pixels, adding simulated random noise to the speckle image according to the model, and normalizing the speckle image between 0 and 1; CNN architecture implementation: According to the overall structure of U-net, it includes a left encoder compression path and a right decoder expansion path; The encoder compression path consists of 4 blocks, each of which uses a 3×3 convolution, a dense concatenation operation, and a 2×2 max pooling layer downsampling with a stride of 2; The initial image is downsampled to reduce the image size. The number of feature channels is doubled after each downsampling. Finally, the encoder outputs a high-dimensional encoded feature map with a smaller size of 16×16 pixels. The decoder expansion path consists of 4 blocks, each of which uses a 3×3 convolution, a dense connection operation, and a 2×2 deconvolution with a stride of 2 for upsampling; The low-resolution high-dimensional feature map is upsampled through 4 2×2 deconvolution operations with a step size of 2. Each deconvolution operation doubles the size of the input feature map and halves the number of feature maps. The cropped feature maps from the corresponding compression path are concatenated through jump connections until the size of the input image is finally restored. A 1×1 additional convolutional layer is used to produce the final network output, completing the restoration of the original image of the dual-channel speckle pattern and generating an image of the occluded object.

6. An active laser radar blind spot detection method for automatic driving direction, characterized in that: include: Illuminate the environment in the direction of vehicle travel; as well as Receive scattered light reflected by the environment after being illuminated, and identify and generate an image of the obscured object based on the scattered light; The active laser radar blind spot detection method for the direction of autonomous driving is suitable for generating an image of an obstructed object using the active laser radar blind spot detection system for the direction of autonomous driving as described in any one of claims 1 to 5.

Images