Vehicle obstacle detection and night vision imaging method and system

Abstract

The invention discloses a vehicle obstacle detection and night vision imaging method and system, and belongs to the technical field of vehicle-mounted environment perception. According to the method, a dynamic threat evaluation model is constructed and cross-modal feature enhancement is realized by fusing millimeter-wave radar and infrared thermal imaging data. Performing cross-modal feature fusion on the point cloud depth information, the RCS reflectivity and the infrared image to generate a multi-dimensional feature matrix; and performing super-resolution reconstruction on the infrared image by using a generative adversarial network (GAN) in combination with a spatial feature transformation module, and outputting an enhanced night vision image. The method adapts to a complex environment through dynamic weight adjustment, combines multi-sensor space-time alignment and condition constraint reconstruction, and improves obstacle detection precision and night vision imaging quality.

Description

Technical Field

[0001] This invention relates to the field of vehicle-mounted environmental perception technology, and more specifically, to a vehicle obstacle detection and night vision imaging method and system. Background Technology

[0002] With the rapid development of autonomous driving technology and advanced driver assistance systems (ADAS), the accuracy and real-time performance requirements for vehicle perception of the surrounding environment are becoming increasingly stringent. Accurately identifying obstacles such as vehicles, pedestrians, and falling rocks ahead of the vehicle during operation is a core element in ensuring driving safety.

[0003] Current vehicle-mounted environmental perception systems primarily rely on sensors such as visible light cameras, millimeter-wave radar, and lidar. However, under complex lighting conditions and adverse weather conditions, such as at night, in tunnels, in fog or glare, the imaging quality of traditional visible light cameras deteriorates significantly or even fails completely, failing to provide effective visual information. To address this issue, infrared thermal imaging technology, due to its independence from ambient light sources, strong smoke and fog penetration capabilities, and ability to perform temperature-difference-based imaging, is increasingly being applied to vehicle-mounted night vision systems.

[0004] Despite the unique advantages of infrared thermal imaging technology in night vision and adverse weather conditions, existing vehicle obstacle detection and night vision imaging technologies still have the following significant drawbacks and limitations:

[0005] Compared to visible light images, infrared thermal images typically have lower resolution and are prone to losing edge contours and texture details of objects, resulting in blurry images that make it difficult for drivers or algorithms to accurately determine the specific category and three-dimensional shape of obstacles. Although existing image super-resolution techniques use interpolation or ordinary deep learning networks to amplify images, they often lack physical constraints and are prone to artifacts or "illusion" details, leading to distortion in the reconstructed image.

[0006] Existing radar and vision fusion solutions mostly remain at the level of simple overlay at the decision-making stage, or only utilize radar to provide distance information. Few technologies can deeply leverage the physical characteristics of millimeter-wave radar to assist in the low-level reconstruction of infrared images, resulting in the ineffective use of radar's spatial structure information to improve the imaging quality of visual sensors.

[0007] Traditional fusion sensing systems typically operate in a "full-time, full-area" state, meaning that infrared cameras continuously acquire and process data at the highest resolution and full frame. However, in actual driving conditions, not all areas pose a threat. High-frequency processing at full frame not only consumes a significant amount of onboard computing resources but also increases the power consumption and heat generation of the infrared detectors, affecting the system's lifespan.

[0008] Therefore, there is an urgent need for a vehicle obstacle detection and night vision imaging method that can combine the all-weather detection advantages of millimeter-wave radar, intelligently trigger infrared imaging through dynamic threat assessment, and use radar features to guide infrared images for super-resolution reconstruction, in order to solve the problems of low imaging clarity, low perception efficiency, and poor fusion effect of existing technologies at night and in harsh environments. Summary of the Invention

[0009] One objective of this invention is to provide a vehicle obstacle detection and night vision imaging method to solve the problems of low imaging clarity, low perception efficiency, and poor fusion effect in existing technologies at night and in adverse environments.

[0010] According to a first aspect of the present invention, a method for vehicle obstacle detection and night vision imaging is provided, comprising the following steps:

[0011] Acquire sparse point cloud data of obstacles collected by millimeter-wave radar, wherein the sparse point cloud data includes obstacle orientation and velocity information;

[0012] A dynamic threat assessment model is constructed based on the radar cross section (RCS) of obstacles to calculate the threat coefficient of obstacles;

[0013] When the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to acquire an infrared thermal image of the obstacle area;

[0014] The sparse point cloud data and the infrared thermal image are fused across modal features to generate a multidimensional feature matrix containing depth information.

[0015] The infrared thermal image is super-resolution reconstructed based on a generative adversarial network (GAN) to output an enhanced night view image, wherein the GAN is constrained by the multidimensional feature matrix.

[0016] Optionally, a dynamic threat assessment model is constructed based on the obstacle's radar cross section (RCS), specifically including:

[0017] The radar cross section is normalized, and obstacles are divided into high-reflectivity and low-reflectivity categories based on the RCS amplitude characteristics, and basic risk weights are matched accordingly.

[0018] The vehicle’s current speed is obtained, and the collision time (TTC) is calculated by combining the location and speed of the obstacle.

[0019] The dynamic threat factor I is calculated using the following formula. threat : ,

[0020] Among them, RCS norm V is the normalized radar cross section value.vis L represents the visibility value. lux Here, α, β, and γ represent the light intensity value, and α, β, and γ are preset environmental impact factor weighting coefficients; when visibility V... vis When the value is below the preset threshold, the weighting coefficient α is automatically increased.

[0021] Optionally, when the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to acquire an infrared thermal image of the obstacle area, specifically including:

[0022] Based on the obstacle's orientation and distance information, calculate the coordinates of its projected area on the imaging plane of the infrared imaging module;

[0023] A region of interest (ROI) window is established with the coordinates of the projected area as the center;

[0024] A window read command is sent to the infrared imaging module to activate only the pixel array within the region of interest for photosensitive processing, generating an initial infrared thermal image of local high-frequency scanning.

[0025] Optionally, after acquiring the sparse point cloud data of obstacles collected by millimeter-wave radar, spatiotemporal alignment processing is also included, specifically including:

[0026] Obtain the time difference Δt between the millimeter-wave radar data acquisition time T1 and the infrared imaging module exposure time T2;

[0027] Calculate the vehicle displacement vector over time Δt using the vehicle's own kinematic parameters;

[0028] The sparse point cloud data is compensated for coordinates based on the vehicle displacement vector, and the compensated point cloud data is projected onto the infrared image plane through a joint calibration matrix to complete spatiotemporal alignment.

[0029] Optionally, generating a multidimensional feature matrix containing depth information specifically includes:

[0030] Create a multi-channel feature tensor, where the first channel is the grayscale data of the initial infrared thermal image;

[0031] The second channel is a sparse depth map, which consists of the distance values ​​of the projected point cloud. Zero values ​​are filled for pixel positions that are not covered by the point cloud.

[0032] The third channel is a sparse reflectance map, which is composed of the RCS values ​​of the projected point cloud.

[0033] Gaussian kernel diffusion is applied to the sparse depth map and sparse reflectance map to generate a denser spatial guiding feature layer. The spatial guiding feature layer is then concatenated with the grayscale data to form the multidimensional feature matrix.

[0034] Optionally, in the step of super-resolution reconstruction of the infrared thermal image based on a generative adversarial network, the generative adversarial network includes a generator and a discriminator, wherein:

[0035] The generator adopts a U-Net architecture and embeds a Spatial Feature Transform (SFT) module between the encoder and decoder;

[0036] The radar spatial structure information in the multidimensional feature matrix is ​​used as a conditional prior and input into the spatial feature transformation module to perform affine transformation modulation on the infrared image features, thereby restoring the edge texture blurred by environmental noise in the feature extraction stage.

[0037] Optionally, the generative adversarial network uses the multidimensional feature matrix as a conditional constraint, specifically reflected in the design of the loss function of the generative adversarial network:

[0038] The loss function includes a geometric consistency loss term L. geo ;

[0039] The depth estimate of the generated night view image in the corresponding multidimensional feature matrix is ​​calculated, and the mean square error between this depth estimate and the original radar ranging value is calculated as the geometric consistency loss term L. geo .

[0040] Optionally, super-resolution reconstruction of the infrared thermal image is performed based on a generative adversarial network to output an enhanced night view image, specifically including:

[0041] The output of the generative adversarial network is connected to a multi-task head.

[0042] The multi-task detection head outputs two sets of data in parallel: one is an RGB or grayscale image reconstructed by super-resolution, and the other is the category probability distribution vector and 3D bounding box regression parameters of obstacles in the image.

[0043] The generated 3D bounding box regression parameters are mapped back to the vehicle coordinate system and verified against the original radar point cloud data. If the overlap ratio (IoU) is lower than the threshold, the image is marked as a low-confidence sample and fed back to the network for online fine-tuning.

[0044] Optionally, the step of calculating the threat coefficient of an obstacle further includes: constructing a dynamic threat assessment model based on the obstacle's radar cross section (RCS) and environmental parameters, wherein the environmental parameters are obtained in the following manner:

[0045] By analyzing the noise floor level and clutter distribution characteristics of millimeter-wave radar, the current rain and fog concentration is estimated as a visibility parameter.

[0046] The present invention also provides a vehicle obstacle detection and night vision imaging system, comprising:

[0047] A millimeter-wave radar module is configured to collect sparse point cloud data of obstacles, wherein the sparse point cloud data includes azimuth and velocity information;

[0048] The threat assessment module is connected to the millimeter-wave radar module and constructs a dynamic threat assessment model based on the radar cross section (RCS) of the obstacle to calculate the threat coefficient.

[0049] An infrared imaging module is connected to the threat assessment module. When the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to perform a local high-frequency scan of the obstacle area to obtain an infrared thermal image.

[0050] The cross-modal fusion module is used to map the sparse point cloud data and the infrared thermal image to a unified coordinate system to generate a multi-dimensional feature matrix containing depth information.

[0051] The adversarial image enhancement module includes a generative adversarial network (GAN) that performs super-resolution reconstruction of the infrared thermal image using the multidimensional feature matrix as a conditional constraint, and outputs an enhanced night view image.

[0052] The vehicle obstacle detection and night vision imaging method and system disclosed herein have the following technical advantages:

[0053] A super-resolution reconstruction technique based on Generative Adversarial Networks (GANs) is employed, using a multi-dimensional feature matrix containing depth information generated by millimeter-wave radar as a conditional constraint. By embedding a Spatial Feature Transform (SFT) module into the GAN, the high-precision spatial structure information and RCS reflectivity features of the radar are used as prior conditions to perform affine transformation modulation on the infrared features. This results in not only improved resolution of the reconstructed night view image but also the recovery of edge textures blurred by environmental noise.

[0054] Furthermore, by introducing the geometric consistency loss term Lgeo, the generated image depth is forced to be consistent with the radar physical ranging value, which effectively avoids the artifacts and "illusion" details that are easy to produce in traditional super-resolution algorithms, and ensures the physical authenticity of the imaging.

[0055] This invention constructs a dynamic threat assessment model based on obstacle RCS. The infrared imaging module is only triggered when the dynamic threat coefficient of an obstacle exceeds a threshold. By reading instructions through a region of interest (ROI) window, only the pixel array corresponding to the obstacle region in the infrared detector is activated, and the integration time is reduced. This mechanism reduces the amount of invalid background data processed, lowering overall system power consumption and heat generation. Furthermore, the smaller reading area significantly improves the imaging frame rate, thereby enabling more sensitive capture of dynamic changes in high-speed moving obstacles.

[0056] By constructing a multi-channel feature tensor containing sparse depth and sparse reflectance maps, the radar's sensitivity to object material properties (RCS) is incorporated into the visual imaging process. Simultaneously, this invention considers the impact of environmental parameters on perception, introducing visibility and illumination intensity factors into the threat coefficient calculation formula, and utilizing radar noise floor inversion to estimate rain and fog concentration. When reduced visibility is detected, the system automatically increases the weighting coefficient α to improve detection sensitivity. This allows the system to maintain highly reliable detection capabilities even in adverse weather conditions where infrared thermal radiation may attenuate, thanks to the radar's penetration characteristics.

[0057] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0058] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0059] Figure 1 This is a flowchart illustrating the vehicle obstacle detection and night vision imaging method provided in an embodiment of the present invention.

[0060] Figure 2 This is a schematic diagram of the vehicle obstacle detection and night vision imaging system provided in an embodiment of the present invention. Detailed Implementation

[0061] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0062] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0063] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0064] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0065] This invention proposes an embodiment of a vehicle obstacle detection and night vision imaging method, specifically, as follows: Figure 1 As shown, it includes the following steps:

[0066] The method involves acquiring sparse point cloud data of obstacles from millimeter-wave radar, which includes obstacle location and velocity information. The millimeter-wave radar transmits a frequency-modulated continuous wave (FMCW) signal, receives the reflected echo from the target, and calculates the distance, velocity, and azimuth of the obstacle using the Doppler effect and time delay difference. Furthermore, based on density-based target clustering, parameters are adaptively adjusted to distinguish between dense obstacles and discrete noise. Kalman filtering is applied to continuous frame point cloud data for tracking, predicting target trajectories, and supplementing positional information between sparse frames.

[0067] A dynamic threat assessment model is constructed based on the radar cross section (RCS) of obstacles to calculate the threat coefficient of obstacles, specifically including:

[0068] The radar cross section is normalized, and obstacles are divided into high-reflectivity and low-reflectivity categories based on the RCS amplitude characteristics. Basic risk weights are matched accordingly. The range method is used to map the RCS value to the [0,1] interval. The high-reflectivity category (RCSnorm≥0.7) corresponds to strong reflective targets such as metal vehicles and buildings, while the low-reflectivity category (RCSnorm<0.7) corresponds to weak reflective targets such as pedestrians, animals, and plastic products.

[0069] The vehicle's current speed is obtained, and the time of collision (TTC) is calculated by combining the location and speed of the obstacle. Specifically, the vehicle's speed is acquired via the CAN bus, and the relative radial velocity V of the obstacle is measured by millimeter-wave radar using the Doppler effect. rel and relative distance D,

[0070] Determine whether the obstacle is within the vehicle's expected driving trajectory based on its location information. If the target is in an adjacent lane and shows no lateral approach tendency, set its TTC to infinity or reduce its priority; for obstacles within the trajectory, calculate... The smaller the TTC value, the more urgent the collision risk.

[0071] The dynamic threat factor I is calculated using the following formula. threat : ,

[0072] Among them, RCS norm V is the normalized radar cross section value. vis L represents the visibility value. lux Here, α, β, and γ represent the light intensity value, and α, β, and γ are preset environmental impact factor weighting coefficients; when visibility V... visWhen the value is below a preset threshold, the weighting coefficient α is automatically increased, enabling dynamic switching of sensor confidence. When vision is obstructed, the radar weighting ratio is proactively increased, fully utilizing the physical characteristics of millimeter-wave radar—its ability to penetrate rain and fog and its immunity to light—ensuring the continuity and reliability of obstacle detection in extreme weather conditions.

[0073] In one embodiment of the present invention, after acquiring sparse point cloud data of obstacles collected by millimeter-wave radar, the method further includes spatiotemporal alignment processing, specifically including:

[0074] Obtain the time difference Δt between the millimeter-wave radar data acquisition time T1 and the infrared imaging module exposure time T2. It should be noted that when the system is ready to process an infrared image frame (time T2), it will search for the radar point cloud frame with the closest timestamp (time T1) in the radar data cache queue and calculate the time difference Δt = T2 − T1 between the two.

[0075] Using the vehicle's own kinematic parameters, the vehicle displacement vector within time interval Δt is calculated. It should be noted that the average longitudinal velocity v and yaw rate ω of the vehicle during time interval Δt are obtained from the vehicle's CAN bus or IMU (Inertial Measurement Unit). The distance the vehicle travels forward is Δx ≈ v × Δt. If the vehicle is turning, the rotation of the vehicle coordinate system Δθ ≈ ω × Δt, as well as the lateral offset caused by turning, must be calculated in conjunction with the yaw rate. Finally, a transformation matrix containing translation and rotation information is output, describing the vehicle's attitude change from time T1 to time T2.

[0076] The sparse point cloud data is coordinate compensated based on the vehicle displacement vector, and the compensated point cloud data is projected onto the infrared image plane through a joint calibration matrix to achieve spatiotemporal alignment. It should be noted that the radar point cloud PT1 acquired at time T1 is transformed to the vehicle coordinate system at time T2 using a transformation matrix to obtain the compensated point cloud.

[0077] Using pre-calibrated extrinsic matrix (rotation and translation relationship from the radar coordinate system to the infrared camera coordinate system) and intrinsic matrix (focal length, principal point, and distortion coefficients of the infrared camera), the compensated point cloud is projected onto the two-dimensional infrared image plane to obtain pixel coordinates (u, v). At this point, each pixel in the infrared image accurately corresponds to the radar depth information at the same physical moment. This scheme corrects the rigidity error caused by the vehicle's motion through kinematic compensation, ensuring that the radar point cloud accurately falls on the obstacle area corresponding to the infrared image, providing a high-precision geometric reference for subsequent feature fusion.

[0078] When the threat level exceeds a preset threshold, the infrared imaging module is triggered to acquire an infrared thermal image of the obstacle area; specifically including:

[0079] Based on the obstacle's orientation and distance information, calculate the coordinates of its projected area on the imaging plane of the infrared imaging module;

[0080] A region of interest (ROI) window is established with the coordinates of the projected area as the center;

[0081] A window readout command is sent to the infrared imaging module, activating only the pixel array within the region of interest (ROI) for photosensitive processing. The integration time is reduced to increase the frame rate, generating an initial infrared thermal image of a localized high-frequency scan. It should be noted that the size of the ROI window is not fixed. The system automatically adjusts the window size based on the obstacle's location and type; for example, a larger window for a radar-classified truck and a smaller window for a pedestrian. Based on the calculated theoretical bounding box, the system automatically expands the surrounding area with a certain proportion of pixel margin to accommodate radar detection positioning errors and minor object displacements during sensor communication delays, preventing targets from "running out" of the field of view. This step does not involve cropping the entire image on the image processing chip (ISP), but rather directly affects the readout circuit (ROIC) of the infrared detector. By sending specific row and column gating commands, charge integration and signal readout are performed only on the photosensitive units within the ROI, ignoring the background area.

[0082] It's important to further explain that the frame rate of infrared detectors is typically limited by the full-pixel readout time. When the readout area shrinks from the full frame to a local ROI, the amount of data decreases, and the readout cycle is significantly shortened. By "reducing the integration time," motion blur is eliminated at its physical source, ensuring sharp edges in the generated initial infrared thermal image. This provides a clear contour foundation for subsequent GAN-based super-resolution reconstruction, resulting in a more realistic and accurate enhanced image.

[0083] The sparse point cloud data and the infrared thermal image are fused across modalities to generate a multidimensional feature matrix containing depth information; specifically including:

[0084] Create a multi-channel feature tensor and construct a data volume with dimensions (H×W×C) in memory, where H and W are the height and width of the image, and C is the number of channels; the first channel is the grayscale data of the initial infrared thermal image; the first channel stores the normalized grayscale values ​​of the infrared thermal image, with pixel values ​​0-255 normalized to 0-1, and this channel provides the two-dimensional contour, texture distribution and thermal radiation features of the object;

[0085] The second channel is a sparse depth map, composed of the distance values ​​of the projected point cloud. Pixels not covered by the point cloud are filled with zero values. Specifically, the distance values ​​of the radar point cloud are projected onto the image coordinates. Apart from a few dozen points with specific distance values, the remaining tens of thousands of pixels, lacking radar echoes, are initialized to a background value of 0, forming a sparse depth map that is "mostly black with scattered bright spots."

[0086] The third channel is a sparse reflectance map, which is composed of the RCS values ​​of the projected point cloud.

[0087] Gaussian kernel diffusion is applied to the sparse depth map and sparse reflectance map to generate a denser spatial guiding feature layer. This spatial guiding feature layer is then concatenated with the grayscale data to form the multidimensional feature matrix. After diffusion, an originally isolated point becomes a "spot of light." For example, a radar point at the center of a vehicle, after diffusion, covers most of the vehicle's image area. At this point, neighboring pixels that were originally 0 are assigned interpolation weights based on distance and RCS. Introducing the RCS reflectance map as an independent channel provides a material prior for image reconstruction and detection. Metallic objects (high RCS) and non-metallic objects (low RCS) are distinctly different in the third channel. The GAN network utilizes this information to selectively sharpen metallic edges (vehicles) or soften biological edges (pedestrians) during reconstruction, thereby more accurately reconstructing target features visually.

[0088] In one embodiment of the present invention, the infrared thermal image is super-resolution reconstructed based on a generative adversarial network (GAN) to output an enhanced night view image, wherein the GAN is constrained by the multidimensional feature matrix; wherein the GAN includes a generator and a discriminator, wherein:

[0089] The generator employs a U-Net (Encoder-Decoder) architecture, embedding a Spatial Feature Transform (SFT) module between the encoder and decoder. It's important to note that traditional fusion methods typically involve directly concatenating radar and image data before inputting them into the network. However, this approach assumes equal importance for both types of data, often resulting in radar information being overwhelmed by image noise. The SFT module, on the other hand, treats radar information as a "condition" to actively modulate image features.

[0090] The radar spatial structure information in the multidimensional feature matrix is ​​used as a conditional prior and input into the spatial feature transformation module to perform affine transformation modulation on the infrared image features, thereby recovering the edge texture blurred by environmental noise during the feature extraction stage. In each layer of U-Net, the network examines the current radar feature map. If the infrared image is blurry in a certain area, the SFT module forces the generator to synthesize a sharp edge at that location based on the radar's high-confidence geometric information, thus achieving "deblurring" and "super-resolution." This ensures that the generated texture details conform to the physical structure detected by the radar, resulting in output night view images with extremely high physical fidelity and avoiding the generation of false targets.

[0091] In one embodiment of the present invention, the generative adversarial network is constrained by the multidimensional feature matrix, specifically reflected in the design of the loss function of the generative adversarial network:

[0092] The loss function includes a geometric consistency loss term L. geo ;

[0093] The depth estimate of the generated night view image in the corresponding multidimensional feature matrix is ​​calculated, and the mean square error between this depth estimate and the original radar ranging value is calculated as the geometric consistency loss term L. geo The algorithm forces the generated image contours to conform to the physical reality of radar detection. Lgeo provides clear directional guidance for the optimization process, reducing the search space for network parameters. This allows the network to converge to the optimal solution more quickly, and even when faced with extremely dark or textureless areas, it can still generate reasonable structures based on the radar's depth constraints, rather than outputting noise.

[0094] In one embodiment of the present invention, the infrared thermal image is super-resolution reconstructed based on a generative adversarial network (GAN) to output an enhanced night view image. Specifically, the output of the GAN is connected to a multi-task head. The multi-task head outputs two sets of data in parallel: one is the super-resolution reconstructed RGB or grayscale image, and the other is the category probability distribution vector and 3D bounding box regression parameters for obstacles in the image. The generated 3D bounding box regression parameters are mapped back to the vehicle coordinate system and verified against the original radar point cloud data. If the overlap ratio (IoU) is lower than a threshold, the image is marked as a low-confidence sample and fed back to the network for online fine-tuning. This online fine-tuning mechanism allows the vehicle to continuously accumulate experience from its mistakes and self-correct during actual driving. As the mileage increases, the system becomes more adaptable to deviations caused by specific environments or sensor aging.

[0095] In another embodiment of the present invention, the step of calculating the threat coefficient of an obstacle further includes: constructing a dynamic threat assessment model based on the obstacle's radar cross section (RCS) and environmental parameters. The environmental parameters are obtained by analyzing the noise floor level and clutter distribution characteristics of millimeter-wave radar to estimate the current rain and fog concentration as a visibility parameter. By inverting visibility through radar, when the radar detects heavy rain (low visibility), the system automatically switches its strategy, increasing the radar weight α, reducing reliance on infrared / visual images, and placing greater trust in the radar's obstacle detection results, thereby improving driving safety in adverse weather conditions.

[0096] The present invention also provides an embodiment of a vehicle obstacle detection and night vision imaging system, specifically as follows: Figure 2 As shown, it includes:

[0097] A millimeter-wave radar module is configured to collect sparse point cloud data of obstacles, wherein the sparse point cloud data includes azimuth and velocity information;

[0098] The threat assessment module is connected to the millimeter-wave radar module and constructs a dynamic threat assessment model based on the radar cross section (RCS) of the obstacle to calculate the threat coefficient.

[0099] The infrared imaging module is connected to the threat assessment module. When the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to perform a local high-frequency scan of the obstacle area to obtain an infrared thermal image. The infrared imaging module does not work continuously, but only triggers a local high-frequency scan when the threat coefficient exceeds the threshold, balancing detection accuracy and energy consumption.

[0100] The cross-modal fusion module is used to map the sparse point cloud data and the infrared thermal image to a unified coordinate system to generate a multi-dimensional feature matrix containing depth information.

[0101] The adversarial image enhancement module includes a generative adversarial network (GAN) that performs super-resolution reconstruction of the infrared thermal image using the multi-dimensional feature matrix as a constraint, outputting an enhanced night view image. The adversarial image enhancement module incorporates a GAN unit to complete the infrared image super-resolution reconstruction using the multi-dimensional feature matrix as a constraint, ensuring the clarity and depth information integrity of the night view image.

[0102] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. It is a system corresponding to the above vehicle obstacle detection and night vision imaging method. All implementation methods in the above method embodiments are applicable to the embodiments of this system. For details on its specific functions and the resulting technical effects, please refer to the method embodiment section. It will not be repeated here.

[0103] This invention acquires sparse point cloud data of obstacles using millimeter-wave radar, constructs a dynamic threat assessment model centered on RCS normalization and dynamic environmental weights, calculates obstacle threat coefficients in real time, and triggers an infrared imaging module to acquire local high-frame-rate thermal images. A cross-modal feature fusion technique is employed to stitch point cloud depth information, RCS reflectivity, and infrared images into a multi-dimensional feature matrix. This matrix is ​​then combined with a generative adversarial network (GAN) for conditionally constrained super-resolution reconstruction, outputting enhanced night vision images. The dynamic threat assessment model achieves accurate risk assessment under complex weather conditions through RCS classification and adaptive adjustment of environmental impact factors. In heavy rain (15m visibility) scenarios, the threat misjudgment rate is only 7%, the night vision image resolution reaches 1280×720, and the single-frame processing latency is <50ms, outperforming traditional single-sensor solutions and providing all-weather, highly reliable environmental perception capabilities for autonomous driving systems.

[0104] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for vehicle obstacle detection and night vision imaging, characterized in that, Includes the following steps: Acquire sparse point cloud data of obstacles collected by millimeter-wave radar, wherein the sparse point cloud data includes obstacle orientation and velocity information; A dynamic threat assessment model is constructed based on the radar cross section of obstacles to calculate the threat coefficient of obstacles. When the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to acquire an infrared thermal image of the obstacle area; The sparse point cloud data and the infrared thermal image are fused across modal features to generate a multidimensional feature matrix containing depth information. The infrared thermal image is super-resolution reconstructed based on a generative adversarial network (GAN) to output an enhanced night view image, wherein the GAN is constrained by the multidimensional feature matrix.

2. The vehicle obstacle detection and night vision imaging method according to claim 1, characterized in that, A dynamic threat assessment model based on the radar cross section of obstacles is constructed, specifically including: The radar cross section is normalized, and obstacles are divided into high-reflectivity and low-reflectivity categories based on the RCS amplitude characteristics, and basic risk weights are matched accordingly. The vehicle’s current speed is obtained, and the collision time (TTC) is calculated by combining the location and speed of the obstacle. The dynamic threat coefficient I is calculated by the following equation threat : ； Wherein, RCS norm is a normalized radar cross section value, V vis is a visibility value, L lux is an illumination intensity value, and α, β, and γ are preset environmental influence factor weight coefficients; when the visibility V vis is lower than a preset threshold, the value of the weight coefficient α is automatically increased.

3. The vehicle obstacle detection and night vision imaging method according to claim 2, characterized in that, When the threat level exceeds a preset threshold, the infrared imaging module is triggered to acquire an infrared thermal image of the obstacle area, specifically including: Based on the obstacle's orientation and distance information, calculate the coordinates of its projected area on the imaging plane of the infrared imaging module; A region of interest window is established centered on the coordinates of the projected region; A window read command is sent to the infrared imaging module to activate only the pixel array within the region of interest for photosensitive processing, generating an initial infrared thermal image of local high-frequency scanning.

4. The vehicle obstacle detection and night vision imaging method according to claim 1, characterized in that, After acquiring sparse point cloud data of obstacles collected by millimeter-wave radar, spatiotemporal alignment processing is also performed, specifically including: Obtain the time difference Δt between the millimeter-wave radar data acquisition time T1 and the infrared imaging module exposure time T2; Calculate the vehicle displacement vector over time Δt using the vehicle's own kinematic parameters; The sparse point cloud data is compensated for coordinates based on the vehicle displacement vector, and the compensated point cloud data is projected onto the infrared image plane through a joint calibration matrix to complete spatiotemporal alignment.

5. The vehicle obstacle detection and night vision imaging method according to claim 1, characterized in that, The generation of a multidimensional feature matrix containing depth information specifically includes: Create a multi-channel feature tensor, where the first channel is the grayscale data of the initial infrared thermal image; The second channel is a sparse depth map, which consists of the distance values ​​of the projected point cloud. Zero values ​​are filled for pixel positions that are not covered by the point cloud. The third channel is a sparse reflectance map, which is composed of the RCS values ​​of the projected point cloud. Gaussian kernel diffusion is applied to the sparse depth map and sparse reflectance map to generate a denser spatial guiding feature layer. The spatial guiding feature layer is then concatenated with the grayscale data to form the multidimensional feature matrix.

6. The vehicle obstacle detection and night vision imaging method according to claim 1, characterized in that, In the step of super-resolution reconstruction of the infrared thermal image based on a generative adversarial network (GAN), the GAN comprises a generator and a discriminator, wherein: The generator adopts a U-Net architecture and embeds a spatial feature transformation module between the encoder and decoder; The radar spatial structure information in the multidimensional feature matrix is ​​used as a conditional prior and input into the spatial feature transformation module to perform affine transformation modulation on the infrared image features, thereby restoring the edge texture blurred by environmental noise in the feature extraction stage.

7. The vehicle obstacle detection and night vision imaging method according to claim 6, characterized in that, The generative adversarial network uses the multidimensional feature matrix as a conditional constraint, which is specifically reflected in the design of the loss function of the generative adversarial network: The loss function includes a geometric consistency loss term L. geo ; The depth estimate of the generated night view image in the corresponding multidimensional feature matrix is ​​calculated, and the mean square error between this depth estimate and the original radar ranging value is calculated as the geometric consistency loss term L. geo .

8. The vehicle obstacle detection and night vision imaging method according to claim 1, characterized in that, The infrared thermal image is super-resolution reconstructed based on a generative adversarial network to output an enhanced night view image, specifically including: The output of the generative adversarial network is connected to a multi-task detection head. The multi-task detection head outputs two sets of data in parallel: one is an RGB or grayscale image reconstructed by super-resolution, and the other is the category probability distribution vector and 3D bounding box regression parameters of obstacles in the image. The generated 3D bounding box regression parameters are mapped back to the vehicle coordinate system and verified against the original radar point cloud data. If the overlap rate is lower than the threshold, the image is marked as a low-confidence sample and fed back to the network for online fine-tuning.

9. The vehicle obstacle detection and night vision imaging method according to any one of claims 1 to 8, characterized in that, The step of calculating the threat coefficient of an obstacle also includes: constructing a dynamic threat assessment model based on the obstacle's radar cross section and environmental parameters, wherein the environmental parameters are obtained in the following way: By analyzing the noise floor level and clutter distribution characteristics of millimeter-wave radar, the current rain and fog concentration is estimated as a visibility parameter.

10. A vehicle obstacle detection and night vision imaging system, characterized in that, include: A millimeter-wave radar module is configured to collect sparse point cloud data of obstacles, wherein the sparse point cloud data includes azimuth and velocity information; The threat assessment module is connected to the millimeter-wave radar module, and constructs a dynamic threat assessment model based on the radar cross section of the obstacle to calculate the threat coefficient. An infrared imaging module is connected to the threat assessment module. When the threat coefficient exceeds a preset threshold, the infrared imaging module is triggered to perform a local high-frequency scan of the obstacle area to obtain an infrared thermal image. The cross-modal fusion module is used to map the sparse point cloud data and the infrared thermal image to a unified coordinate system to generate a multi-dimensional feature matrix containing depth information. The adversarial image enhancement module includes a generative adversarial network that performs super-resolution reconstruction of the infrared thermal image using the multidimensional feature matrix as a conditional constraint, and outputs an enhanced night view image.

Images