Task-driven defogging perception method and system
By using AI smart glasses and in-vehicle systems for collaborative perception, and leveraging the PTP protocol and diffusion model for image enhancement and feature fusion, the problem of blind spots in autonomous driving under fog and haze has been solved, improving the robustness and perception capabilities of autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2026-04-28
- Publication Date
- 2026-07-14
AI Technical Summary
Haze causes light scattering, resulting in reduced image contrast and blurred details. Existing technologies focus on visual restoration while neglecting downstream detection tasks. There are blind spots in perception from a single vehicle's perspective in dense fog. How can we improve the robustness of autonomous driving?
Through collaborative perception between AI smart glasses and vehicle systems, image stream synchronization is achieved using the PTP protocol, enabling image enhancement and feature fusion. Defogging is then performed by combining prior atmospheric physics knowledge with a diffusion model constrained by scene depth gradients.
It achieves panoramic and close-up fusion of the AI smart glasses and the vehicle camera's perspective, identifies near-field obstacles missed by the vehicle camera, improves the reliability and robustness of perception, shortens inference time, and balances computational requirements.
Smart Images
Figure CN122115278B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of vehicle-to-everything (V2X) collaborative perception technology, and in particular relates to a task-driven defogging perception method and system. Background Technology
[0002] In autonomous driving systems, light scattering caused by fog and haze can lead to severe quality degradation, such as reduced image contrast and blurred details, resulting in domain shift issues in deep learning models. Current technologies primarily focus on visual restoration while neglecting downstream detection tasks, and single-vehicle perspectives suffer from blind spots in dense fog. With the development of edge AI technology, AI smart glasses with first-person perception are integrating into the ecosystem as novel mobile edge nodes, providing heterogeneous visual sources and enabling computing power sharing. Therefore, combining the dynamic perspective of AI smart glasses with the vehicle platform to build a task-driven collaborative defogging perception system is crucial for improving the robustness of autonomous driving. Summary of the Invention
[0003] To address the aforementioned technical problems, this invention proposes a task-driven defogging sensing method and system.
[0004] In a first aspect, the present invention proposes a task-driven dehazing perception method, comprising:
[0005] The AI smart glasses' first-view sensor captures a first foggy image stream, while the vehicle system's second-view sensor captures a second foggy image stream. The AI smart glasses are fixed to the driver's head, and the second foggy image stream is synchronized with the first foggy image stream in the time domain.
[0006] A first foggy image is extracted from a first foggy image stream, and processed to obtain a first processed image; a second foggy image is extracted from a second foggy image stream, and processed to obtain a second processed image; the second foggy image and the first foggy image are time-aligned.
[0007] Geometric alignment is performed on the first processed image to obtain a first aligned image, and geometric alignment is performed on the second processed image to obtain a second aligned image;
[0008] The first aligned image is subjected to dehazing and image enhancement processing, and the features of the image after processing the first aligned image are extracted as the first feature; the second aligned image is subjected to dehazing and image enhancement processing, and the features of the image after processing the second aligned image are extracted as the second feature;
[0009] The first and second features are repaired using a diffusion model, and the repaired first and second features are fused together. The final perception result is determined based on the fused features.
[0010] Optionally, the second foggy image stream is synchronized with the first foggy image stream in the time domain, specifically:
[0011] The AI smart glasses utilize the PTP protocol and combine it with the driver's head angular velocity sensed by the AI smart glasses to adjust the synchronization message frequency between the first foggy image stream and the second foggy image stream, thereby achieving time-domain synchronization between the first foggy image stream and the second foggy image stream.
[0012] Optionally, the adjustment of the synchronization message frequency between the first foggy image stream and the second foggy image stream is specifically implemented using the following formula:
[0013] ;
[0014] in, The adjusted synchronization message frequency, The reference frequency for AI smart glasses This is the sensitivity coefficient. The angular velocity of the driver's head. It is a second-order normal form.
[0015] Optionally, the step of processing the first foggy image to obtain the first processed image includes: processing it using a dynamic gain linear stretching method, the specific implementation formula of which is as follows:
[0016] ;
[0017] in, These are the pixels in the first foggy image. The pixel values in the first foggy image are processed using the dynamic gain linear stretching method. These are the original pixel values of the pixels in the first foggy day image. The minimum brightness value among the pixel brightness values of the first foggy day image; This is the maximum brightness value among the pixel brightness values of the first foggy day image.
[0018] Optionally, the step of processing the first foggy image to obtain the first processed image further includes:
[0019] The Gaussian smoothing operator is used to denoise the first foggy image after dynamic gain linear stretching. The adaptive kernel parameters of the Gaussian smoothing operator are described. The following formula is used for calculation:
[0020] ;
[0021] in, Based on the kernel parameters, For adjustment coefficients, This is the gamma correction factor.
[0022] Optionally, the dehazing and image enhancement processing of the first aligned image and the dehazing and image enhancement processing of the second aligned image include:
[0023] Calculate a first transmittance distribution for the first aligned image, and calculate a second transmittance distribution for the second aligned image;
[0024] The first aligned image is graded based on the first transmittance distribution, and the second aligned image is graded based on the second transmittance distribution;
[0025] Based on the grading results, the first aligned image and the second aligned image are subjected to dehazing and image enhancement processing respectively.
[0026] Optionally, the calculation of a first transmittance distribution for the first aligned image and a second transmittance distribution for the second aligned image is performed; the specific implementation formula is as follows:
[0027] t(x)=e (-βd(x)) ;
[0028] Where t(x) is The corresponding transmittance, β is the atmospheric scattering coefficient, and d(x) is the scene depth.
[0029] Optionally, the step of grading the first aligned image based on a first transmittance distribution and grading the second aligned image based on a second transmittance distribution includes:
[0030] The average transmittance of the first aligned image and the second aligned image are determined respectively; based on the average transmittance threshold and the determined average transmittance, the classification result of the image to be calculated is determined.
[0031] The formula for calculating the average transmittance threshold is as follows:
[0032] ;
[0033] This represents the average transmittance threshold at the n1-th iteration step. The average transmittance threshold at the (n1-1)th iteration step. To adjust the step size coefficient, For the preset target accuracy, The accuracy of the average transmittance threshold calculated at the n1-th iteration step.
[0034] Optionally, the step of repairing the first feature and the second feature using a diffusion model includes:
[0035] Based on prior knowledge of atmospheric physics and scene depth gradient constraints, a diffusion model is used to denoise the first feature and the second feature respectively. The diffusion model is a variational autoencoder, and the prior knowledge of atmospheric physics is the prior knowledge of global atmospheric light value and the cooperative transmittance of the image obtained by the AI smart glasses and the image obtained by the vehicle system.
[0036] The formula for the scene depth gradient constraint is:
[0037] ;
[0038] in, The feature gradient corresponding to the haze-free image. Indicates a positive correlation constraint. This represents the geometric gradient corresponding to the feature to be repaired.
[0039] Secondly, a task-driven dehazing perception system is proposed, including:
[0040] The image stream acquisition module is used for the first-view sensor of the AI smart glasses to acquire the first foggy image stream, and the second-view sensor of the vehicle system to acquire the second foggy image stream; the AI smart glasses are fixed to the driver's head, and the second foggy image stream is synchronized with the first foggy image stream in the time domain;
[0041] The image cropping module is used to crop a first foggy image from a first foggy image stream, process the first foggy image to obtain a first processed image; and to crop a second foggy image from a second foggy image stream, process the second foggy image to obtain a second processed image; the second foggy image and the first foggy image are time-aligned images.
[0042] The image alignment module is used to perform geometric alignment on a first processed image to obtain a first aligned image, and to perform geometric alignment on a second processed image to obtain a second aligned image;
[0043] The feature extraction module is used to perform dehazing and image enhancement processing on the first aligned image, and extract the features of the image after processing the first aligned image as the first feature; and to perform dehazing and image enhancement processing on the second aligned image, and extract the features of the image after processing the second aligned image as the second feature;
[0044] The feature repair and fusion module is used to repair the first feature and the second feature respectively using the diffusion model, perform feature fusion on the repaired first feature and the repaired second feature, and determine the final perception result based on the fused features.
[0045] Beneficial technical effects:
[0046] 1. The viewpoint of the AI smart glasses of this invention is combined with the viewpoint of the vehicle camera to form a panoramic and close-up view; the AI smart glasses take advantage of being located at eye level to effectively identify near-field and low-lying obstacles that are easily missed by the vehicle camera.
[0047] 2. This invention utilizes the PTP (Precision Time Protocol) to achieve millisecond-level time synchronization between the image stream acquired by the AI smart glasses and the image stream acquired by the vehicle camera, ensuring high reliability and geometric consistency;
[0048] 3. This invention introduces a routing mechanism to dynamically allocate defogging computing power based on fog concentration; a simple network is used for light fog, while deep repair is activated for moderate to heavy fog, thus shortening the overall inference time.
[0049] 4. This invention balances the low power consumption of AI smart glasses with the high-performance computing requirements of in-vehicle systems. Attached Figure Description
[0050] Figure 1 This is a flowchart of a task-driven dehazing sensing method according to an embodiment of this application;
[0051] Figure 2 This is a schematic diagram of the task-driven defogging sensing system according to an embodiment of this application. Detailed Implementation
[0052] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will now be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.
[0053] Example 1
[0054] This embodiment discloses a task-driven dehazing perception method, such as... Figure 1 As shown, it includes the following steps:
[0055] Step S1: The first-view sensor of the AI smart glasses worn by the driver acquires the first foggy image stream, and the second-view sensor of the vehicle system acquires the second foggy image stream;
[0056] Step S2: Extract a first foggy image from the first foggy image stream and a second foggy image from the second foggy image stream; perform dynamic gain linear stretching and noise reduction processing on the first foggy image and the second foggy image respectively to obtain a first processed image and a second processed image; wherein, the first foggy image and the second foggy image are time-aligned, and the first foggy image stream and the second foggy image stream are synchronized in the time domain.
[0057] Step S3: The AI smart glasses acquire the rotation and translation parameters of the AI smart glasses relative to the vehicle coordinate system, and calculate the spatial homography transformation matrix based on the rotation and translation parameters; perform geometric alignment on the first processed image and the second processed image based on the spatial homography transformation matrix, and the two geometrically aligned images are the first aligned image and the second aligned image, respectively.
[0058] Step S4: Calculate the first transmittance distribution for the first aligned image and the second transmittance distribution for the second aligned image;
[0059] The first aligned image and the second aligned image are classified based on the first transmittance distribution and the second transmittance distribution, respectively.
[0060] Based on the classification results, the first aligned image and the second aligned image are routed to dehazing processing branches with different computational complexities; the dehazing processing branch performs dehazing and image enhancement processing on the input of the dehazing processing branch, and extracts features from the processed input;
[0061] When the input is a first aligned image, the extracted feature is the first feature; when the input is a second aligned image, the extracted feature is the second feature.
[0062] Step S5: Repair the first feature and the second feature respectively using a diffusion model that embeds prior knowledge of atmospheric physics and scene depth gradient constraints; perform feature fusion on the repaired first feature and the repaired second feature to obtain fused features, and determine the final perception result based on the fused features.
[0063] Furthermore, in step S1, after acquiring the first foggy image stream and the second foggy image stream, the AI smart glasses use the PTP protocol and combine it with the driver's head angular velocity sensed by the AI smart glasses to adjust the synchronization message frequency between the first foggy image stream and the second foggy image stream, and synchronize the first foggy image stream and the second foggy image stream in the time domain.
[0064] Furthermore, the AI smart glasses utilize the PTP protocol and combine it with the driver's head angular velocity sensed by the AI smart glasses to adjust the synchronization message frequency between the first foggy image stream and the second foggy image stream, wherein:
[0065] The adjusted synchronization message frequency is as follows:
[0066] ;
[0067] in, The adjusted synchronization message frequency, The reference frequency for AI smart glasses This is the sensitivity coefficient. The angular velocity of the driver's head. It is a second-order normal form.
[0068] In this invention, the vehicle-mounted system is an in-vehicle edge host, which uses vehicle network communication protocols to realize data interaction between the in-vehicle edge host and AI smart glasses.
[0069] Further, in step S2, dynamic gain linear stretching and noise reduction processing are performed on the first foggy image and the second foggy image respectively to obtain a first processed image and a second processed image, wherein:
[0070] Both the first foggy day image and the second foggy day image are used as images to be processed;
[0071] The formula for linear stretching of dynamic gain is as follows:
[0072] ;
[0073] in, For the pixels in the image to be processed, The pixel values in the image to be processed are the result of dynamic gain linear stretching. These are the original pixel values of the pixels in the image to be processed. It is the minimum brightness value among all the brightness values of the pixels in the image to be processed; The maximum brightness value among all pixels in the image to be processed;
[0074] Denoising the image after dynamic gain linear stretching using the Gaussian smoothing operator; adaptive kernel parameters of the Gaussian smoothing operator. for:
[0075] ;
[0076] in, Based on the kernel parameters, For adjustment coefficients, This is the gamma correction factor.
[0077] In this invention, It is usually used as a preset initial kernel size benchmark for subsequent adaptive scaling calculations; It is a dynamic adjustment factor used to scale the contribution of a specific weight; This represents the gamma correction coefficient. The parameter can be predicted in real time by the parameter predictor of a miniature CNN (Convolutional Neural Network) based on the global statistical features of the image.
[0078] In this invention, by using dynamic gain linear stretching, the narrow-band brightness distribution caused by fog occlusion is forcibly mapped to the full dynamic range (0-255), and the image to be processed after dynamic gain linear stretching is denoised, thereby reducing the smoothing intensity in the heavy fog area to preserve the edge details of the target.
[0079] Further, in step S3, the AI smart glasses acquire the rotation and translation parameters of the AI smart glasses relative to the vehicle coordinate system, and calculate the spatial homography transformation matrix, wherein:
[0080] Calculate the spatial homography transformation matrix The formula is as follows:
[0081] ;
[0082] in, This is the intrinsic parameter matrix of the camera in the vehicle system. The intrinsic parameter matrix of the camera for AI smart glasses. This is the real-time rotation matrix of the AI smart glasses relative to the vehicle coordinate system. The planar distance of the current scene measured by the vehicle system. It is a translation vector. The unit normal vector of the plane. It is the transpose of the plane unit normal vector.
[0083] The translation vector represents the three-dimensional spatial displacement of the AI smart glasses relative to the vehicle's camera coordinate system. It describes the relative positional offset of the two sensors in physical space. The direction of the reference plane (such as the road surface) in the eyeglass coordinate system represents the direction of the observation.
[0084] In this invention, the AI smart glasses utilize a built-in attitude perception module to obtain the rotation and translation parameters of the AI smart glasses relative to the vehicle coordinate system. Combined with the inherent intrinsic parameters of the AI smart glasses and the vehicle system, as well as the scene reference depth, a spatial homography transformation matrix is calculated. This spatial homography transformation matrix is then used to geometrically align the first and second processed images to compensate for the viewpoint distortion caused by the driver's head rotation, thereby achieving geometric alignment of multi-view features.
[0085] Further, in step S4, a first transmittance distribution and a second transmittance distribution are calculated for the first aligned image and the second aligned image, respectively, wherein:
[0086] Both the first aligned image and the second aligned image are used as images for which the transmittance is to be calculated.
[0087] The formula for calculating the transmittance distribution of the image to be calculated is as follows:
[0088] t(x) = e(-βd(x));
[0089] Where t(x) is The corresponding transmittance, β is the atmospheric scattering coefficient, and d(x) is the scene depth.
[0090] In this invention, transmittance is defined as the proportion of light that passes through the atmospheric medium and reaches the visual sensor. It reflects the remaining proportion of light radiation energy after it has exponentially decreased with distance and scattering coefficient. β is the atmospheric scattering coefficient, and d(x) is the scene depth, i.e., the distance from the optical center of the visual sensor to the image whose transmittance is to be calculated. The physical distance between them. In high-speed moving scenarios such as autonomous driving, the instantaneous change of d(x) is extremely rapid, and the spatial distribution of fog is often highly non-uniform. Therefore, accurate estimation of the transmittance distribution is a key prerequisite for subsequent restoration of a clear scene.
[0091] In step S4, the first aligned image and the second aligned image are classified based on the first transmittance distribution and the second transmittance distribution, respectively.
[0092] Based on the classification results, the first aligned image and the second aligned image are routed to dehazing processing branches with different computational complexities; the dehazing processing branch performs dehazing and image enhancement processing on the input of the dehazing processing branch, and extracts features from the processed input;
[0093] When the input is a first aligned image, the extracted features are the first features; when the input is a second aligned image, the extracted features are the second features, including:
[0094] Step S41: Use both the first aligned image and the second aligned image as images to be calculated for transmittance;
[0095] Step S42: Determine the average transmittance of the image to be calculated based on the transmittance distribution corresponding to the image to be calculated; determine the grading result of the image to be calculated based on the average transmittance threshold and the average transmittance of the image to be calculated.
[0096] The average transmittance threshold is dynamic, and the formula for calculating the average transmittance threshold is as follows:
[0097] ;
[0098] Indicates the first The average transmittance threshold at the next iteration step. For the first The average transmittance threshold at the next iteration step. To adjust the step size coefficient, For the preset target accuracy, For the first The accuracy of the average transmittance threshold calculated in the next iteration step.
[0099] When the average transmittance of the image to be calculated is greater than the average transmittance threshold, the classification result of the image to be calculated is light fog; otherwise, the classification result of the image to be calculated is moderate to heavy fog. Light fog refers to fog with visibility less than 500 meters within 12 hours, and moderate to heavy fog refers to fog with visibility less than 200 meters within 6 hours.
[0100] Step S43: When the classification result corresponding to the transmittance image to be calculated is light fog, route the transmittance image to be calculated to the first defogging processing branch; when the classification result corresponding to the transmittance image to be calculated is moderate to heavy fog, route the transmittance image to be calculated to the second defogging processing branch.
[0101] The first dehazing branch uses the spatial attention module of a mini CNN to process the image to be calculated for transmittance as follows: recalibrate the contrast of each pixel in the image to be calculated for transmittance, and enhance the edges of the image to be calculated for transmittance using linear operators; extract the features of the processed image to be calculated for transmittance.
[0102] The second dehazing branch processes the image to be calculated as follows: a deep residual shrinking network is used to filter out noise caused by scattering in the image to be calculated; sub-pixel convolutional layers are used to upsample each pixel in the noise-removed image to enhance the high-frequency features lost in the image to be calculated; and features of the processed image to be calculated are extracted.
[0103] When the image to be calculated for transmittance is a first aligned image, the extracted feature is the first feature; when the image to be calculated for transmittance is a second aligned image, the extracted feature is the second feature.
[0104] In this invention, sub-pixel convolutional layers are used for upsampling to compensate for the loss of high-frequency features such as traffic sign details and pedestrian outlines under dense fog, thereby enhancing the image. The step size factor, which is a scaling factor or gain factor used to control the update intensity, determines the magnitude of the correction of the average transmittance threshold.
[0105] Further, in step S5, the first feature and the second feature are repaired respectively using a diffusion model embedded with prior atmospheric physics knowledge and scene depth gradient constraints, including:
[0106] The first and second features are used as input features to the diffusion model in turn. The diffusion model combines prior knowledge of atmospheric physics and scene depth gradient constraints to denoise the input features, and the denoised features are used as the output features of the diffusion model.
[0107] in:
[0108] The diffusion model is a variational autoencoder. The diffusion model embeds atmospheric physical prior knowledge, which is the prior knowledge of global atmospheric light value and the cooperative transmittance of the image obtained by the AI smart glasses and the image obtained by the vehicle system.
[0109] In the diffusion model repair process, both the first and second features are considered as features to be repaired; the scene depth gradient constraint is simplified as follows:
[0110] ;
[0111] in, The feature gradient corresponding to the haze-free image. Indicates a positive correlation constraint. This represents the geometric gradient corresponding to the feature to be repaired.
[0112] In this invention, the diffusion model adopts the ASPDiff (Atrous Spatial Pyramid Diffusion) framework. While diffusion models possess powerful image inpainting capabilities, they may generate artifacts without physical constraints. Therefore, this invention introduces scene depth gradient constraints. These constraints ensure clear lines are produced in areas with drastic depth changes (such as telephone poles and pedestrian edges) and effectively suppress noise interference common in dense fog areas. The diffusion model satisfies atmospheric scattering physics and maintains consistency between the edge gradient of a clear image and the scene depth gradient, effectively preventing the generation of false textures that violate physical laws.
[0113] Furthermore, the loss function of the diffusion model for:
[0114] ;
[0115] in, The ground truth features corresponding to the fog-free image, The features output by the diffusion model, for and The mean square error between them To compare the losses, As the first balancing weight factor, This is the second balancing weight factor.
[0116] Furthermore, step S5 also includes feature fusion of the repaired first feature and the repaired second feature, including:
[0117] The repaired first feature and the repaired second feature are aligned using a flow-guided cosine attention sampler. The semantic correlation between the aligned repaired first feature and the repaired second feature is calculated. The repaired first feature, the repaired second feature and the semantic correlation are then concatenated.
[0118] AI smart glasses have a near-field advantage; because the wearer's perspective is closer to roadside obstacles and pedestrians, their transmittance estimation for near-distance targets is highly accurate. In-vehicle systems have a long-range advantage; the fixed position of the in-vehicle camera provides a wider field of view, making it adept at recognizing distant road outlines and the macroscopic environment. This invention combines information from both perspectives to generate a fused feature that covers the entire field and is accurate at both near and far distances.
[0119] Example 2
[0120] This embodiment proposes a task-driven defogging perception system, such as... Figure 2 As shown, it includes:
[0121] The image stream acquisition module is used for the first-view sensor of the AI smart glasses to acquire the first foggy image stream, and the second-view sensor of the vehicle system to acquire the second foggy image stream; the AI smart glasses are fixed to the driver's head, and the second foggy image stream is synchronized with the first foggy image stream in the time domain;
[0122] The image cropping module is used to crop a first foggy image from a first foggy image stream, process the first foggy image to obtain a first processed image; and to crop a second foggy image from a second foggy image stream, process the second foggy image to obtain a second processed image; the second foggy image and the first foggy image are time-aligned images.
[0123] The image alignment module is used to perform geometric alignment on a first processed image to obtain a first aligned image, and to perform geometric alignment on a second processed image to obtain a second aligned image;
[0124] The feature extraction module is used to perform dehazing and image enhancement processing on the first aligned image, and extract the features of the image after processing the first aligned image as the first feature; and to perform dehazing and image enhancement processing on the second aligned image, and extract the features of the image after processing the second aligned image as the second feature;
[0125] The feature repair and fusion module is used to repair the first feature and the second feature respectively using the diffusion model, perform feature fusion on the repaired first feature and the repaired second feature, and determine the final perception result based on the fused features.
[0126] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.
[0127] This system, through the coordinated operation of its various units, solves the problems of incomplete statistical data, low efficiency, and insufficient accuracy in traditional blasting block size statistics, and can effectively improve the level of intelligence and comprehensive benefits of underground metal mining.
[0128] It should be noted that the overall technical concept on which the methods and systems in the embodiments of the present invention are based is the same, and the technical solutions and technical effects are mutually referential. For the sake of brevity, they will not be described in detail here.
[0129] As can be seen from the above, the intelligent identification and statistics platform for blasted block size in underground metal mines provided by the embodiments of the present invention can efficiently and accurately count the blasted block size in underground metal mines, providing reliable data support for mine blasting effect evaluation and production optimization.
[0130] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0131] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.
Claims
1. A task-driven dehazing perception method, characterized in that, include: The AI smart glasses' first-view sensor captures a first foggy image stream, while the vehicle system's second-view sensor captures a second foggy image stream. The AI smart glasses are fixed to the driver's head, and the second foggy image stream is synchronized with the first foggy image stream in the time domain. Extract a first foggy image from the first foggy image stream, and process the first foggy image to obtain a first processed image; Extract the second foggy image from the second foggy image stream, and process the second foggy image to obtain the second processed image; The second foggy image is time-aligned with the first foggy image. Geometric alignment is performed on the first processed image to obtain a first aligned image, and geometric alignment is performed on the second processed image to obtain a second aligned image; The first aligned image is subjected to dehazing and image enhancement processing, and the features of the image after processing the first aligned image are extracted as the first feature; The second aligned image is subjected to dehazing and image enhancement processing, and the features of the processed second aligned image are extracted as the second features; The first and second features are repaired using a diffusion model, and the repaired first and second features are fused together. The final perception result is determined based on the fused features.
2. The task-driven dehazing sensing method according to claim 1, characterized in that, The second foggy image stream is synchronized with the first foggy image stream in the time domain, specifically as follows: The AI smart glasses utilize the PTP protocol and combine it with the driver's head angular velocity sensed by the AI smart glasses to adjust the synchronization message frequency between the first foggy image stream and the second foggy image stream, thereby achieving synchronization of the first foggy image stream and the second foggy image stream in the time domain.
3. The task-driven dehazing sensing method according to claim 2, characterized in that, The specific formula for adjusting the synchronization message frequency between the first foggy image stream and the second foggy image stream is as follows: ； in, The adjusted synchronization message frequency, The reference frequency for AI smart glasses This is the sensitivity coefficient. The angular velocity of the driver's head. It is a second-order normal form.
4. The task-driven dehazing sensing method according to claim 1, characterized in that, The process of processing the first foggy image to obtain the first processed image includes: processing it using a dynamic gain linear stretching method, the specific implementation formula of which is as follows: ； in, These are the pixels in the first foggy image. The pixel values in the first foggy image are processed using the dynamic gain linear stretching method. These are the original pixel values of the pixels in the first foggy day image. The minimum brightness value among the pixel brightness values of the first foggy day image; This is the maximum brightness value among the pixel brightness values of the first foggy day image.
5. The task-driven dehazing sensing method according to claim 4, characterized in that, The process of processing the first foggy image to obtain the first processed image further includes: The Gaussian smoothing operator is used to denoise the first foggy image after dynamic gain linear stretching. The adaptive kernel parameters of the Gaussian smoothing operator are described. The following formula is used for calculation: ； in, Based on the kernel parameters, For adjustment coefficients, This is the gamma correction factor.
6. The task-driven dehazing sensing method according to claim 1, characterized in that, The dehazing and image enhancement processing of the first aligned image and the dehazing and image enhancement processing of the second aligned image include: Calculate a first transmittance distribution for the first aligned image, and calculate a second transmittance distribution for the second aligned image; The first aligned image is graded based on the first transmittance distribution, and the second aligned image is graded based on the second transmittance distribution; Based on the grading results, the first aligned image and the second aligned image are subjected to dehazing and image enhancement processing respectively.
7. The task-driven dehazing sensing method according to claim 6, characterized in that, The first transmittance distribution is calculated for the first aligned image, and the second transmittance distribution is calculated for the second aligned image; the specific implementation formula is as follows: t(x)=e (-βd(x)) ; Where t(x) is The corresponding transmittance, β is the atmospheric scattering coefficient, and d(x) is the scene depth.
8. The task-driven dehazing sensing method according to claim 6, characterized in that, The step of grading the first aligned image based on a first transmittance distribution and grading the second aligned image based on a second transmittance distribution includes: The average transmittance of the first aligned image and the second aligned image are determined respectively; based on the average transmittance threshold and the determined average transmittance, the classification result of the image to be calculated is determined. The formula for calculating the average transmittance threshold is as follows: ； Indicates the first The average transmittance threshold at the next iteration step. For the first The average transmittance threshold at the next iteration step. To adjust the step size coefficient, For the preset target accuracy, For the first The accuracy of the average transmittance threshold calculated in the next iteration step.
9. The task-driven dehazing sensing method according to claim 1, characterized in that, The method of repairing the first feature and the second feature using a diffusion model includes: Based on prior knowledge of atmospheric physics and scene depth gradient constraints, a diffusion model is used to denoise the first and second features respectively. Among them, the diffusion model is a variational autoencoder, and the atmospheric physics prior knowledge is the prior knowledge of the global atmospheric light value and the cooperative transmittance of the images obtained by the AI smart glasses and the images obtained by the vehicle system. The formula for the scene depth gradient constraint is: ； in, The feature gradient corresponding to the haze-free image. Indicates a positive correlation constraint. This represents the geometric gradient corresponding to the feature to be repaired.
10. A task-driven defogging sensing system, characterized in that, include: The image stream acquisition module is used for the first-view sensor of the AI smart glasses to acquire the first foggy image stream, and the second-view sensor of the vehicle system to acquire the second foggy image stream. The AI smart glasses are fixed to the driver's head, and the second foggy image stream is synchronized with the first foggy image stream in the time domain; The image cropping module is used to crop a first foggy image from a first foggy image stream and process the first foggy image to obtain a first processed image; Extract the second foggy image from the second foggy image stream, and process the second foggy image to obtain the second processed image; The second foggy image is time-aligned with the first foggy image. The image alignment module is used to perform geometric alignment on a first processed image to obtain a first aligned image, and to perform geometric alignment on a second processed image to obtain a second aligned image; The feature extraction module is used to perform dehazing and image enhancement processing on the first aligned image, and extract the features of the image after processing the first aligned image as the first feature; The second aligned image is subjected to dehazing and image enhancement processing, and the features of the processed second aligned image are extracted as the second features; The feature repair and fusion module is used to repair the first feature and the second feature respectively using the diffusion model, perform feature fusion on the repaired first feature and the repaired second feature, and determine the final perception result based on the fused features.