Dust image enhancement method, device and system based on perceptual consistency

Abstract

This invention relates to the field of image perception enhancement technology, specifically disclosing a method, apparatus, and system for enhancing dust images based on perceptual consistency. The method includes: acquiring original image information of an autonomous vehicle; performing a first perceptual inference on the original image information to obtain a first perceptual inference result; performing dust-containing image enhancement processing and a second perceptual inference on the original image information to obtain a second perceptual inference result; performing a perceptual consistency evaluation based on the first and second perceptual inference results to obtain a perceptual consistency evaluation result; and determining a closed-loop control decision for the enhanced image based on the perceptual consistency evaluation result. The dust image enhancement method based on perceptual consistency provided by this invention can improve the visibility and clarity of vehicle-mounted camera images in image degradation scenarios while ensuring that the enhanced dust image does not disrupt the stability and reliability of existing visual perception tasks and remote driving display links.

Description

Technical Field

[0001] This invention relates to the field of image perception enhancement technology, and in particular to a dust image enhancement method based on perception consistency, a dust image enhancement device based on perception consistency, and an unmanned vehicle system. Background Technology

[0002] In applications such as unmanned transport vehicles, engineering vehicles, mining vehicles, tunnel transport vehicles, and remotely driven vehicles, vehicle-mounted cameras are a crucial image source for vehicle environmental perception and remote driving display. When vehicles operate in environments such as mining areas, construction access roads, unpaved roads, and tunnel construction passages, the images captured by the cameras are easily affected by factors such as wheel wear, oncoming traffic, the movement of vehicles ahead, and loading / unloading operations. These factors can lead to degradation issues such as decreased image visibility, blurred target edges, loss of texture details, incomplete target outlines, and difficulty in identifying key areas. Specifically, for example… Figure 1 As shown.

[0003] Existing vehicle vision systems typically possess certain perception and display capabilities, such as target detection, image segmentation, special region recognition, 2.5D image detection, multi-channel video acquisition, surround view stitching, fisheye distortion correction, and remote driving video streaming. These modules collectively form the basic link between vehicle visual perception and remote driving display. In degraded scenarios such as dust or air pollution, directly inputting degraded images into the aforementioned perception or display link may lead to problems such as missed or false detections of targets, detection box offset, segmentation boundary drift, unstable recognition of special regions, and decreased interpretability of remote driving footage.

[0004] To improve the quality of dusty images, existing technologies employ image enhancement methods such as contrast enhancement, dehazing, sharpening, color restoration, deep learning image restoration, and diffusion model editing enhancement to restore or enhance dusty images. However, these methods typically focus on improving the subjective visual effect of the image, and whether the enhancement results truly benefit subsequent perception tasks has not been fully verified. Especially in vehicle-mounted perception and remote driving scenarios, if the image enhancement process alters the original scene structure, target boundaries, color distribution, or local texture, it may result in improved visual effects but deteriorated perception results. For example, the enhancement process may introduce pseudo-textures, amplify local noise, cause target boundary shifts, and alter the contours of special regions, thereby affecting target detection, image segmentation, special region recognition, or remote driver judgment. A comparison of visibility changes in the same image before and after enhancement is provided. Figure 2 As shown.

[0005] Therefore, how to improve the visibility and clarity of vehicle camera images in image degradation scenarios while ensuring that the enhanced dust images do not disrupt the stability and reliability of existing visual perception tasks and remote driving display links has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a dust image enhancement method, a dust image enhancement device, and an unmanned vehicle system based on perceptual consistency, which solves the problem in related technologies that it is impossible to simultaneously improve the visibility and clarity of camera images and the stability and reliability of enhanced dust images in image degradation scenarios.

[0007] As a first aspect of the present invention, a method for enhancing dust images based on perceptual consistency is provided, comprising:

[0008] Acquire raw image information of the autonomous vehicle, wherein the raw image information includes at least dust interference information;

[0009] Perform first perceptual reasoning on the original image information to obtain the first perceptual reasoning result;

[0010] The original image information is subjected to dust-containing image enhancement processing to obtain an enhanced image, and the enhanced image is subjected to second perceptual reasoning to obtain a second perceptual reasoning result;

[0011] Based on the first and second perceptual reasoning results, a perceptual consistency evaluation is performed to obtain a perceptual consistency evaluation result.

[0012] The closed-loop control decision for the enhanced image is determined based on the perceived consistency evaluation results.

[0013] Further, based on the first perceptual reasoning result and the second perceptual reasoning result, a perceptual consistency evaluation is performed to obtain a perceptual consistency evaluation result, including:

[0014] The first perceptual reasoning result is matched with the second perceptual reasoning result using perceptual indicators, which include at least target detection results, image segmentation results, special region recognition results, structural boundary results, and image 2.5D detection results.

[0015] Based on the matching results of the perception indicators, the perception consistency index is calculated to obtain the perception consistency index calculation result. The perception consistency index includes at least the number of targets, target category, target location, segmented region, special region, and structural boundary.

[0016] The perception consistency evaluation result is determined based on the calculation results of the perception consistency index.

[0017] Further, matching the first perceptual reasoning result with the second perceptual reasoning result using perceptual indicators includes:

[0018] When the perception index is the target detection result, a target matching relationship is established based on the target category, detection box overlap, center point distance, box width and height change rate, and confidence change.

[0019] When the perception index is an image segmentation result, it is compared based on the region intersection-union ratio, area change rate, average boundary offset distance, and maximum boundary offset distance.

[0020] When the perception index is a special area identification result, the location, area, boundary and category of one or more of the following areas are compared to determine whether an anomaly has occurred.

[0021] When the perception index is a structural boundary result, compare the degree of offset of key linear or planar structures of road boundary, retaining wall boundary, vehicle outline and obstacle outline.

[0022] When the perception index is the 2.5D image detection result, compare whether the changes in target depth, distance estimation, and spatial position are within the allowable range.

[0023] Further, based on the matching results of the perception indicators, a perception consistency index is calculated to obtain the perception consistency index calculation result, including:

[0024] Based on the matching results of the perception indicators, select perception consistency indicators;

[0025] A perceptual consistency score is calculated based on the perceptual consistency index, wherein the formula for calculating the perceptual consistency score is:

[0026] ,

[0027] in, Indicates the perceived consistency score. This represents the object detection consistency score. This represents the image segmentation consistency score. This indicates the consistency score for a specific region. Indicates the structural boundary consistency score. All represent weights.

[0028] Further, the perceptual consistency evaluation result is determined based on the calculation result of the perceptual consistency index, including:

[0029] The perceptual consistency score is compared with a first preset score threshold.

[0030] If the perceptual consistency score is greater than or equal to the first preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image is reliable.

[0031] If the perceptual consistency score is less than the first preset score threshold and greater than or equal to the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has local anomalies, wherein the second preset score threshold is less than the first preset score threshold.

[0032] If the perceptual consistency score is less than the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has an overall anomaly.

[0033] Further, determining the closed-loop control decision for the enhanced image based on the perceptual consistency evaluation result includes:

[0034] When the perception consistency evaluation result indicates that the enhanced image is reliable, the enhanced image is determined as the dust image enhancement output result.

[0035] When the perception consistency evaluation result indicates that there are local anomalies in the enhanced image, local anomaly processing is performed on the abnormal area to obtain a corrected image, and the corrected image is determined as the dust image enhancement output result.

[0036] When the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the entire region is re-enhanced and the process returns to the step of redetermining the perceptual consistency evaluation result, and the image result after redetermining the perceptual consistency evaluation result is output.

[0037] Furthermore, when the perceptual consistency evaluation result indicates that the enhanced image contains local anomalies, local anomaly processing is performed on the anomaly region to obtain the corrected image, including:

[0038] When the perceptual consistency evaluation result indicates that the enhanced image has local anomalies, a consistency anomaly region mask is generated.

[0039] The enhanced image and the original image information are locally fused based on the mask of the inconsistent abnormal region to obtain the corrected image.

[0040] Furthermore, when the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the step of re-enhancing the entire region and then re-determining the perceptual consistency evaluation result includes:

[0041] When the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the entire region is re-enhanced.

[0042] If the perceptual consistency evaluation result after re-enhancement still indicates that the enhanced image has an overall anomaly, then switch to the conservative enhancement model and perform re-enhancement on the entire region.

[0043] If the perceptual consistency evaluation result after re-enhancing the entire region after switching to the conservative enhancement model is still that the enhanced image has an overall anomaly, then the global enhancement intensity should be reduced and the entire region should be re-enhanced.

[0044] If the perceptual consistency evaluation result after re-enhancing the entire region after reducing the global enhancement intensity still indicates that the enhanced image has an overall anomaly, an enhancement unreliable warning message will be issued.

[0045] As another aspect of the present invention, a dust image enhancement apparatus based on perceptual consistency is provided for implementing the dust image enhancement method based on perceptual consistency described above, wherein the apparatus includes:

[0046] An acquisition module is used to acquire raw image information of the autonomous vehicle, wherein the raw image information includes at least dust interference information;

[0047] The first perception reasoning module is used to perform first perception reasoning on the original image information to obtain the first perception reasoning result;

[0048] The enhancement processing and second perception reasoning module is used to perform dust-containing image enhancement processing on the original image information to obtain an enhanced image, and to perform second perception reasoning on the enhanced image to obtain a second perception reasoning result;

[0049] The perception consistency evaluation module is used to perform a perception consistency evaluation based on the first perception reasoning result and the second perception reasoning result, and obtain a perception consistency evaluation result.

[0050] The closed-loop control decision module is used to determine the closed-loop control decision for the enhanced image based on the perception consistency evaluation result.

[0051] As another aspect of the present invention, an unmanned vehicle system is provided, comprising: a sensing device, an in-vehicle edge computing device, and a remote cockpit server, wherein the sensing device is communicatively connected to the in-vehicle edge computing device, the in-vehicle edge computing device is communicatively connected to the remote cockpit server, and the in-vehicle edge computing device includes the aforementioned dust image enhancement device based on perceptual consistency.

[0052] The sensing device is used to collect raw image information of the unmanned vehicle;

[0053] The dust image enhancement device based on perceptual consistency in the vehicle-mounted edge computing device is used to perform first perceptual reasoning and enhancement processing of the dust-containing image and second perceptual reasoning based on the original image information of the autonomous vehicle, and to make closed-loop control decisions for the enhanced image based on the two perceptual reasoning results.

[0054] The remote cockpit server is used to receive the image processing results of the closed-loop control decision based on the enhanced image output by the vehicle edge computing device.

[0055] The dust image enhancement method based on perceptual consistency provided by this invention obtains a first perceptual inference result by performing a first perceptual inference on the original image information, and then performs enhancement processing on the original image information containing dust and a second perceptual inference to obtain a second perceptual inference result. A perceptual consistency evaluation is then performed based on the first and second perceptual inference results to obtain a perceptual consistency evaluation result. Finally, a closed-loop control decision for the enhanced image is determined based on the perceptual consistency evaluation result. This dust image enhancement method based on perceptual consistency expands the evaluation objective of dust image enhancement from simply improving visual sharpness to a synergistic optimization of visual sharpness improvement and ensuring the stability of perceptual results. By performing perceptual inference before and after image enhancement and performing consistency evaluation based on the two perceptual inference results, it can promptly detect problems caused by the enhancement process, such as target omissions, false detections, detection box offsets, target category anomalies, segmentation boundary drift, special region contour changes, and structural boundary distortion. This avoids directly inputting enhanced images with perceptual risks into subsequent vehicle-mounted perception links or remote driving display links, thereby improving the reliability and practical usability of dust image enhancement results in unmanned transport vehicles, engineering vehicles, mining vehicles, tunnel transport vehicles, and remote driving systems. Therefore, the dust image enhancement method based on perceptual consistency provided by this invention can improve the visibility and clarity of vehicle camera images in image degradation scenarios such as vehicle dust, dust dispersion, partial occlusion, low contrast, and local whitening, while ensuring that the enhanced image does not disrupt the stability and reliability of existing visual perception tasks and remote driving display links. Attached Figure Description

[0056] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof.

[0057] Figure 1 This is a schematic diagram illustrating image degradation in a dusty scene.

[0058] Figure 2 A comparison image showing changes in visibility of the same image before and after enhancement.

[0059] Figure 3A flowchart of the dust image enhancement method based on perceptual consistency provided by the present invention.

[0060] Figure 4 This is a schematic diagram comparing the second perceptual reasoning result with the first perceptual reasoning result provided by the present invention.

[0061] Figure 5 A flowchart for perceptual consistency evaluation provided by the present invention.

[0062] Figure 6 This invention provides a flowchart for determining the perception consistency evaluation result based on the calculation result of the perception consistency index.

[0063] Figure 7 This invention provides a flowchart for determining closed-loop control decisions for enhanced images.

[0064] Figure 8 This is a schematic diagram of the closed-loop control decision-making and backoff correction process provided by the present invention.

[0065] Figure 9 The structural block diagram of the dust image enhancement device based on perceptual consistency provided by the present invention.

[0066] Figure 10 This is a schematic diagram of the onboard deployment of the driverless vehicle provided by the present invention. Detailed Implementation

[0067] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0068] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0069] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0070] This embodiment provides a dust image enhancement method based on perceptual consistency. Figure 3 This is a flowchart of a dust image enhancement method based on perceptual consistency provided according to an embodiment of the present invention, such as... Figure 3 As shown, it includes:

[0071] S100. Obtain the original image information of the unmanned vehicle, wherein the original image information includes at least dust interference information;

[0072] In this embodiment of the invention, the autonomous vehicle is equipped with multiple onboard cameras, which can acquire the original image information of the autonomous vehicle. Specifically, the image containing dust interference captured by the onboard cameras is used as the input image. The input image can be a single image, or the current frame or consecutive frames in a video stream; it can be a front view image, a rear view image, a side view image, a fisheye image, a surround view stitched image, a bird's-eye view image, or a remote driving display image.

[0073] It should be noted that the original image including dust interference information includes, but is not limited to, degraded images caused by factors such as dust generated by vehicles, dust generated by vehicle wheels, dust generated by vehicles ahead, dust generated during loading and unloading operations, changes in tunnel ventilation conditions, dust from construction roads, and dust from mining roads. These degraded images may manifest as overall graying, localized whitening, reduced contrast, blurred target edges, loss of texture details, weakened structural boundaries, and difficulty in identifying specific areas.

[0074] S200: Perform first perceptual reasoning on the original image information to obtain the first perceptual reasoning result;

[0075] In this embodiment of the invention, the original image information is sent to the first perception and reasoning module to obtain the first perception and reasoning result. The first perception and reasoning module can be a perception module in an existing vehicle-mounted autonomous driving system, or it can be an independent perception and reasoning module with the same function as the existing perception module. The specific embodiments of the invention are not limited.

[0076] It should be noted that the first perception result may include at least one or more of the following:

[0077] 1) Target detection results, which may include target category, detection box location, detection box size, detection confidence, number of targets, etc.

[0078] 2) Image segmentation results, which may specifically include road areas, obstacle areas, passable areas, impassable areas, target contour areas, etc.;

[0079] 3) Special area identification results, which may include dusty areas, puddles, pits, soil piles, rocky areas, waterlogged areas, shadowy areas, or other areas that affect vehicle passage and remote judgment;

[0080] 4) Structural boundary extraction results, which may include road boundaries, curb lines, retaining wall boundaries, vehicle outlines, obstacle outlines, unloading area boundaries, construction area boundaries, etc.

[0081] 5) Image 2.5D detection results, which may include depth estimation of the target or region, distance information, approximate three-dimensional spatial position or positional relationship relative to vehicles.

[0082] It should be understood that the first perceptual reasoning result serves as the benchmark result for subsequent reliability assessment, and is used to evaluate whether image enhancement alters or destroys key perceptual information in the original scene.

[0083] S300, Perform dust-containing image enhancement processing on the original image information to obtain an enhanced image, and perform second perceptual reasoning on the enhanced image to obtain a second perceptual reasoning result;

[0084] In this embodiment of the invention, the original image information is sent to the dust image enhancement module to obtain an enhanced image. It should be noted that the dust image enhancement module in this embodiment can specifically employ existing image enhancement algorithms, dust removal algorithms, deep learning image restoration models, diffusion model editing enhancement methods, or other image enhancement models or combinations thereof that may emerge in the future. This embodiment of the invention does not impose any limitations on these methods.

[0085] Specifically, the dust image enhancement module can be implemented in any one or more of the following ways:

[0086] 1) Local enhancement based on dust degradation area detection;

[0087] 2) Traditional enhancement based on image contrast, brightness, color, and edges;

[0088] 3) Image enhancement based on deep learning restoration networks;

[0089] 4) Image restoration based on Transformer, convolutional neural networks, or generative models;

[0090] 5) Local dust removal enhancement based on diffusion models or image editing models;

[0091] 6) Texture and boundary enhancement based on multi-scale feature recovery;

[0092] 7) Selective enhancement based on structural constraints, color constraints, or semantic constraints.

[0093] Preferably, the dust image enhancement module first determines the dust degradation region or region of interest in the input original image information, then performs enhancement processing on the degradation region, and applies conservative constraints to the non-degraded region to maintain the stability of the original scene structure, target boundary, color distribution and texture features as much as possible.

[0094] It should be noted that the embodiments of the present invention are not limited to specific dust image enhancement algorithms. In addition to the dust degradation region detection, local enhancement, structural constraint enhancement, or deep learning restoration methods used in the preferred embodiments, traditional dust removal enhancement algorithms, contrast enhancement algorithms, convolutional neural network restoration models, Transformer structural restoration models, diffusion model editing enhancement, multi-scale local contrast restoration methods, or any combination of the above methods can also be used. As long as the enhancement module can restore or enhance degraded images such as vehicle dust, dust dispersion, local whitening, and low contrast, and output enhanced images that can be used for subsequent perceptual consistency evaluation, it can be used as an alternative implementation of the present invention.

[0095] After the dust image enhancement module outputs the enhanced image, the enhanced image is input into the second perception inference module for second perception inference to obtain the second perception inference result.

[0096] It should be noted that the second perception inference module uses the same model structure, model parameters, input size, confidence threshold, post-processing method and output format as the first perception inference module, so as to directly compare the perception results before and after enhancement.

[0097] In another alternative approach, the second perceptual inference module can also adopt a model that is functionally compatible with the first perceptual inference module but has different parameters, as long as the output results of the two modules can be compared in terms of target category, detection box position, segmentation region, special region recognition result, structural boundary or image 2.5D information.

[0098] It should be noted that the second perceptual reasoning result includes at least one or more types of output corresponding to the original perceptual result, used for subsequent perceptual consistency assessment. A diagram comparing the second perceptual reasoning result with the first perceptual reasoning result is shown below. Figure 4 As shown.

[0099] S400. Perform a perception consistency evaluation based on the first perception reasoning result and the second perception reasoning result to obtain a perception consistency evaluation result.

[0100] In this embodiment of the invention, a matching relationship between the perception objects before and after enhancement is established based on the first perception reasoning result and the second perception reasoning result to obtain a perception consistency evaluation result.

[0101] This invention does not limit the first and second perception inference modules to using identical models. They can use the same model, or they can use functionally compatible perception models with different parameters, structures, or deployment locations. The consistency evaluation object can be any one or more of the following: object detection, image segmentation, special region detection, structural boundary detection, and 2.5D image detection. For the consistency evaluation method, threshold rules, comprehensive scoring functions, expert rule bases, machine learning classifiers, neural network discriminators, or further supplementary indicators such as temporal consistency and multi-view consistency can be used. Any alternative solution is acceptable as long as it can determine whether the enhancement result destroys the original perception information.

[0102] S500. Determine the closed-loop control decision for the enhanced image based on the perceived consistency evaluation result.

[0103] In this embodiment of the invention, the closed-loop control decision for the enhanced image is determined based on the perceptual consistency evaluation result, that is, whether to directly output the enhanced image or to perform correction processing on the enhanced image.

[0104] Therefore, the dust image enhancement method based on perceptual consistency provided by this invention obtains a first perceptual inference result by performing a first perceptual inference on the original image information, and then performs enhancement processing on the original image information containing dust and a second perceptual inference to obtain a second perceptual inference result. A perceptual consistency evaluation is then performed based on the first and second perceptual inference results to obtain a perceptual consistency evaluation result. Finally, a closed-loop control decision for the enhanced image is determined based on the perceptual consistency evaluation result. This dust image enhancement method based on perceptual consistency expands the evaluation objective of dust image enhancement from simply improving visual clarity to a synergistic optimization of visual clarity improvement and ensuring the stability of perceptual results. By performing perceptual inference before and after image enhancement and performing consistency evaluation based on the two perceptual inference results, it can promptly detect problems caused by the enhancement process, such as target omissions, false detections, detection box offsets, target category anomalies, segmentation boundary drift, special region contour changes, and structural boundary distortion. This avoids directly inputting enhanced images with perceptual risks into subsequent vehicle-mounted perception links or remote driving display links, thereby improving the reliability and practical usability of dust image enhancement results in unmanned transport vehicles, engineering vehicles, mining vehicles, tunnel transport vehicles, and remote driving systems. Therefore, the dust image enhancement method based on perceptual consistency provided by this invention can improve the visibility and clarity of vehicle camera images in image degradation scenarios such as vehicle dust, dust dispersion, partial occlusion, low contrast, and local whitening, while ensuring that the enhanced image does not disrupt the stability and reliability of existing visual perception tasks and remote driving display links.

[0105] In this embodiment of the invention, a perceptual consistency evaluation is performed based on the first perceptual reasoning result and the second perceptual reasoning result to obtain a perceptual consistency evaluation result, such as... Figure 5 As shown, it includes:

[0106] S410. Match the first perception reasoning result with the second perception reasoning result using perception indicators, wherein the perception indicators include at least the target detection result, image segmentation result, special region recognition result, structural boundary result, and image 2.5D detection result;

[0107] In this embodiment of the invention, the perceptual indicators can be matched based on the target detection results, image segmentation results, special region recognition results, structural boundary results, and image 2.5D detection results.

[0108] Specifically, matching the first perceptual reasoning result with the second perceptual reasoning result using perceptual indicators includes:

[0109] (1) When the perception index is the target detection result, a target matching relationship is established based on the target category, detection box overlap, center point distance, box width and height change rate and confidence change amount;

[0110] (2) When the perception index is the image segmentation result, it is compared with the region intersection-union ratio, area change rate, average boundary offset distance and maximum boundary offset distance;

[0111] (3) When the perception index is a special area identification result, the location, area, boundary and category of one or more of the following areas are compared to determine whether an anomaly has occurred.

[0112] (4) When the perception index is a structural boundary result, compare the offset of key linear or planar structures of road boundary, retaining wall boundary, vehicle outline and obstacle outline;

[0113] (5) When the perception index is the image 2.5D detection result, compare whether the changes in target depth, distance estimation and spatial position are within the allowable range.

[0114] S420. Calculate the perception consistency index based on the matching result of the perception index to obtain the perception consistency index calculation result. The perception consistency index includes at least the number of targets, target category, target location, segmented region, special region, and structural boundary.

[0115] In this embodiment of the invention, the calculation result of the perception consistency index may specifically include at least one of the following: consistency of target quantity; consistency of target category; consistency of target location; consistency of target scale; consistency of target confidence change; consistency of detection box intersection-union ratio; consistency of segmented region; consistency of special region; consistency of structural boundary; consistency of depth or distance estimation; consistency of multi-channel video or multi-view results; and consistency of continuous frame temporal sequence.

[0116] Specifically, based on the matching results of the perception indicators, a perception consistency index is calculated to obtain the perception consistency index calculation result, including:

[0117] (1) Filter the perception consistency indicators based on the matching results of the perception indicators;

[0118] (2) Calculate the perceptual consistency score based on the perceptual consistency index, wherein the formula for calculating the perceptual consistency score is:

[0119] ,

[0120] in, Indicates the perceived consistency score. This represents the object detection consistency score. This represents the image segmentation consistency score. This indicates the consistency score for a specific region. Indicates the structural boundary consistency score. All represent weights.

[0121] It should be noted that the above weights can be preset or adaptively adjusted according to the application scenario, vehicle operating status, importance of perception tasks, or remote driving display requirements.

[0122] It should also be noted that the perceived consistency score does not necessarily have to use the linear weighting form mentioned above. Threshold rules, decision trees, machine learning classifiers, neural network discriminators, multi-level rule bases, or expert rule bases can also be used for judgment.

[0123] S430. Determine the perception consistency evaluation result based on the calculation result of the perception consistency index.

[0124] In this embodiment of the invention, the abnormal area can be located and the risk level can be judged based on the calculation result of the perception consistency index. That is, when there is an inconsistency between the perception results before and after the enhancement, the location, range and severity of the inconsistent area can be further determined.

[0125] In embodiments of the present invention, the inconsistent region may include at least one of the following:

[0126] 1) The newly added anomaly detection box area after enhancement;

[0127] 2) The original target area that disappeared after enhancement;

[0128] 3) Areas where the detection box position offset exceeds the threshold;

[0129] 4) Areas where the target category has undergone abnormal changes;

[0130] 5) Separate regions where the boundary drift exceeds a threshold;

[0131] 6) Areas with unusual changes in area or boundaries;

[0132] 7) Areas where the structural boundary is fractured, offset, or deformed;

[0133] 8) Local areas in the remote driving display that may cause misjudgment.

[0134] Specifically, the enhancement results can be divided into three levels based on the degree of anomaly: reliable, locally abnormal, and globally abnormal.

[0135] Specifically, the perception consistency evaluation result is determined based on the calculation result of the perception consistency index, such as... Figure 6 As shown, it includes:

[0136] S431. Compare the perceptual consistency score with a first preset score threshold;

[0137] S432. If the perceptual consistency score is greater than or equal to the first preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image is reliable.

[0138] In this embodiment of the invention, when the perceived consistency score Greater than or equal to the first preset scoring threshold At that time, the enhancement result was deemed reliable.

[0139] S433. If the perceptual consistency score is less than the first preset score threshold and greater than or equal to the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has local anomalies, wherein the second preset score threshold is less than the first preset score threshold.

[0140] In this embodiment of the invention, when the perceived consistency score Less than the first preset scoring threshold And greater than or equal to the second preset scoring threshold At that time, it was determined that there were local anomalies in the enhancement results;

[0141] S434. If the perceptual consistency score is less than the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has an overall anomaly.

[0142] In this embodiment of the invention, when the perceived consistency score Less than the second preset scoring threshold At that time, it was determined that the enhancement results showed an overall anomaly.

[0143] It should be noted that the first preset scoring threshold and the second preset scoring threshold can be set according to vehicle type, operating speed, scenario risk level, remote driving needs, perception task type or historical operating data, and the first preset scoring threshold is greater than the second preset scoring threshold.

[0144] In this embodiment of the invention, a closed-loop control decision for the enhanced image is determined based on the perceptual consistency evaluation result, such as... Figure 7 As shown, it includes:

[0145] S510. When the perception consistency evaluation result indicates that the enhanced image is reliable, the enhanced image is determined as the dust image enhancement output result.

[0146] Specifically, when the enhanced image of a dusty image is deemed reliable, the enhanced image is accepted and output as the result of the enhancement process. The specific process is as follows: Figure 8 As shown.

[0147] S520. When the perception consistency evaluation result indicates that there are local anomalies in the enhanced image, local anomaly processing is performed on the abnormal area to obtain a corrected image, and the corrected image is determined as the dust image enhancement output result.

[0148] In this embodiment of the invention, when the enhanced image result containing dust is determined to be a local anomaly, local rollback, local reduction of enhancement intensity, local fusion, or local re-enhancement are performed on the abnormal region, rather than directly abandoning the enhancement of the entire image. The local abnormal region may be formed by abnormal regions of detection boxes, abnormal regions of segmentation boundaries, abnormal regions of special regions, abnormal regions of structural boundaries, or a combination thereof.

[0149] Specifically, when the perceptual consistency evaluation result indicates that the enhanced image has local anomalies, local anomaly processing is performed on the anomaly region to obtain the corrected image, including:

[0150] (1) When the perceptual consistency evaluation result is that there are local anomalies in the enhanced image, a consistency anomaly region mask is generated;

[0151] (2) The enhanced image and the original image information are locally fused according to the mask of the consistency abnormal region to obtain the corrected image.

[0152] In a preferred embodiment, when there are local anomalies in the enhancement result, a mask of the inconsistent anomaly region is generated, and the enhanced image and the original image are locally fused according to the mask to obtain the final output image.

[0153] Specifically, a partial rollback can be represented as:

[0154] ,

[0155] in, This indicates the final output image; Indicates an enhanced image; Represents the original image; Indicates a mask for regions with inconsistent data; This indicates the fusion weight.

[0156] when When the abnormal area completely reverts to the original image; when When anomaly regions are detected, a weighted fusion of the original image and the enhanced image is used; when When the value is close to 0, the enhanced results are mainly retained in the abnormal areas. It should be understood that the fusion weights can be adaptively set according to the degree of anomaly, area type, target category, vehicle speed, or remote driving display requirements.

[0157] S530. When the perception consistency evaluation result is that the enhanced image has an overall anomaly, the overall region is re-enhanced and the process returns to the step of redetermining the perception consistency evaluation result, and the image result after redetermining the perception consistency evaluation result is output.

[0158] In this embodiment of the invention, when the enhanced image result is determined to be abnormal overall, the following actions are performed: re-enhancement, switching to a conservative enhancement model, reducing the global enhancement intensity, outputting the original image, or outputting a risk warning to the remote driving display terminal.

[0159] Specifically, when the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the step of re-enhancing the entire region and then re-determining the perceptual consistency evaluation result is as follows: Figure 8 As shown, it includes:

[0160] (1) When the perception consistency evaluation result is that the enhanced image has an overall anomaly, the overall region is re-enhanced;

[0161] (2) If the perceptual consistency evaluation result after re-enhancement is still that the enhanced image has an overall anomaly, then switch to the conservative enhancement model and perform re-enhancement on the entire region.

[0162] (3) If the perceptual consistency evaluation result after switching to the conservative enhancement model and performing re-enhancement processing on the whole region is still that the enhanced image has an overall anomaly, then reduce the global enhancement intensity and perform re-enhancement processing on the whole region.

[0163] (4) If the perception consistency evaluation result after re-enhancing the entire region after reducing the global enhancement intensity is still that the enhanced image has an overall anomaly, then an enhancement unreliable prompt message will be issued.

[0164] In this embodiment of the invention, the final output image result can be an enhanced image, a locally regressed image, a fused and corrected image, a re-enhanced image, a conservatively enhanced image, or the original image.

[0165] It should be noted that the final output can be used for at least one of the following purposes:

[0166] It serves as input to the vehicle-mounted perception module; as preprocessing results for target detection, image segmentation, special region recognition, or 2.5D image detection; as the display screen for remote driving videos; as input to multi-channel video streaming or surround view stitching links; as a data source for vehicle operation monitoring, work status monitoring, or fault tracing; and as auxiliary information for upper-level scheduling systems, decision planning modules, or safety monitoring modules.

[0167] Preferably, the system can also simultaneously output an enhanced reliability identifier, a perception consistency score, an anomaly area mask, a closed-loop control decision type, and a final output image source identifier, so that the system can record, display, diagnose, or trace these features in the future.

[0168] It should be noted that the embodiments of the present invention do not limit the specific feedback adjustment method and deployment location. When the enhancement result does not meet the requirements of perception consistency, alternative strategies such as overall rollback, local rollback, hierarchical enhancement intensity control, regional fusion output, multi-model switching, re-enhancement, conservative enhancement, or direct output of the original image can be implemented. This solution can also be deployed in the local processing module after the vehicle-side camera captures the image, the vehicle-mounted perception preprocessing node, the edge-side video preprocessing module, the remote driver's cockpit-side video reconstruction preprocessing module, or the unified image preprocessing node before the perception module. As long as the reliability of the enhancement result can be judged based on the consistency of the perception results before and after enhancement, and feedback control is executed in case of anomalies, the same or equivalent inventive objectives as the present invention can be achieved.

[0169] In summary, the dust image enhancement method based on perceptual consistency provided by this invention expands the evaluation objective of dust image enhancement from simply "improving visual clarity" to a synergistic optimization of "improving visual clarity and ensuring the stability of perceptual results." By performing perceptual inference before and after image enhancement and evaluating the consistency of results such as target detection, image segmentation, special region recognition, structural boundary preservation, and 2.5D image detection, this invention can promptly detect problems caused by the enhancement process, such as missed detections, false detections, detection box offsets, abnormal target categories, segmentation boundary drift, special region contour changes, and structural boundary distortion. This avoids directly inputting enhanced images with perceptual risks into subsequent vehicle-mounted perception links or remote driving display links, thereby improving the reliability and practical usability of dust image enhancement results in unmanned transport vehicles, engineering vehicles, mining vehicles, tunnel transport vehicles, and remote driving systems. Meanwhile, through perception consistency evaluation, abnormal region localization, and closed-loop control mechanisms, this invention can automatically execute strategies such as local rollback, local fusion, enhancement intensity reduction, re-enhancement, switching to conservative enhancement mode, or original image output when the enhancement results do not meet preset conditions. This improves the system's adaptability to complex dust, partial occlusion, and dynamically changing scenarios. This solution is not limited to specific image enhancement algorithms and can be used in conjunction with traditional image enhancement methods, deep learning image restoration models, diffusion model editing enhancement methods, etc. It can also connect to vehicle-side image links, perception preprocessing links, remote driving video preprocessing links, or surround-view stitching links without changing the existing overall architecture of unmanned transport vehicles. Simultaneously, it can output perception consistency scores, abnormal region masks, closed-loop control decision types, and final image source identifiers, providing data support for subsequent system diagnosis, operational traceability, and algorithm iteration.

[0170] As another embodiment of the present invention, a dust image enhancement device 100 based on perceptual consistency is provided to implement the dust image enhancement method based on perceptual consistency described above, wherein, as Figure 9 As shown, it includes:

[0171] The acquisition module 110 is used to acquire the original image information of the unmanned vehicle, the original image information including at least dust interference information;

[0172] The first perception reasoning module 120 is used to perform first perception reasoning on the original image information to obtain a first perception reasoning result.

[0173] The enhancement processing and second perception reasoning module 130 is used to perform dust-containing image enhancement processing on the original image information to obtain an enhanced image, and to perform second perception reasoning on the enhanced image to obtain a second perception reasoning result;

[0174] The perception consistency evaluation module 140 is used to perform a perception consistency evaluation based on the first perception reasoning result and the second perception reasoning result, and obtain a perception consistency evaluation result.

[0175] The closed-loop control decision module 150 is used to determine the closed-loop control decision for the enhanced image based on the perception consistency evaluation result.

[0176] The dust image enhancement device based on perceptual consistency provided by this invention obtains a first perceptual inference result by performing a first perceptual inference on the original image information, and then performs enhancement processing on the original image information containing dust and a second perceptual inference to obtain a second perceptual inference result. Based on the first and second perceptual inference results, a perceptual consistency evaluation is performed to obtain a perceptual consistency evaluation result. Finally, a closed-loop control decision for the enhanced image is determined based on the perceptual consistency evaluation result. This dust image enhancement device based on perceptual consistency expands the evaluation objective of dust image enhancement from simply improving visual clarity to a synergistic optimization of visual clarity improvement and ensuring the stability of perceptual results. By performing perceptual inference before and after image enhancement and performing consistency evaluation based on the two perceptual inference results, it can promptly detect problems caused by the enhancement process, such as target omissions, false detections, detection box offsets, target category anomalies, segmentation boundary drift, special region contour changes, and structural boundary distortion. This avoids directly inputting enhanced images with perceptual risks into subsequent vehicle-mounted perception links or remote driving display links, thereby improving the reliability and practical usability of dust image enhancement results in unmanned transport vehicles, engineering vehicles, mining vehicles, tunnel transport vehicles, and remote driving systems. Therefore, the dust image enhancement device based on perceptual consistency provided by the present invention can improve the visibility and clarity of vehicle camera images in image degradation scenarios such as vehicle dust, dust dispersion, partial occlusion, low contrast and local whitening, while ensuring that the enhanced image does not disrupt the stability and reliability of the existing visual perception task and remote driving display link.

[0177] The specific working principle of the dust image enhancement device based on perceptual consistency provided by the present invention can be referred to the description of the dust image enhancement method based on perceptual consistency above, and will not be repeated here.

[0178] As another embodiment of the present invention, an unmanned vehicle system is provided, comprising: a sensing device, an in-vehicle edge computing device, and a remote cockpit server, wherein the sensing device is communicatively connected to the in-vehicle edge computing device, the in-vehicle edge computing device is communicatively connected to the remote cockpit server, and the in-vehicle edge computing device includes the aforementioned dust image enhancement device based on perceptual consistency.

[0179] The sensing device is used to collect raw image information of the unmanned vehicle;

[0180] The dust image enhancement device based on perceptual consistency in the vehicle-mounted edge computing device is used to perform first perceptual reasoning and enhancement processing of the dust-containing image and second perceptual reasoning based on the original image information of the autonomous vehicle, and to make closed-loop control decisions for the enhanced image based on the two perceptual reasoning results.

[0181] The remote cockpit server is used to receive the image processing results of the closed-loop control decision based on the enhanced image output by the vehicle edge computing device.

[0182] In this embodiment of the invention, a schematic diagram of the onboard deployment of a specific driverless vehicle is shown below. Figure 10 As shown, the sensing devices may specifically include: a front-view camera, a rear-view camera, a left-view camera, a right-view camera, and a surround-view camera.

[0183] The autonomous vehicle system provided by this invention employs the aforementioned dust image enhancement device based on perceptual consistency in the vehicle edge computing device. This device can improve the visibility and clarity of vehicle camera images in image degradation scenarios such as vehicle dust, dust dispersion, partial occlusion, low contrast, and local whitening, while ensuring that the enhanced image does not disrupt the stability and reliability of existing visual perception tasks and remote driving display links.

[0184] The specific working principle of the dust image enhancement system based on perceptual consistency provided by this invention can be referred to the description of the dust image enhancement method based on perceptual consistency above, and will not be repeated here.

[0185] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for enhancing dust images based on perceptual consistency, characterized in that, include: Acquire raw image information of the autonomous vehicle, wherein the raw image information includes at least dust interference information; Perform first perceptual reasoning on the original image information to obtain the first perceptual reasoning result; The original image information is subjected to dust-containing image enhancement processing to obtain an enhanced image, and the enhanced image is subjected to second perceptual reasoning to obtain a second perceptual reasoning result; Based on the first and second perceptual reasoning results, a perceptual consistency evaluation is performed to obtain a perceptual consistency evaluation result. The closed-loop control decision for the enhanced image is determined based on the perceived consistency evaluation results.

2. The dust image enhancement method based on perceptual consistency according to claim 1, characterized in that, Based on the first perceptual reasoning result and the second perceptual reasoning result, a perceptual consistency evaluation is performed to obtain a perceptual consistency evaluation result, including: The first perceptual reasoning result is matched with the second perceptual reasoning result using perceptual indicators, which include at least target detection results, image segmentation results, special region recognition results, structural boundary results, and image 2.5D detection results. Based on the matching results of the perception indicators, the perception consistency index is calculated to obtain the perception consistency index calculation result. The perception consistency index includes at least the number of targets, target category, target location, segmented region, special region, and structural boundary. The perception consistency evaluation result is determined based on the calculation results of the perception consistency index.

3. The dust image enhancement method based on perceptual consistency according to claim 2, characterized in that, Matching the first perceptual reasoning result with the second perceptual reasoning result using perceptual indicators includes: When the perception index is the target detection result, a target matching relationship is established based on the target category, detection box overlap, center point distance, box width and height change rate, and confidence change. When the perception index is an image segmentation result, it is compared based on the region intersection-union ratio, area change rate, average boundary offset distance, and maximum boundary offset distance. When the perception index is a special area identification result, the location, area, boundary and category of one or more of the following areas are compared to determine whether an anomaly has occurred. When the perception index is a structural boundary result, compare the degree of offset of key linear or planar structures of road boundary, retaining wall boundary, vehicle outline and obstacle outline. When the perception index is the 2.5D image detection result, compare whether the changes in target depth, distance estimation, and spatial position are within the allowable range.

4. The dust image enhancement method based on perceptual consistency according to claim 2, characterized in that, Based on the matching results of the perception indicators, the perception consistency index is calculated to obtain the perception consistency index calculation result, including: Based on the matching results of the perception indicators, select perception consistency indicators; A perceptual consistency score is calculated based on the perceptual consistency index, wherein the formula for calculating the perceptual consistency score is: ， in, Indicates the perceived consistency score. This represents the object detection consistency score. This represents the image segmentation consistency score. This indicates the consistency score for a specific region. Indicates the structural boundary consistency score. All represent weights.

5. The dust image enhancement method based on perceptual consistency according to claim 4, characterized in that, The perception consistency evaluation result is determined based on the calculation results of the perception consistency index, including: The perceptual consistency score is compared with a first preset score threshold. If the perceptual consistency score is greater than or equal to the first preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image is reliable. If the perceptual consistency score is less than the first preset score threshold and greater than or equal to the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has local anomalies, wherein the second preset score threshold is less than the first preset score threshold. If the perceptual consistency score is less than the second preset score threshold, then the perceptual consistency evaluation result is determined to be that the enhanced image has an overall anomaly.

6. The dust image enhancement method based on perceptual consistency according to any one of claims 1 to 5, characterized in that, Determining closed-loop control decisions for the enhanced image based on the perceptual consistency evaluation results includes: When the perception consistency evaluation result indicates that the enhanced image is reliable, the enhanced image is determined as the dust image enhancement output result. When the perception consistency evaluation result indicates that there are local anomalies in the enhanced image, local anomaly processing is performed on the abnormal area to obtain a corrected image, and the corrected image is determined as the dust image enhancement output result. When the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the entire region is re-enhanced and the process returns to the step of redetermining the perceptual consistency evaluation result, and the image result after redetermining the perceptual consistency evaluation result is output.

7. The dust image enhancement method based on perceptual consistency according to claim 6, characterized in that, When the perceptual consistency evaluation result indicates that the enhanced image has local anomalies, local anomaly processing is performed on the anomaly region to obtain the corrected image, including: When the perceptual consistency evaluation result indicates that the enhanced image has local anomalies, a consistency anomaly region mask is generated. The enhanced image and the original image information are locally fused based on the mask of the inconsistent abnormal region to obtain the corrected image.

8. The dust image enhancement method based on perceptual consistency according to claim 6, characterized in that, When the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the step of re-enhancing the entire region and then re-determining the perceptual consistency evaluation result includes: When the perceptual consistency evaluation result indicates that the enhanced image has an overall anomaly, the entire region is re-enhanced. If the perceptual consistency evaluation result after re-enhancement still indicates that the enhanced image has an overall anomaly, then switch to the conservative enhancement model and perform re-enhancement on the entire region. If the perceptual consistency evaluation result after re-enhancing the entire region after switching to the conservative enhancement model is still that the enhanced image has an overall anomaly, then the global enhancement intensity should be reduced and the entire region should be re-enhanced. If the perceptual consistency evaluation result after re-enhancing the entire region after reducing the global enhancement intensity still indicates that the enhanced image has an overall anomaly, an enhancement unreliable warning message will be issued.

9. A dust image enhancement apparatus based on perceptual consistency, used to implement the dust image enhancement method based on perceptual consistency according to any one of claims 1 to 8, characterized in that, include: An acquisition module is used to acquire raw image information of the autonomous vehicle, wherein the raw image information includes at least dust interference information; The first perception reasoning module is used to perform first perception reasoning on the original image information to obtain the first perception reasoning result; The enhancement processing and second perception reasoning module is used to perform dust-containing image enhancement processing on the original image information to obtain an enhanced image, and to perform second perception reasoning on the enhanced image to obtain a second perception reasoning result; The perception consistency evaluation module is used to perform a perception consistency evaluation based on the first perception reasoning result and the second perception reasoning result, and obtain a perception consistency evaluation result. The closed-loop control decision module is used to determine the closed-loop control decision for the enhanced image based on the perception consistency evaluation result.

10. An unmanned vehicle-mounted system, characterized in that, include: The system includes a sensing device, an in-vehicle edge computing device, and a remote cockpit server. The sensing device is communicatively connected to the in-vehicle edge computing device, and the in-vehicle edge computing device is communicatively connected to the remote cockpit server. The in-vehicle edge computing device includes the dust image enhancement device based on perceptual consistency as described in claim 9. The sensing device is used to collect raw image information of the unmanned vehicle; The dust image enhancement device based on perceptual consistency in the vehicle-mounted edge computing device is used to perform first perceptual reasoning and enhancement processing of the dust-containing image and second perceptual reasoning based on the original image information of the autonomous vehicle, and to make closed-loop control decisions for the enhanced image based on the two perceptual reasoning results. The remote cockpit server is used to receive the image processing results of the closed-loop control decision based on the enhanced image output by the vehicle edge computing device.

Images