Image rain removal method based on prompt learning

Abstract

The application discloses a rain removing method based on prompt learning, and relates to the field of computer vision, which comprises the following steps: selecting a to-be-processed image to obtain a first to-be-identified region; extracting rain mark features from the first to-be-identified region to obtain first feature data; matching a second to-be-identified region according to morphological feature data to obtain a first identification result; comparing second density feature data with the first density feature data; if the difference between the two is within a deviation range, determining that the first identification result and the rain mark of the first to-be-identified region are a rain distribution region of the to-be-processed image, and performing image restoration on the rain distribution region to obtain rain-removed image data; if the difference between the two exceeds the built-in deviation range, performing interference factor judgment, and adjusting the rain mark identification accuracy according to the interference factor until the difference between the two is within the built-in deviation range. The application has the effect of improving rain mark identification accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to an image deraining method based on cue learning. Background Technology

[0002] Image deraining technology is a crucial step in improving image quality. In autonomous driving scenarios, images captured by cameras are easily affected by rain. Raindrops form water droplets or streaks, leading to a decline in target detection and tracking performance. For example, when driving in the rain, raindrops may obscure traffic signs or pedestrians in images captured by onboard cameras, affecting the accuracy of autonomous driving system decisions. In traffic monitoring, blurred images from rainy weather reduce the efficiency of license plate recognition and abnormal behavior detection.

[0003] In related technologies, physics-based methods establish mathematical models by analyzing the impact of raindrops on images. For example, assuming the shape and motion patterns of raindrops, motion compensation techniques are used to remove raindrops from video sequences. In the video, the motion trajectory of the raindrops is estimated to remove them from the image, and noise reduction is applied to the moving areas to restore the original clarity.

[0004] Regarding the aforementioned technologies, before removing raindrops, it is necessary to identify the raindrops in the image. Raindrops in an image are usually identified by matching the features of the image with those of the raindrops. However, due to the diversity of raindrop shapes in different environments, there are deviations in raindrop recognition, resulting in low recognition accuracy. Consequently, the results of subsequent noise reduction and restoration are not ideal, leaving rain streaks and failing to effectively remove raindrop traces from the image. Therefore, there is room for improvement. Summary of the Invention

[0005] To improve the accuracy of rain streak recognition, this application provides an image deraining method based on cue learning.

[0006] This application provides an image deraining method based on cue learning, employing the following technical solution: A cue-based image deraining method includes: Acquire the image to be processed for rain removal, and perform hue selection on the image to be processed. Select the hue region that matches the built-in rain color contrast color and record it as the first region to be identified. Based on the built-in rain color, rain color recognition is performed on the first area to be identified to determine whether there are rain streaks in the first area to be identified; Based on the rain streaks, rain streak features are extracted from the first region to be identified to obtain first feature data; the first feature data includes morphological feature data and first density feature data. The regions in the image to be processed other than the first region to be identified are recorded as the second region to be identified. The second region to be identified is matched according to the morphological feature data, and the region that is successfully matched is recorded as the first recognition result. Dense feature extraction is performed on the first recognition result to obtain the second density feature data of the first recognition result, and the difference between the second density feature data and the first density feature data is calculated to obtain the difference data; If the difference data is within the built-in deviation range, the rain streaks of the first recognition result and the first area to be recognized are determined to be the rain distribution area of ​​the image to be processed; Based on the rain distribution area, the image to be processed is divided into a background area and a rain distribution area. The rain distribution area is then inpainted based on the content of the background area to obtain the rain-removed image data. If the difference data exceeds the built-in deviation range, interference factors are judged for the first and second regions to be identified, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range.

[0007] Preferably, a spatial coordinate system is constructed based on the image to be processed, and depth markings are performed on objects in the image to be processed according to the spatial coordinate system to obtain the depth data of the image to be processed. The depth data of the first region to be identified is obtained, and the density-depth relationship is constructed based on the first density feature data and the corresponding depth data.

[0008] Preferably, the image to be processed is divided based on the depth data of the image to be processed to obtain image sub-regions of the same depth; Based on the first recognition result, the rain streaks in each image sub-region are statistically analyzed to obtain the rain streak density of the corresponding image sub-region; The rain streak densities of each image sub-region are matched to determine whether the rain streak densities of each image sub-region are the same. If the rain streak density is the same across all image sub-regions, the first recognition result is determined to be accurate. Based on the first recognition result and the rain streak in the first recognition region, the rain distribution area of ​​the image to be processed is determined, and the image to be processed is repaired based on the rain distribution area. If there are different rain streak densities among the various image sub-regions, the maximum acquisition depth of the rain streak in the image to be processed is determined based on the depth data of each image sub-region and the corresponding rain streak density. The depth data of the first region to be identified is obtained and compared with the maximum acquisition depth of the rain streaks in the image to be processed. If the depth data of the first region to be identified is greater than the maximum acquisition depth of the rain streaks in the image to be processed, the density-depth relationship is corrected according to the maximum acquisition depth of the region to be processed, and the corrected density-depth relationship is obtained. Based on the corrected density-depth relationship and the depth data of each image sub-region, the first density feature data of the first region to be identified is converted into the third density feature data at the same depth, and the difference between the second density feature data and the third density feature data is calculated to obtain the difference data.

[0009] Preferably, the rain streak densities of each image sub-region are compared to obtain the maximum rain streak density; The depth set is obtained by statistically analyzing the depth data of image sub-regions based on the maximum rain streak density. The depth data of the image sub-region with a density less than the maximum rain streak density is matched with the depth set. If the depth data of the image sub-region with a density less than the maximum rain streak density are all less than the depth set, the depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed. If there are cases in the depth data of image sub-regions with a density less than the maximum rain streak density that are within the depth set, then the image sub-regions in such cases are marked to obtain the image regions to be judged. The image region to be judged is subjected to wind and rain obstruction. If it is determined that there is wind and rain obstruction in the image region to be judged, the difference between the depth data of the image region to be judged and the depth data of the wind and rain obstruction is calculated, and the calculated difference is compared with the depth set. If the calculated difference is less than the minimum value of the depth set, the depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed; otherwise, the calculated difference is taken as the maximum acquisition depth of the area to be processed.

[0010] Preferably, if the difference between the second density feature data and the first density feature data is less than the deviation range, it indicates that the density of rain streaks in the second region to be identified is less than the density of rain streaks in the first region to be identified. Then, a fluid dynamics simulation is performed on the image to be processed with the first region to be identified as the center to determine whether there is a gas vortex in the first region to be identified. If it is determined that there is a gas vortex in the first region to be identified, then the gas vortex is simulated to determine the influence of the gas vortex on the first region to be identified. If the gas vortex has a gathering effect on raindrops, the interference item is determined to be structural specialness, and the image to be processed is reselected according to the built-in rain color contrast color. If the gas vortex does not have a gathering effect on raindrops or there is no gas vortex, then the second area to be identified is matched according to the built-in rain color to determine the low color difference area in the second area to be identified where the color difference value between the rain color and the rain color is within the built-in color difference range. The second region to be identified is further divided based on the low color difference region to obtain a second sub-region to be identified that does not contain the low color difference region. The rain streak density of the second sub-region to be identified is determined to obtain the fourth density feature data. The third density feature data is then compared with the fourth density feature data. If it is determined that the difference between the third density feature data and the fourth density feature data is within the deviation range, the rain streak recognition system performs rain streak feature self-learning based on the third density feature data and the fourth density feature data whose difference is within the deviation range. Based on the self-learning results, the recognition accuracy of the second sub-region to be identified is adjusted until the difference between the recognized result and the third density feature data is within the built-in deviation range.

[0011] Preferably, if the difference between the second density feature data and the first density feature data is greater than the deviation range, it indicates that the second density feature data of the second region to be identified is greater than the first density feature data of the first region to be identified. The image to be processed is then judged for occlusion based on the first region to be identified, to determine whether there are wind and rain obstructions on the first region to be identified. If it is determined that there are wind and rain obstructions around the first area to be identified, the depth data of the wind and rain obstructions is obtained according to the image to be processed, and the density of rain streaks in the first area to be identified is re-judged according to the depth data of the wind and rain obstructions to obtain the fifth density feature data. The difference between the second density feature data and the fifth density feature data is calculated and compared to determine whether the calculated difference is within the built-in deviation range. If the calculated difference is determined to be within the built-in deviation range, the interference item is determined to be caused by wind and rain obstruction. Based on the first recognition result of the second region to be recognized and the rain streaks recognized in the first region to be recognized, the image to be processed is separated into a background region and a rain distribution region. The image of the rain distribution region is then repaired based on the background region. If the calculated difference is determined to be outside the built-in deviation range or there are no obstructions from wind and rain, the rain streak recognition accuracy is adjusted based on the rain streak density of the first area to be recognized until the difference between the rain streak density of the second area to be recognized and the rain streak density of the first area to be recognized is within the built-in deviation range.

[0012] Preferably, when it is determined that there are wind and rain obstructions around the first area to be identified, the difference between the depth data of the wind and rain obstructions and the depth data of the first area to be identified is calculated to obtain the identification depth of the first area to be identified, and the density-depth relationship is verified based on the identification depth and the number of rain streaks to obtain the corrected density-depth relationship. Based on the corrected density-depth relationship and the depth data of the first identification result, the density feature data at the same depth is determined and recorded as the fifth density feature data.

[0013] In summary, this application includes at least one of the following beneficial technical effects: 1. By selecting images from the contrast of rain colors, the contrast between the normal background and rain streaks is increased, improving the accuracy of rain streak recognition. Once rain streaks are detected in the image, rain streak features are extracted and learned from the first region to be identified, enabling the system to accurately identify the second region. Simultaneously, the recognition result of the second region is verified using the density feature data from the first region to further confirm its accuracy. When the difference between the two is within the built-in deviation range, the rain streak recognition system accurately identifies the rain streaks in the image, allowing subsequent image restoration to more accurately repair the background area based on the rain distribution, resulting in a de-rained image and ensuring high-quality restoration. When the difference exceeds the built-in deviation range, interference factors are identified to determine the cause of the excessive deviation. The rain streak recognition system is then adjusted to address these interference factors, ensuring high accuracy of the final rain streak information and more accurate and effective restoration results. 2. By using depth data to divide the image to be processed, and then comparing the rain streak density of image sub-regions at different depths, it is determined whether the rain streak density is the same, so as to determine the recognition accuracy of the second region to be identified. When it is determined that the rain streak density of each image sub-region is the same, the first recognition result is determined to be accurate. When the rain streak density of each image sub-region is different, the maximum acquisition depth of rain streaks on the image to be processed is determined by jointly judging the depth data and the corresponding rain streak density. The maximum acquisition depth is compared with the depth data of the first region to be identified to determine the reliability of the initially constructed density-depth relationship. When it is determined that it is not reliable, the density-depth relationship is corrected according to the maximum acquisition depth to ensure its reliability. After ensuring the reliability of the density-depth relationship, the first density feature data of the first region to be identified is converted into the third density feature data at the corresponding depth. Then the second density feature data is compared with the third density feature data to ensure the reliability and accuracy of the comparison result. 3. By verifying the density-depth relationship constructed in the first area to be identified, it is determined whether there are any wind or rain obstructions causing an imbalance in the density-depth relationship of the first area to be identified. When wind or rain obstructions are determined to exist, the density-depth relationship of the first area to be identified is re-verified based on the depth data of the wind or rain obstructions. Then, based on the verified density-depth relationship, the first density feature data of the first area to be identified is re-evaluated to determine the numerical relationship between the second and fifth density feature data. This numerical relationship is then used to determine the accuracy of the rain streaks identified in the second area to be identified, thus improving the accuracy of the rain streak recognition system. Conversely, when there are no wind or rain obstructions in the first area to be identified, the recognition accuracy of the rain streak recognition system is adjusted while ensuring the accuracy of rain streak recognition in the first area to be identified, thereby increasing the number of rain streaks identified in the second area to be identified. This ensures both the recognition accuracy and the recognition precision of the rain streak recognition system for the second area to be identified. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the steps of the image deraining method based on cue learning in this embodiment. Detailed Implementation

[0015] The following is in conjunction with the appendix Figure 1 This application will be described in further detail.

[0016] This application discloses an image deraining method based on cue learning.

[0017] Example: Figure 1 As shown, the present invention provides an image deraining method based on cue learning, comprising: S1, acquire the image to be processed for rain removal, and perform hue selection on the image to be processed. Select the hue region that matches the built-in rain color contrast color and record it as the first region to be identified. S2, based on the built-in rain color, performs rain color recognition on the first area to be identified to determine whether there are rain streaks in the first area to be identified; by performing rain color recognition on the first area to be identified, it is determined whether there is rain color in the first area to be identified. If there is rain color, it is determined that there are rain streaks in the image to be processed.

[0018] S3, extract rain streak features from the first region to be identified based on the rain streak to obtain first feature data; the first feature data includes morphological feature data and first density feature data; further includes: constructing a spatial coordinate system based on the image to be processed, and performing depth marking on objects in the image to be processed according to the spatial coordinate system to obtain depth data of the image to be processed; obtaining the depth data of the first region to be identified, and constructing a density-depth relationship based on the first density feature data and the corresponding depth data.

[0019] S4, other regions in the image to be processed besides the first region to be identified are recorded as the second region to be identified. The second region to be identified is matched according to the morphological feature data, and the region that is successfully matched is recorded as the first recognition result. S5, perform density feature extraction on the first identification result to obtain the second density feature data, and calculate the difference between the second density feature data and the first density feature data to obtain the difference data; wherein the first identification result is the location of each rain streak, and the second density feature data is the result obtained after statistical analysis of the rain streak.

[0020] S6, if the difference data is within the built-in deviation range, then the rain streaks of the first recognition result and the first area to be recognized are determined to be the rain distribution area of ​​the image to be processed; S7. Based on the rain distribution area, the image to be processed is divided into a background area and a rain distribution area. The rain distribution area is then inpainted according to the content of the background area to obtain the image data after rain removal. S8. If the difference data exceeds the built-in deviation range, the image to be processed is judged for interference factors, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range. Image restoration is performed on the rain distribution area based on the background area. At the same time, the rain streak in the restored rain distribution area can be re-judged to determine whether the rain streak has been completely eliminated, thereby improving the accuracy of the restoration result.

[0021] In this embodiment, by selecting images from the contrast of rain colors, the contrast between the normal background and rain streaks is increased, thereby improving the accuracy of rain streak recognition. Once rain streaks are determined to exist in the image, rain streak features are extracted and learned from the first region to be recognized, enabling the rain streak recognition system to accurately identify the second region. Simultaneously, the recognition result of the second region is verified using the first density feature data identified in the first region to further determine the accuracy of the second region's recognition result. When the difference between the two is determined to be within the built-in deviation range, it indicates... The rain streak recognition system can accurately identify rain streaks in the image to be processed, enabling subsequent image restoration processes to more accurately repair the segmented background area based on the rain distribution region, obtaining de-rained image data and ensuring the accuracy of the restoration result. When the difference between the two is determined to exceed the built-in deviation range, it indicates that the rain streak recognition system has failed to accurately identify the rain streaks in the image to be processed. Therefore, it is necessary to judge the interference factors between the recognition results, determine the cause of the excessive deviation, and then adjust the rain streak recognition system to ensure that the final identified rain streak information has high accuracy, ensuring that the restored result is more accurate and effective.

[0022] For example, when performing rain removal on an image, it is first necessary to determine whether there are rain streaks in the image, and then accurately extract the rain streaks in the image, so as to ensure that the image to be processed is accurately divided into a background layer and a rain streak layer, thereby enabling the subsequent image restoration system to more accurately restore the image.

[0023] When determining whether rain streaks exist in an image, a color tone selection process is performed on the image to be processed. The color region in the image that has a clear color contrast with the rain color is selected as the first region to identify rain streaks, thus ensuring the accuracy of rain streak identification in this region. Since rain streaks are formed due to the scattering of ambient light by raindrops, creating a hazy white visual effect, they appear white (highlights). Therefore, if a white background color is selected, the rain streaks will also appear white, reducing the accuracy of rain streak identification. However, if a black background color is selected, the white rain streaks provide a clear contrast, allowing for accurate identification and improving the overall accuracy of rain streak recognition.

[0024] After accurately identifying rain streaks, feature extraction of the rain streaks is performed on the first region to be identified. This ensures that the rain streak recognition system can learn from the features of the rain streaks in the image, enabling it to more accurately identify rain streaks in other regions, thereby improving the overall accuracy of rain streak recognition in the image being processed. After identifying the second region to be identified based on the morphological features of the rain streaks in the first region to be identified, the recognition result of the second region to be identified (the first recognition result) is verified by using the density of the rain streaks in the first region to be identified, thus ensuring the accuracy of the system in recognizing rain streaks in the image being processed.

[0025] When the difference between the second density feature data and the first density feature data is within the built-in deviation range, it indicates that the rain streak recognition system has accurate recognition results for the second region to be recognized. Therefore, the image to be processed can be separated based on the rain streak recognition results to obtain the background region and the rain distribution region. Then, image restoration can be performed based on the separated background region and rain distribution region to ensure that the restored image is more accurate.

[0026] When the difference between the second density feature data and the first density feature data exceeds the built-in deviation range, it indicates that the rain streak recognition system's recognition result for the second region to be recognized is inaccurate. Therefore, it is necessary to determine the cause of the inaccurate recognition. The interfering factors in this case are: 1. The rain streak recognition system lacks sufficient accuracy in recognizing rain streaks under complex conditions; 2. The selected first region to be recognized contains interference terms in its raindrop recognition, resulting in inaccurate recognition analysis results for the first region, thus leading to deviations in the subsequent verification results.

[0027] Once the interfering factors are identified, targeted adjustments are made based on these factors to ensure the accuracy of rain streak recognition and improve the restoration effect of the subsequent image restoration system.

[0028] In step S5, density feature extraction is performed on the first recognition result to obtain the second density feature data of the first recognition result, and the difference between the second density feature data and the first density feature data is calculated to obtain the difference data, including the following steps: S51, Based on the depth data of the image to be processed, the image to be processed is divided into sub-regions of the same depth; S52, based on the first recognition result, the rain streaks in each image sub-region are statistically analyzed to obtain the rain streak density of the corresponding image sub-region; S53, Match the rain streak densities of each image sub-region to determine whether the rain streak densities of each image sub-region are the same. S54, if the rain streak density is the same between each image sub-region, the first recognition result is determined to be accurate, and the rain distribution area of ​​the image to be processed is determined according to the first recognition result and the rain streak of the first recognition area, and the image to be processed is repaired according to the rain distribution area. S55, If there are different rain streak densities among the various image sub-regions, then determine the maximum acquisition depth of the rain streak in the image to be processed based on the depth data of each image sub-region and the corresponding rain streak density of the image sub-region. S56, acquire the depth data of the first region to be identified, and compare the depth data with the maximum acquisition depth of the rain streaks in the image to be processed. If the depth data of the first region to be identified is greater than the maximum acquisition depth of the rain streaks in the image to be processed, then correct the density-depth relationship according to the maximum acquisition depth of the region to be processed, and obtain the corrected density-depth relationship. S57, based on the corrected density-depth relationship and the depth data of each image sub-region, the first density feature data of the first region to be identified is converted into the third density feature data at the same depth, and the difference between the second density feature data and the third density feature data is calculated to obtain the difference data.

[0029] In this embodiment, the image to be processed is divided using depth data, and the rain streak density of image sub-regions at different depths is compared to determine whether the rain streak density is the same, thereby determining the recognition accuracy of the second region to be identified. When it is determined that the rain streak density of each image sub-region is the same, the first recognition result is determined to be accurate. When the rain streak density of each image sub-region is different, the maximum acquisition depth of rain streaks on the image to be processed is determined by jointly judging the depth data with the corresponding rain streak density, and the maximum acquisition depth is compared with the depth data of the first region to be identified to determine the reliability of the initially constructed density-depth relationship. If the depth data of the first region to be identified is less than or equal to the maximum acquisition depth, the density-depth relationship is determined to be reliable; otherwise, it is determined to be unreliable. When it is determined to be unreliable, the density-depth relationship is corrected according to the maximum acquisition depth to ensure its reliability. After ensuring the reliability of the density-depth relationship, the first density feature data of the first region to be identified is converted into the third density feature data at the corresponding depth, and the second density feature data is compared with the third density feature data to ensure the reliability and accuracy of the comparison result.

[0030] For example, when rain streaks are identified in the second region based on the morphological feature data identified in the first region to be identified, a first identification result is obtained. Then, the second density feature data of the second region to be identified is verified using the first density feature data identified in the first region to be identified, so as to determine the accuracy of the rain streaks identification in the second region to be identified.

[0031] When verifying the second density feature data of the second region to be identified, the image to be processed is first divided according to the depth data to obtain image sub-regions with the same depth. Then, the rain streak density of image sub-regions at different depths is compared. If the rain streak density at different depths is the same, it indicates that the rain streak recognition system can accurately identify rain streaks in the image to be processed. For example, since the appearance of rain streaks varies in different environments, there may be some recognition errors in actual recognition. Also, since the rainfall content varies at different depths, theoretically, the deeper the depth, the greater the rainfall content and the greater the corresponding rain streak density. However, due to the limited acquisition accuracy of the camera acquisition device, it can only acquire raindrop information within a certain depth. Raindrop information beyond this depth cannot be acquired. Therefore, when the depth of the image to be processed is greater than the maximum acquisition depth of the camera acquisition device, the result is that the raindrop acquisition results in each region are consistent, resulting in the same rain streak density at different depths.

[0032] Similarly, when the density of rain streaks displayed at different depths is different, it is necessary to determine the maximum acquisition depth of the camera acquisition device based on the depth data of each image sub-region and the rain streak density at the corresponding depth, and then make a judgment based on the comparison of rain streak densities at the maximum acquisition depth.

[0033] By comparing the maximum acquisition depth with the depth data of the first region to be identified, it is determined whether the density-depth relationship constructed based on the first region to be identified is reliable. If the depth data of the first region to be identified is greater than the maximum acquisition depth, it indicates that the density-depth relationship constructed by the first region to be identified is unreliable and therefore needs to be corrected. Conversely, it indicates that the density-depth relationship is reliable.

[0034] After the judgment, a reliable density-depth relationship is determined. Then, based on this density-depth relationship, the rain streak density at different depths is judged. That is, the original first density feature data is transformed to different depths and then compared accordingly, so as to ensure the accuracy of the judgment result.

[0035] In step S55, if there are different rain streak densities among the various image sub-regions, the maximum acquisition depth of the rain streak in the image to be processed is determined based on the depth data of each image sub-region and the corresponding rain streak density, including the following steps: S551, compare the rain streak densities of each image sub-region to obtain the maximum rain streak density; S552, based on the maximum rain streak density, the depth data of the image sub-region is statistically analyzed to obtain the depth set; S553, match the depth data of the image sub-region with the depth set if the depth data of the image sub-region with the maximum rain streak density is less than the depth set, then take the depth data closest to the depth set as the maximum acquisition depth of the area to be processed. S554, If there is a case in the depth data of the image sub-region with a density less than the maximum rain streak density that is in the depth set, then the image sub-region in this case is marked to obtain the image region to be judged; S555, determine the wind and rain obstruction in the image region to be judged. If it is determined that there is wind and rain obstruction in the image region to be judged, calculate the difference between the depth data of the image region to be judged and the depth data of the wind and rain obstruction, and compare the calculated difference with the depth set. S556, if the calculated difference is less than the minimum value of the depth set, then the depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed; otherwise, the calculated difference is taken as the maximum acquisition depth of the area to be processed.

[0036] In this embodiment, when it is determined that there are different rain streak densities among the various image sub-regions, the rain streak densities of each image sub-region are compared with each other, and the maximum rain streak density is statistically analyzed for depth. Then, the depth is judged. If the depth data corresponding to the image sub-region with a rain streak density less than the maximum rain streak density is also less than the depth set, the result collected by the maximum rain streak density can be directly determined as the final result. The depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed. This avoids the actual acquisition depth of the camera being between the physical depths in the image to be processed, which would cause the constructed density-depth relationship to be inaccurate, thus improving the accuracy of the rain streak recognition system's recognition result judgment.

[0037] For example, assuming the physical depth of the image being processed is greater than the camera's maximum capture depth for raindrops, there will be areas with the same rain streak density where the physical depth is greater than the maximum capture depth. Therefore, this situation can be used to reverse-engineer the camera's maximum capture depth for raindrops. By comparing the rain streak densities, the highest rain streak density is selected. Then, the physical depth of the highest rain streak density is determined to establish the depth range corresponding to that density. Finally, based on this depth range, the depth of image sub-regions with rain streak densities less than the highest density is determined to further verify the accuracy of the selected depth data corresponding to the highest rain streak density. For instance, if regions a and b simultaneously have the highest rain streak densities, then the depth of the selected image sub-region with the highest rain streak density is from depth a to depth b (a < b). Meanwhile, there is also a region c in the image to be processed. If the depth of region c is less than the depth a, then since the rain streak density in region c is less than that in region a, it can be indicated that the maximum acquisition depth of the camera is between the depth c and the depth a. In order to ensure that the density-depth relationship constructed based on the depth data is more accurate, the depth c needs to be taken as the maximum acquisition depth of the camera. This ensures that the constructed density-depth data conforms to the changes in the number of raindrops in each region with a depth less than c. Since the depth a is already at the maximum rain streak density, there is no need to judge the number of rain streaks in other regions with a depth greater than a.

[0038] When the depth of region c is greater than the depth of region a, but the rain streak density of region c is less than that of region a (i.e., region c lies between regions a and b), the following situations may occur: First, some depth in region c may be obscured by wind and rain, resulting in a rain streak depth less than the depth of c. Second, region a may have an increased number of raindrops due to structural reasons, which happens to match the number of raindrops in region b. Therefore, region c needs to be assessed. If region c meets these conditions, the depth closest to region a is taken as the camera's maximum acquisition depth. For example, if region d exists, and its depth is greater than the corrected depth of c but less than the depth of a, then the corrected depth of c can be directly taken as the camera's maximum acquisition depth, thereby improving the accuracy of the density and depth data assessment.

[0039] In step S8, if the difference data exceeds the built-in deviation range, interference factors are judged for the image to be processed, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range. This includes the following steps: S811, if the difference between the second density feature data and the first density feature data is less than the deviation range, it indicates that the density of rain streaks in the second region to be identified is less than the density of rain streaks in the first region to be identified. Fluid dynamics simulation is performed on the image to be processed with the first region to be identified as the center to determine whether there is a gas vortex in the first region to be identified. S812, if it is determined that there is a gas vortex in the first area to be identified, then the gas vortex is simulated to determine the influence of the gas vortex on the first area to be identified. S813, if the gas vortex has a gathering effect on raindrops, the interference item is determined to be structural specialness, and the image to be processed is reselected according to the built-in rain color contrast color. S814, if the gas vortex does not have a gathering effect on raindrops or there is no gas vortex, then the second area to be identified is matched according to the built-in rain color to determine the low color difference area in the second area to be identified where the color difference value between the rain color and the rain color is within the built-in color difference range. S815, the second region to be identified is further divided based on the low color difference region to obtain a second sub-region to be identified that does not contain the low color difference region; S816, the rain streak density is determined for the second sub-region to be identified, and the fourth density feature data is obtained. The third density feature data is compared with the fourth density feature data. If it is determined that the difference between the third density feature data and the fourth density feature data is within the deviation range, the rain streak recognition system performs rain streak feature self-learning based on the third density feature data and the fourth density feature data whose difference is within the deviation range. Based on the self-learning result, the recognition accuracy of the second sub-region to be identified is adjusted until the difference between the recognized result and the third density feature data is within the built-in deviation range.

[0040] In this embodiment, when the difference between the second density feature data and the first density feature data is less than the deviation range, a fluid dynamics simulation is performed on the image to be processed, centered on the first region to be identified, to determine whether the reason why the second density feature data is less than the first density feature data is due to structural specialties causing gas vortices to gather surrounding raindrops, resulting in the first density feature data being greater than the normal value. Based on these specialties, the first region to be identified is reselected and re-evaluated. If it is determined that there are no gas turbines or that gas vortices do not have a gathering effect on raindrops, then the second region to be identified is re-evaluated. The system judges the color tone to determine whether the low recognition volume is due to insufficient accuracy of the rain streak recognition system in recognizing low contrast. If it is determined that the rain streak recognition system has low recognition accuracy in the responsible environment, the system re-identifies and re-compares the second sub-region to be identified that does not contain low color difference areas. Based on the comparison results, the rain streak recognition system performs self-learning to continuously improve the accuracy of the rain streak recognition system in recognizing rain streaks in the current environment. This enables the system to segment the image to be processed based on the accurately identified rain distribution area, ensuring that the subsequent image restoration system can more accurately restore the background area and improve the rain removal effect.

[0041] For example, when the difference between the second density feature data and the first density feature data exceeds the built-in deviation range, there are two scenarios: 1. The second density feature data is less than the first density feature data, and the difference between them exceeds the deviation range; 2. The second density feature data is greater than the first density feature data, and the difference between them exceeds the deviation range. For instance, if the deviation range is ±10, and the first density feature data is 100, then if the second density feature data is 80, the first scenario is satisfied; if the second density feature data is 120, then the second scenario is satisfied.

[0042] For the first scenario, there are two possibilities: 1. There is some reason that causes the number of rain streaks in the first area to be identified to increase, making the first density feature data have regional specificity; 2. There is some reason that causes the identification results in the second area to be identified to be inaccurate, with the identified amount being less than the actual amount, resulting in the second density feature data being less than the actual situation.

[0043] For these two possibilities, firstly, based on the morphological characteristics of the rain streaks identified in the first area to be identified, the wind direction in that area is determined. Then, combined with the three-dimensional model constructed from the image to be processed, the flow form of the corresponding wind direction in the three-dimensional model is determined, i.e., a fluid dynamics simulation is performed to determine whether there is a gas vortex in the first area to be identified that causes the surrounding raindrops to gather in that area. If so, it will lead to an increase in the density of the rain streaks identified in the first area to be identified. Therefore, the interfering factor can be determined to be a structural factor. At this time, it is necessary to remove that area, reselect an area with contrast, and reconstruct a new base sample.

[0044] If no gas vortex is present, the second possibility is assessed to determine if insufficient recognition precision is causing inaccurate results. For example, if the background color of the second area to be identified is white, the contrast with the rain color (white) is insufficient. This means that a rain streak recognition system with the same recognition precision can accurately identify the first area to be identified, which has high contrast, but cannot accurately identify the second area to be identified, which has low contrast, resulting in inaccurate recognition results for the second area.

[0045] Therefore, in this case, by performing a low contrast judgment on the second region to be identified, it is determined whether there is a low contrast region to be identified within the second region to be identified. If it is determined that there is a low contrast region to be identified (low color difference region), then the low color difference region is removed, and rain streak recognition is performed on the second sub-region to be identified that does not contain the low color difference region. The recognition result is then compared with the third density feature data to determine the accuracy of the rain streak recognition system in recognizing rain streak.

[0046] When the difference between the third density feature data and the fourth density feature data is within the deviation range, the rain streak recognition system continuously learns and judges this situation, thereby continuously learning to adjust the recognition accuracy and improve the recognition accuracy of the rain streak recognition system.

[0047] In step S8, if the difference data exceeds the built-in deviation range, interference factors are judged for the image to be processed, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range. This includes the following steps: S821, if the difference between the second density feature data and the first density feature data is greater than the deviation range, it indicates that the second density feature data of the second region to be identified is greater than the first density feature data of the first region to be identified. The image to be processed is judged for occlusion with the first region to be identified as the center, and it is determined whether there is wind and rain occlusion on the first region to be identified. S822, if it is determined that there are wind and rain obstructions around the first area to be identified, then the depth data of the wind and rain obstructions is obtained based on the image to be processed, and the rain streak density of the first area to be identified is re-determined based on the depth data of the wind and rain obstructions to obtain the fifth density feature data; specifically: S822a, when it is determined that there are wind and rain obstructions around the first area to be identified, the difference between the depth data of the wind and rain obstructions and the depth data of the first area to be identified is calculated to obtain the identification depth of the first area to be identified, and the density-depth relationship is verified based on the identification depth and the number of rain streaks to obtain the corrected density-depth relationship. S822b: Based on the corrected density-depth relationship and the depth data of the first identification result, determine the density feature data at the same depth and record it as the fifth density feature data.

[0048] S823, compare the difference between the second density feature data and the fifth density feature data to determine whether the calculated difference is within the built-in deviation range; S824, if the calculated difference is determined to be within the built-in deviation range, the interference item is determined to be caused by wind and rain obstruction, and the image to be processed is separated into a background area and a rain distribution area based on the first recognition result of the second area to be recognized and the rain marks recognized in the first area to be recognized, and the image of the rain distribution area is repaired based on the background area. S825, if it is determined that the calculated difference is not within the built-in deviation range or there is no wind or rain obstruction, the rain streak recognition accuracy is adjusted according to the rain streak density of the first area to be recognized until the difference between the rain streak density of the second area to be recognized and the rain streak density of the first area to be recognized is within the built-in deviation range.

[0049] In this embodiment, when the difference between the second density feature data and the first density feature data is determined to be greater than the deviation range, the density-depth relationship constructed in the first area to be identified is verified to determine whether there are any wind or rain obstructions causing an imbalance in the density-depth relationship of the first area to be identified. If wind or rain obstructions are found, the density-depth relationship of the first area to be identified is re-verified based on the depth data of the wind or rain obstructions, and the first density feature data of the first area to be identified is re-judged based on the verified density-depth relationship to determine the numerical relationship between the second density feature data and the fifth density feature data. Based on this numerical relationship, the accuracy of the rain streaks identified in the second area to be identified is determined, thus improving the accuracy of the rain streak recognition system. Conversely, when there are no wind or rain obstructions in the first area to be identified, the recognition accuracy of the rain streak recognition system is adjusted while ensuring the accuracy of the rain streak recognition in the first area to be identified, thereby increasing the number of recognitions in the second area to be identified. This ensures both the recognition accuracy and the recognition precision of the rain streak recognition system for the second area to be identified.

[0050] For example, when the difference between the second density feature data and the first density feature data is greater than the deviation range, there are two possibilities: 1. There is some reason that causes the number of rain streaks in the first area to be identified to decrease; 2. There is some reason that causes the identification result in the second area to be identified to be inaccurate, and the identified amount is less than the actual amount, resulting in the second density feature data being less than the actual situation.

[0051] To address these two possibilities, firstly, based on the 3D model constructed from the image to be processed, determine whether there are any obstructions or rain-related objects that block the first area to be identified, causing a rainfall area that should have a depth of 'a' meters to become a rainfall area with a depth of only 'ab' meters. For example, if there is a rain shelter above the selected first area to be identified, the area blocked by the rain shelter will not receive rainfall, thus reducing the raindrop content in that area. For instance, if the distance between the first area to be identified and the camera is 5 meters, when identifying rain streaks in the first area to be identified, the identified rain streaks should ideally be traces formed by all raindrops within a 5-meter range. However, if there is a 1-meter rain shelter above the first area to be identified, the identified rain streaks are actually traces formed by all raindrops within a 4-meter range. Consequently, the results of constructing the density-depth relationship are inaccurate, leading to an underestimation of the depth. For example, P represents density, A represents the number of rain streaks in the first area to be identified, and H represents the spatial distance between the camera and the first area to be identified. Therefore, when H is constant, the more rain streaks A there are in the first area to be identified, the greater the density P. Similarly, when the number of rain streaks A in the spatial distance H is constant, a shorter spatial distance H results in a smaller number of rain streaks A, and thus a smaller density. That is, when the rainfall is constant, the greater the spatial distance, the greater the number of rain streaks contained in that space. Therefore, in this case, when A, originally distributed in space H, is distributed in space Ha, the density P will decrease. This leads to an underestimation of the relationship between density and depth when determining the density-depth relationship, resulting in an underestimation of the simulated values ​​at different depths. Consequently, the actual rain streak density in the second area to be identified is greater than the simulated rain streak density.

[0052] Therefore, by judging the wind and rain obstructions in the first region of the 3D model composed of the image to be processed, the accuracy of the constructed density-depth relationship is determined. When it is determined that the constructed density-depth relationship is affected by wind and rain obstructions, that is, there are wind and rain obstructions blocking the raindrops on the first region to be processed. By determining the obstruction depth of the wind and rain obstructions on the first region to be processed, the true depth corresponding to the number of rain streaks identified in the first region to be processed is corrected, and the corrected density-depth relationship is obtained. Then, the first density feature data is transformed into different depths according to the corrected density-depth relationship, and the transformed fifth density feature data is compared with the second density feature data to determine whether the misjudgment is caused by wind and rain obstructions. When it is determined that the misjudgment is caused by wind and rain obstructions, it indicates that the rain streak recognition system has accurate recognition results for the second region to be processed. This enables the subsequent image restoration system to perform rain removal restoration operation on the image to be processed based on the separated background area and rain distribution area, thus improving the rain removal restoration effect.

[0053] When the excessive deviation is not due to wind and rain obstructions, it indicates that the complex environment in the second area to be identified may be causing the rain streak recognition system to be insufficient to accurately identify all the rain streak information in the second area. Therefore, by using the first density feature data of the first area to be identified as a benchmark, the recognition accuracy of the rain streak recognition system is adjusted. By reducing the recognition accuracy, the accuracy of the recognition of the first area to be identified is guaranteed while the error tolerance of the rain streak information in the second area to be identified is increased. For example, the original rain streak recognition system counts a rain streak as one that is 90% similar to the first region to be identified. However, due to the complexity of the environment in the second region to be identified, the number of regions with a similarity of 90% decreases, thus reducing the number of rain streaks counted. When the recognition accuracy is reduced to 80%, the recognition result for the first region to be identified remains unchanged, while the number of regions to be identified in the second region is increased. This continuously narrows the data relationship between the two regions. By utilizing the first region to be identified, the recognition accuracy of the rain streak recognition system is ensured, and the accurate identification of rain streaks in the second region is also ensured, thereby improving the accuracy of rain streak recognition.

[0054] Compared to existing image deraining methods based on cue learning, this invention improves the accuracy of rain streak recognition.

[0055] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. An image deraining method based on cue learning, characterized in that, include: Acquire the image to be processed for rain removal, and perform hue selection on the image to be processed. Select the hue region that matches the built-in rain color contrast color and record it as the first region to be identified. Based on the built-in rain color, rain color recognition is performed on the first area to be identified to determine whether there are rain streaks in the first area to be identified; Based on the rain streaks, rain streak features are extracted from the first region to be identified to obtain first feature data; the first feature data includes morphological feature data and first density feature data. The regions in the image to be processed other than the first region to be identified are recorded as the second region to be identified. The second region to be identified is matched according to the morphological feature data, and the region that is successfully matched is recorded as the first recognition result. Dense feature extraction is performed on the first recognition result to obtain the second density feature data, and the difference between the second density feature data and the first density feature data is calculated to obtain the difference data; If the difference data is within the built-in deviation range, the rain streaks of the first recognition result and the first area to be recognized are determined to be the rain distribution area of ​​the image to be processed; Based on the rain distribution area, the image to be processed is divided into a background area and a rain distribution area. The rain distribution area is then inpainted based on the content of the background area to obtain the rain-removed image data. If the difference data exceeds the built-in deviation range, the image to be processed is judged for interference factors, and the accuracy of rain streak recognition is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range.

2. The image deraining method based on cue learning according to claim 1, characterized in that: The rain streak feature is extracted from the first region to be identified based on the rain streak, and the first feature data of the rain streak in the first region to be identified is obtained. The first feature data includes morphological feature data and first density feature data, and also includes: Based on the image to be processed, a spatial coordinate system is constructed, and depth markings are performed on objects in the image to be processed according to the spatial coordinate system to obtain the depth data of the image to be processed. The depth data of the first region to be identified is obtained, and the density-depth relationship is constructed based on the first density feature data and the corresponding depth data.

3. The image deraining method based on cue learning according to claim 2, characterized in that: The process of extracting density features from the first recognition result to obtain second density feature data of the first recognition result, and calculating the difference between the second density feature data and the first density feature data to obtain difference data, includes: The image to be processed is divided based on its depth data to obtain image sub-regions of the same depth; Based on the first recognition result, the rain streaks in each image sub-region are statistically analyzed to obtain the rain streak density of the corresponding image sub-region; The rain streak densities of each image sub-region are matched to determine whether the rain streak densities of each image sub-region are the same. If the rain streak density is the same across all image sub-regions, the first recognition result is determined to be accurate. Based on the first recognition result and the rain streak in the first recognition region, the rain distribution area of ​​the image to be processed is determined, and the image to be processed is repaired based on the rain distribution area. If there are different rain streak densities among the various image sub-regions, the maximum acquisition depth of the rain streak in the image to be processed is determined based on the depth data of each image sub-region and the corresponding rain streak density. The depth data of the first region to be identified is obtained and compared with the maximum acquisition depth of the rain streaks in the image to be processed. If the depth data of the first region to be identified is greater than the maximum acquisition depth of the rain streaks in the image to be processed, the density-depth relationship is corrected according to the maximum acquisition depth of the region to be processed, and the corrected density-depth relationship is obtained. Based on the corrected density-depth relationship and the depth data of each image sub-region, the first density feature data of the first region to be identified is converted into the third density feature data at the same depth, and the difference between the second density feature data and the third density feature data is calculated to obtain the difference data.

4. The image deraining method based on cue learning according to claim 3, characterized in that: If there are different rain streak densities among the various image sub-regions, the maximum acquisition depth of the rain streak in the image to be processed is determined based on the depth data of each image sub-region and the corresponding rain streak density, including: The rain streak density of each image sub-region is compared with each other to obtain the maximum rain streak density; The depth set is obtained by statistically analyzing the depth data of image sub-regions based on the maximum rain streak density. The depth data of the image sub-region with a density less than the maximum rain streak density is matched with the depth set. If the depth data of the image sub-region with a density less than the maximum rain streak density are all less than the depth set, the depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed. If there are cases in the depth data of image sub-regions with a density less than the maximum rain streak density that are within the depth set, then the image sub-regions in such cases are marked to obtain the image regions to be judged. The image region to be judged is subjected to wind and rain obstruction. If it is determined that there is wind and rain obstruction in the image region to be judged, the difference between the depth data of the image region to be judged and the depth data of the wind and rain obstruction is calculated, and the calculated difference is compared with the depth set. If the calculated difference is less than the minimum value of the depth set, the depth data closest to the depth set is taken as the maximum acquisition depth of the area to be processed; otherwise, the calculated difference is taken as the maximum acquisition depth of the area to be processed.

5. The image deraining method based on cue learning according to claim 3, characterized in that: If the difference data exceeds the built-in deviation range, then the image to be processed is judged for interference factors, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range, including: If the difference between the second density feature data and the first density feature data is less than the deviation range, it indicates that the density of rain streaks in the second region to be identified is less than the density of rain streaks in the first region to be identified. Fluid dynamics simulation is performed on the image to be processed with the first region to be identified as the center to determine whether there is gas vortex in the first region to be identified. If it is determined that there is a gas vortex in the first region to be identified, then the gas vortex is simulated to determine the influence of the gas vortex on the first region to be identified. If the gas vortex has a gathering effect on raindrops, the interference item is determined to be structural specialness, and the image to be processed is reselected according to the built-in rain color contrast color. If the gas vortex does not have a gathering effect on raindrops or there is no gas vortex, then the second area to be identified is matched according to the built-in rain color to determine the low color difference area in the second area to be identified where the color difference value between the rain color and the rain color is within the built-in color difference range. The second region to be identified is further divided based on the low color difference region to obtain a second sub-region to be identified that does not contain the low color difference region. The rain streak density of the second sub-region to be identified is determined to obtain the fourth density feature data. The third density feature data is then compared with the fourth density feature data. If it is determined that the difference between the third density feature data and the fourth density feature data is within the deviation range, the rain streak recognition system performs rain streak feature self-learning based on the third density feature data and the fourth density feature data whose difference is within the deviation range. Based on the self-learning results, the recognition accuracy of the second sub-region to be identified is adjusted until the difference between the recognized result and the third density feature data is within the built-in deviation range.

6. The image deraining method based on cue learning according to claim 1, characterized in that: If the difference data exceeds the built-in deviation range, then the image to be processed is judged for interference factors, and the rain streak recognition accuracy is adjusted according to the interference factors until the difference between the second density feature data and the first density feature data is within the built-in deviation range. This also includes: If the difference between the second density feature data and the first density feature data is greater than the deviation range, it indicates that the second density feature data of the second region to be identified is greater than the first density feature data of the first region to be identified. The image to be processed is then judged for occlusion based on the first region to be identified, to determine whether there are wind and rain obstructions on the first region to be identified. If it is determined that there are wind and rain obstructions around the first area to be identified, the depth data of the wind and rain obstructions is obtained according to the image to be processed, and the density of rain streaks in the first area to be identified is re-judged according to the depth data of the wind and rain obstructions to obtain the fifth density feature data. The difference between the second density feature data and the fifth density feature data is calculated and compared to determine whether the calculated difference is within the built-in deviation range. If the calculated difference is determined to be within the built-in deviation range, the interference item is determined to be caused by wind and rain obstruction. Based on the first recognition result of the second region to be recognized and the rain streaks recognized in the first region to be recognized, the image to be processed is separated into a background region and a rain distribution region. The image of the rain distribution region is then repaired based on the background region. If the calculated difference is determined to be outside the built-in deviation range or there are no obstructions from wind and rain, the rain streak recognition accuracy is adjusted based on the rain streak density of the first area to be recognized until the difference between the rain streak density of the second area to be recognized and the rain streak density of the first area to be recognized is within the built-in deviation range.

7. The image deraining method based on cue learning according to claim 6, characterized in that: If it is determined that there are wind and rain obstructions around the first area to be identified, then the depth data of the wind and rain obstructions is obtained based on the image to be processed, and the rain streak density of the first area to be identified is re-determined based on the depth data of the wind and rain obstructions to obtain the fifth density feature data, including: When it is determined that there are wind and rain obstructions around the first area to be identified, the difference between the depth data of the wind and rain obstructions and the depth data of the first area to be identified is calculated to obtain the identification depth of the first area to be identified. The density-depth relationship is then verified based on the identification depth and the number of rain streaks identified to obtain the corrected density-depth relationship. Based on the corrected density-depth relationship and the depth data of the first identification result, the density feature data at the same depth is determined and recorded as the fifth density feature data.

Images