An image processing method, apparatus, device and medium for an all-sky imager

Abstract

This invention relates to an image processing method, apparatus, device, and medium for an all-sky imager, belonging to the field of image processing technology. The method includes: acquiring visible light and infrared images at the same time; identifying the effective observation area of ​​the sky in the images; determining a first insect-occupied region based on the difference between the visible light image and the target clear-sky image at the corresponding time; if the first insect-occupied region exists, determining and removing static occupied regions based on the infrared images at consecutive times; then, using a radial brightness expectation model, determining a second insect-occupied region in the infrared image by analyzing the residual between the actual pixel brightness and the model's expected value; finally, repairing the insect-occupied regions in both images. This invention can accurately identify and repair interference caused by insect occlusion in all-sky images, effectively eliminating cloud detection misjudgments caused by this, improving image quality, and providing a reliable data foundation for subsequent accurate prediction of solar radiation and photovoltaic power.

Description

Technical Field

[0001] This invention relates to the field of graphics processing technology, and in particular to an image processing method, apparatus, device and medium for an all-sky imager. Background Technology

[0002] To improve the accuracy of ultra-short-term solar radiation and photovoltaic power prediction under grid-connected photovoltaic power generation, continuous sky observation data with high spatiotemporal resolution is required. All-sky imagers can simultaneously acquire sky images in the visible and infrared bands, providing key inputs for prediction models based on thresholding, optical flow, and machine learning. The image quality directly affects the reliability of subsequent predictions.

[0003] Currently, preprocessing methods for all-sky imagers primarily focus on eliminating static or structural interference. Common practices include: removing fixed terrain features in the background through image registration and difference techniques; identifying and repairing occlusion areas caused by lens mounts; and restoring images of areas affected by direct sunlight. These processes mainly target visible light images and aim to improve the overall visual quality and usability of the images.

[0004] However, centralized photovoltaic power plants are mostly located in open, outdoor areas, and all-sky imagers are typically installed at high altitudes, leading to frequent entry of flying insects into the field of view, creating dynamic occlusion. This occlusion alters the color ratio and infrared radiation information of local image areas. When cloud detection is performed based on the red-blue ratio characteristics of visible light images or brightness temperature data from infrared images, the abnormal color and brightness temperature values ​​caused by insect occlusion exhibit characteristics similar to small, fragmented clouds, easily leading to misidentification. This introduces errors into cloud cover statistics, cloud motion tracking, and subsequent radiometric inversion, limiting further improvements in the accuracy of prediction models. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an image processing method, apparatus, device and medium for an all-sky imager, aiming to solve at least one of the above-mentioned technical problems.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: In a first aspect, this application provides an image processing method for an all-sky imager, employing the following technical solution: An image processing method for an all-sky imager includes: Acquire image information from an all-sky imager at the same time, including the current visible light image and the current infrared image; Image recognition is performed on the current visible light image and the current infrared image respectively to obtain the effective observation area in the current visible light image and the current infrared image; Based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time, the first insect occlusion area in the current visible light image is determined. In response to the determination that the first insect occlusion area exists, a static occlusion area is determined based on the current infrared image, the first infrared image, and the second infrared image, and the static occlusion area is removed from the current infrared image to obtain the target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. Based on the target infrared image, a radial brightness model of the effective observation area is established, and based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model, the second insect occlusion area in the current infrared image is determined. Based on the first insect occlusion area, the current visible light image is repaired, and based on the second insect occlusion area and the radial brightness expectation model, the current infrared image is repaired to obtain the repaired image information.

[0007] The beneficial effects of this invention are: by constructing a collaborative processing scheme for visible light and infrared dual-channel images, it is possible to intelligently identify insect-occupied areas with uncertain area and position. The principle lies in utilizing the characteristics of insects causing grayscale anomalies in visible light images and brightness-temperature anomalies in infrared images. The difference between the visible light image and a clear-sky reference is used to directly locate insect occlusion, thereby triggering the processing flow of the infrared image. Static interference is eliminated through multi-temporal registration, and a radial brightness expectation model is established to detect abnormal brightness-temperature points caused by insect body temperature. Through image processing and feature analysis algorithms, insects and clouds can be accurately distinguished in both bands, achieving automatic identification and location of typical insect-occupied areas.

[0008] Building upon this foundation, the present invention further eliminates anomalies in red-blue ratio and brightness temperature caused by insect occlusion, reducing the false positive rate of cloud detection. This improves the reliability and data quality of all-sky images, enabling them to more accurately reflect the true sky conditions, thereby providing more reliable data input for subsequent radiation calculations and ultra-short-term photovoltaic power prediction.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Further, determining the first insect occlusion region in the current visible light image based on the current visible light image and the target clear sky image at the corresponding time includes: Based on the acquisition timestamp of the visible light image, the target clear sky image at the corresponding time is obtained from a pre-built clear sky image database; The visible light image and the target clear sky image are converted into grayscale images respectively to obtain the grayscale values ​​of the visible light image and the target clear sky image; Calculate the difference image between the grayscale values ​​of the visible light image and the grayscale values ​​of the target clear sky image; The region in the difference image whose gray value is less than a preset negative gray value threshold is determined as the first insect occlusion region.

[0011] The beneficial effects of adopting the above-mentioned further scheme are as follows: by using a clear-sky reference image that is strictly matched with the current acquisition time as a benchmark, the background sky brightness determined by the fixed solar altitude angle and atmospheric conditions is effectively stripped away, making the local gray-scale anomalies caused by dynamic and minute insect occlusion more prominent. Using gray-scale difference calculation and preset negative gray-scale threshold determination, specific dark area features caused by insect body occlusion, distinct from cloud changes, can be sensitively and accurately captured. This overcomes the limitation of traditional methods that only handle static occlusion, and achieves automatic detection of flying insects whose position, shape, and area are uncertain.

[0012] Furthermore, determining the static occlusion area based on the current infrared image, the first infrared image, and the second infrared image includes: Within the effective observation area, color distribution matching is performed on the first infrared image and the second infrared image to obtain the first infrared image and the second infrared image after distribution matching. Based on the preset Farnebäck optical flow method, the second infrared image after distribution matching is image registered with the first infrared image, so that the second infrared image is aligned to the coordinate system of the first infrared image, and a registered image pair is obtained. Within the preset edge ring area of ​​the registered image pair, the color difference of each pixel corresponding to the same spatial position in the color space is calculated, and each region with a color difference less than a preset color difference threshold is taken as the static occlusion region.

[0013] The beneficial effects of adopting the above-mentioned further scheme are as follows: By matching the color distribution of infrared images at different times, the overall illumination and color temperature differences caused by time intervals are eliminated; image registration using the Farnebäck optical flow method effectively compensates for pixel displacement caused by the movement of dynamic sky features such as clouds, ensuring strict alignment of image pairs in spatial coordinates. By calculating the pixel-level color difference of the aligned image pairs within a preset edge ring, local areas where color features do not change with time in natural scenes (such as clouds and illumination) can be sensitively detected. This allows for the robust automatic identification and extraction of stationary occlusions such as lens brackets and dust stains, which can then be pre-emptively removed as known interference in subsequent processing, avoiding confusion caused by these static factors in the subsequent dynamic insect occlusion detection based on the brightness model.

[0014] Further, within the effective observation area, color distribution matching is performed on the first infrared image and the second infrared image to obtain a first infrared image and a second infrared image after distribution matching, including: The pixels of the first infrared image and the pixels of the second infrared image within the effective observation area are converted to the CIE-Lab color space. The pixel value distribution of the first infrared image and the second infrared image in each channel of the CIE-Lab color space is statistically analyzed to obtain the mean and standard deviation of the first infrared image in each channel, and the mean and standard deviation of the second infrared image in each channel. Based on the mean and standard deviation of the first infrared image in each channel and the mean and standard deviation of the second infrared image in each channel, the pixel values ​​of the first infrared image in each channel are matched to the distribution of the second infrared image to obtain the first infrared image and the second infrared image after distribution matching.

[0015] The beneficial effects of adopting the above-mentioned further scheme are as follows: By converting the infrared image to the CIE-Lab color space, which conforms to the perceptual characteristics of the human eye, and by statistically analyzing the global mean and standard deviation of the images at different time points in each channel, the overall color distribution shift caused by differences in lighting conditions and environmental radiation can be accurately quantified. Based on this statistical characteristic, the pixel values ​​of the earlier first infrared image are standardized and matched to align its color distribution with the global features of the later second infrared image, thereby effectively eliminating the systematic brightness and chromaticity deviation between the two frames at the pixel level. This provides a stable and reliable comparison benchmark for subsequent static occlusion detection that relies on color consistency, ensuring that subsequent color difference calculations are not affected by changes in global illumination over time intervals.

[0016] Furthermore, establishing the radial brightness model of the effective observation area based on the target infrared image includes: The effective observation area of ​​the target infrared image is divided into multiple annular bands at equal intervals along the radial direction; For each of the annular bands, calculate the representative value of all pixels within the annular band in the luminance channel; Based on the representative values ​​of all the aforementioned annular bands, an initial brightness curve varying along the radial direction is generated; The initial brightness curve is smoothed to obtain a smoothed radial brightness expectation model, wherein the radial brightness expectation model is used to output the expected brightness value corresponding to the pixel based on the radius position of any pixel.

[0017] The beneficial effects of adopting the above-mentioned further scheme are as follows: It considers the physical law that the brightness of the sky background typically changes continuously and smoothly with the viewing angle (radius) in all-sky fisheye lens imaging. By dividing the effective observation area into annular bands along the radial direction and extracting representative brightness values, the normal radial distribution characteristics of the sky background brightness can be accurately captured. Furthermore, a robust brightness expectation model can be constructed through smoothing processing. This model provides an expected value for each pixel location that conforms to the natural brightness gradient of the sky, thereby effectively separating local abnormal brightness points caused by insect body temperature from normal sky brightness changes. Residual analysis provides a reliable and adaptive judgment benchmark for accurately detecting the second insect occlusion area, improving the detection accuracy of small, random insect occlusion in infrared images.

[0018] Furthermore, determining the second insect occlusion region in the current infrared image based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model includes: Based on the residuals corresponding to all pixels, determine the median and the mean absolute value of the residuals; Based on the residuals corresponding to all pixels, the median of the residuals, and the mean absolute value of the residuals, candidate abnormal regions are identified. From the candidate abnormal regions, connected regions with an area larger than a preset minimum area are selected as the second insect occlusion region.

[0019] The beneficial effects of adopting the above-mentioned further scheme are as follows: By calculating the residual between the actual brightness of each pixel and the expected value of the radial model representing the gradual change of normal brightness in the sky, it is possible to effectively amplify and capture localized minor brightness and temperature anomalies caused by insect body temperature. Using the median and distribution characteristics (such as mean absolute deviation) of all pixel residuals as an adaptive threshold, rather than a fixed threshold, allows the anomaly detection to dynamically adapt to the overall statistical characteristics of the current image, improving the algorithm's universality and robustness to images under different times and weather conditions. Furthermore, by setting a minimum area threshold to filter candidate anomaly connected regions, false detections caused by image noise, transient interference, or minor pixel defects can be effectively filtered out, ensuring that the finally identified second insect occlusion region accurately corresponds to the real, significantly influential insect target. This achieves high-precision and high-reliability automated identification of insect occlusion regions in infrared images.

[0020] Furthermore, the repaired image information includes a repaired visible light image and a repaired infrared image. The repair process involves performing repair processing on the current visible light image based on the first insect occlusion region, and on the current infrared image based on the second insect occlusion region and the radial brightness expectation model, to obtain the repaired image information, including: The pixel values ​​within the area obscured by the first insect are marked as invalid, resulting in a repaired visible light image, or... Based on the previous and next images of the current visible light image, the filling value of the first insect occlusion area is generated by optical flow method, and the original pixel value in the first insect occlusion area is replaced by the filling value to obtain the repaired visible light image. For each pixel within the second insect occlusion area, the corresponding expected brightness value is obtained from the radial brightness expectation model based on the radius position of the pixel; The brightness values ​​of each pixel within the area occluded by the second insect are replaced with the corresponding desired brightness values ​​to obtain the repaired infrared image.

[0021] The beneficial effects of adopting the above-mentioned further scheme are: it provides adapted restoration strategies for the different characteristics of visible light and infrared images. For visible light images, by directly marking invalid values ​​or using optical flow information from images at different times for intelligent filling, it is possible to restore the texture and structural information of the occluded area to the maximum extent while removing insect occlusion, maintaining the continuity of the image in the spatiotemporal sequence. For infrared images, it utilizes an established radial expectation model that conforms to the physical brightness distribution of the sky to replace the abnormal brightness temperature value of the occluded area with the reasonable expectation value that should exist at that location, thereby efficiently and accurately correcting the radiation distortion introduced by the insect's body temperature. This differentiated restoration strategy eliminates pixel-level anomalies caused by insect occlusion in dual-band images, directly improving the data quality and physical consistency of the restored image, and providing reliable and interference-free data input for subsequent cloud detection and radiation prediction based on the fusion of multiple features such as red-blue ratio and brightness temperature.

[0022] Secondly, this application provides an image processing device for an all-sky imager, which adopts the following technical solution: An all-sky image processing device, comprising: The image acquisition module is used to acquire image information collected by the all-sky imager at the same time, including the current visible light image and the current infrared image; The region recognition module is used to perform image recognition on the current visible light image and the current infrared image respectively, and to obtain the effective observation area in the current visible light image and the current infrared image; The visible light occlusion recognition module is used to determine the first insect occlusion area in the current visible light image based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time. An infrared image processing module is used to determine a static occlusion area based on the current infrared image, the first infrared image, and the second infrared image in response to the determination that the first insect occlusion area exists, and to remove the static occlusion area from the current infrared image to obtain a target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. An infrared light occlusion recognition module is used to establish a radial brightness model of the effective observation area based on the target infrared image, and to determine the second insect occlusion area in the current infrared image based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model. The image restoration module is used to restore the current visible light image based on the first insect occlusion area, and to restore the current infrared image based on the second insect occlusion area and the radial brightness expectation model, so as to obtain restored image information.

[0023] Thirdly, this application provides an electronic device that adopts the following technical solution: An electronic device includes a memory and a processor, wherein the memory stores a computer program capable of being loaded by the processor and executing the all-sky image processing method of any one of the first aspects.

[0024] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium storing a computer program capable of being loaded by a processor and executing the all-sky imager image processing method according to any one of the first aspects.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0026] Figure 1 A flowchart illustrating an all-sky image processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an all-sky image processing device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0029] This application provides an image processing method for an all-sky imager. This method can be executed by an electronic device, which can be a server or a mobile terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The mobile terminal device can be a laptop computer, a desktop computer, etc., but is not limited to these.

[0030] like Figure 1 As shown, an all-sky image processing method mainly includes: S1, acquire image information collected by the all-sky imager at the same time, the image information including the current visible light image and the current infrared image; In this embodiment, the all-sky imager is a dedicated optical imaging device, typically employing a fisheye lens, used to capture and record the complete view of the entire hemispherical sky, and its output is a circular image containing the entire sky view.

[0031] Simultaneous timing indicates that the acquisition of visible light and infrared images is strictly synchronized in time. This is usually controlled by the imager's internal clock or an external trigger signal to ensure that the two images reflect the sky state at exactly the same moment. This serves as the time reference for subsequent multispectral joint analysis.

[0032] Current visible light images refer to images acquired simultaneously by an imager using a visible light band (typically 380-780 nm) sensor. Their pixel values ​​primarily reflect the intensity distribution of sunlight reflection and scattering from clouds, aerosols, etc., and are expressed as grayscale or color information discernible to the human eye. Current infrared images refer to images simultaneously acquired by an imager using an infrared band (e.g., thermal infrared) sensor. Their pixel values ​​reflect the thermal radiation intensity (brightness temperature) of the observed target (such as clouds, clear-sky atmosphere, or ground features), and can be used to retrieve temperature information.

[0033] S2, perform image recognition on the current visible light image and the current infrared image respectively, and obtain the effective observation area in the current visible light image and the current infrared image; In this embodiment, visible light and infrared images are distinguished by naming, the image disk area is identified, and the image resolution is obtained.

[0034] Specifically, using image segmentation algorithms or fixed optical parameters from an all-sky imager, the center coordinates and radius of the circular region are calculated or detected. Once the circular boundary is determined, a binary mask is generated, where pixels within the circular region are marked as valid, and pixels outside the circle are marked as invalid. Ultimately, this mask represents the valid observation area. After performing this operation on both visible light and infrared images, the valid observation areas for each image are obtained, ensuring that all subsequent pixel-level calculations are confined to the pixel range corresponding to the real sky, eliminating background interference.

[0035] S3, based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time, determine the first insect occlusion area in the current visible light image; In this embodiment of the application, S3 includes the following sub-steps: S31, Based on the acquisition timestamp of the visible light image, obtain the target clear sky image at the corresponding time from the pre-built clear sky image database; S32, convert the visible light image and the target clear sky image into grayscale images respectively to obtain the grayscale values ​​of the visible light image and the target clear sky image; S33, calculate the difference image between the grayscale value of the visible light image and the grayscale value of the target clear sky image; S34, the region in the difference image whose gray value is less than a preset negative gray value threshold is determined as the first insect occlusion region.

[0036] In this embodiment, the region outside the effective observation area is set as a NaN value that does not participate in subsequent calculations.

[0037] Iterate through all historical data, select cloudless visible light images from different times, and pre-build a time-continuous clear sky database. Extract the acquisition timestamps of the visible light images, and retrieve target clear sky images with the same acquisition timestamp from the clear sky database. Convert the target clear sky images and visible light images at the same time into grayscale images.

[0038] The grayscale values ​​of the target's clear sky image and visible light image at the same time are represented by grayscale values ​​respectively. csl,t with gray t This indicates the difference between the two. Then it is: ; In static occlusion Smaller, therefore can When the grayscale level drops significantly, it is identified as an area obscured by insects, and an abnormal occlusion mask A1 is obtained.

[0039] S4, in response to determining that the first insect occlusion area exists, based on the current infrared image, the first infrared image and the second infrared image, a static occlusion area is determined, and the static occlusion area is removed from the current infrared image to obtain the target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. In this embodiment of the application, if it is determined from the visible light image that there is an area blocked by insects, then the infrared image at the same acquisition time is further processed.

[0040] Considering that insects do not stay for long periods of time, it is necessary to import both the first infrared image and the second infrared image from before and after the current collection time.

[0041] Specifically, determining the static occlusion area based on the current infrared image, the first infrared image, and the second infrared image includes: Within the effective observation area, color distribution matching is performed on the first infrared image and the second infrared image to obtain the first infrared image and the second infrared image after distribution matching. Based on the preset Farnebäck optical flow method, the second infrared image after distribution matching is image registered with the first infrared image, so that the second infrared image is aligned to the coordinate system of the first infrared image, and a registered image pair is obtained. Within the preset edge ring area of ​​the registered image pair, the color difference of each pixel corresponding to the same spatial position in the color space is calculated, and each region with a color difference less than a preset color difference threshold is taken as the static occlusion region.

[0042] In this embodiment, color distribution matching is an image preprocessing technique that aims to adjust the global color statistical characteristics (such as mean and variance) of an image through mathematical transformations, making its color distribution more consistent with that of another image, thereby eliminating the overall color cast caused by differences in lighting and atmospheric conditions due to different acquisition times. Farnebäck's optical flow method is a classic algorithm for calculating dense optical flow fields. It approximates the local image structure through polynomial expansion, thereby estimating the motion vector of each pixel in the image between two consecutive frames. Image registration refers to the process of geometrically aligning two images of the same scene acquired at different times, from different perspectives, or by different sensors, so that points representing the same physical location in the two images have the same image coordinates. The preset edge ring region refers to a ring-shaped region near the edge of the circular area within the effective observation area of ​​the entire sky. This region is usually chosen because static obstructions (such as supports) often appear at the edges of the image.

[0043] Because the two images were acquired at different times, the overall brightness and color may differ due to variations in illumination. Therefore, color distribution matching is necessary to adjust the color statistical characteristics of the first infrared image to match those of the second infrared image, eliminating systematic color deviations. Next, even with color consistency, moving clouds between the two frames can cause the position of the same object in the image to shift. Therefore, the Farnebäck optical flow method is used to perform pixel-level geometric deformation on the second infrared image to ensure complete spatial alignment with the first infrared image, resulting in a registered image pair. Finally, within a predefined edge ring region where static occlusions are most likely to occur, the color difference between the two aligned images is calculated pixel-by-pixel. Regions with a color difference less than a preset color difference threshold are identified as static occlusion regions.

[0044] In this embodiment of the application, the step of performing color distribution matching on the first infrared image and the second infrared image within the effective observation area to obtain the first infrared image and the second infrared image after distribution matching includes: The pixels of the first infrared image and the pixels of the second infrared image within the effective observation area are converted to the CIE-Lab color space. The pixel value distribution of the first infrared image and the second infrared image in each channel of the CIE-Lab color space is statistically analyzed to obtain the mean and standard deviation of the first infrared image in each channel, and the mean and standard deviation of the second infrared image in each channel. Based on the mean and standard deviation of the first infrared image in each channel and the mean and standard deviation of the second infrared image in each channel, the pixel values ​​of the first infrared image in each channel are matched to the distribution of the second infrared image to obtain the first infrared image and the second infrared image after distribution matching.

[0045] In the above implementation, firstly, within the identified effective observation area, the CIE-Lab color space pixels of the image are calculated within the disk: ; Where L, A, and B represent brightness, red-green axis, and yellow-blue axis, respectively.

[0046] The distribution characteristics of the three channels of Lab were statistically analyzed, and the standard deviation was obtained. and mean To eliminate the overall brightness deviation caused by the time difference, the Lab distribution of the first infrared image is matched to the second infrared image, taking L as an example (the same applies to A and B): ; in, A very small constant (e.g., 10⁻⁶) is used to prevent algorithm instability caused by a zero denominator. Based on this, the Farnebäck optical flow method is used to register the second infrared image to the coordinate system of the first infrared image.

[0047] The color difference between the calibrated first and second infrared images is calculated near the edge ring, and static occlusion areas are selected using a set color difference threshold. These static occlusion areas are then removed and do not participate in subsequent image processing.

[0048] S5. Based on the target infrared image, establish a radial brightness model of the effective observation area, and based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model, determine the second insect occlusion area in the current infrared image. In this embodiment of the application, a radial brightness model of the effective observation area is established based on the target infrared image, including: The effective observation area of ​​the target infrared image is divided into multiple annular bands at equal intervals along the radial direction; For each of the annular bands, calculate the representative value of all pixels within the annular band in the luminance channel; Based on the representative values ​​of all the aforementioned annular bands, an initial brightness curve varying along the radial direction is generated; The initial brightness curve is smoothed to obtain a smoothed radial brightness expectation model, wherein the radial brightness expectation model is used to output the expected brightness value corresponding to the pixel based on the radius position of any pixel.

[0049] In the above implementation, based on the center and maximum radius of the effective observation area, a set of concentric circles with increasing radii are generated, dividing the circular area into multiple continuous annular bands. Next, for each annular band, the infrared brightness channel values ​​of all pixels within it are extracted, and a statistical representative value, such as the median, that resists outlier interference is calculated. This value represents the typical brightness of the sky background in that annular area. Then, using the average radius of the annular band as the abscissa and its brightness representative value as the ordinate, a series of discrete data points are obtained. Connecting these points in radial order forms the initial brightness curve. The initial brightness curve initially depicts the brightness variation trend from the image center to the edge, but may contain spikes due to residual noise, thin clouds, or minor interference that has not been completely removed. Finally, a smoothing algorithm, such as one-dimensional Gaussian filtering, Savitzky-Golay filtering, or fitting with a smooth spline function, is applied to the initial brightness curve to filter out high-frequency fluctuations, resulting in a smooth and continuous radial brightness expectation model. This model encapsulates the background field of brightness distribution under the current sky conditions. Any pixel with a given radius can obtain its expected brightness value at that location by querying this model.

[0050] In this embodiment of the application, the second insect occlusion region in the current infrared image is determined based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model, including: Based on the residuals corresponding to all pixels, determine the median and the mean absolute value of the residuals; Based on the residuals corresponding to all pixels, the median of the residuals, and the mean absolute value of the residuals, candidate abnormal regions are determined. From the candidate abnormal regions, connected regions with an area larger than a preset minimum area are selected as the second insect occlusion region.

[0051] In this embodiment, the expected brightness value and deviation are first calculated. The disk image of the target infrared image is divided into thin rings at equal radius intervals, and representative values ​​are statistically analyzed on each ring. (L can be either the mode or the median, and A and B can be the median), and the curve is smoothed using Gaussian smoothing along the radial direction to obtain the smoothed radial Lab curve. .

[0052] For each pixel Expected value of brightness for: ; Calculate the residual within the disk after removing the static occlusion area. : ; Further statistical analysis yielded its corresponding median. Then the mean absolute deviation (MAD) can be expressed as: ; Finally, the areas with abnormally high brightness (when it exceeds a certain multiple of MAD k, it is recommended to set it to 1.5) are extracted as insect abnormal occlusion mask A2 (the second insect occlusion area): ; For A2, the minimum area Amin of the mask can be set to ensure that A2>Amin in order to eliminate misjudgments of the sun and image noise.

[0053] S6, based on the first insect occlusion area, the current visible light image is repaired, and based on the second insect occlusion area and the radial brightness expectation model, the current infrared image is repaired to obtain the repaired image information.

[0054] In this embodiment of the application, the repaired image information includes a repaired visible light image and a repaired infrared image.

[0055] The restoration process involves repairing the current visible light image based on the first insect occlusion region and repairing the current infrared image based on the second insect occlusion region and the radial brightness expectation model, to obtain restored image information, including: The pixel values ​​within the area obscured by the first insect are marked as invalid, resulting in a repaired visible light image, or... Based on the previous and next images of the current visible light image, the filling value of the first insect occlusion area is generated by optical flow method, and the original pixel value in the first insect occlusion area is replaced by the filling value to obtain the repaired visible light image. For each pixel within the second insect occlusion area, the corresponding expected brightness value is obtained from the radial brightness expectation model based on the radius position of the pixel; The brightness values ​​of each pixel within the area occluded by the second insect are replaced with the corresponding desired brightness values ​​to obtain the repaired infrared image.

[0056] For the boundaries of the repaired regions in the restored visible light and infrared images, the fusion weights are determined based on distance transformation and then feathering is performed.

[0057] In this application embodiment, two repair strategies are provided for visible light images: one is the invalid value marking method, which simply marks the pixels in the occluded area as special values ​​and declares that their data is missing. This is suitable for scenarios where the requirements for data continuity are not strict or the occlusion is minimal. The second method is optical flow filling, which involves interpolating and filling occluded areas using optical flow. First, within the unoccluded area or the effective sky domain, based on the CIE-Lab space, the Lab distribution of the first and second visible light images at consecutive time points is matched to the current visible light image. Then, the dense optical flow from the first visible light image to the second visible light image is calculated. And resampling is performed using bilinear interpolation to align with the current visible light acquisition time: ; ; Further, the imputed value is generated by taking the average: .

[0058] For infrared images, the restoration strategy is model replacement. Since a desired model describing the radial distribution of normal sky brightness has been established, for each pixel within the occluded area, the expected brightness value at that location is obtained by querying the model based on its radius position, and this value is directly used to replace the actual value contaminated by the abnormal brightness and temperature of the insect. This restoration, based on physical principles, effectively corrects the radiance values.

[0059] Finally, boundary feathering is applied as an optimization step to the two repaired images. Since direct replacement or filling may produce discontinuous jumps at the boundary between the repaired area and the normal area, the distance from the pixel to the boundary of the repaired area is calculated, and fusion weights are assigned to pixels near the boundary. Then, the repaired pixel values ​​are blended with the original background pixel values ​​according to the weights, thereby generating a visually smooth final image with jagged edges.

[0060] Export the processed image in the same format as the input image, and add a suffix to the name to distinguish the original image from the processed image.

[0061] This method constructs a collaborative processing scheme for visible and infrared dual-channel images, enabling intelligent identification of insect-occupied areas with uncertain area and location. The principle lies in utilizing the characteristics of insects causing grayscale anomalies in visible light images and brightness-temperature anomalies in infrared images. By leveraging the difference between the visible light image and a clear-sky reference, insect occlusion is directly located, triggering the processing flow of the infrared image. Static interference is eliminated through multi-temporal registration, and a radial brightness expectation model is established to detect abnormal brightness-temperature points caused by insect body temperature. Through image processing and feature analysis algorithms, insects and clouds can be accurately distinguished in both bands, achieving automatic identification and localization of typical insect-occupied areas.

[0062] Figure 2 A schematic diagram of the structure of an all-sky imager image processing device 200 is shown.

[0063] like Figure 2 As shown, an all-sky image processing device 200 mainly includes: Image acquisition module 201 is used to acquire image information collected by the all-sky imager at the same time, the image information including the current visible light image and the current infrared image; The region recognition module 202 is used to perform image recognition on the current visible light image and the current infrared image respectively, and to obtain the effective observation area in the current visible light image and the current infrared image; The visible light occlusion recognition module 203 is used to determine the first insect occlusion area in the current visible light image based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time. The infrared image processing module 204 is used to determine a static occlusion area based on the current infrared image, the first infrared image, and the second infrared image in response to determining the existence of the first insect occlusion area, and to remove the static occlusion area from the current infrared image to obtain the target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. The infrared light occlusion recognition module 205 is used to establish a radial brightness model of the effective observation area based on the target infrared image, and to determine the second insect occlusion area in the current infrared image based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model. The image restoration module 206 is used to restore the current visible light image based on the first insect occlusion area, and to restore the current infrared image based on the second insect occlusion area and the radial brightness expectation model, so as to obtain restored image information.

[0064] In one example, the module in any of the above devices may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0065] For example, when modules in a device can be implemented via a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0066] In this application, various objects such as messages / information / devices / network elements / systems / apparatus / actions / operations / processes / concepts may be named. It is understood that these specific names do not constitute a limitation on the relevant objects. The names may be changed depending on the scenario, context, or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from their functions and technical effects embodied / performed in the technical solution.

[0067] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0068] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] Figure 3 This is a structural block diagram of an electronic device 300 according to an embodiment of this application.

[0070] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302, and may further include one or more of an information input / output (I / O) interface 303, a communication component 304, and a communication bus 305.

[0071] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps in the above-described all-sky image processing method. The memory 302 stores various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0072] I / O interface 303 provides an interface between processor 301 and other interface modules, such as keyboards, mice, and buttons. These buttons can be virtual or physical. Communication component 304 is used to test wired or wireless communication between electronic device 300 and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 304 may include a Wi-Fi component, a Bluetooth component, and an NFC component.

[0073] The communication bus 305 may include a path for transmitting information between the aforementioned components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0074] The electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute an all-sky image processing method for an image processing device as described in the above embodiments.

[0075] The following describes the computer-readable storage medium provided in the embodiments of this application. The computer-readable storage medium described below can be referred to in correspondence with the image processing method of an all-sky imager described above.

[0076] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described all-sky image processing method for an image processing device.

[0077] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0079] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. An image processing method for an all-sky imager, characterized in that, include: Acquire image information from an all-sky imager at the same time, including the current visible light image and the current infrared image; Image recognition is performed on the current visible light image and the current infrared image respectively to obtain the effective observation area in the current visible light image and the current infrared image; Based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time, the first insect occlusion area in the current visible light image is determined. In response to the determination that the first insect occlusion area exists, a static occlusion area is determined based on the current infrared image, the first infrared image, and the second infrared image, and the static occlusion area is removed from the current infrared image to obtain the target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. Based on the target infrared image, a radial brightness model of the effective observation area is established, and based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model, the second insect occlusion area in the current infrared image is determined. Based on the first insect occlusion area, the current visible light image is repaired, and based on the second insect occlusion area and the radial brightness expectation model, the current infrared image is repaired to obtain the repaired image information.

2. The all-sky image processing method according to claim 1, characterized in that, The step of determining the first insect occlusion region in the current visible light image based on the current visible light image and the target clear sky image at the corresponding time includes: Based on the acquisition timestamp of the visible light image, the target clear sky image at the corresponding time is obtained from a pre-built clear sky image database; The visible light image and the target clear sky image are converted into grayscale images respectively to obtain the grayscale values ​​of the visible light image and the target clear sky image; Calculate the difference image between the grayscale values ​​of the visible light image and the grayscale values ​​of the target clear sky image; The region in the difference image whose gray value is less than a preset negative gray value threshold is determined as the first insect occlusion region.

3. The all-sky image processing method according to claim 1, characterized in that, The determination of the static occlusion area based on the current infrared image, the first infrared image, and the second infrared image includes: Within the effective observation area, color distribution matching is performed on the first infrared image and the second infrared image to obtain the first infrared image and the second infrared image after distribution matching. Based on the preset Farnebäck optical flow method, the second infrared image after distribution matching is image registered with the first infrared image, so that the second infrared image is aligned to the coordinate system of the first infrared image, and a registered image pair is obtained. Within the preset edge ring area of ​​the registered image pair, the color difference of each pixel corresponding to the same spatial position in the color space is calculated, and each region with a color difference less than a preset color difference threshold is taken as the static occlusion region.

4. The all-sky image processing method according to claim 3, characterized in that, Within the effective observation area, color distribution matching is performed on the first infrared image and the second infrared image to obtain a first infrared image and a second infrared image after distribution matching, including: The pixels of the first infrared image and the pixels of the second infrared image within the effective observation area are converted to the CIE-Lab color space. The pixel value distribution of the first infrared image and the second infrared image in each channel of the CIE-Lab color space is statistically analyzed to obtain the mean and standard deviation of the first infrared image in each channel, and the mean and standard deviation of the second infrared image in each channel. Based on the mean and standard deviation of the first infrared image in each channel and the mean and standard deviation of the second infrared image in each channel, the pixel values ​​of the first infrared image in each channel are matched to the distribution of the second infrared image to obtain the first infrared image and the second infrared image after distribution matching.

5. The all-sky image processing method according to claim 3, characterized in that, The step of establishing a radial brightness model of the effective observation area based on the target infrared image includes: The effective observation area of ​​the target infrared image is divided into multiple annular bands at equal intervals along the radial direction; For each of the annular bands, calculate the representative value of all pixels within the annular band in the luminance channel; Based on the representative values ​​of all the aforementioned annular bands, an initial brightness curve varying along the radial direction is generated; The initial brightness curve is smoothed to obtain a smoothed radial brightness expectation model, wherein the radial brightness expectation model is used to output the expected brightness value corresponding to the pixel based on the radius position of any pixel.

6. The all-sky image processing method according to claim 1, characterized in that, The residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model is used to determine the second insect occlusion area in the current infrared image, including: Based on the residuals corresponding to all pixels, determine the median and the mean absolute value of the residuals; Based on the residuals corresponding to all pixels, the median of the residuals, and the mean absolute value of the residuals, candidate abnormal regions are determined. From the candidate abnormal regions, connected regions with an area larger than a preset minimum area are selected as the second insect occlusion region.

7. The all-sky image processing method according to claim 1, characterized in that, The repaired image information includes a repaired visible light image and a repaired infrared image. The repair process involves performing repair processing on the current visible light image based on the first insect occlusion region, and on the current infrared image based on the second insect occlusion region and the radial brightness expectation model, to obtain the repaired image information, including: The pixel values ​​within the area obscured by the first insect are marked as invalid, resulting in a repaired visible light image, or... Based on the previous and next images of the current visible light image, the filling value of the first insect occlusion area is generated by optical flow method, and the original pixel value in the first insect occlusion area is replaced by the filling value to obtain the repaired visible light image. For each pixel within the second insect occlusion area, the corresponding expected brightness value is obtained from the radial brightness expectation model based on the radius position of the pixel; The brightness values ​​of each pixel within the area occluded by the second insect are replaced with the corresponding desired brightness values ​​to obtain the repaired infrared image.

8. An image processing device for an all-sky imager, characterized in that, include: The image acquisition module is used to acquire image information collected by the all-sky imager at the same time, including the current visible light image and the current infrared image; The region recognition module is used to perform image recognition on the current visible light image and the current infrared image respectively, and to obtain the effective observation area in the current visible light image and the current infrared image; The visible light occlusion recognition module is used to determine the first insect occlusion area in the current visible light image based on the effective observation area of ​​the current visible light image and the target clear sky image at the corresponding time. An infrared image processing module is used to determine a static occlusion area based on the current infrared image, the first infrared image, and the second infrared image in response to the determination that the first insect occlusion area exists, and to remove the static occlusion area from the current infrared image to obtain a target infrared image. The first infrared image represents the infrared image at the previous moment of the current infrared image acquisition time, and the second infrared image represents the infrared image at the next moment of the current infrared image acquisition time. An infrared light occlusion recognition module is used to establish a radial brightness model of the effective observation area based on the target infrared image, and to determine the second insect occlusion area in the current infrared image based on the residual between the actual brightness of each pixel in the effective observation area of ​​the target infrared image and the expected brightness of the radial brightness model. The image restoration module is used to restore the current visible light image based on the first insect occlusion area, and to restore the current infrared image based on the second insect occlusion area and the radial brightness expectation model, so as to obtain restored image information.

9. An electronic device, characterized in that, Includes a processor, which is coupled to a memory; The processor is configured to execute a computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images