A solar tracking dataset generation method based on gradient algorithm

Abstract

The application discloses a solar tracking dataset generation method based on a gradient algorithm, which is applied to the field of image processing and aims at the problem that existing datasets cannot effectively reflect the cloud thickness and the relationship between the cloud thickness and the spatial position of the sun. The cloud blocks of different weathers, thicknesses and moments are extracted, the Sobel gradient of all the cloud blocks is calculated, the intervals are divided according to the gradient values according to the requirement, and the cloud blocks of different thicknesses are classified. Then, a cloud block image in each thickness category is randomly selected, the source image is distinguished in front and back backgrounds, and the cloud ROI region in the source image is extracted. Next, the cloud ROI region is further binarized to obtain a binary image mask of the cloud block ROI. Then, the mask of the cloud block and the position information of the sun in the target image are used to perform image fusion of different coverage degrees. Finally, the cloud sun-shading images under different weather conditions, different cloud block thicknesses and different coverage degrees are obtained.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to a solar tracking dataset generation technique. Background Technology

[0002] Image fusion technology is an important subfield of image processing, aiming to achieve seamless fusion between a selected target object and its background in a source image, resulting in a more natural synthesized image. When there are significant differences in texture features between the source and target images, the directly synthesized image will have obvious boundaries. Currently, image fusion techniques mainly include optical flow-based image fusion, optimal fusion line-based image fusion, and generative adversarial network-based fusion algorithms. Optical flow-based methods achieve good registration results, but their real-time performance is poor due to the need to calculate dense optical flow. While optimal fusion line-based images resolve ghosting issues, this algorithm usually requires further processing of exposure differences between different images using other algorithms, increasing its complexity. Generative adversarial network-based algorithms offer good fusion results, but their benefits are limited in terms of real-time performance and computational complexity. Poisson fusion, an algorithm that solves for pixel-optimal solutions by constructing a Poisson equation, achieves good results and, based on the characteristics of the gradient domain, can achieve smooth transitions at image boundaries. The algorithm's complexity is also well optimized thanks to the OpenCV library.

[0003] With the rapid development of the AIoT era, people's lifestyles and production methods have undergone tremendous changes. Deep learning and machine learning have been applied to various fields. Image detection and object tracking systems have gradually developed, and for supervised learning deep learning models, the existence of high-quality training datasets is particularly important for improving model recognition accuracy. However, in areas such as images of cloud cover, plant diseases and pests, rare lesions, and rail cracks, manually collecting actual images is labor-intensive and yields limited benefits. Furthermore, it's impossible to determine the label location of objects in occluded images, thus limiting supervised learning methods. While supervised learning is prevalent in single-object tracking, datasets of images showing cloud cover obscuring the sun are particularly lacking. This hinders the effective application of powerful deep learning in solar-related fields, such as the impact of different solar orientations on photovoltaic reception efficiency, the influence of urban building lighting decisions, and the impact on agricultural crop growth. Therefore, developing a simple algorithm for generating image datasets with good fusion results is of great significance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a method for generating solar tracking datasets based on gradient algorithms.

[0005] The technical solution adopted in this invention is: a method for generating solar tracking datasets based on gradient algorithms, comprising:

[0006] S1. Use a camera to record a video of the sun throughout the day under different weather conditions, such as sunny and cloudy weather, and use a Python script to process the obtained video images into video frames to obtain a sequence of solar source images.

[0007] S2. Randomly extract cloud blocks from the solar source image multiple times, perform Sobel gradient calculation on the obtained cloud blocks, and then use the gradient value of a cloud block of a certain thickness as a benchmark to divide cloud block intervals of different thicknesses into different gradient intervals, classify all cloud block images, and scale all images to the set size using the resize function of the OpenCV library.

[0008] S3. Randomly select one cloud block image from each thickness category, and use the grabcut method on the source image corresponding to its name index to distinguish the foreground information of the cloud block ROI and the background information of the non-cloud block ROI, so as to obtain the image of the outline foreground and the completely black background information of the cloud block ROI.

[0009] S4. Threshold segmentation is performed on the image processed in step S3 to obtain a binarized mask image for subsequent Poisson fusion processing.

[0010] S5. Using the Poisson fusion principle, combine the label information of the sun's position in the target image, the coverage requirement, and the mask image obtained in step S4 to perform Poisson fusion to obtain images of cloud blocks with different thicknesses and coverage that obscure the sun.

[0011] S6. Repeat steps S3, S4, and S5 until the image in each class has been selected once, and then end to obtain a solar tracking dataset with different coverage rates of the sun by cloud layers of different thicknesses.

[0012] The beneficial effects of this invention are as follows: For the fusion image of the sun and cloud patches, this invention ensures the diversity of cloud patches by randomly selecting different cloud patches multiple times. Then, Sobel gradients are used to calculate the selected cloud patches, dividing them into different gradient value ranges according to requirements to distinguish cloud patches of different thicknesses, fully reflecting the diversity of clouds in the natural environment. Furthermore, the Grabcut method is used to extract cloud foreground information and non-cloud background information from the source image corresponding to the cloud patches, obtaining contour information whose boundaries fit the shape of the clouds. After thresholding the extracted image, Poisson fusion based on the gradient domain is used to achieve seamless fusion of cloud patches and the sun target under different occlusion conditions, ensuring that the tone and illumination are consistent with the scene area in the sun target image. Simultaneously, before the Poisson fusion begins, the center coordinates of different coverages are calculated based on the sun target's position information in the image, thus performing fusion of different coverages to realistically simulate the scene of the sun entering and leaving the clouds in a natural environment.

[0013] The method proposed in this invention aims to simply and efficiently expand scarce datasets, especially solar tracking datasets, providing an efficient and simple image processing method for the training phase of deep learning models. Furthermore, by generating images with different coverage levels, different illumination conditions are created, which also has positive implications for other fields such as maximizing photovoltaic energy efficiency, rationalizing urban building lighting, and maximizing agricultural photosynthesis. Attached Figure Description

[0014] Figure 1 This is a flowchart of the solution of the present invention;

[0015] Figure 2 Image 1 of the sun obtained by processing video recorded by a camera in frame segmentation, as provided in an embodiment of the present invention;

[0016] Figure 3 Image 2 of the sun obtained by processing video recorded by a camera in frame segmentation, as provided in an embodiment of the present invention;

[0017] Figure 4 This invention provides cloud blocks of varying thicknesses randomly extracted using the selectROI function, as provided in an embodiment of the invention.

[0018] Figure 5 The Sobel gradient algorithm is used to calculate the gradient values ​​of each cloud block in this embodiment of the invention.

[0019] Figure 6 The cloud block ROI outline image provided in this embodiment of the invention has cloud foreground information and non-cloud background information extracted by Grabcut;

[0020] Figure 7This invention provides masks for different cloud blocks obtained after threshold segmentation in its embodiments.

[0021] Figure 8 This is a seamless fusion result image of a thick cloud block obtained by Poisson fusion with different coverage of solar image 1, provided in an embodiment of the present invention.

[0022] Figure 9 The image provided in this embodiment of the invention shows a seamless fusion result of thick cloud blocks obtained by Poisson fusion over solar image 2 with different coverage levels. Detailed Implementation

[0023] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0024] Example 1

[0025] like Figure 1 As shown, the present invention provides a method for generating a solar tracking dataset based on a gradient algorithm, comprising:

[0026] S1: First, use a camera to record videos of sunny and cloudy weather, and then use a Python script to process the obtained videos into frames, setting the fps to 30, to obtain a sequence of solar images.

[0027] S2: Read the solar image sequence using the OpenCV library in Python. The `selectROI` function is called for manual interaction, randomly selecting multiple cloud ROIs to be moved and merged. The Sobel gradient values ​​of all cloud ROIs are calculated. Then, using the gradient value of a cloud ROI of a certain thickness as a standard, different gradient value intervals are defined according to requirements to distinguish the thickness of the cloud ROIs. This embodiment sets three gradient intervals, corresponding to thick clouds, relatively thin clouds, and thin clouds, respectively. In this embodiment, thick clouds have a Sobel gradient value greater than 7; relatively thin clouds have a Sobel gradient value between 5 and 7; and thin clouds have a Sobel gradient value between 0 and 5.

[0028] S3: Randomly select one image from each of the three types of cloud block images, and index the corresponding solar image sequence based on the name of the cloud block. Use the Grabcut method on this source image to extract foreground information of the cloud block and background information of the non-cloud block, as well as possible foreground and background information. Call the grabCut method in the OpenCV library, setting the iteration count to 11, to obtain the contour image of the cloud block foreground. For example... Figure 6 The image shown contains foreground contour information of cloud blocks.

[0029] S4: Threshold segment the image obtained in S3 that contains the foreground contour information of the cloud block, so that the cloud block part is completely white and the rest is completely black, as the mask image of Poisson fusion.

[0030] S5: Based on the sun's position information obtained from labelme software and the required coverage, the center coordinates of different coverage levels are calculated. The Poisson fusion method (seamlessClone) from the OpenCV library is used for seamless image fusion. The coverage level is determined based on the actual continuous coverage of the sun entering the cloud layer from 0% to 100%. This invention sets four coverage levels: 0%, 25%, 50%, and 100%, and calculates the corresponding sun center coordinates accordingly. Poisson fusion is then performed to ensure that the cloud layer blocks obscure the sun's target position with varying degrees of coverage.

[0031] Example 2

[0032] In step S1, a Hikvision camera is used for recording and storage. The stored video file is processed into frames, with one frame captured every 10 seconds. Preset points for the camera are set according to different times of day, and the lens is rotated between morning and afternoon to ensure the sun is always visible in the video image. The captured results, after frame processing, are as follows: Figure 2 , Figure 3 As shown.

[0033] Example 3

[0034] In step S2, this invention uses the Sobel gradient value to characterize the ROI of cloud blocks of different thicknesses, such as... Figure 4 As shown. The gradient images in the X and Y directions are calculated separately, and then summed with average weights. The calculation formula is:

[0035]

[0036] g x This represents the response of the Sobel operator to the image in the horizontal (x-direction); g y The response of the Sobel operator in the vertical direction (y-direction) of the image is represented; f represents the gray value of the image; z1-z9: the pixel values ​​of the nine pixels in the image, forming a 3*3 neighborhood around the current pixel position.

[0037] The Sobel gradient calculation results for the four cloud block ROIs are as follows: Figure 5As shown, the Sobel gradient of the thick cloud layer Black_thick_cloud is 8.029072667096159, the Sobel gradient of the relatively thick cloud layer thick_cloud is 8.067581432512378, the Sobel gradient of the relatively thin cloud layer thin_cloud is 5.6733174574513585, and the Sobel gradient of the thin cloud layer thin_cloud2 is 2.8823510850074907.

[0038] Example 4

[0039] The Grabcut algorithm in step S3, based on Graph Cut, employs a Gaussian Mixture Model (GMM) with RGB three channels. It replaces the single-pass energy minimization segmentation with an iterative process of segmentation estimation and model parameter learning. Through iteration, the algorithm optimizes the GMM parameters for target and background modeling in each iteration, resulting in better image segmentation.

[0040] First, the cloud patch is randomly selected by calling the selectROI function in the OpenCV library, resulting in an initial trimap T, where all pixels outside the bounding box are used as background pixels T. B All pixels within the box are considered as possible target pixels T. U For T B For each pixel n within the range, initialize the label α of pixel n. n =1;T U The label of each pixel within is set to α. n =0, so that the GMM model of the target and the background can be estimated by pixels.

[0041] In the RGB color space, the foreground and background are each modeled using a K-dimensional Gaussian mixture model with full covariance. The resulting Gaussian Mixture Module (GMM) is:

[0042]

[0043]

[0044] π i μ represents the weight of the i-th Gaussian component, i.e., the proportion of that component in the entire mixture model. i Let μ represent the mean vector of the i-th Gaussian component, containing k elements, each element corresponding to a dimension, μ representing one of the components, d representing the dimension of the data, and T representing the transpose of the vector.

[0045] The iterative energy minimization segmentation method used for foreground and background segmentation is as follows:

[0046] (1) Assign Gaussian components from the GMM to each pixel:

[0047]

[0048] Among them, D n As a metric, z can be understood as a function that measures the distance or difference between pixel n and each component in the GMM; n Let be the feature vector of pixel n. Here, we use the color features of three channels; := is the assignment operator.

[0049] (2) For the image Z to be fused obtained in step S2, learn and optimize the parameters of the GMM:

[0050] θ:=argmin θ U(α,k,θ,z)

[0051] (3) Segmentation estimation:

[0052]

[0053] Then the iteration process begins. Through experimentation, this invention determined the optimal segmentation to be achieved after 11 iterations, as shown in the following results. Figure 6 As shown in the figure, thick_cloud_grabcut1 represents the contour extraction result of the thick cloud layer ROI, thick_cloud_grabcut2 represents the contour extraction result of the relatively thick cloud layer ROI, thin_cloud_grabcut1 represents the contour extraction result of the relatively thin cloud layer, and thin_cloud_grabcut2 represents the extraction result of the thin cloud layer. Figure 6 The third image, thin_cloud_grabcut1, is pure black because the selected target area is entirely the foreground area of ​​thin clouds.

[0054] Example 5

[0055] This invention uses the `threshold` method from the OpenCV library to set values ​​greater than 70 to 255 and values ​​less than or equal to 70 to 0 in the cloud foreground image obtained through the Grabcut algorithm, thus creating a mask image. The result is as follows: Figure 7As shown in the image, thick_cloud_mask1 is the binarized result of contour extraction for the thick cloud ROI, thick_cloud_mask2 is the binarized result of contour extraction for the relatively thick cloud ROI, thin_cloud_mask1 is the binarized result of contour extraction for the relatively thin cloud ROI, and thin_cloud_mask2 is the binarized result of extraction for the thin cloud ROI. The reason why thin_cloud_mask1 in the third image is completely black is that the cloud block region randomly selected by the selectROI method is entirely a foreground thin cloud region.

[0056] The threshold function expression is: cv2.threshold(src,thresh,maxval,type). The first parameter src represents the input image; the second parameter thresh represents the threshold value, which ranges from 0 to 255; the third parameter maxval represents the fill color, which also ranges from 0 to 255; and the fourth parameter type represents the type of algorithm used for binarization.

[0057] The fourth parameter of the threshold function is set to THRESH_BINARY. The purpose of THRESH_BINARY is as follows:

[0058]

[0059] Where src(x,y) is the gray value in the x-th row and y-th column of the gray value matrix of the input image, and maxval is the gray value that needs to be set when src(x,y) is greater than the threshold thresh, where thresh is the judgment threshold. In this embodiment, thresh is set to 70 and maxval is set to 255.

[0060] Example 6

[0061] This invention first uses the labelme labeling software to annotate the sun's position, obtaining the sun's center position information in the solar image. Then, it uses a Python script to extract the position coordinates from the JSON file obtained by labelme labeling. Based on four set coverage levels, it calculates the changes in the center position coordinates of cloud blocks under different coverage levels, thus obtaining the position coordinates of the cloud block targets embedded in the solar image required by the Poisson fusion algorithm.

[0062] The following formula provides a variational explanation of Poisson image fusion. The variational equation signifies that the Poisson-based image editing fusion method uses the gradient field within the source image patch as a guide to smoothly diffuse the differences between the cloud patch ROI target scene and the solar source image at the fusion boundary into the image patch. In this way, the fused cloud image patch can be seamlessly integrated into the solar image, and its tone and illumination are consistent with the solar image.

[0063]

[0064]

[0065]

[0066] in, I represents the gradient of the fused image patch. * This represents the optimal image we are looking for, such that the gradient field... With a given gradient field over the entire image domain Ω The square of the difference is minimized, Ω represents the domain of image I, and represents the region of the entire image. B Given known image boundary values, F represents the pixel values ​​at the image boundaries; f , The symbol is used in Euler's formula. div represents divergence, and the divergence of the gradient field is obtained by calculating the divergence.

[0067] This invention utilizes the `seamlessClone` function from the OpenCV library for Poisson fusion. The function expression is: `void cv::seamlessClone(inputArray src, inputArray dst, inputArray mask, Pointp, OutputArray blend, int flags)`. Its first parameter, `src`, is the source image from which the cloud patch is extracted; the second parameter, `dst`, is the background image of the target sun; the third parameter, `mask`, is the masked region image of the cloud patch obtained through thresholding; the fourth parameter, `p`, is the position where the cloud patch is placed within the target sun image; the fifth parameter is the output image; and the sixth parameter is the cloning method. In this embodiment, the `MIXED_CLONE` type is selected for seamless fusion. The experimental results are as follows: Figure 8 , Figure 9 As shown.

[0068] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method for generating solar tracking datasets based on gradient algorithm, characterized in that, include: S1. Use a camera to record solar videos under different weather conditions, and use a Python script to perform video frame segmentation to obtain source images at different times under different weather conditions; S2. Randomly extract cloud patches from the source image multiple times, and perform Sobel gradient calculation on the obtained cloud patch ROI. Divide cloud patch intervals of different thicknesses according to the gradient value and classify all cloud patch images. S3. Randomly select one cloud block image from each thickness category, index the corresponding source image according to its name, use the grabcut method on the source image to distinguish the foreground information of the cloud block ROI and the background information of the non-cloud block ROI, and obtain the outline image of the cloud block foreground. S4. Threshold segmentation is performed on the contour image of the cloud block foreground to obtain a binarized mask image; S5. Using the Poisson fusion principle, combine the label information of the sun's position in the target image, the coverage requirement, and the mask image obtained in step S4 to perform Poisson fusion to obtain images of cloud blocks with different thicknesses and coverage that obscure the sun. S6. Repeat steps S3, S4, and S5 until the image in each class has been selected once, and then end to obtain a solar tracking dataset with different coverage rates of the sun by cloud layers of different thicknesses.

2. The method for generating a solar tracking dataset based on a gradient algorithm according to claim 1, characterized in that, Step S2, which involves randomly extracting cloud patches from the source image multiple times, specifically involves randomly selecting an image from the source image and using the selectROI method in the OpenCV library to randomly select cloud patch ROIs multiple times from the source image.

3. The method for generating a solar tracking dataset based on a gradient algorithm according to claim 1, characterized in that, The implementation process of the grabcut method in step S3 is as follows: The `selectROI` function in the OpenCV library is used to randomly select a cloud patch, with all pixels outside the selected area treated as background pixels (T). B All pixels within the box are considered as possible target pixels T. U Initialize T B Label α of each pixel within n =1; Initialize T U Label α of each pixel within n =0, thus obtaining a GMM model that estimates the target and background through pixels; The GMM model parameters for target and background modeling are obtained more optimally during the iteration process, resulting in better image segmentation; Once the set number of iterations is reached, the outline image of the cloud block foreground is obtained.

4. The method for generating a solar tracking dataset based on a gradient algorithm according to claim 1, characterized in that, Step S4 specifically involves calling the threshold method in the OpenCV library. For the outline image of the cloud block foreground obtained in step S3, the pixel values ​​in the image containing the cloud foreground target that are greater than the threshold are set to 255, and the pixel values ​​that are less than or equal to the threshold are set to 0, thus obtaining the mask image.

5. The method for generating a solar tracking dataset based on a gradient algorithm according to claim 1, characterized in that, Step S5 calls the seamlessClone function in the OpenCV library to perform Poisson fusion.

Images