A method for measuring the height of objects in a power environment based on image data processing

Abstract

This invention provides a method for measuring the height of objects in a power environment based on image data processing. The method involves photographing the environment of a substation to obtain substation environmental images, generating three-dimensional point cloud data from these images, creating spatial image blocks, constructing a priority queue of these spatial image blocks, selecting high-priority spatial image blocks from the queue to represent the current environmental conditions, training a neural network model, and using the neural network to predict the height of objects within the substation environment, thereby achieving monitoring of the substation environment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method for measuring the height of objects in a power environment based on image data processing. Background Technology

[0002] With the rapid development of the power industry, the number of substations is increasing rapidly. Substations below 220 kV are often unmanned, and the environment inside the substation affects the safe operation of the equipment. The presence of other objects within the safe energized distance of the equipment can pose safety hazards. Existing technologies all rely on sensors to measure the substation's own sensor data and monitor its operation through this data, but they cannot monitor the environment inside the substation. Summary of the Invention

[0003] To address the aforementioned shortcomings in existing technologies, this invention provides a method for measuring the height of objects in the power environment based on image data processing, which solves the problem of the lack of a method for monitoring the environment within substations.

[0004] To achieve the aforementioned objectives, the present invention employs the following technical solution: a method for measuring the height of objects in a power environment based on image data processing, comprising:

[0005] The environment surrounding the substation was photographed to obtain images of the substation environment;

[0006] Multiple substation environment images of the same scene are converted into 3D point cloud data;

[0007] Based on the 3D point cloud data, spatial image blocks are generated, and a priority queue of spatial image blocks is constructed.

[0008] Select high-priority spatial image patches from the spatial image patch priority queue, train the neural network model, and obtain the trained neural network model;

[0009] The trained neural network model is used to process the substation environment image to obtain the height of objects in the power environment.

[0010] Furthermore, the conversion of multiple substation environmental images of the same scene into three-dimensional point cloud data includes:

[0011] Based on each substation environment image of the same scene, extract sparse feature points of pixels;

[0012] Depth and normal vectors of sparse feature points of pixels are reconstructed to obtain three-dimensional point cloud data.

[0013] Furthermore, the step of generating spatial image patches based on 3D point cloud data and constructing a spatial image patch priority queue includes:

[0014] Based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain the 3D point region;

[0015] Each 3D point region is filtered to obtain a filtered 3D point region.

[0016] Calculate HOG or Harris features for each filtered 3D point region;

[0017] Based on the HOG or Harris features of the filtered 3D point region, find the similar HOG or Harris features of another filtered 3D point region along the epipolar line to obtain a 3D point matching pair.

[0018] Triangulation of the 3D point matching pairs yields spatial image patches;

[0019] The positions within the spatial image patch are optimized to obtain an optimized spatial image patch;

[0020] Calculate the confidence score of the optimized spatial image patch;

[0021] Based on the confidence level, the optimized spatial image patches are sorted to obtain a spatial image patch queue;

[0022] The depth of sparse feature points in the 3D points of each optimized spatial image block in the spatial image block queue is calculated to obtain the depth value of the sparse feature points.

[0023] Based on the depth values ​​of sparse feature points, nonlinear depth optimization is performed on the 3D points in each optimized spatial image block to obtain a queue of spatial image blocks with optimized 3D point depth.

[0024] If the depth of the central 3D point in each optimized spatial image block in the 3D point depth-optimized spatial image block queue is greater than the confidence threshold, then the 3D point depth-optimized spatial image block queue is a completed spatial image block priority queue. Otherwise, the depth value of the next sparse feature point is searched, and nonlinear depth optimization is continued until the judgment condition is met.

[0025] Furthermore, the step of generating spatial image patches based on 3D point cloud data and constructing a spatial image patch priority queue includes:

[0026] Based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain the 3D point region;

[0027] Each 3D point region is filtered to obtain a filtered 3D point region.

[0028] Calculate the distance from each 3D point in the filtered 3D point region to its adjacent 3D points;

[0029] Based on the distance of each 3D point to its neighboring 3D points, find 3D points in different filtered 3D point regions whose distance difference is lower than the distance threshold, and form 3D point matching pairs.

[0030] Based on the 3D point matching pairs, the 3D point matching pairs are triangulated to obtain multiple spatial image patches;

[0031] Calculate the proportion of 3D points of the same type of power environment object in each spatial image patch to the total number of 3D points in the entire single spatial image patch;

[0032] Multiple spatial image patches are sorted according to their proportion to obtain a spatial image patch priority queue, where the higher the proportion, the higher the priority.

[0033] The beneficial effects of the above-mentioned further scheme are as follows: clustering the three-dimensional points within the neighborhood is equivalent to dividing the three-dimensional point cloud data into various regions to obtain three-dimensional point regions. By calculating the distance between each three-dimensional point and its adjacent three-dimensional points, two three-dimensional points with close distances are found, and three-dimensional points with similar surrounding environments are identified to form three-dimensional point matching pairs. Spatial image blocks are established on the three-dimensional point matching pairs to find the range of the same environmental region. Based on the proportion of three-dimensional points of the same type of power environment objects in the overall single spatial image block, the distribution of the same type of power environment objects in an image block is determined. Thus, image blocks with high proportions are selected to train the neural network to predict the overall object height of the environment, thereby enabling the monitoring of the substation environment.

[0034] Furthermore, the neural network model is the PointNet++ neural network model.

[0035] Furthermore, the training of the neural network model to obtain the trained neural network model includes:

[0036] Input high-priority spatial image patches and their corresponding labels into the neural network model to obtain the output of the neural network model for this training.

[0037] Calculate the training loss value based on the output of the neural network model in this training iteration;

[0038] Determine whether the current training loss value is lower than the loss threshold. If yes, the neural network model training is complete. If not, update the weights of the neural network model based on the current training loss value to obtain the updated neural network model.

[0039] The updated neural network model is trained again using new high-priority spatial image patches until the loss value is lower than the loss threshold.

[0040] Furthermore, the formula for calculating the training loss value for this iteration is:

[0041]

[0042] Where Loss is the training loss value for this iteration, h is the output height of the neural network model for this iteration, h′ is the label height for this iteration, and th... + For positive height difference, th _ The height difference is negative.

[0043] The beneficial effect of the above further scheme is: when hh′≥th + At that time, through Measuring distance (hh′) 2 and ratio The weights between h and h′ are determined by considering the distance (hh′) more as the difference between h and h′ increases. 2 The percentage, determined by distance (hh′) 2 When rapidly adjusting the parameters of the neural network, the ratio of h to h′ should be the primary consideration when the difference between them is small. The weight, through the ratio Fine-tuning the parameters of the neural network to make them more accurately approximate the target value; in the case of h′-h≥th _ hour, When the value is negative, the greater the difference between h and h′, the more important the distance (hh′) should be. 2 The percentage, determined by distance (hh′) 2 When rapidly adjusting the parameters of the neural network, the ratio of h to h′ should be the primary consideration when the difference between them is small. The weight, through the ratio Fine-tuning the parameters of a neural network makes the parameters more accurately approximate the target values.

[0044] Furthermore, the formula for updating the weights of the neural network model based on the current training loss value is as follows:

[0045]

[0046] Where W is the updated weight, W′ is the original weight, Loss is the training loss value, h is the output height of the neural network model in the current training, and h′ is the label height in the current training.

[0047] The beneficial effect of the above further scheme is that when hh′≥0, the weights are adjusted downwards, and through... The ratio of h to h′ measures the difference between the output height and the label height. The larger the value, the faster the rate of decrease; when h′-h≥0, the weights are adjusted upwards. The larger the value, the greater the upward adjustment of the weight. Through bidirectional adjustment, the output height h approaches the label height h′.

[0048] The technical solution of this invention has at least the following advantages and beneficial effects: This invention obtains substation environment images by taking pictures of the substation environment, obtains three-dimensional point cloud data based on the substation environment images, generates spatial image blocks, constructs a spatial image block priority queue, selects high-priority spatial image blocks from the spatial image block priority queue to represent the current environmental situation, trains a neural network model, and uses the neural network to predict the height of objects in the substation environment, thereby realizing the monitoring of the substation environment. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method for measuring the height of objects in a power environment based on image data processing. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0051] like Figure 1 As shown, a method for measuring the height of objects in a power environment based on image data processing includes the following steps:

[0052] S1. Take pictures of the environment where the substation is located to obtain images of the substation environment;

[0053] S2. Convert multiple substation environment images of the same scene into 3D point cloud data;

[0054] In step S2, converting multiple substation environmental images of the same scene into three-dimensional point cloud data includes:

[0055] Based on each substation environment image of the same scene, extract sparse feature points of pixels;

[0056] Depth and normal vectors of sparse feature points of pixels are reconstructed to obtain three-dimensional point cloud data.

[0057] In step S2, other existing methods can also be used to convert substation environmental images into three-dimensional point cloud data.

[0058] S3. Generate spatial image blocks based on 3D point cloud data and construct a spatial image block priority queue;

[0059] In step S3, one implementation method is as follows:

[0060] Based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain the 3D point region;

[0061] Each 3D point region is filtered to obtain a filtered 3D point region.

[0062] Calculate HOG or Harris features for each filtered 3D point region;

[0063] Based on the HOG or Harris features of the filtered 3D point region, find the similar HOG or Harris features of another filtered 3D point region along the epipolar line to obtain a 3D point matching pair.

[0064] In this embodiment, "similar" means that the Harris feature of a 3D point in one filtered 3D point region differs from the Harris feature of a 3D point in another filtered 3D point region by less than the Harris feature threshold, or the HOG feature of a 3D point in one filtered 3D point region differs from the HOG feature of a 3D point in another filtered 3D point region by less than the HOG feature threshold.

[0065] Triangulation of the 3D point matching pairs yields spatial image patches;

[0066] The positions within the spatial image patch are optimized to obtain an optimized spatial image patch;

[0067] In this embodiment, "optimization" in the context of optimizing the position in a spatial image block means correcting or fine-tuning the position in the spatial image block.

[0068] Calculate the confidence score of the optimized spatial image patch;

[0069] Based on the confidence level, the optimized spatial image patches are sorted to obtain a spatial image patch queue, where the higher the confidence level, the higher the priority.

[0070] The depth of sparse feature points in the 3D points of each optimized spatial image block in the spatial image block queue is calculated to obtain the depth value of the sparse feature points.

[0071] Based on the depth values ​​of sparse feature points, nonlinear depth optimization is performed on the 3D points in each optimized spatial image block to obtain a queue of spatial image blocks with optimized 3D point depth.

[0072] Determine whether there are sparse feature points in the neighborhood of the central 3D point of each optimized spatial image block in the 3D point depth-optimized spatial image block queue, and whether the difference between the confidence of each central 3D point and the confidence of the 3D points in the neighborhood is greater than the confidence threshold. If both are true, the 3D point depth-optimized spatial image block queue is a completed spatial image block priority queue. If not, find the depth value of the next sparse feature point and continue nonlinear depth optimization until the judgment condition is met.

[0073] In this embodiment, the judgment condition is: there are sparse feature points in the neighborhood of the central 3D point of each optimized spatial image block in the spatial image block queue after 3D point depth optimization, and the difference between the confidence of each central 3D point and the confidence of the 3D points in the neighborhood is greater than the confidence threshold.

[0074] In step S3, another implementation method is as follows:

[0075] Based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain the 3D point region;

[0076] Each 3D point region is filtered to obtain a filtered 3D point region.

[0077] Calculate the distance from each 3D point in the filtered 3D point region to its adjacent 3D points;

[0078] Based on the distance of each 3D point to its neighboring 3D points, find 3D points in different filtered 3D point regions whose distance difference is lower than the distance threshold, and form 3D point matching pairs.

[0079] Based on the 3D point matching pairs, the 3D point matching pairs are triangulated to obtain multiple spatial image patches;

[0080] Calculate the proportion of 3D points of the same type of power environment object in each spatial image patch to the total number of 3D points in the entire single spatial image patch;

[0081] Multiple spatial image patches are sorted according to their proportion to obtain a spatial image patch priority queue, where the higher the proportion, the higher the priority.

[0082] Clustering 3D points within a neighborhood is equivalent to dividing the 3D point cloud data into various regions, resulting in 3D point regions. By calculating the distance of each 3D point to its neighboring 3D points, two 3D points with similar distances are found, and 3D points with similar surrounding environments are identified, forming 3D point matching pairs. Spatial image patches are then built on these 3D point matching pairs to find the range of areas with the same environment. Based on the proportion of 3D points of the same type of power environment objects in a single spatial image patch, the distribution of the same type of power environment objects in an image patch is determined. Image patches with high proportions are then selected to train a neural network, enabling the prediction of the overall object height in the environment, thereby achieving monitoring of the substation environment.

[0083] S4. Select high-priority spatial image blocks from the spatial image block priority queue, train the neural network model, and obtain the trained neural network model.

[0084] In step S4, the neural network model is the PointNet++ neural network model.

[0085] The trained neural network model, resulting in a fully trained neural network model, includes:

[0086] Input high-priority spatial image patches and their corresponding labels into the neural network model to obtain the output of the neural network model for this training.

[0087] Calculate the training loss value based on the output of the neural network model in this training iteration;

[0088] Determine whether the current training loss value is lower than the loss threshold. If yes, the neural network model training is complete. If not, update the weights of the neural network model based on the current training loss value to obtain the updated neural network model.

[0089] The updated neural network model is trained again using new high-priority spatial image patches until the loss value is lower than the loss threshold.

[0090] The formula for calculating the loss value of the current training iteration is as follows:

[0091]

[0092] Where Loss is the training loss value for this iteration, and h is the output height of the neural network model for this iteration. ′ The label height for this training session, th + For positive height difference, th - The height difference is negative.

[0093] In hh′≥th + At that time, through Measuring distance (hh′)2 and ratio The weights between h and h′ are determined by considering the distance (hh′) more as the difference between h and h′ increases. 2 The percentage, determined by distance (hh′) 2 When rapidly adjusting the parameters of the neural network, the ratio of h to h′ should be the primary consideration when the difference between them is small. The weight, through the ratio Fine-tuning the parameters of the neural network to make them more accurately approximate the target value; in the case of h′-h≥th _ hour, When the value is negative, the greater the difference between h and h′, the more important the distance (hh′) should be. 2 The percentage, determined by distance (hh′) 2 When rapidly adjusting the parameters of the neural network, the ratio of h to h′ should be the primary consideration when the difference between them is small. The weight, through the ratio Fine-tuning the parameters of a neural network makes the parameters more accurately approximate the target values.

[0094] exist Greater than 1, or When the value is less than -1, the neural network is initially trained using the cross-entropy loss function. Less than 1 and Between -1 and 1.

[0095] The formula for updating the weights of the neural network model based on the current training loss value is as follows:

[0096]

[0097] Where W is the updated weight, W′ is the original weight, Loss is the training loss value, h is the output height of the neural network model in the current training, and h′ is the label height in the current training.

[0098] When hh′≥0, the weights are adjusted downwards, and through... The ratio of h to h′ measures the difference between the output height and the label height. The larger the value, the faster the rate of decrease; when h′-h≥0, the weights are adjusted upwards. The larger the value, the greater the upward adjustment of the weight. Through bidirectional adjustment, the output height h approaches the label height h′.

[0099] S5. The trained neural network model is used to process the substation environment image to obtain the height of objects in the power environment.

[0100] The technical solution of this invention has at least the following advantages and beneficial effects: This invention obtains substation environment images by taking pictures of the substation environment, obtains three-dimensional point cloud data based on the substation environment images, generates spatial image blocks, constructs a spatial image block priority queue, selects high-priority spatial image blocks from the spatial image block priority queue to represent the current environmental situation, trains a neural network model, and uses the neural network to predict the height of objects in the substation environment, thereby realizing the monitoring of the substation environment.

[0101] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for measuring the height of objects in a power environment based on image data processing, characterized in that, include: The environment surrounding the substation was photographed to obtain images of the substation environment; Multiple substation environment images of the same scene are converted into 3D point cloud data; Based on the 3D point cloud data, spatial image patches are generated, and a priority queue of spatial image patches is constructed; specifically, based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain 3D point regions; Each 3D point region is filtered to obtain a filtered 3D point region. Calculate HOG or Harris features for each filtered 3D point region; Based on the HOG or Harris features of the filtered 3D point region, find the similar HOG or Harris features of another filtered 3D point region along the epipolar line to obtain a 3D point matching pair. Triangulation of the 3D point matching pairs yields spatial image patches; The positions within the spatial image patch are optimized to obtain an optimized spatial image patch; Calculate the confidence score of the optimized spatial image patch; Based on the confidence level, the optimized spatial image patches are sorted to obtain a spatial image patch queue; The depth of sparse feature points in the 3D points of each optimized spatial image block in the spatial image block queue is calculated to obtain the depth value of the sparse feature points. Based on the depth values ​​of sparse feature points, nonlinear depth optimization is performed on the 3D points in each optimized spatial image block to obtain a queue of spatial image blocks with optimized 3D point depth. If the depth of the central 3D point of each optimized spatial image block in the queue of spatial image blocks after 3D point depth optimization is greater than the confidence threshold, then the queue of spatial image blocks after 3D point depth optimization is a completed spatial image block priority queue. Otherwise, the depth value of the next sparse feature point is searched, and nonlinear depth optimization is continued until the judgment condition is met. Select high-priority spatial image patches from the spatial image patch priority queue, train the neural network model, and obtain the trained neural network model; The trained neural network model is used to process the substation environment image to obtain the height of objects in the power environment.

2. The method for measuring the height of objects in a power environment based on image data processing according to claim 1, characterized in that, The conversion of multiple substation environmental images of the same scene into three-dimensional point cloud data includes: Based on each substation environment image of the same scene, extract sparse feature points of pixels; Depth and normal vectors of sparse feature points of pixels are reconstructed to obtain three-dimensional point cloud data.

3. The method for measuring the height of objects in a power environment based on image data processing according to claim 1, characterized in that, The neural network model is the PointNet++ neural network model.

4. The method for measuring the height of objects in a power environment based on image data processing according to claim 1, characterized in that, The trained neural network model, resulting in a fully trained neural network model, includes: Input high-priority spatial image patches and their corresponding labels into the neural network model to obtain the output of the neural network model for this training. Calculate the training loss value based on the output of the neural network model in this training iteration; Determine whether the current training loss value is lower than the loss threshold. If yes, the neural network model training is complete. If not, update the weights of the neural network model based on the current training loss value to obtain the updated neural network model. The updated neural network model is trained again using new, high-priority spatial image patches until the loss value is below the loss threshold.

5. The method for measuring the height of objects in a power environment based on image data processing according to claim 4, characterized in that, The formula for calculating the loss value of the current training iteration is as follows: in, This is the training loss value for this session. The output height of the neural network model during this training session. The label height for this training session. A positive height difference. The height difference is negative.

6. The method for measuring the height of objects in a power environment based on image data processing according to claim 5, characterized in that, The formula for updating the weights of the neural network model based on the current training loss value is as follows: in, For the updated weights, The weights before the update. This is the training loss value for this session. The output height of the neural network model during this training session. This represents the label height for the current training session.

7. A method for measuring the height of objects in a power environment based on image data processing, characterized in that, include: The environment surrounding the substation was photographed to obtain images of the substation environment; Multiple substation environment images of the same scene are converted into 3D point cloud data; Based on the 3D point cloud data, spatial image patches are generated, and a priority queue of spatial image patches is constructed; specifically, based on the location of the 3D point cloud data, the 3D points within the neighborhood are clustered to obtain 3D point regions; Each 3D point region is filtered to obtain a filtered 3D point region. Calculate the distance from each 3D point in the filtered 3D point region to its adjacent 3D points; Based on the distance of each 3D point to its neighboring 3D points, find 3D points in different filtered 3D point regions whose distance difference is lower than the distance threshold, and form 3D point matching pairs. Based on the 3D point matching pairs, the 3D point matching pairs are triangulated to obtain multiple spatial image patches; Calculate the proportion of 3D points of the same type of power environment object in each spatial image patch to the total number of 3D points in the entire single spatial image patch; Multiple spatial image patches are sorted according to their proportion to obtain a spatial image patch priority queue, where the higher the proportion, the higher the priority. Select high-priority spatial image patches from the spatial image patch priority queue, train the neural network model, and obtain the trained neural network model; The trained neural network model is used to process the substation environment image to obtain the height of objects in the power environment.

8. The method for measuring the height of objects in a power environment based on image data processing according to claim 7, characterized in that, The conversion of multiple substation environmental images of the same scene into three-dimensional point cloud data includes: Based on each substation environment image of the same scene, extract sparse feature points of pixels; Depth and normal vectors of sparse feature points of pixels are reconstructed to obtain three-dimensional point cloud data.

9. The method for measuring the height of objects in a power environment based on image data processing according to claim 7, characterized in that, The neural network model is the PointNet++ neural network model.

10. The method for measuring the height of objects in a power environment based on image data processing according to claim 7, characterized in that, The trained neural network model, resulting in a fully trained neural network model, includes: Input high-priority spatial image patches and their corresponding labels into the neural network model to obtain the output of the neural network model for this training. Calculate the training loss value based on the output of the neural network model in this training iteration; Determine whether the current training loss value is lower than the loss threshold. If yes, the neural network model training is complete. If not, update the weights of the neural network model based on the current training loss value to obtain the updated neural network model. The updated neural network model is trained again using new, high-priority spatial image patches until the loss value is below the loss threshold.

11. The method for measuring the height of objects in a power environment based on image data processing according to claim 10, characterized in that, The formula for calculating the loss value of the current training iteration is as follows: in, This is the training loss value for this session. The output height of the neural network model during this training session. The label height for this training session. A positive height difference. The height difference is negative.

12. The method for measuring the height of objects in a power environment based on image data processing according to claim 11, characterized in that, The formula for updating the weights of the neural network model based on the current training loss value is as follows: in, For the updated weights, The weights before the update. This is the training loss value for this session. The output height of the neural network model during this training session. This represents the label height for the current training session.

Images