Laser aiming and safety warning method of laser obstacle clearing instrument based on visual method

Abstract

The application relates to a laser aiming and safety warning method of a laser obstacle clearing instrument based on a visual method, and belongs to the technical field of laser obstacle clearing. The method first constructs a YOLOv7 floating object and personnel and animal detection model; then an unmanned aerial vehicle is hung with an edge intelligent computing terminal and a laser obstacle clearing instrument to perform line inspection flight, the edge intelligent computing terminal acquires video stream data collected by a camera of the unmanned aerial vehicle in real time, and real-time detection is performed by using the YOLOv7 floating object and personnel and animal detection model; after floating objects are detected, the edge intelligent computing terminal performs calculation and controls a holder to rotate corresponding horizontal and vertical angles alpha and beta; then detection is performed by using the detection model, if no personnel or animals are detected, the edge intelligent computing terminal sends a signal for starting the laser obstacle clearing instrument once, and the laser obstacle clearing instrument clears the detected floating objects. The application greatly improves the inspection efficiency, saves manpower, and reduces the risks faced by workers in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of laser obstacle removal technology, specifically relating to a laser aiming and safety warning method for a vision-based laser obstacle removal device. Background Technology

[0002] The laser obstacle removal device mainly consists of a light source unit, an obstacle removal transmitter, a motorized pan-tilt unit, and a controller. The collimated laser beam strikes the surface of the obstacle. Upon absorbing the laser energy, the obstacle's temperature rises rapidly, reaching or exceeding its melting point, ignition point, or even boiling point, causing it to melt, burn, or vaporize and eventually fall off, thus removing the obstacle. This laser-based method allows for remote, rapid, and precise removal of obstacles without damaging the equipment. It is safe, fast, and can be used while the power is on. The laser obstacle removal device overcomes the drawbacks of traditional manual methods, such as long operating times and high safety risks during power outages or live-line work, significantly improving removal efficiency.

[0003] Traditional laser obstacle clearing devices rely on manual aiming, using the same principles as traditional firearms aiming, and are highly dependent on manual aiming and tracking. For example, CN 105242392 B describes an aiming device and method for remote laser removal of foreign objects from overhead power lines; when there are people or animals in the direction of the target, manual observation is the primary method, with emergency shutdown and safety precautions implemented to avoid danger.

[0004] Manual aiming using traditional firearm aiming principles is inefficient, requiring high-powered telescopes and other equipment for manual aiming, which is time-consuming and labor-intensive. At greater distances, it's easy to miss floating objects or other foreign objects. When high-voltage power towers are nearby, the safety of on-site workers is highly dangerous. Manually determining whether people or animals are within the target's field of vision is a limited and simplistic method, unable to provide effective and rapid response to hazard levels, making it quite dangerous. Therefore, overcoming the shortcomings of existing technologies is a pressing issue that needs to be addressed in the field of laser obstacle removal technology. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies and provide a laser aiming and safety warning method for laser obstacle clearing devices based on a vision approach. This method utilizes the YOLOv7 algorithm for automatic aiming, tracking, and emergency safety protection of the laser obstacle clearing device, overcoming the problems of traditional methods relying on manual labor and having low detection efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A vision-based laser obstacle clearance device laser aiming and safety warning method includes the following steps:

[0008] Step (1): Collect visible light images of floating objects on the power tower and visible light images of people and animals. Label the categories and locations of these targets, such as floating objects on the power tower, people and animals, to build a dataset. Divide the constructed dataset into a training set and a test set.

[0009] Step (2): Using the training set, with visible light images as input and the corresponding category and location as output, the YOLOv7 model is trained; after training, the test set is used for testing to obtain the YOLOv7 floating object and human / animal detection model.

[0010] Step (3): The UAV is equipped with an edge intelligent computing terminal and a laser obstacle clearing device to conduct line inspection flight. The edge intelligent computing terminal acquires the video stream data collected by the UAV camera in real time and uses the YOLOv7 floating object and human and animal detection model obtained in step (2) for real-time detection.

[0011] Step (4): After detecting the floating object, the edge intelligent computing terminal detects the position (X,Y) of the center point of the floating object within the camera's field of view; then calculates the offsets Δx and Δy between (X,Y) and the center (x,y) of the image; and then converts Δx and Δy into offset angles α and β in the horizontal and vertical directions through coordinate transformation.

[0012] Step (5): The edge intelligent computing terminal controls the gimbal to rotate at the corresponding horizontal angle α and vertical angle β. Then, the edge intelligent computing terminal acquires the video stream data collected by the drone camera in real time and uses the YOLOv7 floating object and human / animal detection model obtained in step (2) for real-time detection. If no human or animal is detected, the edge intelligent computing terminal sends a signal to start the laser obstacle clearing device once, and the laser obstacle clearing device clears the detected floating objects.

[0013] In step (6), the edge intelligent computing terminal continues to track the detected floating object and determines whether the floating object has been cleared. If it has been cleared, the obstacle clearing task is completed and the UAV flies to the next target waypoint.

[0014] Furthermore, preferably, in step (1), a drone is used to collect visible light images; the floating objects of the power tower include kites, plastic bags, and advertising banners; the animals include birds; and the ratio of the training set to the test set is 8:2.

[0015] Furthermore, preferably, in step (2), the training process is repeated, and the test set loss value decreases until it remains stable, at which point the YOLOv7 model is successfully trained.

[0016] Furthermore, preferably, in step (4), Δx = xX; Δy = yY.

[0017] Furthermore, preferably, in step (4), α = tan -1 (Δx / f); β=tan -1 (Δy / f); where f is the focal length.

[0018] Furthermore, preferably, in step (5), if a person and / or animal is detected, the edge intelligent computing terminal sends a signal to control the drone to adjust its position, return to step (3), re-detect the floating object, and repeat steps (4) and (5).

[0019] Furthermore, preferably, in step (6), if the floating object is not completely removed, then in step (3), the floating object is re-detected for not being removed, and steps (4) to (6) are repeated.

[0020] This invention does not modify the structure of existing laser obstacle removal devices. Typically, laser obstacle removal devices mainly consist of modules such as a light source host, a foreign object removal transmitter, an electric pan-tilt unit, and a controller.

[0021] With the development of drone inspection technology, drones are carrying more and more weight and sensors, and their endurance is also getting stronger. This invention uses an airborne laser obstacle clearing device to handle foreign objects. By using the YOLOv7 algorithm, it can detect targets and automatically aim and track floating objects. It also realizes emergency safety protection functions when people or animals appear in the target's field of view.

[0022] This invention uses the current state-of-the-art object detection algorithm YOLOv7, which is already quite mature and convenient to implement and use, and has the best speed and accuracy. This not only improves the precision and recall of object detection, but also saves time in model development, leaving enough time for engineers to process important logic in the subsequent stages.

[0023] The main advantages of the YOLOv7 model are as follows:

[0024] The Effective Layer Aggregation Network (ELAN) algorithm, by controlling the shortest and longest gradient paths, enables deeper models to learn and converge more effectively. However, the ELAN model reaches an equilibrium with appropriate gradient paths and the number of computational modules. Adding more computational modules disrupts this equilibrium, leading to decreased parameter utilization. YOLOv7, building upon this structure, proposed Extended-ELAN (E-ELAN). E-ELAN employs a similar feature aggregation and feature transfer process to ELAN, but incorporates ShuffleNet-like grouped convolutions, dilation modules, and shuffling modules within the computational modules. Features are then fused through an aggregation module. This approach yields more diverse features while improving parameter computation and utilization efficiency.

[0025] The main purpose of model scaling is to obtain a model structure that meets different inference speed requirements by adjusting the scaling parameters of the base model. The composite model scaling method, after modifying the depth factor, calculates the proportion of width change corresponding to the transition layer, and then applies the same width scaling. In other words, the width scaling factor is calculated based on the model change caused by the depth scaling factor. The advantage of this approach is that it maintains the original model design and preserves the optimal structural model.

[0026] Compared with the prior art, the beneficial effects of this invention are as follows:

[0027] This invention utilizes UAV inspection images and the advanced YOLOv7 target detection algorithm to detect floating objects, people, and animals, significantly improving inspection efficiency, saving manpower, and reducing the risks faced by workers in complex environments. The YOLOv7 algorithm boasts high accuracy, ensuring precise detection.

[0028] Specifically, it can be divided into the following aspects:

[0029] 1. This invention presents a method for using an automatic positioning, aiming, and tracking floating object clearing device based on deep learning. The method utilizes the detection results of a model and rigorous logical judgment to analyze the floating object situation, achieving automatic target angle determination for clearing obstacles and ensuring the safety of personnel and animals, significantly reducing the time cost and safety risks for maintenance personnel.

[0030] 2. It adopts the advanced target detection algorithm YOLOv7, which overcomes the problems of single features and difficult design in traditional digital image processing. It can easily expand the categories detected by the model, and has high scalability and high accuracy.

[0031] 3. Photogrammetric techniques and methods were used to determine and calculate the gimbal rotation angle, and combined with deep learning target detection methods, automated control of the laser obstacle clearing device was achieved. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the laser aiming and safety warning process of the laser obstacle clearing device based on the vision method of the present invention.

[0033] Figure 2 This is a schematic diagram of the offset angle;

[0034] Figure 3 This is a structural diagram of the ELAN module;

[0035] Figure 4 This is a structural diagram of the MP-1 module;

[0036] Figure 5 Here is a structural diagram of the ELAN-W module;

[0037] Figure 6 Here is a diagram of the SPPCS module structure;

[0038] Figure 7 This is a diagram of the network structure of the YOLOv7 model. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the embodiments.

[0040] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0041] like Figure 1 As shown, the laser aiming and safety warning method for a vision-based laser obstacle clearing device is characterized by the following steps:

[0042] Step (1): Collect visible light images of floating objects on the power tower and visible light images of people and animals. Label the categories and locations of these targets, such as floating objects on the power tower, people and animals, to build a dataset. Divide the constructed dataset into a training set and a test set.

[0043] Step (2): Using the training set, with visible light images as input and the corresponding category and location as output, the YOLOv7 model is trained; after training, the test set is used for testing to obtain the YOLOv7 floating object and human / animal detection model.

[0044] Step (3): The UAV is equipped with an edge intelligent computing terminal and a laser obstacle clearing device to conduct line inspection flight. The edge intelligent computing terminal acquires the video stream data collected by the UAV camera in real time and uses the YOLOv7 floating object and human and animal detection model obtained in step (2) for real-time detection.

[0045] Step (4): After detecting the floating object, the edge intelligent computing terminal detects the position (X,Y) of the center point of the floating object within the camera's field of view; then calculates the offsets Δx and Δy between (X,Y) and the center (x,y) of the image; and then converts Δx and Δy into offset angles α and β in the horizontal and vertical directions through coordinate transformation.

[0046] Step (5): The edge intelligent computing terminal controls the gimbal to rotate at the corresponding horizontal angle α and vertical angle β. Then, the edge intelligent computing terminal acquires the video stream data collected by the drone camera in real time and uses the YOLOv7 floating object and human / animal detection model obtained in step (2) for real-time detection. If no human or animal is detected, the edge intelligent computing terminal sends a signal to start the laser obstacle clearing device once, and the laser obstacle clearing device clears the detected floating objects.

[0047] In step (6), the edge intelligent computing terminal continues to track the detected floating object and determines whether the floating object has been cleared. If it has been cleared, the obstacle clearing task is completed and the UAV flies to the next target waypoint.

[0048] In step (1), visible light images are collected using drones; the floating objects on the power tower include kites, plastic bags, and advertising banners; the animals include birds; and the ratio of the training set to the test set is 8:2.

[0049] In step (2), the training process is repeated, and the test set loss value decreases until it stabilizes. At this point, the YOLOv7 model has been successfully trained. Specifically, the training process is repeated, and the hyperparameters are adjusted to improve the accuracy on the test set. Both the training and test set losses decrease until they stabilize, thus achieving the set iteration goal. Based on experience, a stable output loss value indicates that the algorithm has converged stably.

[0050] In step (4), Δx = xX; Δy = yY.

[0051] In step (4), α = tan -1 (Δx / f); β=tan -1 (Δx / f); where f is the focal length.

[0052] In step (5), if a person and / or animal is detected, the edge intelligent computing terminal sends a signal to control the drone to adjust its position, return to step (3), re-detect the floating object, and repeat steps (4) and (5).

[0053] In step (6), if the floating object is not completely removed, then in step (3), the floating object is re-detected for not being removed, and steps (4) to (6) are repeated.

[0054] This invention uses a test set to test the trained model and evaluates the model using recall and precision (when the trained model achieves a recall and precision of over 90% on the open sample dataset, the model is considered successfully built; otherwise, it returns to iterative training). If the model does not meet the requirements after evaluation, the dataset is expanded (i.e., visible light images of floating objects on power towers, as well as visible light images of people and animals are collected again, and the categories and locations of these targets are labeled), and training is performed again.

[0055] like Figure 2 As shown, S is the center of the photograph, f is the focal length, S'(x,y) is the center point of the photograph, a(X,Y) is the principal point of the image, A is the principal point of the ground (i.e. the actual ground object target point), and Δx,Δy are the offsets of X,Y from the center x,y of the photograph.

[0056] The formulas for calculating the horizontal angle α and the vertical angle β are:

[0057] α = tan -1 (Δx / f);

[0058] β = tan -1 (Δy / f).

[0059] Application Examples

[0060] A vision-based laser obstacle clearance device laser aiming and safety warning method includes the following steps:

[0061] Step (1): Collect visible light images of floating objects on the power tower and visible light images of people and animals. Using annotation tools, collect visible light images of floating objects (kites, kite strings, plastic sheets, advertising cloths, etc.) on the power tower and visible light images of people and animals (birds) collected by the drone. Save the images as XML files, which contain category information and location information. Divide the dataset into 80% training set A and 20% test set B.

[0062] Step (2): Using the training set, with visible light images as input and the corresponding category and location as output, the YOLOv7 model is trained; after training, the test set is used for testing to obtain the YOLOv7 floating object and human / animal detection model.

[0063] The YOLOv7 model is iteratively trained using the training data in datasets A and B. The training process is repeated, and the hyperparameters are adjusted to improve the accuracy of the validation set and reduce the loss of both the training and validation sets until they stabilize, thus achieving the set iterative goal.

[0064] The backbone network of the YOLOv7 model consists of ELAN and MP-1 modules. It extracts image features through a combination of various convolution operations with kernel sizes of 3×3 and 1×1, activation functions, max pooling operations, and normalization operations.

[0065] The ELAN module is a highly efficient feature extraction module, and its module structure is as follows: Figure 3 As shown, the system consists of two interconnected branches: a short branch and a long branch. The short branch extracts shallow features, while the long branch extracts deep features. Shallow features retain the texture and appearance information of the feature map, while deep features contain abstract semantic information. The ELAN module then concatenates the feature maps obtained from both branches to obtain a feature map that simultaneously contains both shallow and deep feature information. The CBS modules on each branch work together to complete the extraction of image features. The short branch consists of one CBS module, and the long branch consists of five CBS modules. Each CBS module consists of a convolutional layer (Conv), a batch normalization layer (Batch Normalization), and an activation function (Silu).

[0066] The MP-1 module also consists of two branches, but unlike the ELAN module, the two branches of the MP-1 are of equal length (e.g., ...). Figure 4 (As shown). The upper branch consists of a max pooling layer (MaxPool) and a CBS module, while the lower branch consists of two CBS modules. The purpose of using the MaxPool layer is to downsample and obtain key feature information from the feature map while reducing parameters. The role of the MP-1 module is to minimize feature information loss as much as possible during the downsampling process of the feature map, while reducing parameters.

[0067] The detection part of the YOLOv7 model mainly consists of the ELAN-W module, MP-2 module, and SPPCSPC module, which predict the features maps processed by the backbone network. The detection part has three detection heads, one for large, one for medium, and one for small targets in the image. This is because large feature maps retain more spatial information, including features of small targets. Therefore, large feature maps in large detection heads are typically used to detect small targets, while small feature maps in small detection heads, having undergone several convolutions, have more abstract information and are more suitable for detecting large targets.

[0068] The ELAN-W module is structurally very similar to the ELAN module, employing a similar number of CBS modules. However, a slight difference is that the ELAN-W module places greater emphasis on the fusion of shallow and deep features, such as... Figure 5 As shown. The structure of the MP-2 module is exactly the same as that of the MP-1 module, and the number of channels in the feature map obtained by the MP-2 module is twice that of the MP-1 module.

[0069] The SPPC module is responsible for aggregating feature maps with different receptive fields by performing max pooling operations at different scales. This allows the network model to adapt to images of different resolutions without increasing the network's parameters, thus ensuring the network's lightweight nature. Figure 6 As shown.

[0070] After training, the results were tested using a test set. The performance of the training on the test set for pedestrian, animal, and floating object targets is shown in Table 1.

[0071] Table 1

[0072] Detection target P accuracy R Recall Rate mAP@0.5 pedestrians, animals 0.942 0.924 0.919 floating objects 0.987 0.964 0.983

[0073] Step (3): The UAV is equipped with an edge intelligent computing terminal and a laser obstacle clearing device to conduct line inspection flight. The edge intelligent computing terminal acquires the video stream data collected by the UAV camera in real time and uses the YOLOv7 floating object and human and animal detection model obtained in step (2) for real-time detection.

[0074] Step (4): After detecting the floating object, the edge intelligent computing terminal detects the position (X,Y) of the center point of the floating object within the camera's field of view; then calculates the offsets Δx and Δy between (X,Y) and the center (x,y) of the image; and then converts Δx and Δy into offset angles α and β in the horizontal and vertical directions through coordinate transformation.

[0075] Step (5): The edge intelligent computing terminal controls the gimbal to rotate at the corresponding horizontal angle α and vertical angle β. Then, the edge intelligent computing terminal acquires the video stream data collected by the drone camera in real time and uses the YOLOv7 floating object and human / animal detection model obtained in step (2) for real-time detection. If no human or animal is detected, the edge intelligent computing terminal sends a signal to start the laser obstacle clearing device once, and the laser obstacle clearing device clears the detected floating objects.

[0076] In step (6), the edge intelligent computing terminal continues to track the detected floating object and determines whether the floating object has been cleared. If it has been cleared, the obstacle clearing task is completed and the UAV flies to the next target waypoint.

[0077] Using the COCO dataset, the YOLOv7 model of this invention is compared with existing YOLOv5, SSD, and YOLOv4 models. The results are shown in Table 2.

[0078] Table 2

[0079] Model COCO dataset performance (mAP50) YOLOv7 0.697 YOLOv5 0.641 SSD 0.288 YOLOv4 0.628

[0080] The laser aiming and safety warning method of this invention, based on a vision-based approach, is used for detecting floating objects, people, and animals. This significantly improves patrol efficiency, saves manpower, and reduces the risks faced by workers in complex environments. The YOLOv7 algorithm has high accuracy, ensuring the precision of the detection.

[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A laser aiming and safety warning method for a vision-based laser obstacle clearing device, characterized in that, Includes the following steps: Step (1): Collect visible light images of floating objects on the power tower and visible light images of people and animals. Label the categories and locations of these targets, such as floating objects on the power tower, people and animals, to build a dataset. Divide the constructed dataset into a training set and a test set. Step (2): Using the training set, with visible light images as input and the corresponding category and location as output, the YOLOv7 model is trained; after training, the test set is used for testing to obtain the YOLOv7 floating object and human / animal detection model. Step (3): The UAV is equipped with an edge intelligent computing terminal and a laser obstacle clearing device to conduct line inspection flight. The edge intelligent computing terminal acquires the video stream data collected by the UAV camera in real time and uses the YOLOv7 floating object and human and animal detection model obtained in step (2) for real-time detection. Step (4): After detecting the floating object, the edge intelligent computing terminal detects the position (X,Y) of the center point of the floating object within the camera's field of view; then calculates the offsets Δx and Δy between (X,Y) and the center (x,y) of the image; and then converts Δx and Δy into offset angles α and β in the horizontal and vertical directions through coordinate transformation. Step (5): The edge intelligent computing terminal controls the gimbal to rotate at the corresponding horizontal angle α and vertical angle β. Then, the edge intelligent computing terminal acquires the video stream data collected by the drone camera in real time and uses the YOLOv7 floating object and human / animal detection model obtained in step (2) for real-time detection. If no human or animal is detected, the edge intelligent computing terminal sends a signal to start the laser obstacle clearing device once, and the laser obstacle clearing device clears the detected floating objects. In step (6), the edge intelligent computing terminal continues to track the detected floating object and determines whether the floating object has been cleared. If it has been cleared, the obstacle clearing task is completed and the UAV flies to the next target waypoint.

2. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (1), visible light images are collected using drones; the floating objects on the power tower include kites, plastic bags, and advertising banners; the animals include birds; the ratio of the training set to the test set is 8:

2.

3. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (2), the training process is repeated, and the test set loss value decreases until it remains stable. At this point, the YOLOv7 model is successfully trained.

4. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (4), Δx = xX; Δy = yY.

5. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (4), α = tan -1 (Δx / f); β=tan -1 (Δy / f); where f is the focal length.

6. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (5), if a person and / or animal is detected, the edge intelligent computing terminal sends a signal to control the drone to adjust its position, return to step (3), re-detect the floating object, and repeat steps (4) and (5).

7. The laser aiming and safety warning method for a vision-based laser obstacle clearing device according to claim 1, characterized in that, In step (6), if the floating object is not completely removed, then in step (3), the floating object is re-detected for not being removed, and steps (4) to (6) are repeated.

Images