Patrol control method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a method and device for inspection control, an electronic device and a computer readable storage medium. The method comprises: acquiring a first image collected by a flight device at a specified image collection position; identifying whether an inspection object exists in the first image through a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses a DW convolution, a 3*3 convolution kernel and a structured pruning method; if the first image has the inspection object, determining image collection control information of the flight device on the inspection object based on the specified image collection position and a position of the inspection object; and sending an image collection instruction to the flight device based on the image collection control information, so that the flight device collects a second image with the inspection object based on the image collection instruction. The technical solution reduces the labor cost and environmental factor interference in the inspection object identification process, improves the identification efficiency of the inspection object, and helps the smooth progress of power inspection.

Description

[Technical Field]

[0001] This application relates to the field of power inspection technology, and in particular to an inspection control method and device, electronic equipment and computer-readable storage medium. [Background Technology]

[0002] With the nationwide expansion of power grids, drones are poised for significant growth in the field of power line inspection. Currently, power line inspections typically involve manual operation of drones to precisely target the inspection object and collect images for testing. However, this method is labor-intensive, costly, and inefficient.

[0003] In addition, related technologies can also use drones to automatically collect images at designated locations. However, because the movement of drones is greatly constrained by the environment, the automatically collected inspection images often have a lot of redundancy, such as the image content deviating from the inspection target or the inspection target being obscured by obstacles. Therefore, manual screening of the inspection images collected by drones is still required to reduce data redundancy, which consumes a lot of manpower, is costly, and inefficient.

[0004] Therefore, how to enable drones to conduct adaptive inspections without relying on human intervention and the environment has become a pressing technical problem that needs to be solved. [Summary of the Invention]

[0005] This application provides an inspection control method and apparatus, electronic device and computer-readable storage medium, aiming to solve the technical problem that the reliability of UAV inspection is seriously constrained by labor costs and environmental factors in related technologies.

[0006] In a first aspect, embodiments of this application provide an inspection control method, comprising: acquiring a first image captured by a flight device at a designated image acquisition location; identifying whether there is an inspection object in the first image using a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, a 3*3 convolution kernel, and a structured pruning method; if the inspection object is present in the first image, determining image acquisition control information of the flight device for the inspection object based on the designated image acquisition location and the location of the inspection object; and sending an image acquisition command to the flight device based on the image acquisition control information, so that the flight device can acquire a second image containing the inspection object based on the image acquisition command.

[0007] In one embodiment of this application, optionally, the process of identifying whether there is an inspection object in the first image using a preset yolov4-tiny model includes: calculating the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model; determining, among all the 3*3 convolutional kernels in the preset yolov4-tiny model, a target convolutional kernel whose confidence level satisfies a specified pruning condition; and deleting the target convolutional kernel and the feature map corresponding to the target convolutional kernel in each convolutional layer using the structured pruning method.

[0008] In one embodiment of this application, optionally, calculating the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model includes: obtaining the weights of each 3*3 convolutional kernel in the preset yolov4-tiny model; determining the confidence level of the 3*3 convolutional kernel based on the weights and weight adjustment coefficients, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolutional kernel and the confidence level.

[0009] In one embodiment of this application, optionally, before determining the credibility of the 3*3 convolutional kernel based on the weights and weight adjustment coefficients, the method further includes: determining a weight matrix based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model, wherein the element in the nth row and mth column of the weight matrix is ​​the weight product of the mth 3*3 convolutional kernel and the (m-1)th 3*3 convolutional kernel in the nth convolutional layer of the preset yolov4-tiny model; for each 3*3 convolutional kernel in the preset yolov4-tiny model, determining the target element corresponding to the 3*3 convolutional kernel in the weight matrix; obtaining the mean of the target element and at least one adjacent element of the target element; and determining the difference between the weight of the 3*3 convolutional kernel and the mean as the weight adjustment coefficient of the 3*3 convolutional kernel.

[0010] In one embodiment of this application, optionally, before acquiring the first image acquired by the flight device at the specified image acquisition location, the method further includes: setting the specified pruning conditions, wherein the specified pruning conditions include: the confidence level is lower than a specified confidence level threshold; and / or after sorting the confidence levels of all the 3*3 convolution kernels in ascending order, the confidence level is located in the first specified ranking position.

[0011] In one embodiment of this application, optionally, determining the image acquisition control information of the flight device for the inspected object based on the specified image acquisition location and the location of the inspected object includes: determining shooting deviation information based on a preset image of the inspected object and the first image, wherein the shooting deviation information includes: shooting angle deviation value, sharpness deviation value, and center point deviation distance of the inspected object; and determining the image acquisition control information of the flight device for the inspected object based on the shooting deviation information, wherein the image acquisition control information includes the updated position of the specified image acquisition location, the shooting angle of the shooting module of the flight device for the inspected object, and focus and zoom information.

[0012] Secondly, embodiments of this application provide an inspection control device, comprising: a first image acquisition unit, configured to acquire a first image acquired by a flight device at a designated image acquisition location; an inspection object identification unit, configured to identify whether there is an inspection object in the first image using a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, a 3*3 convolution kernel, and a structured pruning method; a control information generation unit, configured to, if the first image contains the inspection object, determine image acquisition control information of the flight device for the inspection object based on the designated image acquisition location and the location of the inspection object; and a second image acquisition unit, configured to send an image acquisition command to the flight device based on the image acquisition control information, so that the flight device can acquire a second image containing the inspection object based on the image acquisition command.

[0013] In one embodiment of this application, optionally, the inspection object identification unit includes: a confidence calculation unit, used to calculate the confidence of each 3*3 convolutional kernel in the preset yolov4-tiny model; a pruning judgment unit, used to determine, among all the 3*3 convolutional kernels in the preset yolov4-tiny model, a target convolutional kernel whose confidence satisfies a specified pruning condition; and a pruning processing unit, used to delete the target convolutional kernel and the feature map corresponding to the target convolutional kernel in each convolutional layer through the structured pruning method.

[0014] In one embodiment of this application, optionally, the credibility calculation unit is used to: obtain the weights of each 3*3 convolutional kernel in the preset yolov4-tiny model; and determine the credibility of the 3*3 convolutional kernel based on the weights and weight adjustment coefficients, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolutional kernel and the credibility.

[0015] In one embodiment of this application, optionally, the inspection control device further includes: a weight matrix determination unit, configured to determine a weight matrix based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model before determining the confidence level of the 3*3 convolutional kernel, wherein the element in the nth row and mth column of the weight matrix is ​​the weight product of the mth 3*3 convolutional kernel and the (m-1)th 3*3 convolutional kernel in the nth convolutional layer of the preset yolov4-tiny model; a target element determination unit, configured to determine the target element corresponding to each 3*3 convolutional kernel in the weight matrix for each 3*3 convolutional kernel in the preset yolov4-tiny model; a mean acquisition unit, configured to acquire the mean of the target element and at least one adjacent element of the target element; and a weight adjustment coefficient acquisition unit, configured to determine the difference between the weight of the 3*3 convolutional kernel and the mean as the weight adjustment coefficient of the 3*3 convolutional kernel.

[0016] Optionally, in one embodiment of this application, the inspection control device further includes: a pruning condition setting unit, used to set the specified pruning conditions before acquiring the first image acquired by the flight device at the specified image acquisition location, wherein the specified pruning conditions include: the confidence level is lower than a specified confidence level threshold; and / or after sorting the confidence levels of all the 3*3 convolution kernels in ascending order, the confidence level is located in the first specified ranking position.

[0017] In one embodiment of this application, optionally, the second image acquisition unit includes: a first information determination unit, configured to determine shooting deviation information based on a preset image of the inspection object and the first image, wherein the shooting deviation information includes: a shooting angle deviation value, a sharpness deviation value, and a deviation distance of the center point of the inspection object; and a second information determination unit, configured to determine image acquisition control information of the flight device for the inspection object based on the shooting deviation information, wherein the image acquisition control information includes the update position of the specified image acquisition position, the shooting angle of the shooting module of the flight device for the inspection object, and focus and zoom information.

[0018] Thirdly, embodiments of this application provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in the first aspect above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the method described in the first aspect above.

[0020] The above technical solution addresses the technical problem that the reliability of UAV inspection is severely constrained by labor costs and environmental factors. First, it acquires the first image captured by the flight device at the designated image acquisition location.

[0021] The flight device can be a drone, or any inspection device other than a drone that is capable of autonomous driving and has image acquisition capabilities. The designated image acquisition locations are pre-set image acquisition points for the flight device, where images of the inspected objects are expected to be captured. Optionally, the flight device follows a predetermined flight path with one or more designated image acquisition locations along this path. When the flight device reaches a designated image acquisition location, it automatically captures a first image.

[0022] Next, the presence of an inspection object in the first image is identified using a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, 3*3 convolution kernel, and structured pruning.

[0023] The YOLOv4-tiny model boasts advantages such as high detection efficiency and high prediction accuracy. Employing the YOLOv4-tiny model allows for direct control of the flight device to automatically capture images and effectively identify inspection targets, reducing human involvement in both image capture and target identification. Furthermore, when the images acquired by the flight device are affected by environmental factors, such as when inspection targets are obscured by obstacles, the YOLOv4-tiny model can effectively identify the absence of valid inspection targets in the image, facilitating the removal of redundant data.

[0024] The default yolov4-tiny model uses DW convolution, 3*3 convolution kernels, and structured pruning, which can further reduce weight and improve the efficiency of object recognition during the inspection process.

[0025] Furthermore, if the inspection object is present in the first image, the image acquisition control information of the flight device for the inspection object is determined based on the specified image acquisition location and the location of the inspection object.

[0026] Specifically, if the first image contains the inspection object, the relevant attribute information of the flight device for image acquisition can be adjusted based on the specified image acquisition location and the location of the inspection object, so that the flight device can acquire images of the inspection object more accurately and effectively after adjustment.

[0027] Finally, based on the obtained image acquisition control information, an image acquisition command is generated for the flight device. Based on this image acquisition command, the flight device performs a second image capture according to the image acquisition control information, thereby acquiring a second image containing the inspected object.

[0028] The above technical solution introduces a preset yolov4-tiny model using DW convolution, 3*3 convolution kernel, and structured pruning to identify inspection objects in images acquired by the flight device. While accurately identifying inspection objects, it reduces the labor costs and environmental interference in the inspection object identification process, improves the identification efficiency of inspection objects, and helps to ensure the smooth progress of power inspection. [Attached Image Description]

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart of an inspection control method according to an embodiment of this application is shown;

[0031] Figure 2 A schematic diagram of traditional convolution in related technologies is shown;

[0032] Figure 3 A schematic diagram of DW convolution according to an embodiment of this application is shown;

[0033] Figure 4 A schematic diagram of structured pruning according to an embodiment of this application is shown;

[0034] Figure 5 A block diagram of an inspection control device according to an embodiment of this application is shown;

[0035] Figure 6 A flowchart of an inspection control method according to another embodiment of this application is shown;

[0036] Figure 7 A block diagram of an inspection control device according to another embodiment of this application is shown;

[0037] Figure 8 A block diagram of an electronic device according to an embodiment of this application is shown.

Detailed Implementation Methods

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Figure 1 A flowchart of an inspection control method according to an embodiment of this application is shown.

[0040] like Figure 1 As shown, an inspection control method according to an embodiment of this application includes:

[0041] Step 102: Acquire the first image captured by the flight device at the designated image acquisition location.

[0042] The flight device can be a drone, or any inspection device other than a drone that is capable of autonomous driving and has image acquisition capabilities.

[0043] The designated image acquisition location is a pre-set image acquisition point for the flight device, where it is expected to capture images of the inspected object. Optionally, the flight device has a predetermined flight path during flight, along which there are one or more designated image acquisition locations. When the flight device reaches a designated image acquisition location, it will automatically capture a first image.

[0044] Step 104: Identify whether there is an inspection object in the first image using a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, 3*3 convolution kernel and structured pruning.

[0045] In one possible design, this application may install an edge terminal on the flight device, and set the yolov4-tiny model in the edge terminal, through which the edge terminal performs real-time data processing for the flight device.

[0046] In another possible design, this application can directly set up the yolov4-tiny model in the flight device to perform real-time data processing for the flight device.

[0047] After the flight device automatically captures the first image, the flight device or edge terminal can identify whether there is an inspection object in the first image through a preset yolov4-tiny model.

[0048] To address the technical problem that the reliability of UAV inspections is severely constrained by labor costs and environmental factors, this application employs the lightweight YOLOv4-tiny model to identify whether inspection objects are present in images captured by the flight device. The YOLOv4-tiny model offers advantages such as high detection efficiency and high prediction accuracy. Using the YOLOv4-tiny model allows for direct control of the flight device to automatically take pictures and effectively identify inspection objects, reducing human involvement in both the flight device's image capture and object identification processes.

[0049] Furthermore, when the images acquired by the flight device are affected by the environment, such as when the inspection object in the image is obscured by obstacles, the yolov4-tiny model can effectively identify that there is no valid inspection object in the image at this time, which facilitates the removal of redundant data.

[0050] In one possible design, the preset yolov4-tiny model uses DW convolution.

[0051] like Figure 2 As shown, in traditional convolution, each convolution kernel operates on each channel of the input image simultaneously.

[0052] like Figure 3 As shown, in DW convolution, one convolutional kernel is responsible for one channel, and one channel is convolved by only one convolutional kernel. In other words, in DW convolution, the number of groups equals the number of convolutional channels; it performs channel-by-channel convolution, with one convolutional kernel responsible for one channel. (Comparison) Figure 2 and Figure 3 It can be seen that using DW convolution can reduce the amount of computation by several times compared with traditional convolution, thereby achieving the goal of reducing the number of model parameters and improving the computation speed.

[0053] Therefore, the default yolov4-tiny model uses DW convolution, which can further reduce weight and improve the efficiency of object recognition during the inspection process.

[0054] In one possible design, the preset yolov4-tiny model uses a 3x3 convolutional kernel.

[0055] The smaller the convolution kernel, the fewer parameters are required for convolution, resulting in lower model complexity and faster computational efficiency. Therefore, the preset yolov4-tiny model uses a smaller 3*3 convolution kernel, which can further reduce weight and improve the efficiency of object recognition during the inspection process.

[0056] In one possible design, the preset yolov4-tiny model uses a structured pruning method.

[0057] like Figure 4As shown, structured pruning can be performed using a filter-wise approach. Specifically, in the i-th convolutional layer, the 2nd and 5th convolutional kernels are pruned. This reduces the number of kernels while keeping the output tensor of each kernel unchanged. When the feature map of layer i-1 is convolved by the i-th convolutional matrix, the i-th feature map is obtained, and the 2nd and 5th channels are also removed. To match the change in channel dimension of the i-th feature map, the weights of the 2nd and 5th channels of each convolutional kernel in the (i+1)-th convolutional layer are removed. At this point, the number of kernels remains the same, but the output tensor of each convolution changes. This structured pruning method compresses the model weight parameters, reducing the model file size, optimizing inference time, and improving recognition efficiency.

[0058] Step 106: If the inspection object is present in the first image, determine the image acquisition control information of the flight device for the inspection object based on the specified image acquisition location and the location of the inspection object.

[0059] If the inspection object is present in the first image, the relevant attribute information of the flight device for image acquisition can be adjusted based on the specified image acquisition location and the location of the inspection object, so that the flight device can acquire images of the inspection object more accurately and effectively after adjustment.

[0060] In one possible design, based on a preset image of the inspected object and the first image, shooting deviation information is determined, wherein the shooting deviation information includes: shooting angle deviation value, sharpness deviation value, and center point deviation distance of the inspected object; based on the shooting deviation information, image acquisition control information of the flight device for the inspected object is determined, wherein the image acquisition control information includes the update position of the specified image acquisition position, the shooting angle of the shooting module of the flight device for the inspected object, and focus and zoom information.

[0061] Due to environmental factors such as equipment relocation, the originally set image acquisition location may not accurately capture images of the inspected object. To address this, a preset image of the inspected object can be pre-captured, showing the object in a state that is easy to identify and observe. After obtaining the first image, the preset image is compared with the first image; the difference between the two represents the difference between the first image captured by the flying device and the image required for effective detection of the inspected object.

[0062] Furthermore, based on the preset image of the inspection object and the first image, shooting deviation information such as shooting angle deviation value, sharpness deviation value, and center point deviation distance of the inspection object is obtained.

[0063] Among them, the shooting angle deviation value reflects the angular difference between the shooting angle of the preset image and the actual shooting angle of the flight device at the specified image acquisition position. If the shooting angle deviation value is greater than the specified angle difference, it means that the actual shooting angle of the flight device at the specified image acquisition position deviates too much from the shooting angle of the preset image, and it is impossible to obtain an image of the inspection object with a reasonable angle.

[0064] The sharpness deviation value reflects the difference between the sharpness of the preset image and the sharpness of the first image actually captured by the flight device. If the sharpness deviation value is greater than the specified sharpness difference, it means that the first image is relatively blurry and its sharpness cannot support the effective inspection of the inspection object.

[0065] In the preset image, the object to be inspected is usually located in a designated, easily observable position, such as the center of the image. Correspondingly, the center point of the object to be inspected is also relatively fixed in the preset image, such as the center of the image. If the deviation between the center point of the object to be inspected in the first image actually captured by the flight device and the center point of the object to be inspected in the preset image exceeds a specified distance, it indicates that the flight device's capture has deviated from the center point of the object to be inspected, posing a risk of not fully capturing the object in the first image and affecting further inspection of the object.

[0066] Based on this, the camera angle deviation value can be used to adjust the specified image acquisition position and / or the shooting angle of the flight device's camera module towards the inspected object, enabling the flight device to optimize the shooting angle and obtain a more accurate image of the inspected object. Simultaneously, based on the sharpness deviation value, the focus and zoom information of the flight device's camera module can be adjusted to capture clearer images, improving the image sharpness of the inspected object.

[0067] Step 108: Based on the image acquisition control information, send an image acquisition command to the flight device so that the flight device can acquire a second image containing the inspection object based on the image acquisition command.

[0068] Finally, based on the obtained image acquisition control information, an image acquisition command is generated for the flight device. Based on this image acquisition command, the flight device performs a second image capture according to the image acquisition control information, thereby acquiring a second image containing the inspected object.

[0069] The above technical solution introduces a preset yolov4-tiny model using DW convolution, 3*3 convolution kernel, and structured pruning to identify inspection objects in images acquired by the flight device. While accurately identifying inspection objects, it reduces the labor costs and environmental interference in the inspection object identification process, improves the identification efficiency of inspection objects, and helps to ensure the smooth progress of power inspection.

[0070] In one possible design, the process of applying the preset yolov4-tiny model to identify whether there is an inspection object in the first image in step 104 includes: calculating the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model; determining the target convolutional kernel whose confidence level meets the specified pruning conditions among all the 3*3 convolutional kernels in the preset yolov4-tiny model; and deleting the target convolutional kernel and the feature map corresponding to the target convolutional kernel in each convolutional layer through the structured pruning method.

[0071] The confidence level of each 3x3 convolutional kernel reflects the effectiveness of the features it represents. Higher confidence levels indicate more critical features and a greater impact on the output of the predefined YOLOv4-tiny model. Conversely, lower confidence levels indicate a weaker correlation between the features represented by the kernel and the output of the predefined YOLOv4-tiny model.

[0072] Furthermore, if the confidence level is sufficiently low, it indicates that the features represented by the 3x3 convolution kernel have extremely low correlation with the output of the preset YOLOv4-tiny model. Therefore, these convolution kernels can be deleted to simplify the calculation process of the preset YOLOv4-tiny model. This deletion operation is called pruning.

[0073] In one possible design, the specified pruning conditions can be preset, wherein the specified pruning conditions include: the confidence level is lower than a specified confidence level threshold; and / or after sorting the confidence levels of all the 3*3 convolutional kernels in ascending order, the confidence level is located in the first specified ranking position.

[0074] The specified confidence threshold refers to the minimum confidence level of the 3x3 convolutional kernel when the features represented by it have a valid impact on the output of the preset YOLOv4-tiny model. If the confidence level of the 3x3 convolutional kernel is lower than the specified confidence threshold, it means that the features represented by the 3x3 convolutional kernel have such low correlation with the output of the preset YOLOv4-tiny model that they do not need to be considered. Therefore, the convolutional kernel can be deleted, and the corresponding feature map can be deleted accordingly.

[0075] In one possible design, the target convolutional kernel and its corresponding feature map are deleted from each convolutional layer using the structured pruning method. For example... Figure 4As shown, in the i-th convolutional layer, the 2nd and 5th convolutional kernels are pruned. This reduces the number of kernels while keeping the output tensor of each kernel unchanged. When the feature map of layer i-1 is convolved by the i-th convolutional matrix, the i-th feature map is obtained, and the 2nd and 5th channels are also removed. To match the change in channel dimension of the i-th feature map, the weights of the 2nd and 5th channels of each convolutional kernel in the (i+1)-th convolutional layer are removed. At this point, the number of kernels remains the same, but the output tensor of each convolution changes. Thus, this structured pruning method compresses the model weight parameters, reducing the model file size, optimizing inference time, and improving recognition efficiency.

[0076] In one possible design, calculating the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model includes: for each 3*3 convolutional kernel in the preset yolov4-tiny model, obtaining the weights of the 3*3 convolutional kernel; and determining the confidence level of the 3*3 convolutional kernel based on the weights and weight adjustment coefficients, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolutional kernel and the confidence level.

[0077] In other words, the confidence of a 3x3 convolution kernel is the correction value of its weights, which is obtained by adjusting the weights by the weight adjustment coefficient.

[0078] Further, the method for obtaining the weight adjustment coefficient includes: determining a weight matrix based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model, wherein the element in the nth row and mth column of the weight matrix is ​​the weight product of the mth 3*3 convolutional kernel and the (m-1)th 3*3 convolutional kernel in the nth convolutional layer of the preset yolov4-tiny model; for each 3*3 convolutional kernel in the preset yolov4-tiny model, determining the target element corresponding to the 3*3 convolutional kernel in the weight matrix; obtaining the mean of the target element and at least one adjacent element of the target element; and determining the difference between the weight of the 3*3 convolutional kernel and the mean as the weight adjustment coefficient of the 3*3 convolutional kernel.

[0079] In other words, a weight matrix is ​​constructed with the number of convolutional layers as the number of rows and the number of 3x3 convolutional kernels within each convolutional layer as the number of columns. The element in the nth row and mth column of the weight matrix is ​​the weight product of the mth and (m-1)th 3x3 convolutional kernels of the nth convolutional layer in the preset yolov4-tiny model. That is, each element in the weight matrix is ​​the product of the current convolutional kernel corresponding to that element and the preceding convolutional kernel in the same convolutional layer. Therefore, each element reflects not only the feature information corresponding to its current convolutional kernel but also the feature information corresponding to the preceding convolutional kernel in the same convolutional layer; in other words, each element can reflect feature information in two dimensions.

[0080] At least one neighboring element of the target element includes, but is not limited to, neighboring elements located above, below, to the left, to the right, to the upper left, to the upper right, to the lower left, and to the lower right within the weight matrix. The weight adjustment coefficient is determined based on the mean of the target element and at least one of its neighboring elements. This fully considers the combined effect of the features corresponding to the convolutional kernel and the features corresponding to related convolutional kernels in the layers above, below, and within the current layer. This mean accurately and comprehensively reflects the influence of multi-dimensional features on a single feature, thereby reflecting the deviation level between the weight of that single feature and its actual importance.

[0081] Furthermore, this mean can accurately and comprehensively reflect the correlation between multi-dimensional features and the recognition results of the preset yolov4-tiny model, which helps the preset yolov4-tiny model output more accurate and reliable recognition results.

[0082] Figure 5 A block diagram of an inspection control device according to an embodiment of this application is shown.

[0083] like Figure 5 As shown, in an embodiment of the inspection control device according to this application, the power supply module is divided into two parts: a power supply and a control supply. The power supply refers to enabling the motors 1, 2, ..., n of the drive module to operate, thereby supplying power to the entire inspection control device. The control supply refers to supplying power to the detection module and the control module.

[0084] The rotational speed of each rotor can be controlled by adjusting the rotation of the motors to provide different lift and achieve various attitudes. Each motor is connected to a control module, and its rotational speed is adjusted via control signals from the module. An inertial measurement unit (IMU) provides attitude calculation data to the flight controller of the control module. Barometers, electronic compasses, and other sensors provide direct data on the flight device's position and attitude, enabling it to control flight attitude and adjust camera focus in complex environments. The flight device can be a quadcopter drone.

[0085] Figure 6 A flowchart of an inspection control method according to another embodiment of this application is shown.

[0086] like Figure 6 As shown, in another embodiment of the inspection control method according to this application, firstly, it is detected whether the drone has reached the data collection point.

[0087] If the drone reaches the data collection point, it acquires the first image and uses the component model to perform target recognition on the first image, outputting the recognition information. The component model is a preset yolov4-tiny model.

[0088] Next, based on the output recognition information, it is determined whether the required target exists in the first image, that is, whether there is an inspection object.

[0089] If the result indicates that the target is not present in the first image, the drone is moved to the next acquisition point to take another image, so as to determine whether the target is present in the newly acquired image.

[0090] If the judgment result indicates that the desired target exists in the first image, combined with... Figure 5 In the illustrated embodiment, the control module can send adjustment commands to the drone. These adjustment commands are determined based on the location information of the target in the identification information, and include at least the shooting angle of the drone's shooting module towards the inspected object, focus and zoom information, etc.

[0091] Finally, following the adjustment instructions, the drone adjusted its gimbal to precisely align with the desired target and took another picture, obtaining a second image. This completed the data acquisition process.

[0092] The above technical solution introduces a preset YOLOv4-tiny model to identify inspection objects in images collected by the flight device. While accurately identifying inspection objects, it can also adjust parameters such as the drone's shooting angle and focus / zoom information. Overall, it reduces the labor cost and environmental interference in the inspection object identification process, improves the efficiency of inspection object identification, and helps the smooth progress of power line inspection.

[0093] Figure 7 A block diagram of an inspection control device according to another embodiment of this application is shown.

[0094] like Figure 7As shown, an inspection control device 700 according to another embodiment of this application includes: a first image acquisition unit 702, used to acquire a first image acquired by a flight device at a designated image acquisition location; an inspection object identification unit 704, used to identify whether there is an inspection object in the first image through a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, 3*3 convolution kernel and structured pruning; a control information generation unit 706, used to determine image acquisition control information of the flight device for the inspection object based on the designated image acquisition location and the location of the inspection object if the inspection object is in the first image; and a second image acquisition unit 708, used to send an image acquisition command to the flight device based on the image acquisition control information, so that the flight device can acquire a second image containing the inspection object based on the image acquisition command.

[0095] In one embodiment of this application, optionally, the inspection object identification unit 704 includes: a confidence calculation unit, used to calculate the confidence of each 3*3 convolutional kernel in the preset yolov4-tiny model; a pruning judgment unit, used to determine, among all the 3*3 convolutional kernels in the preset yolov4-tiny model, a target convolutional kernel whose confidence satisfies a specified pruning condition; and a pruning processing unit, used to delete the target convolutional kernel and the feature map corresponding to the target convolutional kernel in each convolutional layer through the structured pruning method.

[0096] In one embodiment of this application, optionally, the credibility calculation unit is used to: obtain the weights of each 3*3 convolutional kernel in the preset yolov4-tiny model; and determine the credibility of the 3*3 convolutional kernel based on the weights and weight adjustment coefficients, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolutional kernel and the credibility.

[0097] In one embodiment of this application, the inspection control device 700 may optionally further include: a weight matrix determination unit, configured to determine a weight matrix based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model before determining the confidence level of the 3*3 convolutional kernel, wherein the element in the nth row and mth column of the weight matrix is ​​the weight product of the mth 3*3 convolutional kernel and the (m-1)th 3*3 convolutional kernel in the nth convolutional layer of the preset yolov4-tiny model; a target element determination unit, configured to determine the target element corresponding to each 3*3 convolutional kernel in the weight matrix for each 3*3 convolutional kernel in the preset yolov4-tiny model; a mean acquisition unit, configured to acquire the mean of the target element and at least one adjacent element of the target element; and a weight adjustment coefficient acquisition unit, configured to determine the difference between the weight of the 3*3 convolutional kernel and the mean as the weight adjustment coefficient of the 3*3 convolutional kernel.

[0098] In one embodiment of this application, optionally, the inspection control device 700 further includes: a pruning condition setting unit, used to set the specified pruning conditions before acquiring the first image acquired by the flight device at the specified image acquisition location, wherein the specified pruning conditions include: the confidence level is lower than a specified confidence level threshold; and / or after sorting the confidence levels of all the 3*3 convolution kernels in ascending order, the confidence level is located in the first specified ranking position.

[0099] In one embodiment of this application, optionally, the second image acquisition unit 708 includes: a first information determination unit, configured to determine shooting deviation information based on a preset image of the inspection object and the first image, wherein the shooting deviation information includes: shooting angle deviation value, sharpness deviation value, and center point deviation distance of the inspection object; and a second information determination unit, configured to determine image acquisition control information of the flight device for the inspection object based on the shooting deviation information, wherein the image acquisition control information includes the update position of the specified image acquisition position, the shooting angle of the shooting module of the flight device for the inspection object, and focus and zoom information.

[0100] The inspection control device 700 uses the solution described in any one of the above embodiments, and therefore has all the above-mentioned technical effects, which will not be repeated here.

[0101] Figure 8 A block diagram of an electronic device according to an embodiment of this application is shown.

[0102] like Figure 8As shown, an electronic device 800 according to one embodiment of this application includes at least one memory 802; and a processor 804 communicatively connected to the at least one memory 802; wherein the memory stores instructions executable by the at least one processor 804, the instructions being configured to execute the scheme described in any of the above embodiments. Therefore, this electronic device 800 has the same technical effects as any of the above embodiments, and will not be repeated here.

[0103] The electronic devices in this application embodiments exist in various forms, including but not limited to:

[0104] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0105] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0106] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players (such as iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0107] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0108] (5) Devices with autonomous driving capabilities, such as autonomous vehicles and drones;

[0109] (6) Other electronic devices with data interaction functions.

[0110] In addition, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for performing the following steps: acquiring a first image captured by a flight device at a designated image acquisition location; identifying whether an inspection object exists in the first image using a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, a 3*3 convolution kernel, and structured pruning; if the inspection object exists in the first image, determining image acquisition control information for the inspection object by the flight device based on the designated image acquisition location and the location of the inspection object; and sending an image acquisition instruction to the flight device based on the image acquisition control information, so that the flight device can acquire a second image containing the inspection object based on the image acquisition instruction.

[0111] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or electronic device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0112] The technical solution of this application has been described in detail above with reference to the accompanying drawings. Through the technical solution of this application, a preset yolov4-tiny model using DW convolution, 3*3 convolution kernel and structured pruning is introduced to identify inspection objects in images collected by the flight device. While accurately identifying inspection objects, it reduces the labor cost and environmental interference in the process of inspection object identification, improves the identification efficiency of inspection objects, and helps to ensure the smooth progress of power inspection.

[0113] Furthermore, the improved YOLOv4-tiny model enables real-time identification of target components, improving accuracy. Simultaneously, the improved YOLOv4-tiny model is more streamlined. During training, the feature extraction part of the YOLOv4-tiny model uses DW convolution instead of traditional convolution, and replaces large convolution kernels with 3x3 small convolution kernels to build a lightweight target detection model. Coupled with structured pruning, this maximizes the compression of weight parameters in the YOLOv4-tiny model, reducing model file size, optimizing inference time, and improving recognition efficiency.

[0114] In addition, the edge terminal sends various commands to complete operations such as precise alignment, zoom magnification, precise focusing, and snapshot of the gimbal and payload of the flight device, thereby achieving data collection, reducing the labor costs and environmental interference in the process of identifying inspection objects, and improving the efficiency of inspection object identification.

[0115] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0116] It should be understood that although the terms "first," "second," etc., may be used to describe images in the embodiments of this application, these images should not be limited to these terms. These terms are only used to distinguish the images from each other. For example, without departing from the scope of the embodiments of this application, a first image may also be referred to as a second image, and similarly, a second image may also be referred to as a first image.

[0117] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0118] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A patrol inspection control method, characterized in that, include: Acquire the first image captured by the flight device at the designated image acquisition location; The first image is identified by a preset yolov4-tiny model to determine whether there is an inspection object. The preset yolov4-tiny model uses DW convolution, 3*3 convolution kernel and structured pruning. If the inspection object is present in the first image, the image acquisition control information of the flight device for the inspection object is determined based on the specified image acquisition location and the location of the inspection object; Based on the image acquisition control information, an image acquisition command is sent to the flight device so that the flight device can acquire a second image containing the inspection object based on the image acquisition command; The process of identifying whether there is an inspection object in the first image using a preset yolov4-tiny model includes: Calculate the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model; Among all the 3*3 convolutional kernels in the preset yolov4-tiny model, determine the target convolutional kernel whose confidence satisfies the specified pruning condition; The target convolutional kernel and its corresponding feature map are deleted in each convolutional layer using the structured pruning method described above. The calculation of the confidence level of each 3*3 convolutional kernel in the preset yolov4-tiny model includes: For each 3*3 convolutional kernel in the preset yolov4-tiny model, obtain the weights of the 3*3 convolutional kernel; Based on the weights and weight adjustment coefficients, the confidence level of the 3*3 convolution kernel is determined, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolution kernel and the confidence level. Before determining the confidence level of the 3x3 convolution kernel based on the weights and weight adjustment coefficients, the method further includes: Based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model, a weight matrix is ​​determined, wherein the element in the nth row and mth column of the weight matrix is ​​the product of the weights of the mth 3*3 convolutional kernel of the nth convolutional layer and the (m-1)th 3*3 convolutional kernel of the preset yolov4-tiny model. For each 3*3 convolutional kernel in the preset yolov4-tiny model Determine the target element corresponding to the 3*3 convolution kernel in the weight matrix; Obtain the mean of the target element and at least one of its neighboring elements; The difference between the weights of the 3x3 convolution kernel and the mean is determined as the weight adjustment coefficient of the 3x3 convolution kernel.

2. The inspection control method according to claim 1, characterized in that, Before acquiring the first image captured by the flight device at the designated image acquisition location, the method further includes: The specified pruning conditions are set, wherein the specified pruning conditions include: The credibility level is lower than a specified credibility threshold; and / or After sorting the confidence scores of all the 3*3 convolution kernels from smallest to largest, the confidence score is placed in the first specified ranking position.

3. The inspection control method according to claim 1, characterized in that, The step of determining the image acquisition control information of the flight device for the inspected object based on the specified image acquisition location and the location of the inspected object includes: Based on the preset image of the inspection object and the first image, shooting deviation information is determined, wherein the shooting deviation information includes: shooting angle deviation value, sharpness deviation value and center point deviation distance of the inspection object; Based on the shooting deviation information, the image acquisition control information of the flight device for the inspected object is determined, wherein the image acquisition control information includes the update position of the specified image acquisition position, the shooting angle of the shooting module of the flight device for the inspected object, and focus and zoom information.

4. An inspection control device, characterized in that, include: The first image acquisition unit is used to acquire a first image acquired by the flight device at a designated image acquisition location; The inspection object identification unit is used to identify whether there is an inspection object in the first image through a preset yolov4-tiny model, wherein the preset yolov4-tiny model uses DW convolution, 3*3 convolution kernel and structured pruning method; A control information generation unit is used to determine image acquisition control information of the flight device for the inspection object based on the specified image acquisition location and the location of the inspection object if the inspection object is present in the first image; The second image acquisition unit is used to send an image acquisition command to the flight device based on the image acquisition control information, so that the flight device can acquire a second image containing the inspection object based on the image acquisition command; The inspection object identification unit includes: a credibility calculation unit, used to calculate the credibility of each 3*3 convolutional kernel in the preset yolov4-tiny model; a pruning judgment unit, used to determine the target convolutional kernel whose credibility satisfies the specified pruning condition among all the 3*3 convolutional kernels in the preset yolov4-tiny model; and a pruning processing unit, used to delete the target convolutional kernel and the feature map corresponding to the target convolutional kernel in each convolutional layer through the structured pruning method. The credibility calculation unit is used to: obtain the weights of each 3*3 convolution kernel in the preset yolov4-tiny model; and determine the credibility of the 3*3 convolution kernel based on the weights and weight adjustment coefficients, wherein the weight adjustment coefficients are used to reflect the degree of deviation between the weights of the 3*3 convolution kernel and the credibility. The inspection control device further includes: a weight matrix determination unit, used to determine a weight matrix based on the weights of each 3*3 convolutional kernel in all convolutional layers of the preset yolov4-tiny model before determining the confidence level of the 3*3 convolutional kernel, wherein the element in the nth row and mth column of the weight matrix is ​​the weight product of the mth 3*3 convolutional kernel and the (m-1)th 3*3 convolutional kernel in the nth convolutional layer of the preset yolov4-tiny model; a target element determination unit, used to determine the target element corresponding to each 3*3 convolutional kernel in the weight matrix for each 3*3 convolutional kernel in the preset yolov4-tiny model; a mean acquisition unit, used to acquire the mean of the target element and at least one of its adjacent elements; and a weight adjustment coefficient acquisition unit, used to determine the difference between the weight of the 3*3 convolutional kernel and the mean as the weight adjustment coefficient of the 3*3 convolutional kernel.

5. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for performing the method as described in any one of claims 1 to 3.

Images