A wrench switch recognition method based on a target detection model

Abstract

The application discloses a wrench switch recognition method based on a target detection model, and relates to the technical field of instrument and meter scene switch state monitoring. The method comprises the following steps: a coal mine underground substation scene experimental site is built; a substation intelligent inspection robot is intelligently inspected and a cloud platform is set; the inspection robot is fixed at the instrument horizontal position, the cloud platform information and the recognition type are sent to a visual recognition service through an upper computer interface; the visual recognition service analyzes the parameters transmitted by the upper computer interface and intercepts the image of the point; a yolov5 target detection model is used to locate the wrench and switch position and type; the target frame of the target detection is used to judge the vertical direction height and the horizontal direction length, and the wrench switch state is obtained; the image preprocessing method is used to segment the switch base and the switch, and then a straight line fitting method is used to fit the switch handle to obtain the knob switch state; finally, the visual recognition result is saved as a dictionary type and transmitted to the upper computer.

Description

Technical Field

[0001] This invention relates to the field of instrument and meter scene switch status monitoring technology, and in particular to a wrench switch identification method based on a target detection model. Background Technology

[0002] Unmanned operation of substations is a crucial aspect of substation intelligent transformation, and the identification of the operating status of key electrical cabinet switchgear is essential for the safe and stable operation of the substation. To ensure the normal operation of the substation, it is necessary to monitor the opening and closing status of the substation switchgear.

[0003] Currently, substations rely on manual inspections to monitor the operational status of electrical control cabinet switches. This manual method requires significant manpower and resources to periodically inspect the electrical control cabinets and verify the electrical switches. With the continuous development of power networks, the number of electrical control cabinets in substations is increasing, significantly increasing the workload for staff. This makes it highly susceptible to situations where individual equipment malfunctions or incorrect switch operation are not detected in time, potentially leading to major power accidents. Summary of the Invention

[0004] Therefore, it is necessary to provide a wrench switch recognition method based on a target detection model to address the problems mentioned in the background technology.

[0005] The objective of this invention can be achieved through the following technical solution: a wrench switch recognition method based on a target detection model, comprising the following steps:

[0006] S1: Build a simulation environment to simulate the substation scenario in a coal mine;

[0007] S2: Intelligent inspection and pan-tilt-zoom (PTZ) setup of substation intelligent inspection robot. The inspection robot is positioned at a fixed point on the instrument horizontal position to perform image acquisition and visual recognition tasks on the substation cabinet.

[0008] S3: The inspection robot is automatically integrated with the visual recognition service to complete the automatic inspection of the location and to acquire and analyze image information through visual recognition technology;

[0009] S4: Location and type identification of wrenches and switches based on the YOLOv5 target detection model;

[0010] S5: Based on the detection results of the target detection model, the state of the wrench switch is determined by calculating the vertical height and horizontal length of the wrench target frame;

[0011] S6: Determine the state of the knob switch through image preprocessing and line fitting;

[0012] S7: The results obtained through visual recognition are saved as a dictionary and uploaded to the host computer via an interface. Finally, the host computer parses and displays the results.

[0013] In some embodiments, the S1 process is as follows:

[0014] S1-1: Create an experimental site that matches the environment of an underground coal mine substation. Based on experience with underground coal mine substation scenarios, determine that switch instruments are usually located on the upper level of the substation cabinet.

[0015] S1-2: Design a scenario that includes a transformer cabinet and related instruments and meters, and embed devices such as wrench switches and rotary switches;

[0016] S1-3: Based on the layout of the underground substation in the coal mine, the substation cabinets need to be arranged neatly on the site, either on one side or both sides, to simulate the real environment of the underground coal mine.

[0017] In some embodiments, the S2 process is as follows:

[0018] S2-1: Prepare the inspection robot and debug the host computer program. The inspection robot includes functions such as autonomous navigation and intelligent inspection. The host computer function debugging includes robot control, data processing and visual recognition interface.

[0019] S2-2: Set the inspection robot locations and adjust the gimbal. Analyze and plan the positions of each location to obtain a location path map. Control the inspection robot to collect images of each location according to the location path map. Adjust the height and focal length of the gimbal so that the camera can be aimed at the switch position of the instrument and take horizontal pictures.

[0020] S2-3: Call the host computer interface and visual recognition service.

[0021] In some embodiments, a point path map is obtained by analyzing and planning the location of each point. The specific analysis and planning process is as follows:

[0022] 401: Obtain the layout map of the substation, mark the locations where image acquisition is required on the layout map, and determine the inspection locations;

[0023] 402: Obtain the quantity and type of each target object at each location, and based on this, conduct a comprehensive analysis of the importance of target object collection at each location to obtain the target value of each location for the target object;

[0024] 403: Mark the corresponding points of each target value on the substation layout map to obtain the starting point of the inspection robot; therefore, the constraint condition of this path planning can be obtained as: starting point - each point and its corresponding target value - starting point. Based on the constraint condition, dynamic path planning is performed to obtain the point path map.

[0025] 404: Control the inspection robot to collect images of each point according to the point path map.

[0026] In some embodiments, the specific process of comprehensively analyzing the importance of target object acquisition at each location is as follows:

[0027] Obtain the quantity and type of each target object at each location; assign an importance coefficient to each target object; extract the quantity and type of each target object at each location, and compare them with all the set target object types to match the corresponding importance coefficient, and record it as Yi, where i = 1, 2, 3... I, I takes the value of a positive integer, I represents the total quantity of target objects at the location, and i represents the number of any one of the target objects corresponding to the location;

[0028] A quality coefficient is set for image acquisition of different types of target objects. The target object types of each point are extracted and compared with all the set target object types to match the corresponding quality coefficient, which is denoted as Li.

[0029] Obtain the historical number of failures for each target object corresponding to the point and record it as Gi;

[0030] The importance coefficient Yi, quality coefficient Li, and historical failure count Gi of each target object at the location are normalized and their values ​​are taken. The target value YG of the location is then calculated and analyzed using a formula. The specific calculation formula is as follows:

[0031]

[0032] Wherein λ1, λ2, and λ3 are set proportional constants, the values ​​of which can be set by those skilled in the art according to actual needs.

[0033] In some embodiments, the specific process of dynamic path planning based on constraints is as follows:

[0034] Let D be a subset containing all visited points, where n and m are any two points in the set M. Let dp[S][n] be the minimum path length from the starting point, passing through all points in the set D, to finally reach point n.

[0035] Starting point: Set the starting point to 0, which is the initial position of the inspection robot;

[0036] The core of dynamic programming is the state transition equation, where dp[D][n] is the minimum path length from the starting point, through all points in the set D, to finally reach point n.

[0037]

[0038] Where dp[D\{n}][m] represents the minimum path length from point n to point m in the point set D;

[0039] Initialization: dp[0][0] = 0, indicating that the path length from the starting point to itself is 0;

[0040] Calculate the shortest path containing different subsets of points in turn until all points are traversed. In each round of calculation, select all points n and m, and use the state transition equation to gradually optimize the path value.

[0041] For each point n, its corresponding target value is denoted as YGn. A weighting factor is added in the path planning. The specific formula for calculating the weighted path is as follows:

[0042]

[0043] Where γ is the set weighting factor;

[0044] The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is as follows:

[0045]

[0046] Where dp[All][n] represents the shortest path value that visits all points and finally stops at point n, and dn0 represents the path length from point n back to the starting point.

[0047] In some embodiments, the S3 process is as follows:

[0048] S3-1: Set up the inspection points on the host computer interface and configure these points as automatic inspection tasks;

[0049] S3-2: Whenever the inspection robot arrives at a preset point, the host computer will automatically trigger the start of the visual recognition service and send the camera's RTSP stream and the current point information to the visual recognition service.

[0050] S3-3: Visual recognition service parses parameters transmitted from the host computer interface;

[0051] S3-4: The visual recognition service uses the screenshot function to capture images of the current location via the RTSP stream from the robot's pan-tilt camera.

[0052] In some embodiments, the S4 process is as follows:

[0053] S4-1: Organize the datasets of wrench switches and knob switches, label them, split the datasets, and generate the required format;

[0054] S4-2: Train the model using the dataset. The model training part mainly includes data preprocessing and data augmentation, model network design, training process, post-processing process, and model evaluation.

[0055] S4-3: Apply the model to the newly acquired image. The model application part mainly includes image acquisition, image preprocessing, model input detection and result display. If the result type is a wrench switch, execute S5; if the result type is a knob switch, execute S6.

[0056] In some embodiments, the S5 process is as follows:

[0057] S5-1: Using the results output by S4-3 above, including the target type and the coordinates of the top left and bottom right points of the target box, calculate the vertical height and horizontal length of the box.

[0058] S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is "Off"; if the length is less than the height, the output result is "On".

[0059] In some embodiments, the S6 process is as follows:

[0060] S6-1: Based on the coordinates of the top left and bottom right points of the target box output in S4-3, extract the target image containing the rotary switch from the original image;

[0061] S6-2: Convert the captured image to a grayscale image, use binarization or other image preprocessing methods to separate the switch base from the rotary switch, and extract the rotary part from the base;

[0062] S6-3: Filter out the black knob switch points in the image and use the fitLine function in OpenCV to fit a straight line; the fitLine function fits a straight line using the least squares method, so that the distance between all points and the line is minimized.

[0063] S6-4: By calculating the slope of the fitted straight line, the angle between the straight line and the y-axis is further calculated. Based on the slope and the angle, the current position of the rotary switch is determined, thereby inferring the state of the rotary switch.

[0064] In some embodiments, the S7 process is as follows:

[0065] S7-1: The visual recognition service saves the recognition results as a dictionary type and uploads them to the host computer via the interface;

[0066] S7-2: The host computer parses and displays the results.

[0067] Compared with the prior art, the beneficial effects of the present invention are:

[0068] 1. The method provided by this invention proposes a hardware construction approach. This approach constructs an experimental site and uses a wheeled inspection robot for the experiment, which can better reproduce the robot inspection scenario in an underground coal mine substation. The early debugging is completed in the experimental scenario, which greatly shortens the on-site debugging cycle and provides more time and possibilities for robot trial operation and stability testing.

[0069] 2. The method provided by this invention forms an algorithmic structure for scene target detection and region of interest identification. This structure has high accuracy and low false recognition rate. Compared with traditional algorithms, it is more stable. The framework using this algorithmic structure has strong transferability, is easy to deploy, shortens the overall debugging cycle, and has strong operability.

[0070] 3. The method provided by this invention performs a linear fitting operation on the rotary switch, which can effectively retain the switch information and prepare for subsequent calculation of the switch angle to determine the gear position.

[0071] 4. This invention obtains an optimal point path map by dynamically planning the paths of each point that needs image acquisition, and points with larger target values ​​in the path are visited first; the inspection robot is controlled to acquire images of the target objects accordingly, ensuring that each predetermined point can be accessed by the inspection robot and images are acquired, reducing redundant paths, optimizing the inspection route, shortening the inspection time, and giving priority to equipment with higher image quality requirements and higher failure risk, thus achieving optimal inspection efficiency and quality. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of the principle of the present invention;

[0074] Figure 2 This is a flowchart of step S1 in an embodiment of the present invention;

[0075] Figure 3 This is a flowchart of step S2 in an embodiment of the present invention;

[0076] Figure 4 This is a flowchart illustrating step S3 in an embodiment of the present invention.

[0077] Figure 5 This is a flowchart illustrating step S4 in an embodiment of the present invention.

[0078] Figure 6 This is a flowchart of step S5 in an embodiment of the present invention;

[0079] Figure 7 This is a flowchart illustrating step S6 in an embodiment of the present invention.

[0080] Figure 8 This is a flowchart of step S7 in an embodiment of the present invention. Detailed Implementation

[0081] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0082] like Figures 1-2 As shown, a wrench switch recognition method based on a target detection model includes the following steps:

[0083] S1: An experimental site with wrenches, switches, etc., is set up to simulate and recreate the substation in an underground coal mine. This provides an experimental site that matches the environment of an underground coal mine substation for the debugging and testing of the robot vision system, ensuring that the robot can be successfully put into actual work.

[0084] S1-1: Based on images of wrench switches, rotary switches, etc. in the substation scene of a coal mine, it can be known from experience that switch instruments are mainly distributed on the upper layer of the substation cabinet.

[0085] S1-2: Based on the experience summarized in S1-1, design a set of instrument and meter scenes that can be restored and displayed in the factory area. It includes a complete cabinet similar to a transformer cabinet, and embeds wrench switches and rotary switches on the cabinet. It should be noted that the entire design is produced by the manufacturer and mailed to the factory area.

[0086] S1-3: Factory site design, with the cabinets arranged neatly, and the substation cabinets placed on one or both sides, mimicking the substation scene.

[0087] It should be noted that, in this embodiment of the application, the main function of the underground coal mine substation is that its main facilities include power distribution equipment, power transformers, control equipment, automatic protection devices, communication facilities and compensation devices, etc. Therefore, the substation stores instruments and meters representing various parameters. The inspection robot is used for intelligent inspection to replace manual collection of instrument parameter images. The visual algorithm is used to analyze the images to obtain readings and status for reference by the staff. Since the coal mine site is relatively complex and it is very difficult to go down the mine most of the time, the inspection robot cannot be visually debugged before formal operation. Therefore, it is necessary to perform visual debugging as much as possible before the robot enters the site. The construction conditions are to imitate the underground coal mine substation, including the design of the substation cabinet with the same length, width and height, and to embed wrenches, switches, etc. into the upper layer of the substation cabinet.

[0088] like Figure 3 As shown, S2: Intelligent inspection and pan-tilt-zoom (PTZ) setup of the substation intelligent inspection robot. The inspection robot is positioned at a fixed point on the horizontal position of the instrument, performing image acquisition and visual recognition tasks on the substation cabinet. Specifically:

[0089] It should be noted that in this application example, the substation intelligent inspection robot consists of a mobile carrier, communication equipment, and testing equipment, and adopts remote control or fully autonomous operation mode. It is a mobile inspection device used for substation equipment inspection operations. The robot is equipped with a Hikvision camera as a pan-tilt unit to dynamically collect instrument images, use visual algorithms to analyze and calculate the readings and status of instruments on the substation cabinet, and return the results to the host computer for display. The pan-tilt unit is set to set the status of each point, the pan-tilt unit lifting height is for horizontal shooting of the target, and the pan-tilt unit focal length is for content that is clearly visible to the human eye.

[0090] S2-1: Prepare the inspection robot and debug the host computer program. Specifically, prepare a wheeled inspection robot with functions such as autonomous navigation and intelligent inspection to ensure that it can automatically or remotely inspect substation equipment. The host computer program of the robot is debugged and completed, including the setting of functions such as robot control, data processing and visual recognition interface.

[0091] S2-2: Set up the inspection robot's locations and adjust the pan-tilt unit. Analyze and plan the positions of each location to obtain a location path map. Control the inspection robot to collect images from each location according to the location path map. Adjust the pan-tilt unit's height and focus to ensure the camera can horizontally capture images of the instrument's switch positions, ensuring the captured content is clearly visible to the human eye and that the wide-angle lens covers both sides of the cabinet in the image. The specific process of location analysis and route planning is as follows:

[0092] Obtain the layout map of the substation, mark the locations where image acquisition is required on the layout map, and determine the inspection locations. It should be noted that each inspection location corresponds to one or more targets that need to be imaged (such as switches, knobs, instruments, etc.).

[0093] Obtain the quantity and type of each target object at each location; assign an importance coefficient to each target object. It should be noted that different types of target objects (such as main switches and circuit breakers in a substation) have different importance, especially when they malfunction, as they will have varying degrees of impact on the operation of the entire power system. Therefore, their corresponding importance coefficients should differ. The specific importance coefficient values ​​are set by those skilled in the art based on the criticality of the equipment; for example, importance coefficients range from 1 to 10, with higher importance coefficients for main switches and circuit breakers, and lower weight values ​​for some auxiliary equipment. Extract the quantity and type of each target object at each location, and compare them with all the set target object types to match the corresponding importance coefficient. Record this as Yi, where i = 1, 2, 3…I, where I is a positive integer, I represents the total quantity of target objects at the location, and i represents the number of any one of the target objects corresponding to the location.

[0094] A quality coefficient is assigned to each type of target object for image acquisition. It should be noted that the image acquisition quality requirements differ for different target objects. For devices requiring precise detail identification (such as the angle of a knob), the acquired image must have high resolution and clarity, which is helpful for subsequent image analysis (such as calculating the knob angle). For example, a rotary switch requires a high-precision image to fit the knob angle and determine the current switch position based on the angle. This requires high image clarity to accurately capture the shape and relative position of the knob. The target object types at each location are extracted and compared with all the set target object types to match the corresponding quality coefficient, which is denoted as Li.

[0095] Obtain the historical failure count of each target object corresponding to the location and record it as Gi; normalize the importance coefficient Yi, quality coefficient Li, and historical failure count Gi of each target object at the location and take their values. Perform formulaic calculation and analysis on these values ​​to obtain the target value YG of the location. The specific calculation formula is as follows:

[0096]

[0097] Where λ1, λ2, and λ3 are set proportional constants, the values ​​of which can be set by those skilled in the art according to actual needs; for example, λ1 is 1.531, λ2 is 1.082, and λ3 is 1.294; as can be seen from the formula, when the target objects at the point have high requirements for the quality of the acquired images, the target value is larger; when the target objects at the point are more important to the substation, or when their fault risk is greater (more faults), the target value is larger; thus, the target value for each point can be obtained;

[0098] Mark the corresponding points of each target value YG on the substation layout diagram to obtain the starting point of the inspection robot (usually the starting point is the initial position of the inspection robot when it enters the coal mine; specifically, in this simulated substation, the starting point of the inspection robot is set by those skilled in the art). Therefore, the constraints of this path planning can be obtained as: starting point - each point and its corresponding target value - starting point.

[0099] Let D be a subset containing all visited points (including the starting point), where n and m are any two points in the set M. Let dp[S][n] be the minimum path length from the starting point, through all points in the set D, to finally reach point n.

[0100] Starting point: Set the starting point to 0, which is the initial position of the inspection robot;

[0101] The core of dynamic programming is the state transition equation, where dp[D][n] is the minimum path length from the starting point, through all points in the set D, to finally reach point n.

[0102]

[0103] Where dp[D\{n}][m] represents the minimum path length from point n to point m in the point set D;

[0104] Initialization: dp[0][0] = 0, indicating that the path length from the starting point to itself is 0; for other dp[0][n], it can be set to a very large value, because in the initial state, other points have not been traversed yet;

[0105] Calculate the shortest path containing different subsets of points in turn until all points are traversed. In each round of calculation, select all possible points n and m, and use the state transition equation to gradually optimize the path value.

[0106] For each point n, its corresponding target value is denoted as YGn. A weighting factor is added to the path planning so that points with larger target values ​​are visited more frequently. The specific formula for calculating the weighted path is as follows:

[0107]

[0108] Here, γ is a weighting factor that is set to adjust the priority of selecting points with larger target values. Its value can be set by those skilled in the art. In this way, points with larger target values ​​are prioritized for access.

[0109] The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is as follows:

[0110]

[0111] Where dp[All][n] represents the shortest path value that visits all points and finally stops at point n, and dn0 represents the path length from point n back to the starting point;

[0112] Based on the above dynamic path planning, an optimal point path map can be obtained, and points with larger target values ​​in the path will be visited first. The inspection robot is controlled to collect images of each point accordingly, ensuring that each predetermined point can be visited by the inspection robot and images can be collected, reducing redundant paths, optimizing the inspection route, shortening the inspection time, and giving priority to equipment with higher image quality requirements and higher failure risk, thus achieving optimal inspection efficiency and quality.

[0113] S2-3: Calling the host computer interface and visual recognition service. Specifically, the host computer actively initiates a request, sending the gimbal settings (including gimbal angle, focal length, etc.) and the target type to be recognized (e.g., wrench switch, knob switch, etc.) to the visual recognition service. The visual recognition service parses and captures images taken by the inspection robot at designated locations based on the received parameters. Real-time camera footage is obtained by calling the RTSP stream (typically using the RTSP stream protocol of Hikvision cameras). OpenCV's VideoCapture function is used to connect to the camera and acquire image data from the current gimbal location. This function can be used to capture video streams, supporting real-time image acquisition and processing from the camera.

[0114] It should be noted that this is an active request API, initiated by the host computer. It's suitable for business scenarios where a single recognition task requires multiple recognitions, and the returned result array contains all recognition results from that single call. The API request method is HTTP POST, and the returned format is JSON. Image retrieval uses RTSP streaming; robot gimbals typically use Hikvision cameras for RTSP streaming. OpenCV's VideoCapture API can be used to capture the gimbal's image points. VideoCapture is an API (Application Programming Interface) used to open video files, network videos, and video capture devices (cameras).

[0115] like Figure 4 As shown, S3: The inspection robot is automatically integrated with visual recognition services to complete the automatic inspection task of the location, and image information is acquired and analyzed through visual recognition technology; the specific process includes:

[0116] S3-1: Set up the inspection points on the host computer interface and configure these points as automatic inspection tasks. This means that the robot will automatically move and inspect according to the set points during the inspection process.

[0117] S3-2: Whenever the inspection robot arrives at a preset point, the host computer will automatically trigger the start of the visual recognition service. The host computer will send the RTSP stream of the camera (for real-time video acquisition) and the information of the current point to the visual recognition service to provide necessary data support for subsequent image acquisition and analysis.

[0118] S3-3: After receiving a request from the host computer, the visual recognition service will parse the parameters transmitted in the interface, including the RTSP stream and point information. By parsing the parameters transmitted by the host computer interface, the recognition service can locate the robot's current inspection position and prepare to execute the corresponding image processing task.

[0119] S3-4: The visual recognition service uses the screenshot function (RTSP screenshot) to capture the image of the current location through the RTSP stream of the robot's pan-tilt camera. It should be noted that this image is a real-time image based on the current inspection task and includes the status and readings of the instruments.

[0120] like Figure 5 As shown, S4: Location and type identification of wrenches and switches based on the YOLOv5 target detection model, specifically:

[0121] S4-1: Prepare an image dataset containing wrench switches and knob switches, add labels to the images in the dataset, label the type of each object (wrench switch or knob switch) and their location in the image; divide the dataset into training set, validation set and test set, and convert it into the format required by the YOLOv5 model;

[0122] S4-2: Data preprocessing, such as image resizing, normalization, and data augmentation (e.g., Mosaic enhancement, random scaling, cropping), is performed to improve the model's ability to detect small objects while reducing overfitting. YOLOv5 is selected as the object detection model, and its network structure consists of three main parts: Backbone (feature extraction), Neck (feature fusion), and Output (predicted bounding box generation). During training, the model calculates predicted bounding boxes through forward propagation, uses a loss function to calculate bias, and updates weights through backpropagation to minimize the loss value until the stopping condition is met (until the model converges). The YOLOv5 model uses an adaptive anchor box calculation method to dynamically adjust the size of the anchor boxes for more accurate target box prediction. Adaptive image scaling is also used to reduce computation and improve detection speed. The trained model is tested on an evaluation dataset, and performance metrics such as accuracy, recall, and F1 score are calculated and analyzed.

[0123] Furthermore, in this embodiment, the model evaluation index is calculated using the following formula:

[0124]

[0125] mAP = mean(AP) 扳手开关 +AP 旋钮钮开 )

[0126] The confusion matrices for TP, TN, FP, and FN are as follows:

[0127]

[0128] Where TP (True Positive): correctly detected targets, TN (True Negative): correctly identified regions as non-target areas, FP (False Positive): incorrectly identified non-target regions as targets, and FN (False Negative): missed target regions; based on the evaluation results, analyze the model's errors, discuss the issues, and adjust the model or data accordingly.

[0129] S4-3: Acquire the video stream via IP or camera serial number, and preprocess the image (e.g., resize, normalize, etc.) to adapt it to the input requirements of the YOLOv5 model. Input the preprocessed image into the trained YOLOv5 model for object detection, identifying the position, category, and confidence of wrench switches and knob switches in the image. Draw the bounding boxes, categories, and confidence of the detected wrench switches or knob switches in a new image and display it. If the detection result is a wrench switch, proceed to step S5 (analyze the wrench switch state); if it is a knob switch, proceed to step S6 (analyze the knob switch state).

[0130] like Figure 6 As shown, S5: Based on the detection results of the target detection model, the state of the wrench switch (on or off) is determined by calculating the vertical height and horizontal length of the wrench target frame. Specifically:

[0131] S5-1: Using the results output in step S4-3 above, including the target type and the coordinates of the top left and bottom right points of the target box, calculate the vertical height of the box (i.e., the vertical length of the box). Similarly, calculate the horizontal length of the target box (i.e., the horizontal length of the box).

[0132] S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is "Off"; if the length is less than the height, the output result is "On".

[0133] S6: The state of the knob switch is determined through image preprocessing and line fitting, specifically:

[0134] S6-1: Based on the coordinates of the top left and bottom right points of the target box output in step S4-3, extract the target image containing the rotary switch from the original image;

[0135] S6-2: Convert the captured image to a grayscale image, use binarization or other image preprocessing methods to separate the switch base from the rotary switch, and extract the rotary part from the base;

[0136] S6-3: Filter out the black knob switch points in the image (usually the knob handle), and use the `fitLine` function in OpenCV to fit a straight line. The `fitLine` function fits a straight line using the least squares method, minimizing the distance between all points and the line. The parameters of the fitted line (slope k and intercept b) can be expressed in point-slope form, specifically described by the following formula:

[0137]

[0138] Where (vx,vy) is the normalized vector collinear with the line, and (x0,y0) is an arbitrary point on the fitted line.

[0139] S6-4: By calculating the slope k of the fitted straight line, the angle between the straight line and the y-axis is further calculated (usually using the arctangent function atan). Based on the slope k and the angle, the current position of the rotary switch (e.g., horizontal, vertical, or other angles) is determined, thereby inferring the state of the rotary switch.

[0140] S7: The results obtained through visual recognition are saved as a dictionary and uploaded to the host computer via an interface. The host computer then parses and displays the results. Specifically:

[0141] S7-1: After completing target detection and status determination, the visual recognition service saves all recognition results (such as wrench status, switch type and status, knob switch position, etc.) as a dictionary-type data structure. The dictionary usually includes information such as the target type, position (coordinates), and status.

[0142] S7-2: After receiving the dictionary-type recognition result, the host computer parses the result. The parsing process includes reading various data in the dictionary and displaying them visually on the host computer interface, such as displaying the wrench's on / off status, the knob's gear position, the switch's position, and other information.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A wrench switch recognition method based on a target detection model, characterized in that, Includes the following steps: S1: Build a simulation environment to simulate the substation scenario in a coal mine; S2: Intelligent inspection and pan-tilt-zoom (PTZ) setup of substation intelligent inspection robot. The inspection robot is positioned at a fixed point on the instrument horizontal position to perform image acquisition and visual recognition tasks on the substation cabinet. S3: The inspection robot is automatically integrated with the visual recognition service to complete the automatic inspection of the location and to acquire and analyze image information through visual recognition technology; S4: Location and type identification of wrenches and switches based on the YOLOv5 target detection model; S5: Based on the detection results of the target detection model, the state of the wrench switch is determined by calculating the vertical height and horizontal length of the wrench target frame; S6: Determine the state of the knob switch through image preprocessing and line fitting; The process of step S6 is as follows: S6-1: Based on the coordinates of the top left and bottom right points of the target box output in step S4-3, extract the target image containing the rotary switch from the original image; S6-2: Convert the captured image to a grayscale image, use binarization or other image preprocessing methods to separate the switch base from the rotary switch, and extract the rotary part from the base; S6-3: Filter out the black knob switch points in the image and use the fitLine function in OpenCV to fit a straight line; the fitLine function fits a straight line using the least squares method, so that the distance between all points and the line is minimized. S6-4: By calculating the slope of the fitted straight line, calculate the angle between the straight line and the y-axis. Based on the slope and the angle, determine the current position of the rotary switch, and thus infer the state of the rotary switch. S7: The results obtained through visual recognition are saved as a dictionary type and uploaded to the host computer through the interface. Finally, the host computer parses and displays the results. The process of step S7 is as follows: S7-1: The visual recognition service saves the recognition results as a dictionary type and uploads them to the host computer via the interface; S7-2: The host computer parses and displays the results.

2. The wrench switch recognition method based on a target detection model according to claim 1, characterized in that, The process of step S1 is as follows: S1-1: Create an experimental site that matches the environment of an underground coal mine substation. Based on experience with underground coal mine substation scenarios, determine the distribution of switchgear instruments on the upper layer of the substation cabinet. S1-2: Design a scenario that includes a transformer cabinet and related instruments, and embed wrench switches and rotary switches. S1-3: Based on the layout of the underground substation in the coal mine, the substation cabinets need to be arranged neatly on the site, either on one side or both sides, to simulate the real environment of the underground coal mine.

3. The wrench switch recognition method based on a target detection model according to claim 1, characterized in that, The process of step S2 is as follows: S2-1: Prepare the inspection robot and debug the host computer program. The inspection robot includes autonomous navigation and intelligent inspection functions. The host computer function debugging includes robot control, data processing and visual recognition interface. S2-2: Set the inspection robot locations and adjust the gimbal. Analyze and plan the positions of each location to obtain a location path map. Control the inspection robot to collect images of each location according to the location path map. Adjust the height and focal length of the gimbal so that the camera can be aimed at the switch position of the instrument and take horizontal pictures. S2-3: Call the host computer interface and visual recognition service.

4. The wrench switch recognition method based on a target detection model according to claim 3, characterized in that, A point path map is obtained by analyzing and planning the location of each point. The specific analysis and planning process is as follows: 401: Obtain the layout map of the substation, mark the locations where image acquisition is required on the layout map, and determine the inspection locations; 402: Obtain the quantity and type of each target object at each location, and based on this, conduct a comprehensive analysis of the importance of target object collection at each location to obtain the target value of each location for the target object; 403: Mark the corresponding points of each target value on the substation layout diagram to obtain the starting point of the inspection robot; therefore, the constraint condition of this path planning can be obtained as: starting point - each point and its corresponding target value - starting point. Based on the constraint condition, dynamic path planning is performed to obtain the point path map. 404: Control the inspection robot to collect images of each point according to the point path map.

5. The wrench switch recognition method based on a target detection model according to claim 4, characterized in that, The specific process of comprehensively analyzing the importance of target object acquisition at each location is as follows: Obtain the quantity and type of each target object at each location; assign an importance coefficient to each target object; extract the quantity and type of each target object at each location and compare them with all the set target object types to match the corresponding importance coefficient; A quality coefficient is set for image acquisition of different types of target objects. The target object types of each point are extracted and compared with all the set target object types to match the corresponding quality coefficient. Obtain the historical number of failures for each target object corresponding to the location; The importance coefficient, quality coefficient, and historical failure count of each target object at the location are normalized and their values ​​are taken. The values ​​are then used to perform formulaic calculations and analyses to obtain the target value of the location.

6. The wrench switch recognition method based on a target detection model according to claim 5, characterized in that, The specific process of dynamic path planning based on constraints is as follows: Let D be a subset containing all visited points, where n and m are any two points in the set M. Let dp[S][n] be the minimum path length from the starting point, passing through all points in the set D, to finally reach point n. Starting point: Set the starting point to 0, which is the initial position of the inspection robot; The core of dynamic programming is the state transition equation, where dp[D][n] is the minimum path length from the starting point, through all points in the set D, to finally reach point n. ； in, This represents the minimum path length from point n to point m in the point set D. Initialization: dp[0][0]=0, indicating that the path length from the starting point to itself is 0; Calculate the shortest path containing different subsets of points in turn until all points are traversed. In each round of calculation, select all points n and m, and use the state transition equation to gradually optimize the path value. For any point n, its corresponding target value is denoted as YGn. A weighting factor is added in the path planning. The specific formula for calculating the weighted path is as follows: Where γ is the set weighting factor; The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is as follows: ； in This represents the shortest path value that visits all points and finally stops at point n. This represents the path length from point n back to the starting point.

7. The wrench switch recognition method based on a target detection model according to claim 1, characterized in that, The process of step S3 is as follows: S3-1: Set up the inspection points on the host computer interface and configure these points as automatic inspection tasks; S3-2: Whenever the inspection robot arrives at a preset point, the host computer automatically triggers the start of the visual recognition service and sends the camera's RTSP stream and the current point information to the visual recognition service. S3-3: Visual recognition service parses parameters transmitted from the host computer interface; S3-4: The visual recognition service uses the screenshot function to capture images of the current location via the RTSP stream from the robot's pan-tilt camera.

8. The wrench switch recognition method based on a target detection model according to claim 1, characterized in that, The process of step S4 is as follows: S4-1: Organize the datasets of wrench switches and knob switches, label them, split the datasets, and generate the required format; S4-2: Train the model using the dataset. The model training part includes data preprocessing and data augmentation, model network design, training process, post-processing process, and model evaluation. S4-3: Apply the model to the newly acquired image. The model application part includes image acquisition, image preprocessing, model input detection and result display. If the result type is a wrench switch, execute step S5. If the result type is a knob switch, execute step S6.

9. The wrench switch recognition method based on a target detection model according to claim 1, characterized in that, The process of step S5 is as follows: S5-1: Using the results output in step S4-3, including the target type and the coordinates of the top left and bottom right points of the target box, calculate the vertical height and horizontal length of the box. S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is off; if the length is less than the height, the output result is on.

Images