Wrench switch identification method based on target detection model

Abstract

The invention discloses a wrench switch identification method based on a target detection model, and relates to the technical field of instrument scene switch state monitoring. Comprising the following steps: constructing an experiment site imitating a coal mine underground substation scene; performing intelligent inspection of the substation intelligent inspection robot and setting a holder; the inspection robot is fixed at the horizontal position of the instrument, and the upper computer interface is used for calling and sending the holder information and the identification type to the visual identification service; the visual identification service analyzes the parameters transmitted by the upper computer interface and intercepts an image of the point location; positioning the positions and types of a wrench and a switch by using a yolov5 target detection model; judging the vertical height and the horizontal length of the wrench type by using a target frame of target detection to obtain the on-off state of the wrench; the knob switch type is divided into a switch base and a switch by using an image preprocessing means, and then a switch handle is fitted by using a straight line fitting mode to obtain a knob switch state; and finally, storing a visual identification result as a dictionary type and transmitting the dictionary type to an upper computer.

Description

Technical Field

[0001] The present invention relates to the technical field of instrument scene switch status monitoring, and in particular to a spanner switch recognition method based on a target detection model. Background Art

[0002] Unmanned substation is an important part of the intelligent transformation of substation, and the operating status identification of key electrical cabinet switchgear is crucial for the safe and stable operation of substation. In order to ensure the normal operation of substation, it is necessary to monitor the opening and closing status of substation switchgear.

[0003] At present, substations use manual inspections to monitor the operating status of electrical cabinet switches. The manual method requires staff to spend a lot of manpower and material resources to periodically inspect the electrical control cabinets and check the electrical switches. With the continuous development of power networks, there are more and more electrical control cabinets in substations, and the workload of staff has increased significantly. It is very easy for individual equipment to fail or the switch working status to be wrong without being discovered in time, causing major power accidents. Summary of the invention

[0004] Based on this, it is necessary to provide a wrench switch recognition method based on a target detection model to address the problems mentioned in the above background technology.

[0005] The purpose of the present invention can be achieved by the following technical solution: A method for identifying a wrench switch based on a target detection model, comprising the following steps:

[0006] S1: Build a simulation environment to simulate the substation scene in a coal mine;

[0007] S2: Intelligent inspection and pan / tilt setting of the substation intelligent inspection robot. The inspection robot is fixed at the horizontal position of the instrument to perform image acquisition and visual recognition tasks on the substation cabinet;

[0008] S3: Automatically coordinate the inspection robot with the visual recognition service to complete the automatic inspection task of points, and obtain and analyze image information through visual recognition technology;

[0009] S4: Positioning and type recognition of wrenches and switches based on the YOLOv5 target detection model;

[0010] S5: Based on the detection result of the target detection model, the state of the wrench switch is determined by calculating the vertical height and horizontal length of the wrench target frame;

[0011] S6: judging the state of the knob switch through image preprocessing and straight line fitting;

[0012] S7: The results obtained through visual recognition are saved as a dictionary type and uploaded to the host computer through the interface. Finally, the host computer parses and displays the results.

[0013] In some embodiments, the S1 process is as follows:

[0014] S1-1: Create an experimental site that matches the environment of a coal mine underground substation. Combined with the experience of coal mine underground substation scenarios, it is determined that switch instruments are usually distributed on the upper layer of the substation cabinet;

[0015] S1-2: Design a scenario that includes a substation cabinet and related instruments and meters, and embed devices such as wrench switches and knob switches;

[0016] S1-3: According to the layout of the underground substation in the coal mine, the substation cabinets need to be neatly arranged in the venue, and can be placed on one side or both sides to simulate the real environment of the underground coal mine.

[0017] In some embodiments, the S2 process is as follows:

[0018] S2-1: Prepare the inspection robot and debug the host computer program. The inspection robot includes functions such as autonomous navigation and intelligent inspection. The host computer function debugging includes robot control, data processing and visual recognition interface;

[0019] S2-2: Set the inspection robot's position and pan / tilt adjustment, obtain the position path map by analyzing and planning the position of each point, and control the inspection robot to collect images of each point according to the point path map; adjust the up and down height and focal length of the pan / tilt so that the camera can aim at the switch position of the instrument for horizontal shooting;

[0020] S2-3: Call the host computer interface and visual recognition service.

[0021] In some embodiments, a point path map is obtained by analyzing and planning the positions of each point. The specific analysis and planning process is as follows:

[0022] 401: Obtain a layout diagram of the substation, mark the points where image collection is required on the layout diagram, and determine the inspection points;

[0023] 402: Obtain the number of targets corresponding to each point and the type of each target, and based on this, conduct a comprehensive analysis on the importance of target collection at each point to obtain a target value of each point with respect to the target;

[0024] 403: Mark the points corresponding to each target value in the layout diagram of the substation one by one to obtain the starting point of the inspection robot; therefore, the constraint conditions of this path planning are: starting point-each point and its corresponding target value-starting point, and perform dynamic path planning based on the constraint conditions to obtain a point path diagram;

[0025] 404: Control the inspection robot to collect images of each point according to the point path map.

[0026] In some embodiments, the specific process of comprehensively analyzing the importance of collecting the target objects at each point is as follows:

[0027] Obtain the number of targets corresponding to each point and the type of each target; set different targets to correspond to an important coefficient respectively; extract the number of targets at each point and the type of each target, and compare them with all the set target types to match the corresponding important coefficient, and record it as Yi, where i = 1, 2, 3...I, I is a positive integer, I represents the total number of targets at the point, and i represents the number of any target corresponding to the point;

[0028] Set the image acquisition of different types of targets to correspond to a quality coefficient respectively, extract each target type at the point, and compare it with all the set target types to match the corresponding quality coefficient, which is recorded as Li;

[0029] Obtain the historical failure times of each target object corresponding to the point and record it as Gi;

[0030] The important coefficient Yi, quality coefficient Li and historical fault number Gi of each target object at the point are normalized and their values ​​are taken. The numerical values ​​are calculated and analyzed by formula to obtain the target value YG of the point. The specific calculation formula is:

[0031]

[0032] Among them, λ1, λ2, and λ3 are respectively set proportional constants, and their values ​​are set by those skilled in the art according to actual needs.

[0033] In some embodiments, the specific process of performing dynamic path planning based on constraint conditions is as follows:

[0034] Let D be a subset, where D contains all the points visited, where n and m are any two points in the point set M, and let dp[S][n] be the minimum path length from the starting point, through all the points in the point set D, and finally to the point n;

[0035] Starting point: Set the starting point to 0, which is the initial position of the inspection robot;

[0036] The core of dynamic programming is the state transfer equation, which sets dp[D][n] as the minimum path length from the starting point, through all points in the point set D, and finally to point n;

[0037]

[0038] Among them, dp[D\{n}][m] represents the minimum path length to point m after removing point n from point set D;

[0039] Initialization: dp[0][0] = 0, indicating that the path length from the starting point to itself is 0;

[0040] Calculate the shortest paths containing different point subsets in sequence until all points are traversed. In each round of calculation, select all points n and m, and use the state transfer equation to gradually optimize the path value.

[0041] For each point n, the corresponding target value is recorded as YGn. A weighting factor is added to the path planning. The specific weighted path calculation formula is:

[0042]

[0043] Where γ is the set weighting factor;

[0044] The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is:

[0045]

[0046] Where dp[All][n] represents the shortest path value that visits all points and finally stops at point n, and dn0 represents the length of the path from point n back to the starting point.

[0047] In some embodiments, the S3 process is as follows:

[0048] S3-1: Complete the setting of inspection points on the host computer interface and configure these points as automatic inspection tasks;

[0049] S3-2: Whenever the inspection robot reaches a preset point, the host computer will automatically trigger the start of the visual recognition service, and the host computer will send the camera's RTSP stream and the current point information to the visual recognition service;

[0050] S3-3: The visual recognition service parses the parameters transmitted by the host computer interface;

[0051] S3-4: Through the RTSP stream of the robot's gimbal camera, the visual recognition service uses the screenshot function to collect the image of the current point.

[0052] In some embodiments, the S4 process is as follows:

[0053] S4-1: Arrange the wrench switch and knob switch datasets, label them, split the datasets, and generate the required format;

[0054] S4-2: The data set is used for model training. The model training part mainly includes data preprocessing and data enhancement, model network design, training process, post-processing process and model evaluation;

[0055] S4-3: Apply the model to the newly acquired image. The model application part mainly includes image acquisition, image pre-processing, model input detection and result display. If the result type is a wrench switch, execute S5; if the result type is a knob switch, execute S6.

[0056] In some embodiments, the S5 process is as follows:

[0057] S5-1: Using the result outputted by S4-3, including the target type and the coordinates of the upper left point and the lower right point of the target box, calculate the vertical height and horizontal length of the box;

[0058] S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is "off", if the length is less than the height, the output result is "on".

[0059] In some embodiments, the S6 process is as follows:

[0060] S6-1: According to the coordinates of the upper left point and the lower right point of the target frame output in S4-3, the target image containing the knob switch is cut out from the original image;

[0061] S6-2: convert the captured image into a grayscale image, use binarization or other image preprocessing methods to separate the switch base and the knob switch, and extract the knob part from the base;

[0062] S6-3: Filter out the black knob switch points in the image and use the fitLine function in Opencv to fit a straight line; the fitLine function uses the least squares method to fit a straight line so that the distance between all points and the straight line is the minimum;

[0063] S6-4: By calculating the slope of the fitting line, further calculating the angle between the line and the y-axis, judging the current gear position of the knob switch according to the slope and the angle, thereby inferring the state of the knob switch.

[0064] In some embodiments, the S7 process is as follows:

[0065] S7-1: The visual recognition service saves the recognition results as a dictionary type and uploads them to the host computer through the interface;

[0066] S7-2: The host computer analyzes the results and displays them.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The method provided by the present invention proposes a hardware construction method. This method constructs an experimental site and uses a wheeled inspection robot for experiments, which can better restore the robot inspection scene of the coal mine underground substation. The early debugging is completed in the experimental scene, which greatly shortens the on-site debugging cycle and provides longer time and possibility for robot trial operation and stability testing;

[0069] 2. The method provided by the present invention forms an algorithm structure for scene target detection and target region of interest recognition. The structure has a high accuracy rate and a low misrecognition rate. Compared with the traditional algorithm, it is more stable. The framework using the algorithm structure has strong portability, convenient deployment, shortened overall debugging cycle, and strong operability.

[0070] 3. The method provided by the present invention performs a straight line fitting operation on the knob switch, which can effectively retain the switch information and prepare for the subsequent calculation of the switch angle to determine the gear position;

[0071] 4. The present invention obtains an optimal point path map by dynamically planning the path for each point that requires image acquisition, and the points with large target values ​​in the path will be visited first; the inspection robot is controlled to acquire images of the target object accordingly, ensuring that each predetermined point can be visited by the inspection robot and image acquisition is performed, thereby reducing redundant paths, optimizing inspection routes, shortening inspection time, and giving priority to equipment with higher image quality requirements and greater failure risks, thereby achieving optimal inspection efficiency and quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0073] Figure 1 It is a principle block diagram of the present invention;

[0074] Figure 2 is a specific flow chart of step S1 in an embodiment of the present invention;

[0075] Figure 3 This is a specific flow chart of step S2 in an embodiment of the present invention;

[0076] Figure 4 This is a specific flow chart of step S3 in an embodiment of the present invention;

[0077] Figure 5 This is a specific flow chart of step S4 in an embodiment of the present invention;

[0078] Figure 6 is a specific flow chart of step S5 in an embodiment of the present invention;

[0079] Figure 7 is a specific flow chart of step S6 in an embodiment of the present invention;

[0080] Figure 8 4 is a specific flow chart of step S7 in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0082] like Figure 1-Figure 2 As shown, a method for identifying a wrench switch based on a target detection model includes the following steps:

[0083] S1: An experimental site with wrenches, switches, etc. is built based on the scene of a coal mine underground substation to simulate and restore the substation underground in the coal mine, providing an experimental site that matches the environment of the coal mine underground substation for the debugging and testing of the robot vision system, ensuring that the robot can be put into actual work smoothly;

[0084] S1-1: Based on the images of wrench switches, knob switches, etc. in the underground substation scene of a coal mine, it is known from accumulated experience that switch instruments are mainly distributed on the upper layer of the substation cabinet in the substation;

[0085] S1-2: Based on the experience summarized in S1-1, a set of instrumentation scenes that can be restored and displayed in the factory area is designed, including a complete box similar to a transformer cabinet, and the wrench switch and knob switch are embedded in the box. It should be noted that the entire set of designs is produced by the manufacturer and mailed to the factory area;

[0086] S1-3: Factory site design, arranging the boxes neatly, placing the substation cabinets on one or both sides to imitate the substation scene;

[0087] It should be noted that in the embodiment of the present application, the main role and function of the coal mine underground substation is that its main facilities include distribution devices, power transformers, control equipment, automatic protection devices, communication facilities and compensation devices, etc., so the substation stores instruments and equipment representing various parameters, and uses patrol robots for intelligent patrol instead of manual collection of instrument parameter images, and uses visual algorithms to parse the images to obtain indications and status for reference by staff; since the coal mine site is relatively complex and it is very difficult to go down the mine most of the time, the patrol robot cannot be visually debugged before formal operation, so it is necessary to perform visual debugging as much as possible before the robot enters the site, and the construction conditions are to imitate the coal mine underground substation, including the design of a substation cabinet with the same proportion of length, width and height, and wrenches, switches, etc. are embedded in the upper layer of the substation cabinet.

[0088] like Figure 3 As shown, S2: intelligent inspection and pan / tilt setting of the substation intelligent inspection robot, using the inspection robot to fix the horizontal position of the instrument, to implement image acquisition and visual recognition tasks on the substation cabinet, specifically:

[0089] It should be noted that in the present application example, the substation intelligent inspection robot is composed of a mobile carrier, communication equipment and detection equipment, etc., and adopts a remote control or fully autonomous operation mode, and is a mobile inspection device for substation equipment inspection operations; the robot is equipped with a Hikvision camera as a pan / tilt, dynamically collects instrument images, uses a visual algorithm to analyze and calculate the instrument readings and status on the substation cabinet, and returns the results to the host computer and displays them, the pan / tilt is set to set the status of the pan / tilt at each point, the pan / tilt lifting height is a horizontal shooting target, and the pan / tilt focal length is the content that is clearly visible to the human eye;

[0090] S2-1: Prepare the inspection robot and debug the host computer program, specifically: prepare a wheeled inspection robot, which includes autonomous navigation, intelligent inspection and other functions to ensure that it can automatically or remotely inspect substation equipment; the robot host computer program is debugged, including the settings of robot control, data processing and visual recognition interface functions;

[0091] S2-2: Set the inspection robot's position and pan / tilt adjustment, obtain the position path map by analyzing and planning the position of each point, control the inspection robot to collect images of each point according to the point path map, adjust the up and down height and focal length of the pan / tilt, so that the camera can aim at the switch position of the instrument for horizontal shooting, ensure that the shooting content is within the range that the human eye can clearly identify, and the wide angle of the image can cover the left and right sides of the cabinet; the specific process of point analysis and route planning is as follows:

[0092] Obtain the layout diagram of the substation, mark the points where image acquisition is required on the layout diagram, and determine the inspection points. It should be noted that each inspection point corresponds to one or more targets (such as switches, knobs, instruments, etc.) that require image acquisition;

[0093] Obtain the number of targets corresponding to each point and the type of each target; set an important coefficient corresponding to different targets. It should be noted that different types of targets (such as main switches, circuit breakers and other equipment in substations) have different importance. In particular, when they fail, they will have different degrees of impact on the operation of the entire power system. The corresponding important coefficients should be different. The specific important coefficient values ​​are set by personnel in this field according to the criticality of the equipment. For example, the important coefficient values ​​range from 1 to 10. The important coefficients of equipment such as main switches and circuit breakers can be set to be higher, while the weight values ​​of some auxiliary equipment can be lower. Extract the number of targets at each point and the type of each target, and compare them with all the set target types to match the corresponding important coefficients, and record them as Yi, where i = 1, 2, 3...I, I is a positive integer, I represents the total number of targets at the point, and i represents the number of any one of the targets corresponding to the point;

[0094] Set the image acquisition of different types of targets to correspond to a quality coefficient respectively. It should be noted that the image acquisition quality requirements of different targets are different. For devices that need to accurately identify details (such as the angle of the knob, etc.), the acquired images are required to have high resolution and clarity, which is helpful for subsequent image analysis (such as calculating the angle of the knob); for example, a knob switch requires a high-precision image to fit the angle of the knob, and judge which gear the current switch is in by the angle, which requires a high image clarity to accurately capture the shape and relative position of the knob; extract each target type at the point, and compare it with all the set target types to match the corresponding quality coefficient, and record it as Li;

[0095] The historical failure times of each target object corresponding to the point are obtained and recorded as Gi; the important coefficient Yi, quality coefficient Li and historical failure times Gi of each target object at the point are normalized and their values ​​are taken, and the values ​​are calculated and analyzed by formula to obtain the target value YG of the point. The specific calculation formula is:

[0096]

[0097] Among them, λ1, λ2, and λ3 are respectively set proportional constants, and their values ​​are set by the personnel in this field according to actual needs; for example, the value of λ1 is 1.531, the value of λ2 is 1.082, and the value of λ3 is 1.294; It can be seen from the formula that when the target objects at the point have higher requirements for the quality of the collected image, the larger the target value; when the importance of the target objects at the point to the substation is greater, or the risk of its failure is greater (the more failures), the larger the target value; thus, the target value of each point can be obtained;

[0098] Mark the corresponding points of each target value YG in the layout diagram of the substation one by one to obtain the starting point of the inspection robot (usually the starting point is the initial position of the inspection robot entering the coal mine. Specifically, in this simulated substation, the starting point of the inspection robot is set by personnel in this field); therefore, the constraint conditions of this path planning are: starting point-each point and its corresponding target value-starting point;

[0099] Let D be a subset, where D contains all visited points (including the starting point), where n and m are any two points in the point set M, and let dp[S][n] be the minimum path length from the starting point, through all points in the point set D, and finally to point n;

[0100] Starting point: Set the starting point to 0, which is the initial position of the inspection robot;

[0101] The core of dynamic programming is the state transfer equation, which sets dp[D][n] as the minimum path length from the starting point, through all points in the point set D, and finally to point n;

[0102]

[0103] Among them, dp[D\{n}][m] represents the minimum path length to point m after removing point n from point set D;

[0104] Initialization: dp[0][0] = 0, indicating that the path length from the starting point to itself is 0; for other dp[0][n], it can be set to a very large value, because other points have not been traversed in the initial state;

[0105] Calculate the shortest paths containing different point subsets in sequence until all points are traversed. In each round of calculation, select all possible points n and m, and use the state transfer equation to gradually optimize the path value.

[0106] For each point n, its corresponding target value is recorded as YGn. A weighting factor is added to the path planning so that points with larger target values ​​are visited more preferentially. The specific weighted path calculation formula is:

[0107]

[0108] Where γ is a set weighting factor, which is used to adjust the degree of priority selection of points with large target values. Its value is set by personnel in this field. In this way, points with large target values ​​are visited first;

[0109] The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is:

[0110]

[0111] Where dp[All][n] represents the shortest path value that visits all points and finally stops at point n, and dn0 represents the path length from point n back to the starting point;

[0112] According to the above dynamic path planning, an optimal point path map can be obtained, and the points with large target values ​​in the path will be visited first; the inspection robot is controlled to collect images of each point accordingly, ensuring that each predetermined point can be visited by the inspection robot and image collection is performed, reducing redundant paths, optimizing inspection routes, shortening inspection time, and giving priority to equipment with higher image quality requirements and greater failure risks, so as to achieve the best inspection efficiency and quality;

[0113] S2-3: Call the host computer interface and visual recognition service, specifically: use the host computer interface to actively initiate a request, send the parameters set by the PTZ (including the angle and focal length of the PTZ), and the target type to be identified (such as wrench switch, knob switch, etc.) to the visual recognition service; the visual recognition service parses and captures the image taken by the inspection robot at the specified point based on the received parameters. Get the real-time camera image by calling the RTSP stream (usually using the RTSP stream protocol of the Hikvision camera); use OpenCV's VideoCapture function to connect the camera and obtain the image data of the current PTZ point. This function can be used to capture video streams and support real-time image acquisition and processing of the camera;

[0114] It should be noted that the interface call is an active request interface, which is initiated by the host computer. It is suitable for business scenarios where a single call recognition task needs to be recognized multiple times, and the returned result array contains all the single recognition results. The interface request method is http.post, and the return format is json; the rtsp stream acquisition method used to obtain the image, the robot pan-tilt generally uses the Hikvision camera rtsp stream acquisition. Use Opencv's VideoCapture to obtain the pan-tilt point image. VideoCapture is an API (program application interface) used to open video files, network videos, and video capture devices (cameras).

[0115] like Figure 4 As shown, S3: the inspection robot is automatically coordinated with the visual recognition service to complete the automatic inspection task of the point, and the image information is obtained and analyzed through the visual recognition technology; the specific process includes:

[0116] S3-1: Complete the setting of inspection points on the host computer interface and configure these points as automatic inspection tasks, which means that the robot will automatically move and inspect according to the set points during the inspection process;

[0117] S3-2: Whenever the inspection robot reaches a preset point, the host computer will automatically trigger the start of the visual recognition service. The host computer sends the camera's RTSP stream (for real-time video acquisition) and the current point information to the visual recognition service to provide necessary data support for subsequent image acquisition and analysis;

[0118] S3-3: After receiving the request from the host computer, the visual recognition service will parse the parameters transmitted in the interface, including RTSP stream and point information; by parsing the parameters transmitted by the host computer interface, the recognition service can locate the current inspection position of the robot and prepare to perform the corresponding image processing task;

[0119] S3-4: Through the RTSP stream of the robot's pan-tilt camera, the visual recognition service uses the screenshot function (RTSP screenshot) to collect the image of the current point. It should be noted that this image is a real-time image based on the current inspection task, which includes the status and indications of instruments.

[0120] like Figure 5 As shown, S4: Positioning and type recognition of wrenches and switches based on the YOLOv5 target detection model, specifically:

[0121] S4-1: Prepare an image dataset containing wrench switches and knob switches, add labels to the images in the dataset, mark the type of each object (wrench switch or knob switch), and their location in the image; divide the dataset into training set, validation set, and test set, and convert them into the format required by the YOLOv5 model;

[0122] S4-2: Preprocess the data, such as adjusting the image size, normalization, data enhancement (such as Mosaic enhancement, random scaling, cropping, etc.) to improve the model's ability to detect small targets and reduce overfitting; select YOLOv5 as the target detection model and design the network. The network structure of YOLOv5 includes three main parts: Backbone (feature extraction), Neck (feature fusion), and Output (prediction box generation); during the training process, the model calculates the prediction box through forward propagation, uses the loss function to calculate the deviation, and updates the weights through back propagation to minimize the loss value until the stopping condition is met (until the model converges); the YOLOv5 model uses an adaptive anchor box calculation method to dynamically adjust the size of the anchor box to more accurately predict the target box. At the same time, the adaptive image scaling technology is used to reduce the amount of calculation and improve the detection speed; test the trained model on the evaluation data set, and calculate and analyze the performance indicators of the model, such as accuracy, recall rate, and F1 value;

[0123] Furthermore, in this embodiment, the following formula is used to calculate the model evaluation index:

[0124]

[0125] mAP = mean(AP 扳手开关 +AP 旋钮钮开 )

[0126] Among them, TP, TN, FP, and FN are as follows:

[0127]

[0128] TP (True Positive): correctly detected targets, TN (True Negative): correctly identified as non-target areas, FP (False Positive): incorrectly identified as targets, and FN (False Negative): missed target areas. Based on the evaluation results, analyze the model errors, discuss the problems, and adjust the model or data accordingly.

[0129] S4-3: Obtain the video stream through the IP or camera serial number, and preprocess the image (such as resizing, normalization, etc.) to adapt it to the input requirements of the YOLOv5 model, input the preprocessed image into the trained YOLOv5 model, perform target detection, and identify the position, category and confidence of the wrench switch and knob switch in the image; draw the position box, classification and confidence of the detected wrench switch or knob switch in a new image and display it; if the detection result is a wrench switch, execute step S5 (analyze the wrench switch state); if it is a knob switch, execute step S6 (analyze the knob switch state).

[0130] like Figure 6 As shown, S5: Based on the detection result of the target detection model, the state of the wrench switch (on or off) is determined by calculating the vertical height and horizontal length of the wrench target frame, specifically:

[0131] S5-1: Using the result outputted from the above step S4-3, including the target type and the coordinates of the upper left point and the lower right point of the target box, calculate the height of the box in the vertical direction (i.e., the longitudinal length of the box). Similarly, calculate the length of the target box in the horizontal direction (i.e., the lateral length of the box);

[0132] S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is "off", if the length is less than the height, the output result is "on".

[0133] S6: Determine the state of the knob switch through image preprocessing and straight line fitting, specifically:

[0134] S6-1: According to the coordinates of the upper left point and the lower right point of the target frame output in step S4-3, the target image including the knob switch is cut out from the original image;

[0135] S6-2: convert the captured image into a grayscale image, use binarization or other image preprocessing methods to separate the switch base and the knob switch, and extract the knob part from the base;

[0136] S6-3: Filter out the black knob switch points in the image (usually the knob handle), and use the fitLine function in Opencv to fit a straight line; the fitLine function fits a straight line by the least squares method so that the distance between all points and the straight line is minimized; the parameters of the fitted straight line (slope k and intercept b) can be expressed by the point slope formula, and the following formula is used to describe the fitted straight line:

[0137]

[0138] Among them, (vx,vy) is a normalized vector collinear with the line, and (x0,y0) is an arbitrary point on the fitted line.

[0139] S6-4: By calculating the slope k of the fitting line, further calculate the angle between the line and the y-axis (usually calculated using the inverse tangent function atan), and based on the slope k and the angle, determine the current position of the knob switch (for example, horizontal, vertical or other angles), thereby inferring the state of the knob switch.

[0140] S7: The results obtained through visual recognition are saved as a dictionary type and uploaded to the host computer through the interface. Finally, the host computer parses and displays the results; specifically:

[0141] S7-1: After completing target detection and state judgment, the visual recognition service saves all recognition results (such as wrench state, switch type and state, knob switch position, etc.) as a dictionary-type data structure. The dictionary usually includes information such as the type, position (coordinates), and state of the target;

[0142] S7-2: After receiving the dictionary type recognition result, the host computer parses the result; the parsing process includes reading various data in the dictionary and visually displaying them on the host computer interface, such as displaying the switch status of the wrench, the gear position of the knob, the position of the switch and other information.

[0143] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for identifying a wrench switch based on a target detection model, characterized in that: The following steps are involved: S1: Build a simulation environment to simulate the substation scene in a coal mine; S2: Intelligent inspection and pan / tilt setting of the substation intelligent inspection robot. The inspection robot is fixed at the horizontal position of the instrument to perform image acquisition and visual recognition tasks on the substation cabinet; S3: Automatically coordinate the inspection robot with the visual recognition service to complete the automatic inspection task of points, and obtain and analyze image information through visual recognition technology; S4: Positioning and identifying wrenches and switches based on the YOLOv5 target detection model; S5: Based on the detection result of the target detection model, the state of the wrench switch is determined by calculating the vertical height and horizontal length of the wrench target frame; S6: judging the state of the knob switch through image preprocessing and straight line fitting; S7: The results obtained through visual recognition are saved as a dictionary type and uploaded to the host computer through the interface. Finally, the host computer parses and displays the results.

2. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S1 is as follows: S1-1: Create an experimental site that matches the environment of a coal mine underground substation. Combined with the experience of coal mine underground substation scenarios, it is determined that switch instruments are usually distributed on the upper layer of the substation cabinet; S1-2: Design a scenario that includes a substation cabinet and related instruments, and embed wrench switches and knob switch devices; S1-3: According to the layout of the underground substation in the coal mine, the substation cabinets need to be neatly arranged in the venue, and can be placed on one side or both sides to simulate the real environment of the underground coal mine.

3. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S2 is as follows: S2-1: Prepare the inspection robot and debug the host computer program. The inspection robot includes autonomous navigation and intelligent inspection functions. The host computer function debugging includes robot control, data processing and visual recognition interface; S2-2: Set the inspection robot's position and pan / tilt adjustment, obtain the position path map by analyzing and planning the position of each point, and control the inspection robot to collect images of each point according to the point path map; adjust the up and down height and focal length of the pan / tilt so that the camera can aim at the switch position of the instrument for horizontal shooting; S2-3: Call the host computer interface and visual recognition service.

4. The method for identifying a wrench switch based on a target detection model according to claim 3, characterized in that: The point path map is obtained by analyzing and planning the location of each point. The specific analysis and planning process is as follows: 401: Obtain a layout diagram of the substation, mark the points where image collection is required on the layout diagram, and determine the inspection points; 402: Obtain the number of targets corresponding to each point and the type of each target, and based on this, conduct a comprehensive analysis on the importance of target collection at each point to obtain a target value of each point with respect to the target; 403: Mark the points corresponding to each target value in the layout diagram of the substation one by one to obtain the starting point of the inspection robot; therefore, the constraint conditions of this path planning are: starting point - each point and its corresponding target value - starting point, and perform dynamic path planning based on the constraint conditions to obtain a point path diagram; 404: Control the inspection robot to collect images of each point according to the point path map.

5. The method for identifying a wrench switch based on a target detection model according to claim 4, characterized in that: The specific process of comprehensive analysis of the importance of collecting targets at each point is as follows: Obtain the number of targets corresponding to each point and the type of each target; set different targets to correspond to an important coefficient respectively; extract the number of targets at each point and the type of each target, and compare them with all the set target types to match the corresponding important coefficient; Set the image acquisition of different types of targets to correspond to a quality coefficient respectively, extract each target type at the point, and compare it with all the set target types to match the corresponding quality coefficient; Get the historical failure times of each target object corresponding to the point; The important coefficients, quality coefficients and historical failure times of each target object at the point are normalized and their values ​​are taken. The values ​​are calculated and analyzed in a formula to obtain the target value of the point.

6. The method for identifying a wrench switch based on a target detection model according to claim 5, characterized in that: The specific process of dynamic path planning based on constraints is as follows: Let D be a subset, where D contains all the points visited, where n and m are any two points in the point set M, and let dp[S][n] be the minimum path length from the starting point, through all the points in the point set D, and finally to the point n; Starting point: Set the starting point to 0, which is the initial position of the inspection robot; The core of dynamic programming is the state transfer equation, which sets dp[D][n] as the minimum path length from the starting point, through all points in the point set D, and finally to point n; Among them, dp[D\{n}][m] represents the minimum path length to point m after removing point n from point set D; Initialization: dp[0][0] = 0, indicating that the path length from the starting point to itself is 0; Calculate the shortest paths containing different point subsets in sequence until all points are traversed. In each round of calculation, select all points n and m, and use the state transfer equation to gradually optimize the path value. For any point n, its corresponding target value is recorded as YGn. A weighting factor is added to the path planning. The specific weighted path calculation formula is: Where γ is the set weighting factor; The ultimate goal is to calculate the shortest path length back to the starting point after visiting all points. The specific calculation formula is: Where dp[All][n] represents the shortest path value that visits all points and finally stops at point n, and dn0 represents the length of the path from point n back to the starting point.

7. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S3 is as follows: S3-1: Complete the setting of inspection points on the host computer interface and configure these points as automatic inspection tasks; S3-2: Whenever the inspection robot reaches a preset point, the host computer automatically triggers the start of the visual recognition service, and the host computer sends the camera's RTSP stream and the current point information to the visual recognition service; S3-3: The visual recognition service parses the parameters transmitted by the host computer interface; S3-4: Through the RTSP stream of the robot's gimbal camera, the visual recognition service uses the screenshot function to collect the image of the current point.

8. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S4 is as follows: S4-1: Arrange the wrench switch and knob switch datasets, label them, split the datasets, and generate the required format; S4-2: Use the data set to train the model. The model training part includes data preprocessing and data enhancement, model network design, training process and post-processing process and model evaluation; S4-3: Apply the model to the newly acquired image. The model application part includes image acquisition, image pre-processing, model input detection and result display. If the result type is a wrench switch, execute step S5. If the result type is a knob switch, execute step S6.

9. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S5 is as follows: S5-1: Using the result outputted from step S4-3, including the target type and the coordinates of the upper left point and the lower right point of the target box, calculate the vertical height and horizontal length of the box; S5-2: By comparing the vertical height with the horizontal length, if the length is greater than the height, the output result is off; if the length is less than the height, the output result is on.

10. The method for identifying a wrench switch based on a target detection model according to claim 1, characterized in that: The process of step S6 is as follows: S6-1: According to the coordinates of the upper left point and the lower right point of the target frame output in step S4-3, the target image including the knob switch is cut out from the original image; S6-2: convert the captured image into a grayscale image, use binarization or other image preprocessing methods to separate the switch base and the knob switch, and extract the knob part from the base; S6-3: Filter out the black knob switch points in the image and use the fitLine function in Opencv to fit a straight line; the fitLine function uses the least squares method to fit a straight line so that the distance between all points and the straight line is the minimum; S6-4: by calculating the slope of the fitting line, further calculating the angle between the straight line and the y-axis, judging the current gear position of the knob switch according to the slope and the angle, thereby inferring the state of the knob switch; The process of step S7 is as follows: S7-1: The visual recognition service saves the recognition results as a dictionary type and uploads them to the host computer through the interface; S7-2: The host computer analyzes the results and displays them.

Images