Valve position abnormity identification method based on deep learning

Abstract

The invention provides a valve position abnormity identification method based on deep learning, and the method comprises the steps: obtaining the images of various valve handles at the vehicle bottom through a subway vehicle bottom inspection robot, carrying out the detection of specific key points of the valve handles through a key point detection network based on a deep neural network, and calculating a rotation angle. And then, the system compares the handle angle difference between the task image and the template image, compares the handle angle difference with a set threshold value to judge whether the position of the valve is abnormal or not, and triggers an alarm if necessary. According to the method, deep learning and computer vision technologies are combined, efficient and accurate valve state detection can be achieved in a complex environment, and the intelligent level of subway operation and maintenance is improved.

Description

Technical Field

[0001] The present invention relates to the field of valve handle detection, and in particular to a method for identifying valve position anomalies based on deep learning. Background Art

[0002] During the long-term operation of subway trains, various valves under the train (such as air cylinder valves, air valves, oil valves, and electrical valves) must be maintained in the correct open and close states to ensure safe and stable operation. Currently, subway maintenance relies on manual inspections, which primarily confirm valve status through visual inspection and manual recording. However, the complex undercarriage environment and confined working space make manual inspections inefficient, labor-intensive, and prone to omissions, making them difficult to meet the needs of efficient and accurate operation and maintenance. Therefore, automated intelligent inspection based on subway undercarriage inspection robots has become an important development direction.

[0003] The rotation angle of a valve handle is crucial for determining whether its status is abnormal. Due to the wide variety of valves installed at varying angles under subway cars, traditional object detection methods struggle to accurately identify the rotational state of valve handles. Summary of the Invention

[0004] In order to solve the problems of the existing technology, the present invention provides a valve position anomaly identification method based on deep learning. By combining deep learning and computer vision technology, it can realize efficient and accurate valve status detection in complex environments, thereby improving the intelligent level of subway operation and maintenance.

[0005] The present invention provides a method for identifying valve position abnormality based on deep learning, comprising the following steps:

[0006] Step 1) Data acquisition and preprocessing: The vehicle inspection robot acquires images of the valve handle, including template images and task images, and preprocesses the acquired images;

[0007] Step 2) Extracting key points of the valve handle in the template image and the task image using a key point detection network: Using the key point detection network to perform neural network training and inference, identify the coordinates of specific key points in the template image and the task image, wherein the key points include the rotation axis point and the valve handle endpoint of the valve handle;

[0008] Step 3) Valve handle angle calculation and abnormality judgment: Fit the key points of the valve handle in the task diagram and the template diagram into straight lines respectively, and finally judge whether the position of the valve handle is abnormal by comparing the angle between the two straight lines and the set threshold range.

[0009] Further improvement, the data collection and preprocessing process in step 1) is as follows:

[0010] Step 1.1) Template image acquisition: The inspection system pre-collects a standard undercarriage image of a subway vehicle in normal operation and stores it as a template image;

[0011] Step 1.2) Task map acquisition: When performing an inspection task, the underbody inspection robot collects the current vehicle underbody image in real time and performs image registration processing;

[0012] Step 1.3) Preprocess the collected images, including denoising and distortion correction.

[0013] Further improvement, the image registration process in step 1.2) is specifically as follows: obtaining the homography transformation matrix between the two images to ensure that subsequent key point detection is performed in a unified coordinate system. The homography transformation matrix H defines the projective transformation relationship from the template image to the task image:

[0014]

[0015] in:

[0016]

[0017] (x,y) is the pixel coordinate in the template image, (x',y') is the corresponding pixel coordinate in the task image, and w is the supplementary variable that converts the non-homogeneous coordinate to the homogeneous coordinate.

[0018] Further improvement, the existing door handle key point extraction process in step 2) is as follows:

[0019] Step 2.1) Valve handle dataset construction:

[0020] Step 2.11) Valve handle image acquisition: RGB color images of different types of valve handles on the bottom of the subway vehicle are acquired;

[0021] Step 2.12) Dataset Annotation: Use a dataset annotation tool to annotate key points of the collected image data. Based on the appearance characteristics of different valve handles, select specific locations of the valve handles as key points, including rotation axis points, valve handle endpoints, or other auxiliary feature points with obvious characteristics;

[0022] Step 2.2) Keypoint detection model training: Use a deep learning keypoint detection network to train a model to extract specific keypoints of the valve handle;

[0023] Step 2.3) Keypoint detection reasoning;

[0024] Step 2.31) Input the task graph, use the trained keypoint detection model, output the heatmap and parse the keypoint coordinates:

[0025] Step 2.32) Post-processing: Use non-maximum suppression (NMS) to eliminate redundant key points. If the key point spacing exceeds a reasonable range, combine prior knowledge to remove outliers. The output key point coordinates are used for valve handle angle calculation and anomaly judgment to ensure the final detection accuracy.

[0026] As a further improvement, in the process of extracting the door handle key points in step 2), settings are made according to different types of valve handles, and a multi-scale feature extraction method is combined to improve the adaptability to valve handles of different sizes and shapes.

[0027] Further improvement, the key point detection model training in step 2.2) follows the following principles:

[0028] Step 2.21) Network Model Architecture: Use a convolutional neural network or an attention-based Transformer network as the image feature extraction network to represent the high-level semantic features of the image according to a certain data structure for subsequent processing;

[0029] Step 2.22) Key point characterization method: Characterize key points in the form of heat maps, and generate a Gaussian distribution heat map for each key point Its objective function is Where (x t ,y t ) is the coordinate of the t-th key point, and σ controls the response range;

[0030] Step 2.23) For any key point detection network, its mapping relationship can be uniformly described as:

[0031] P=Φ(I;Θ)={(x t ,y t ,s t )|t=1,…,T},

[0032] Where: I∈R H×W×3 is the input image, H×W×3 represents the width, height and number of channels of the image respectively, Θ is the network parameter, (x t ,y t )∈[0,W]×[0,H] is the coordinate of the t-th key point, s t ∈[0,1] is the confidence score.

[0033] Further improvement, step 3) of calculating the valve handle angle and judging abnormality is specifically as follows:

[0034] 3.1) Key point straight line fitting: Based on the key point coordinates detected in step 2), the slope of the straight line is calculated using the two-point straight line equation:

[0035] Assume the coordinates of the rotation axis are (x1, y1) and the coordinates of the endpoints are (x2, y2). Then the straight line equation of the handle satisfies:

[0036]

[0037] The slope k of the straight line is directly calculated as:

[0038] (If x2≠x1)

[0039] If x2 = x1, that is, the lines are perpendicular, then the slope is infinite and requires special treatment;

[0040] 3.2) Angle calculation: Assumption: The slope of the straight line of the valve handle in the task diagram is k t , the slope of the valve handle straight line in the template diagram is k m , the angle θ between the two straight lines is calculated by the following formula:

[0041]

[0042] If the straight lines of the task graph or template graph are vertical, that is, x2=x1, which leads to k being infinite, you can use

[0043]

[0044] If the two lines are parallel, that is, k t =k m , then θ=0°.

[0045] 3.3) Valve handle abnormality judgment: According to industry standards or equipment requirements, set the allowable angle deviation threshold θ threshold , judgment logic:

[0046] If θ≤θ threshold , the valve handle is in normal condition;

[0047] If θ>θ threshold , the valve handle position is abnormal.

[0048] The beneficial effects of the present invention are:

[0049] 1. Use a key point detection network (including but not limited to the hourglass network and RSN network) to accurately locate the specific key points of the valve handle, and calculate the rotation angle of the valve handle based on the key points to identify its state.

[0050] 2. Able to adapt to valves of different shapes, angles and lighting conditions to improve the robustness of detection.

[0051] 3. By calculating the handle angle difference between the task image and the template image and comparing it with the set threshold, the system can automatically determine whether there is any abnormality in the valve and trigger an alarm.

[0052] 4. This invention combines key point detection technology with deep learning to achieve high-precision position recognition and anomaly detection for valve handles on subway trains. Compared to traditional manual inspection methods, this invention significantly improves detection efficiency and accuracy, providing reliable technical support for intelligent maintenance of subway trains. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 Execute flowchart for software;

[0055] Figure 2 This is a schematic diagram of the valve handle on the bottom of a subway vehicle. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The purpose of the present invention is to provide a valve handle position abnormality recognition system based on deep learning. The system obtains images of various valve handles under the subway car through the inspection robot (such as Figure 2 (as shown in the figure), a keypoint detection network based on a deep neural network detects specific key points on the valve handle and calculates the rotation angle. The system then compares the handle angle difference between the task image and the template image and compares it with a set threshold to determine whether the valve position is abnormal and trigger an alarm if necessary. This method, combining deep learning and computer vision technologies, can achieve efficient and accurate valve status detection in complex environments, improving the intelligent level of subway operations and maintenance.

[0058] The technical solution adopted by the present invention is specifically as follows Figure 1 Steps to implement:

[0059] Step 1: Data collection and preprocessing: The undercar inspection robot obtains images of each valve handle under the subway vehicle and performs preprocessing such as denoising and distortion correction on the collected images. The main purpose is to enhance image quality and highlight details.

[0060] Step 2: A key point detection network extracts key points of the valve handle. This network (including but not limited to hourglass networks and RSN networks) is used for neural network training and inference to identify specific key points for each valve handle. These key points are typically set based on the type of valve handle, such as the handle's endpoints and rotation axis. This is combined with multi-scale feature extraction methods to improve adaptability to valve handles of varying sizes and shapes.

[0061] Step 3: Calculate the valve handle angle and judge abnormalities. In step 2, key point detection is performed on the valve handles of the task diagram and the template diagram respectively, and the straight lines where the valve handles are located in the task diagram and the template diagram are fitted respectively. Finally, the angle between the two straight lines is compared with the set threshold range to judge whether the position of the valve handle is abnormal.

[0062] A more specific plan is as follows:

[0063] Step 1) Data collection and preprocessing:

[0064] Step 1.1) Data Collection:

[0065] Template image collection: The inspection system collects standard vehicle bottom images of subway vehicles in normal operation in advance and stores them as template images.

[0066] Task map acquisition: When performing inspection tasks, the underbody inspection robot collects the current vehicle underbody image in real time. Since the inspection environment may have problems such as camera angle offset and lighting changes, there may be geometric differences in perspective, scale, rotation, etc. between the task map and the template map, so image registration processing is required.

[0067] Step 1.2) Alignment of the task graph and the template graph:

[0068] The task image and the template image may have changes in angle, scale, and perspective. Therefore, they need to be aligned using an image registration algorithm to obtain the homography matrix between the two images to ensure that subsequent key point detection is performed in a unified coordinate system. The homography matrix H defines the projective transformation relationship from the template image to the task image:

[0069]

[0070] in:

[0071]

[0072] (x,y) is the pixel coordinate in the template image, (x',y') is the corresponding pixel coordinate in the task image, and w is the supplementary variable that converts the non-homogeneous coordinate to the homogeneous coordinate.

[0073] Step 1.3) Task graph preprocessing:

[0074] The task image is preprocessed by denoising, distortion correction, etc., the main purpose of which is to enhance the image quality and highlight the details.

[0075] Step 2) The key point detection network extracts the key points of the valve handle:

[0076] Step 2.1) Valve handle dataset construction:

[0077] Valve handle image acquisition: We collected RGB color images of various valve handles on the underside of subway vehicles. We focused on capturing images of various valve models, rotation angles, and wear levels. We also adjusted the camera position to capture images of valve handles from different perspectives, angles, and lighting conditions to enhance the diversity of the dataset.

[0078] Dataset Annotation: Use dataset annotation tools to annotate key points in the captured image data. Based on the appearance characteristics of different valve handles, select specific locations on the valve handles as key points. Typically, these locations are the rotation axis, the end points of the valve handle, and some other significant auxiliary feature points.

[0079] Step 2.2) Keypoint detection model training

[0080] The keypoint detection task is essentially a regression problem: given an image input, the network needs to predict the coordinate positions or heatmap distribution of keypoints. This paper uses a deep learning Keypoint Detection Network training model to extract specific keypoints of the valve handle. Its specific implementation follows the following general design principles:

[0081] Network model architecture: A convolutional neural network or an attention-based Transformer network can be used as an image feature extraction network to represent the high-level semantic features of the image according to a certain data structure for subsequent processing.

[0082] Key point characterization method: Characterize key points in the form of heat maps. Generate a Gaussian distribution heat map for each key point Its objective function is Where (x t ,y t ) is the coordinate of the t-th key point, and σ controls the response range (preferably σ = 2 to 7 pixels).

[0083] For any key point detection network, its mapping relationship can be uniformly described as:

[0084] P=Φ(I;Θ)={(x t ,y t ,s t )|t=1,…,T},

[0085] in:

[0086] I∈R H×W×3 is the input image;

[0087] H×W×3 represents the width, height and number of channels of the image respectively;

[0088] Θ is the network parameter;

[0089] (x t ,y t )∈[0,W]×[0,H] is the coordinate of the tth key point;

[0090] s t ∈[0,1] is the confidence score.

[0091] Step 2.3) Keypoint detection reasoning

[0092] 1. Input the task graph, use the trained key point detection model, output the heat map and parse the key point coordinates:

[0093] 2. Post-processing:

[0094] Non-maximum suppression (NMS) is used to eliminate redundant key points.

[0095] Combined with prior knowledge, outliers are removed (such as key point spacing outside a reasonable range).

[0096] The output key point coordinates will be used for valve handle angle calculation and abnormality judgment to ensure the final detection accuracy. Step 3: Valve handle angle calculation and abnormality judgment

[0097] 1. Key point straight line fitting:

[0098] The slope of the line is calculated using the two-point line equation using the coordinates of the key points detected in step 2.

[0099] Assume the coordinates of the rotation axis are (x1, y1) and the coordinates of the endpoints are (x2, y2). Then the straight line equation of the handle satisfies:

[0100]

[0101] Among them, the slope k of the straight line can be directly calculated as:

[0102] (If x2≠x1)

[0103] If x2 = x1 (i.e. the lines are perpendicular), then the slope is infinite and requires special treatment.

[0104] 2. Angle calculation:

[0105] Assumptions:

[0106] The slope of the valve handle line in the task map (actual inspection image) is k t

[0107] The slope of the valve handle straight line in the template image (reference image under normal state) is k m

[0108] The angle θ between the two straight lines is calculated using the following formula:

[0109]

[0110] Special situation handling:

[0111] If the straight lines of the task graph or template graph are vertical (i.e. x2 = x1, resulting in k being infinite), you can use

[0112]

[0113] If two lines are parallel (i.e. k t =k m ), then θ=0°.

[0114] 3. Judgment of abnormal valve handle:

[0115] According to industry standards or equipment requirements, set the allowable angle deviation threshold θ threshold .

[0116] Judgment logic:

[0117] If θ≤θ threshold , the valve handle is in normal condition.

[0118] If θ>θ threshold , the valve handle position is abnormal, and may be in an open, closed or loose state.

[0119] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, the above is only a preferred embodiment of the present invention. Since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited to this. Any technical personnel familiar with this technical field is within the technical scope disclosed by the present invention. For ordinary technical personnel in this technical field, changes or replacements that can be easily thought of should be covered within the protection scope of the present invention without departing from the principle of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A valve position abnormality recognition method based on deep learning, characterized in that The following steps are involved: Step 1) Data acquisition and preprocessing: The vehicle inspection robot acquires images of the valve handle, including template images and task images, and preprocesses the acquired images; Step 2) Extracting key points of the valve handle in the template image and the task image using a key point detection network: Using the key point detection network to perform neural network training and inference, identify the coordinates of specific key points in the template image and the task image, wherein the key points include the rotation axis point and the valve handle endpoint of the valve handle; Step 3) Valve handle angle calculation and abnormality judgment: Fit the key points of the valve handle in the task diagram and the template diagram into straight lines respectively, and finally judge whether the position of the valve handle is abnormal by comparing the angle between the two straight lines and the set threshold range.

2. The method for identifying abnormal valve position based on deep learning according to claim 1, characterized in that: The data collection and preprocessing process in step 1) is as follows: Step 1.1) Template image acquisition: The inspection system pre-collects a standard undercarriage image of a subway vehicle in normal operation and stores it as a template image; Step 1.2) Task map acquisition: When performing an inspection task, the underbody inspection robot collects the current vehicle underbody image in real time and performs image registration processing; Step 1.3) Preprocess the collected images, including denoising and distortion correction.

3. The method for identifying abnormal valve position based on deep learning according to claim 2, characterized in that: The image registration process in step 1.2) is specifically as follows: obtaining the homography transformation matrix between the two images to ensure that subsequent key point detection is performed in a unified coordinate system. The homography transformation matrix H defines the projective transformation relationship from the template image to the task image: in: (x,y) is the pixel coordinate in the template image, (x',y') is the corresponding pixel coordinate in the task image, and w is the supplementary variable that converts the non-homogeneous coordinate to the homogeneous coordinate.

4. The method for identifying abnormal valve position based on deep learning according to claim 1, characterized in that: The existing door handle key point extraction process described in step 2) is as follows: Step 2.1) Valve handle dataset construction: Step 2.11) Valve handle image acquisition: RGB color images of different types of valve handles on the bottom of the subway vehicle are acquired; Step 2.12) Dataset Annotation: Use a dataset annotation tool to annotate key points of the collected image data. Based on the appearance characteristics of different valve handles, select specific locations of the valve handles as key points, including rotation axis points, valve handle endpoints, or other auxiliary feature points with obvious characteristics; Step 2.2) Keypoint detection model training: Use a deep learning keypoint detection network to train a model to extract specific keypoints of the valve handle; Step 2.3) Keypoint detection reasoning; Step 2.31) Input the task graph, use the trained keypoint detection model, output the heatmap and parse the keypoint coordinates: Step 2.32) Post-processing: Use non-maximum suppression (NMS) to eliminate redundant key points. If the key point spacing exceeds a reasonable range, remove outliers based on prior knowledge. The output key point coordinates are used for valve handle angle calculation and abnormality judgment to ensure the ultimate detection accuracy.

5. The method for identifying abnormal valve position based on deep learning according to claim 1 or 4, characterized in that: In the process of extracting the door handle key points in step 2), settings are made according to different types of valve handles, and a multi-scale feature extraction method is combined to improve the adaptability to valve handles of different sizes and shapes.

6. The method for identifying abnormal valve position based on deep learning according to claim 4, characterized in that: The key point detection model training in step 2.2) follows the following principles: Step 2.21) Network Model Architecture: Use a convolutional neural network or an attention-based Transformer network as the image feature extraction network to represent the high-level semantic features of the image according to a certain data structure for subsequent processing; Step 2.22) Key point characterization method: Characterize key points in the form of heat maps, and generate a Gaussian distribution heat map for each key point Its objective function is Where (x t ,y t ) is the coordinate of the t-th key point, and σ controls the response range; Step 2.23) For any key point detection network, its mapping relationship can be uniformly described as: P=Φ(I;Θ)={(x t ,y t ,s t )|t=1,…,T}, Where: I∈R H×W×3 is the input image, H×W×3 represents the width, height and number of channels of the image respectively, Θ is the network parameter, (x t ,y t )∈[0,W]×[0,H] is the coordinate of the t-th key point, s t ∈[0,1] is the confidence score.

7. The method for identifying abnormal valve position based on deep learning according to claim 1, characterized in that: Step 3) The valve handle angle calculation and abnormality judgment are specifically as follows: 3.1) Key point straight line fitting: Based on the key point coordinates detected in step 2), the slope of the straight line is calculated using the two-point straight line equation: Assume the coordinates of the rotation axis are (x1, y1) and the coordinates of the endpoints are (x2, y2). Then the straight line equation of the handle satisfies: The slope k of the straight line is directly calculated as: If x2 = x1, that is, the lines are perpendicular, then the slope is infinite and requires special treatment; 3.2) Angle calculation: Assumption: The slope of the straight line of the valve handle in the task diagram is k t , the slope of the valve handle straight line in the template diagram is k m , the angle θ between the two straight lines is calculated by the following formula: If the straight lines of the task graph or template graph are vertical, that is, x2=x1, which leads to k being infinite, use: If the two lines are parallel, that is, k t =k m , then θ=0°; 3.3) Valve handle abnormality judgment: According to industry standards or equipment requirements, set the allowable angle deviation threshold θ threshold , judgment logic: If θ≤θ threshold , the valve handle is in normal condition; If θ>θ threshold , the valve handle position is abnormal.

Images