Track health detection method and system based on 3D vision
By utilizing deployed 3D cameras to acquire three-dimensional images, the system automatically detects the lifting and tilting of the track, solving the problem of increased cost and complexity of existing equipment and achieving efficient track health detection.
Patent Information
- Application Number
- CN202511175169.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-21
- Publication Date
- 2025-11-28
AI Technical Summary
Existing methods for monitoring orbital health require additional equipment, increasing costs and complexity.
By using deployed 3D cameras to acquire 3D images, and through image pixel block recognition and feature vector analysis, the elevation and tilt of the track can be automatically detected, reducing reliance on sensors or 2D cameras.
It improved equipment utilization, reduced additional costs, simplified the inspection process, and enabled automated track health status analysis.
Smart Images
Figure CN121032984A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of track inspection technology, and in particular to a track health inspection method and system based on 3D vision. Background Technology
[0002] Introducing three-dimensional (3D) vision technology into the track inspection process, 3D cameras operate synchronously with track equipment. For example, a 3D vision track inspection robot can perform uninterrupted track inspections across the entire route, preventing sudden and random accidents and enabling the detection and monitoring of foreign objects on the track surface. The normal operation of track equipment depends on track health; when the track experiences subsidence, elevation, tilting, or other unhealthy conditions, the normal operation of the track equipment will inevitably be affected.
[0003] Currently, there are two main categories of methods for detecting track health: one uses sensors to directly detect the track's condition, and the other uses 2D cameras to photograph the track and applies computer vision technology to analyze the track's condition from the captured images. Both methods require additional equipment, increasing both cost and complexity. Summary of the Invention
[0004] This application provides a 3D vision-based method and system for track health detection, which solves the technical problems of existing technologies that require additional equipment, increasing implementation costs and usage complexity.
[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a 3D vision-based method for track health detection is provided, comprising: acquiring an image of the target track to be detected and a set of point coordinates constituting each target pixel block in the image to be detected using a 3D camera; comparing the image to be detected with a reference image of the target track in a track health state to determine a reference pixel block in the reference image that has the same image content as the target pixel block; calculating the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates; and calculating the elevation and tilt of the target track based on the plane normal vector of the target pixel block, the camera distance, and the plane normal vector of the reference pixel block and the camera distance to determine the health state of the target track.
[0006] Based on the above technical solution, the 3D vision-based track health detection method provided in this application utilizes 3D images acquired by 3D cameras already deployed on the equipment during track inspection, increasing the utilization rate of the deployed equipment and eliminating the need to deploy sensors or 2D cameras, thus reducing additional costs. Then, by analyzing the track's condition using the 3D data, automatic analysis and detection can be achieved, further reducing the complexity of use.
[0007] In conjunction with the first aspect above, in one possible implementation, the method for comparing the image to be detected with a reference image of the target orbit under orbital health conditions to determine a reference pixel block in the reference image that has the same image content as the target pixel block specifically includes: inputting the image to be detected and the reference image into an image pixel block recognition network, and outputting the feature vector of each pixel block; and determining the reference pixel block corresponding to the target pixel block from the reference image based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image.
[0008] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: acquiring sample images of the target trajectory multiple times using a 3D camera; extracting sample pixel blocks from the sample images and classifying the sample pixel blocks, identifying the classification results; the classification rule is that the image content of pixel blocks of the same type is the same object, and the image content of pixel blocks of different types is different; training an initial recognition network based on the sample images and classification results to obtain an image pixel block recognition network.
[0009] In conjunction with the first aspect above, in one possible implementation, the method for training an initial recognition network based on sample images and classification results to obtain an image pixel block recognition network specifically includes: sampling three groups of pixel blocks from the classified sample pixel blocks; in the three groups of pixel blocks, the first group of pixel blocks and the second group of pixel blocks are of the same class, and the third group of pixel blocks is of different class from both the first group of pixel blocks and the second group of pixel blocks; inputting the three groups of pixel blocks into the initial recognition network to obtain feature vectors for the three groups of pixel blocks; calculating the triplet loss based on the feature vectors of the three groups of pixel blocks; the formula for calculating the triplet loss is: ; This indicates taking the maximum value between A and B; This represents the cosine of the angle between A and B; This is the feature vector of the first group of pixels; This is the feature vector of the second group of pixel blocks; This is the feature vector of the third group of pixels; These are empirical parameters; The triplet loss is used; the triplet loss is minimized using a gradient iterative algorithm until the learnable parameters of the initial recognition network converge, resulting in the image pixel block recognition network; the convergence of learnable parameters is expressed as: These are learnable parameters.
[0010] In conjunction with the first aspect above, in one possible implementation, the method for determining the reference pixel block corresponding to the target pixel block from the reference image based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image specifically includes: calculating the cosine similarity between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; when the cosine similarity corresponding to any pixel block in the reference image is greater than a preset similarity, determining any pixel block as the reference pixel block.
[0011] In conjunction with the first aspect above, in one possible implementation, the method for determining the reference pixel block corresponding to the target pixel block from the reference image based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image specifically includes: calculating the Euclidean distance between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; when the Euclidean distance corresponding to any pixel block in the reference image is less than a preset distance, determining any pixel block as the reference pixel block.
[0012] In conjunction with the first aspect above, in one possible implementation, the method for calculating the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates specifically includes: determining the average of the coordinates of multiple points in the vertical direction in the set of point coordinates as the camera distance; fitting the plane of the target pixel block using the least squares method to obtain the plane equation and calculating the plane normal vector.
[0013] In conjunction with the first aspect above, in one possible implementation, the method for calculating the elevation and tilt of the target track based on the plane normal vector of the target pixel block and the camera distance, as well as the plane normal vector of the reference pixel block and the camera distance, and determining the health status of the target track, specifically includes: calculating the absolute value of the difference between the camera distances between the target pixel block and the reference pixel block, and the similarity of the plane normal vectors between the target pixel block and the reference pixel block; when the absolute value of the difference between the camera distances is greater than a preset displacement threshold, the health status of the target track is output as elevation abnormality; when the similarity of the plane normal vectors is less than a preset tilt threshold, the health status of the target track is output as tilt abnormality.
[0014] In conjunction with the first aspect above, in one possible implementation, the method further includes: sending an alarm message when the health status of the target track is abnormal in elevation or tilt; the alarm message includes: location information of the abnormality on the target track.
[0015] Secondly, a 3D vision-based track health detection system is provided, comprising: a 3D camera, an image recognition module, and a health analysis module; the 3D camera is used to acquire an image of the target track to be detected, and a set of point coordinates constituting each target pixel block in the image to be detected; the image recognition module is used to compare the image to be detected with a reference image of the target track in a track health state, and to determine the reference pixel block in the reference image that has the same image content as the target pixel block; the health analysis module is used to calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates; the health analysis module is also used to calculate the degree of ascent and descent and the degree of tilt of the target track based on the plane normal vector of the target pixel block and the camera distance, and the plane normal vector of the reference pixel block and the camera distance, to determine the health state of the target track.
[0016] Thirdly, this application provides a 3D vision-based track health detection device, comprising: a processor and a storage medium; the storage medium includes instructions, and the processor is configured to execute the instructions to implement the method described in the first aspect and any possible implementation thereof. This 3D vision-based track health detection device can be an electronic device or a chip within an electronic device.
[0017] Fourthly, this application provides a track health detection device based on 3D vision, comprising: a processing unit; the processing unit is configured to acquire an image of a target track to be detected and a set of point coordinates constituting each target pixel block in the image to be detected via a 3D camera; the processing unit is further configured to compare the image to be detected with a reference image of the target track in a track health state, and determine a reference pixel block in the reference image that has the same image content as the target pixel block; the processing unit is further configured to calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates; the processing unit is further configured to calculate the degree of elevation and tilt of the target track based on the plane normal vector of the target pixel block and the camera distance, and the plane normal vector of the reference pixel block and the camera distance, and determine the health state of the target track.
[0018] Fifthly, this application provides a computer-readable storage medium storing instructions that, when executed on a 3D vision-based track health detection device, cause the 3D vision-based track health detection device to perform the methods described in the first aspect and any possible implementation thereof.
[0019] In a sixth aspect, this application provides a computer program product containing instructions that, when the computer program product is run on a 3D vision-based track health detection device, cause the 3D vision-based track health detection device to perform the methods described in the first aspect and any possible implementation thereof.
[0020] This application provides a 3D vision-based method and system for track health inspection. It utilizes 3D images captured by 3D cameras already deployed on the equipment during track inspection, increasing the utilization rate of the deployed equipment and eliminating the need for additional sensors or 2D cameras, thus reducing extra costs. Then, by analyzing the track's condition using the 3D data, automatic analysis and inspection can be achieved, further reducing the complexity of use.
[0021] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description
[0022] Figure 1 A system architecture diagram of a 3D vision-based track health detection system provided in this application embodiment; Figure 2 A schematic flowchart of a 3D vision-based track health detection method provided in an embodiment of this application; Figure 3 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 4 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 5 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 6 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 7 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 8 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 9 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 10 A schematic flowchart of another 3D vision-based track health detection method provided in this application embodiment; Figure 11 A schematic diagram of a track health detection device based on 3D vision provided in an embodiment of this application; Figure 12 This is a schematic diagram of the hardware structure of a track health detection device based on 3D vision, provided in an embodiment of this application. Detailed Implementation
[0023] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.
[0024] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.
[0025] The 3D vision-based track health detection method provided in this application embodiment can be applied to a 3D vision-based track health detection system 100, such as... Figure 1 As shown, the 3D vision-based track health detection system 100 includes: a 3D camera 101, an image recognition module 102, and a health analysis module 103.
[0026] Among them, the 3D camera 101 is used to acquire three-dimensional images of the target track and the coordinates of each point in the image.
[0027] The image recognition module 102 is connected to the 3D camera 101 and is used to extract features from the images acquired by the 3D camera 101.
[0028] The health analysis module 103 is connected to the 3D camera 101 and the image recognition module 102 respectively. It is used to calculate and analyze the point coordinate set collected by the 3D camera 101 and the feature vector extracted by the image recognition module 102, and output the track health status.
[0029] In some implementations, the 3D vision-based track health detection system 100 also includes a model training module 104.
[0030] The model training module 104 is connected to the 3D camera 101 and the image recognition module 102 respectively, and is used to train the model based on the sample images collected by the 3D camera 101. The trained model is applied to the feature extraction of the image recognition module 102.
[0031] Furthermore, the modules in the 3D vision-based track health detection system 100 can be functional modules integrated into a 3D vision track inspection robot. Alternatively, the 3D camera 101 can be deployed on the 3D vision track inspection robot, while the image recognition module 102, health analysis module 103, and model training module 104 can be deployed on a backend cloud or server. This application embodiment does not impose any limitations on this.
[0032] To address the technical problems of existing technologies requiring additional equipment, increasing implementation costs and operational complexity, this application provides a 3D vision-based track health detection method. This method utilizes 3D images captured by 3D cameras already deployed on the equipment during track inspection, increasing the utilization rate of the deployed equipment and eliminating the need for additional sensors or 2D cameras, thus reducing additional costs. Then, by analyzing the track's state through the 3D data, automatic analysis and detection can be achieved, further reducing operational complexity.
[0033] like Figure 2 As shown in the embodiments of this application, the track health detection method based on 3D vision includes: S201. Obtain the image of the target track to be detected by a 3D camera, as well as the set of point coordinates that constitute each target pixel block in the image to be detected.
[0034] In some implementations, the 3D camera is a binocular structure that can simultaneously acquire depth images, use camera intrinsics to map 2D pixel coordinates to 3D world coordinates, and then segment and extract the coordinates of the target pixel block.
[0035] S202. Compare the image to be detected with the reference image of the target track under the track health condition, and determine the reference pixel block in the reference image that has the same image content as the target pixel block.
[0036] Specifically, the reference image contains 3D scan data of the track in a healthy state, including complete point clouds and textures of the track head, web, and bottom. The comparison is used to locate the wear areas in the image to be inspected, which need to be matched to the corresponding positions in the reference image.
[0037] S203. Based on the set of point coordinates, calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera.
[0038] S204. Based on the plane normal vector of the target pixel block and the camera distance, as well as the plane normal vector of the reference pixel block and the camera distance, calculate the degree of elevation and tilt of the target track to determine the health status of the target track.
[0039] Specifically, the plane normal vector is perpendicular to the plane, and by comparison, the degree of plane tilt can be obtained, thus reflecting the degree of tilt of the track in the pixel block.
[0040] The deployment of 3D cameras on track equipment is usually perpendicular to the ground, so the distance between the cameras can reflect the track height, and thus the degree of track elevation can be reflected by comparison.
[0041] Based on the above technical solution, the track health detection method based on 3D vision provided in this application utilizes three-dimensional images acquired by 3D cameras already deployed on the equipment during track inspection, increasing the utilization rate of the deployed equipment and eliminating the need to deploy sensors or 2D cameras, thus reducing additional costs. Then, by analyzing the track's state through the three-dimensional data, automatic analysis and detection can be achieved, further reducing the complexity of use.
[0042] In one possible implementation, combining Figure 2 ,like Figure 3 As shown, the method in S202 described above can be specifically implemented through the following S301 to S302, which are explained in detail below: S301. Input the image to be detected and the reference image into the image pixel block recognition network respectively, and output the feature vector of each pixel block.
[0043] In some implementations, deep learning models (such as convolutional neural networks (CNNs)) or traditional algorithms (such as scale invariant feature transform (SIFT) and speedup robust features (SURF)) are used to extract feature vectors from images.
[0044] In some implementations, pre-trained models can be used to extract features from fully connected or convolutional layers. Alternatively, keypoints and descriptors can be extracted and then aggregated into a global feature vector.
[0045] In some implementations, feature vectors can include multimodal features, such as geometric features, texture features, and material features.
[0046] Furthermore, the extracted feature vectors can be normalized to reduce the influence of factors such as lighting and scale.
[0047] S302. Based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image, determine the reference pixel block corresponding to the target pixel block from the reference image.
[0048] In some implementations, cosine similarity, Euclidean distance, Manhattan distance, and other methods can be used to compare the similarity of feature vectors, thereby obtaining a baseline pixel block.
[0049] Based on the above technical solution, the accuracy of image content comparison can be improved by comparing feature vectors, thereby reducing the matching error of the reference pixel block and improving the accuracy of subsequent health status assessment.
[0050] In one possible implementation, combining Figure 3 ,like Figure 4 As shown, the above method also includes the following steps S401 to S403, which are described in detail below: S401. Acquire sample images of the target orbit multiple times using a 3D camera.
[0051] S402. Extract sample pixel blocks from the sample image, classify the sample pixel blocks, and label the classification results.
[0052] The classification rule is as follows: images of the same type of pixel blocks contain the same object, while images of different types of pixel blocks contain different objects.
[0053] In some implementations, the classification results can serve as weakly supervised labels, reducing the cost of manual annotation.
[0054] S403. Train the initial recognition network based on the sample images and classification results to obtain the image pixel block recognition network.
[0055] In some implementations, the initial recognition network can be a deep learning model (such as CNN) or a traditional algorithm (such as SIFT, SURF).
[0056] Based on the above technical solution, an image pixel block recognition network can be trained, enabling the extracted image feature vectors to be used for accurate classification.
[0057] In one possible implementation, combining Figure 4 ,like Figure 5 As shown, the method in S403 above can be specifically implemented through the following S501 to S504, which are explained in detail below: S501. Sample three groups of pixel blocks from the categorized sample pixel blocks.
[0058] Among the three groups of pixel blocks, the first group of pixel blocks and the second group of pixel blocks are of the same type, while the third group of pixel blocks is of a different type from both the first group of pixel blocks and the second group of pixel blocks.
[0059] S502. Input the three sets of pixel blocks into the initial recognition network to obtain the feature vectors of the three sets of pixel blocks.
[0060] S503. Calculate the triplet loss based on the feature vectors of the three sets of pixel blocks.
[0061] The formula for calculating the triplet loss is as follows: ; This indicates taking the maximum value between A and B; This represents the cosine of the angle between A and B; This is the feature vector of the first group of pixels; This is the feature vector of the second group of pixel blocks; This is the feature vector of the third group of pixels; These are empirical parameters; This is the loss of the triplet.
[0062] S504. Minimize the triplet loss using a gradient iteration algorithm until the learnable parameters of the initial recognition network converge, thus obtaining the image pixel block recognition network.
[0063] The convergence of the learnable parameters is expressed as: ; These are learnable parameters.
[0064] Based on the above technical solution, the triplet loss, which clusters similar samples in the feature space and keeps dissimilar samples away, is superior to the direct classification loss and can learn more discriminative features. The triplet loss significantly improves the ability to distinguish similar components in track detection through explicit constraints on the feature space.
[0065] In one possible implementation of the embodiments of this application, combined with Figure 3 ,like Figure 6 As shown, the method in S302 described above can be specifically implemented through the following steps S601 to S602, which are explained in detail below: S601. Calculate the cosine similarity between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image.
[0066] S602. When the cosine similarity of any pixel block in the reference image is greater than the preset similarity, the arbitrary pixel block is determined as the reference pixel block.
[0067] Specifically, cosine similarity measures the consistency of vector directions, and the formula is: .
[0068] If the similarity is higher than the threshold, such as cosine similarity > 0.9, then the image content is considered to be consistent.
[0069] Based on the above technical solution, the relative distribution of features in space can be measured to achieve similarity comparison of image content.
[0070] In one possible implementation of the embodiments of this application, combined with Figure 3 ,like Figure 7 As shown, the method in S302 described above can be implemented through the following steps S701 to S702, which are explained in detail below: S701. Calculate the Euclidean distance between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image.
[0071] S702. When the Euclidean distance corresponding to any pixel block in the reference image is less than the preset distance, the arbitrary pixel block is determined as the reference pixel block.
[0072] Specifically, Euclidean distance measures the distance between vectors; the smaller the distance, the more similar they are. The formula is: .
[0073] A maximum allowable distance needs to be set. If the Euclidean distance is less than 0.1, the image content is considered to be consistent.
[0074] Based on the above technical solution, the geometric distance between vectors can be directly calculated, the absolute difference of features can be captured, and the similarity comparison of image content can be realized.
[0075] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 8 As shown, the method in S203 described above can be implemented through the following steps S801 to S802, which are explained in detail below: S801. The average value of the coordinates of multiple points in the vertical direction in the point coordinate set is determined as the camera distance.
[0076] Specifically, since the deployment of 3D cameras on track equipment is usually perpendicular to the ground, the coordinates in the vertical direction can reflect the track height, and the average z-axis coordinate of each point represents the camera distance of that pixel block.
[0077] S802. Fit the plane of the target pixel block using the least squares method to obtain the plane equation and calculate the plane normal vector.
[0078] Based on the above technical solution, the camera distance and plane normal vector of each pixel block can be obtained, providing a data basis for subsequent comparison.
[0079] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 9 As shown, the method of S204 described above can be specifically implemented through the following steps S901 to S903, which will be explained in detail below: S901. Calculate the absolute value of the difference in camera distance between the target pixel block and the reference pixel block, and the similarity of the plane normal vectors between the target pixel block and the reference pixel block.
[0080] S902. When the absolute value of the difference between camera distances is greater than the preset displacement threshold, the health status of the target track is output as abnormal rise and fall.
[0081] Since the camera distance reflects the vertical height, comparing the height difference with that in a healthy state can accurately reflect the degree of rise or fall of the track.
[0082] S903. When the similarity of the plane normal vectors is less than the preset tilt threshold, the health status of the target orbit is output as tilt anomaly.
[0083] Since the plane normal vector can reflect the plane angle, comparing its similarity with the plane normal vector in a healthy state can accurately reflect the degree of track tilt.
[0084] Based on the above technical solutions, the accuracy of track health status analysis can be improved by comparing camera distances to determine whether track elevation is abnormal and by comparing plane normal vectors to determine whether track tilt is abnormal.
[0085] In one possible implementation of the embodiments of this application, combined with Figure 9 ,like Figure 10 As shown, the above method also includes the following S1001, which will be explained in detail below: S1001. When the health status of the target track is abnormal in elevation or tilt, send an alarm message.
[0086] The alarm messages include: location information of an anomaly occurring on the target orbit.
[0087] Based on the above technical solution, an alarm message can be sent in a timely manner after an anomaly is detected, and the location of the anomaly can be marked in the alarm message, so that maintenance personnel can quickly and accurately reach the location of the anomaly for maintenance.
[0088] The above primarily describes the solutions of the embodiments of this application from the perspective of device implementation. It is understood that each device, such as a 3D vision-based track health detection device, includes at least one of the hardware structures and software modules corresponding to each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0089] This application embodiment can divide the 3D vision-based track health detection device into functional units according to the above method example. For example, each function can be divided into separate functional units, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0090] When using integrated units, Figure 11 A possible structural schematic diagram of the 3D vision-based track health detection device 1100 involved in the above embodiments is shown. The 3D vision-based track health detection device 1100 includes a processing unit 1101, and may also include a communication unit 1102 and a storage unit 1103. Figure 11 The structural diagram shown can be used to illustrate the structure of the 3D vision-based track health detection device involved in the above embodiments.
[0091] when Figure 11 The schematic diagram shown illustrates the structure of the track health detection device based on 3D vision involved in the above embodiments. The processing unit 1101 is used to control and manage the operation of the track health detection device based on 3D vision, the communication unit 1102 is used for the track health detection device based on 3D vision to communicate with other devices, and the storage unit 1103 is used to store the program code and data of the track health detection device based on 3D vision.
[0092] For example, processing unit 1101; The processing unit 1101 is used to acquire the image to be detected of the target track and the set of point coordinates that constitute each target pixel block in the image to be detected through a 3D camera.
[0093] The processing unit 1101 is also used to compare the image to be detected with a reference image of the target track in a healthy track state, and to determine a reference pixel block in the reference image that has the same image content as the target pixel block.
[0094] The processing unit 1101 is also used to calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates.
[0095] The processing unit 1101 is also used to calculate the degree of elevation and tilt of the target track based on the plane normal vector of the target pixel block and the camera distance, as well as the plane normal vector of the reference pixel block and the camera distance, and to determine the health status of the target track.
[0096] In one possible implementation, the processing unit 1101 is specifically used to input the image to be detected and the reference image into the image pixel block recognition network respectively, and output the feature vector of each pixel block; based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image, determine the reference pixel block corresponding to the target pixel block from the reference image.
[0097] In one possible implementation, the processing unit 1101 is further configured to acquire sample images of the target track multiple times using a 3D camera; extract sample pixel blocks from the sample images and classify the sample pixel blocks, identifying the classification results; the classification rule is that the image content of pixel blocks of the same type is the same object, and the image content of pixel blocks of different types is different; and train an initial recognition network based on the sample images and classification results to obtain an image pixel block recognition network.
[0098] In one possible implementation, the processing unit 1101 is specifically used to sample three groups of pixel blocks from the classified sample pixel blocks; in the three groups of pixel blocks, the first group of pixel blocks and the second group of pixel blocks are of the same type, and the third group of pixel blocks is of different type from the first group of pixel blocks and the second group of pixel blocks, respectively; the three groups of pixel blocks are input into the initial recognition network to obtain the feature vectors of the three groups of pixel blocks; the triplet loss is calculated based on the feature vectors of the three groups of pixel blocks; the triplet loss is minimized through a gradient iterative algorithm until the learnable parameters of the initial recognition network converge, thus obtaining the image pixel block recognition network.
[0099] In one possible implementation, the processing unit 1101 is specifically used to calculate the cosine similarity between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; when the cosine similarity corresponding to any pixel block in the reference image is greater than a preset similarity, the arbitrary pixel block is determined as the reference pixel block.
[0100] In one possible implementation, the processing unit 1101 is specifically used to calculate the Euclidean distance between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; when the Euclidean distance corresponding to any pixel block in the reference image is less than a preset distance, the arbitrary pixel block is determined as the reference pixel block.
[0101] In one possible implementation, the processing unit 1101 is specifically used to determine the average of the coordinates of multiple points in the vertical direction in the point coordinate set as the camera distance; to fit the plane of the target pixel block by the least squares method, to obtain the plane equation and to calculate the plane normal vector.
[0102] In one possible implementation, the processing unit 1101 is specifically used to calculate the absolute value of the difference between the camera distances between the target pixel block and the reference pixel block, and the similarity of the plane normal vectors between the target pixel block and the reference pixel block; when the absolute value of the difference between the camera distances is greater than a preset displacement threshold, the health status of the target track is output as abnormal elevation; when the similarity of the plane normal vectors is less than a preset tilt threshold, the health status of the target track is output as abnormal tilt.
[0103] In one possible implementation, the processing unit 1101 is further configured to send an alarm message when the health status of the target track is abnormal in elevation or tilt; the alarm message includes: location information of the abnormality on the target track.
[0104] The processing unit 1101 can be a processor or a controller, and the communication unit 1102 can be a communication interface, transceiver, transceiver circuit, transceiver device, etc. The term "communication interface" is a general term and may include one or more interfaces. The storage unit 1103 can be a memory. When the 3D vision-based track health detection device 1100 is a chip, the processing unit 1101 can be a processor or a controller, and the communication unit 1102 can be an input interface and / or an output interface, pins, or circuits, etc. The storage unit 1103 can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip (e.g., read-only memory (ROM), random access memory (RAM, etc.).
[0105] Figure 11If the integrated units in the process are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. Storage media for storing computer software products include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.
[0106] Figure 11 The units in the process can also be called modules; for example, a processing unit can be called a processing module.
[0107] This application also provides a hardware structure diagram of a track health detection device based on 3D vision, see [link / reference]. Figure 12 The 3D vision-based track health detection device 1200 includes a processor 1201, and optionally, a memory 1202 connected to the processor 1201.
[0108] In the first possible implementation, see Figure 12 The 3D vision-based track health detection device 1200 also includes a transceiver 1203. The processor 1201, memory 1202, and transceiver 1203 are connected via a bus. The transceiver 1203 is used to communicate with other devices or communication networks. Optionally, the transceiver 1203 may include a transmitter and a receiver. The device in the transceiver 1203 that implements the receiving function can be considered as a receiver, which is used to perform the receiving steps in the embodiments of this application. The device in the transceiver 1203 that implements the transmitting function can be considered as a transmitter, which is used to perform the transmitting steps in the embodiments of this application.
[0109] Based on the first possible implementation method Figure 12 The structural diagram shown can be used to illustrate the structure of the 3D vision-based track health detection device involved in the above embodiments.
[0110] in, Figure 12 The diagram also illustrates the system chip in a 3D vision-based track health detection device. In this case, the actions performed by the aforementioned 3D vision-based track health detection device can be implemented by this system chip; the specific actions performed are described above and will not be repeated here.
[0111] In implementation, each step of the method provided in this embodiment can be completed by integrated logic circuits in the processor or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.
[0112] The processor in this application may include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., which are various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor may be a separate semiconductor chip or integrated with other circuits into a single semiconductor chip. For example, it may be integrated with other circuits (such as encoding / decoding circuits, hardware acceleration circuits, or various bus and interface circuits) to form a System-on-a-Chip (SoC), or it may be integrated as a built-in processor within an ASIC. The ASIC with the integrated processor may be packaged separately or together with other circuits. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.
[0113] The memory in the embodiments of this application may include at least one of the following types: read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; or electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory may also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0114] This application also provides a computer-readable storage medium including instructions that, when run on a computer, cause the computer to perform any of the methods described above.
[0115] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.
[0116] This application also provides a chip including a processor and an interface circuit. The interface circuit is coupled to the processor. The processor is used to run computer programs or instructions to implement the above-described method. The interface circuit is used to communicate with other modules outside the chip.
[0117] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).
[0118] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0119] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.
Claims
1. A method for detecting track health based on 3D vision, characterized in that, include: The image to be detected of the target track and the set of point coordinates that constitute each target pixel block in the image to be detected are obtained by using a 3D camera. By comparing the image to be detected with a reference image of the target orbit in a healthy orbit state, a reference pixel block in the reference image with the same image content as the target pixel block is determined; Based on the set of point coordinates, calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera; Based on the plane normal vector of the target pixel block and the camera distance, as well as the plane normal vector of the reference pixel block and the camera distance, the elevation and tilt of the target track are calculated to determine the health status of the target track.
2. The method according to claim 1, characterized in that, The step of comparing the image to be detected with a reference image of the target orbit under a healthy orbital condition, and determining a reference pixel block in the reference image that has the same image content as the target pixel block, includes: The image to be detected and the reference image are respectively input into the image pixel block recognition network, and the feature vector of each pixel block is output. Based on the feature vector of the target pixel block and the feature vector of the pixel block in the reference image, the reference pixel block corresponding to the target pixel block is determined from the reference image.
3. The method according to claim 2, characterized in that, Also includes: The 3D camera acquires sample images of the target orbit multiple times. Extract sample pixel blocks from the sample image, classify the sample pixel blocks, and identify the classification results; the classification rule is: the image content of the same type of pixel blocks is the same object, and the image content of the different type of pixel blocks is different; The initial recognition network is trained based on the sample images and the classification results to obtain the image pixel block recognition network.
4. The method according to claim 3, characterized in that, The step of training an initial recognition network based on the sample image and the classification result to obtain the image pixel block recognition network includes: Three groups of pixel blocks are sampled from the categorized sample pixel blocks; in the three groups of pixel blocks, the first group of pixel blocks and the second group of pixel blocks are the same type of pixel blocks, and the third group of pixel blocks is the different type of pixel blocks from the first group of pixel blocks and the second group of pixel blocks, respectively. The three sets of pixel blocks are input into the initial recognition network to obtain the feature vectors of the three sets of pixel blocks; Based on the feature vectors of the three sets of pixel blocks, the triplet loss is calculated; the formula for calculating the triplet loss is as follows: ； This indicates taking the maximum value between A and B; This represents the cosine of the angle between A and B; The feature vector of the first group of pixel blocks; This is the feature vector of the second group of pixel blocks; The feature vector of the third group of pixel blocks; These are empirical parameters; The loss is the triplet loss; The triplet loss is minimized using a gradient iterative algorithm until the learnable parameters of the initial recognition network converge, thus obtaining the image pixel block recognition network; the convergence of the learnable parameters is expressed as: ； The learnable parameters are as described above.
5. The method according to claim 2, characterized in that, The step of determining the reference pixel block corresponding to the target pixel block from the reference image based on the feature vector of the target pixel block and the feature vector of a pixel block in the reference image includes: Calculate the cosine similarity between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; When the cosine similarity of any pixel block in the reference image is greater than a preset similarity, the arbitrary pixel block is determined to be the reference pixel block.
6. The method according to claim 2, characterized in that, The step of determining the reference pixel block corresponding to the target pixel block from the reference image based on the feature vector of the target pixel block and the feature vector of a pixel block in the reference image includes: Calculate the Euclidean distance between the feature vector of the target pixel block and the feature vector of the pixel block in the reference image; When the Euclidean distance corresponding to any pixel block in the reference image is less than a preset distance, the arbitrary pixel block is determined to be the reference pixel block.
7. The method according to claim 1, characterized in that, The step of calculating the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates includes: The average of the coordinates of multiple points in the set of point coordinates in the vertical direction is determined as the camera distance; The plane of the target pixel block is fitted by least squares method to obtain the plane equation and the plane normal vector is calculated.
8. The method according to claim 1, characterized in that, The step of calculating the elevation and tilt of the target track based on the plane normal vector and camera distance of the target pixel block, and the plane normal vector and camera distance of the reference pixel block, and determining the health status of the target track, includes: Calculate the absolute value of the difference in camera distance between the target pixel block and the reference pixel block, and the similarity of the plane normal vectors between the target pixel block and the reference pixel block; When the absolute value of the difference between the camera distances is greater than a preset displacement threshold, the health status of the target track is output as abnormal rise and fall. When the similarity of the plane normal vectors is less than a preset tilt threshold, the health status of the target orbit is output as tilt anomaly.
9. The method according to claim 8, characterized in that, Also includes: When the health status of the target track is abnormal in elevation or tilt, an alarm message is sent. The alarm message includes: location information of the anomaly occurring on the target track.
10. A track health detection system based on 3D vision, characterized in that, include: 3D camera, image recognition module, and health analysis module; The 3D camera is used to acquire the image to be detected of the target track, and the set of point coordinates that constitute each target pixel block in the image to be detected. The image recognition module is used to compare the image to be detected with a reference image of the target track in a healthy track state, and to determine a reference pixel block in the reference image that has the same image content as the target pixel block; The health analysis module is used to calculate the plane normal vector of the target pixel block and the camera distance between the target pixel block and the 3D camera based on the set of point coordinates. The health analysis module is further configured to calculate the elevation and tilt of the target track based on the plane normal vector and camera distance of the target pixel block, and the plane normal vector and camera distance of the reference pixel block, and determine the health status of the target track.