High-precision intelligent monitoring and detection method of turnout point rail structure status based on binocular vision

Abstract

The embodiment of the present invention discloses a high-precision intelligent monitoring and detection method for the structure state of a turnout point rail based on binocular vision. The embodiment of the present invention acquires images of machine-recognizable marks set on the point rail and the base rail by setting a binocular camera, acquires spatial coordinate information of the machine-recognizable marks through images at different positions, and thus realizes monitoring of the displacement state of the point rail state based on the spatial coordinate information. At the same time, under a specific state, a point cloud image is generated according to the acquired image, and a point cloud image in which each pixel is marked with a point rail category or a base rail category is obtained through semantic segmentation of the point cloud image. The point cloud image is detected based on a deep learning model to determine the state of the point rail. In parallel, the image is detected based on another deep learning model to determine the state of the point rail. Through the outputs of two different models, they are calibrated with each other to obtain accurate information on the state of the point rail.

Description

Technical Field

[0001] The present invention relates to the fields of rail transportation and image processing, and in particular to a high-precision intelligent monitoring and detection method for the structural state of a turnout point rail based on binocular vision. Background Art

[0002] The point rail is an important part of the turnout, used to switch the track state of the turnout and guide the train into the main line or siding direction. The point rail is a movable section of rail with a gradually tapering tip, usually installed on one side of the turnout. It cooperates with the fixed base rail and changes its position through connecting devices (such as switches or switchmen). When the point rail is close to the base rail on one side, the train continues to move along the straight track; when the point rail moves to the other side and is close to it, it guides the train into the siding or branch line. In the railway system, it is necessary to effectively monitor the state of the point rail to avoid danger. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a high-precision intelligent monitoring and detection method for the structural state of a turnout point rail based on binocular vision, so as to accurately monitor the state of the point rail and avoid danger.

[0004] The embodiment of the present invention provides a high-precision intelligent monitoring and detection method for the structure state of a turnout point rail based on binocular vision, wherein the method comprises:

[0005] Continuously acquiring a first image and a second image from a first binocular camera, and continuously acquiring a third image and a fourth image from a second binocular camera, wherein a first machine-identifiable mark is provided on the side of the first point rail and the second point rail of the target turnout, a second machine-identifiable mark is provided on the side of the first base rail and the second base rail of the target turnout, the first binocular camera is arranged at a position capable of photographing the first point rail, the first base rail, and the corresponding first machine-identifiable mark and the second machine-identifiable mark, and the second binocular camera is arranged at a position capable of photographing the second point rail, the second base rail, and the corresponding first machine-identifiable mark and the second machine-identifiable mark;

[0006] determining first spatial coordinates of the first machine identifiable mark of the first point rail and second spatial coordinates of the second machine identifiable mark of the first base rail based on positions of the first machine identifiable mark and the second machine identifiable mark in the first image and the second image;

[0007] determining third spatial coordinates of the first machine identifiable mark of the second point rail and fourth spatial coordinates of the second machine identifiable mark of the second base rail based on the positions of the first machine identifiable mark and the second machine identifiable mark in the third image and the fourth image;

[0008] Determine the displacement state information of the two pointed rails and each basic rail of the chain according to the first space coordinate, the second space coordinate, the third space coordinate and the fourth space coordinate;

[0009] In response to the displacement state information indicating that the turnout is in a point rail close contact state or a point rail repelling state, determining a first point cloud image of the target turnout according to the first image, the second image, the third image, and the fourth image;

[0010] Performing point cloud semantic segmentation on the first point cloud image, marking the point cloud belonging to the point track and the point cloud belonging to the base track to determine a second point cloud image;

[0011] Inputting the second point cloud image into a first detection model to determine a corresponding first point rail state category, wherein the first point rail state category belongs to a predetermined point rail state category set, and each point rail state category in the point rail state category set corresponds to a set of point rail state information; wherein the first detection model is a pre-trained classification model using point cloud data as input, and the point rail state information includes relative position parameters of the point rail and the base rail;

[0012] Inputting the first image and the third image into a second detection model to determine a corresponding second point rail state category, the second point rail state category belonging to the point rail state category set; wherein the second detection model is a pre-trained classification model that takes image data as input;

[0013] In response to the first point rail state information and the second point rail state information being consistent, determining that the point rail state information corresponding to the second point rail state category is initial point rail state information;

[0014] The initial switch state information is corrected according to the displacement state information to determine the switch state of the target turnout.

[0015] In some embodiments, the displacement state information includes at least one of the relative displacement of the tip of the first point rail and the second point rail, the lowering value of the tip of the point rail, the gauge of the point rail, the displacement, speed and acceleration of the point rail when it is turned, and the gauge of the basic rail.

[0016] In some embodiments, the method further comprises:

[0017] Acquire the first spatial coordinate sequence;

[0018] Obtaining a subsequence of a changed portion of the first spatial coordinate sequence;

[0019] Calculating the similarity between the subsequence and a standard subsequence;

[0020] The first defect information of the point rail is determined according to the similarity, where the first defect information is used to characterize the defects of the point rail during the movement.

[0021] In some embodiments, the method further comprises:

[0022] The second image and the fourth image are used as input, and second defect information of the first and second point rails is determined based on the third detection model, wherein the second defect information includes at least one of point rail crushing, point rail block falling, point rail fat edge, and point rail breakage.

[0023] In some embodiments, determining a first point cloud image of the target turnout according to the first image, the second image, the third image, and the fourth image includes:

[0024] Determine a first initial point cloud in the coordinates of a first binocular camera according to the first image and the second image;

[0025] Determine a second initial point cloud in the coordinates of a second binocular camera according to the third image and the fourth image;

[0026] Performing coordinate transformation on the second initial point cloud to determine a second transformed point cloud corresponding to the coordinates of the first binocular camera;

[0027] The first initial point cloud and the second transformed point cloud are fused to determine the first point cloud image.

[0028] In some embodiments, performing point cloud semantic segmentation on the first point cloud image, annotating the point cloud belonging to the point track and the point cloud belonging to the base track to determine the second point cloud image includes:

[0029] Based on the point cloud semantic segmentation model, performing point cloud semantic segmentation on the first point cloud image to output the second point cloud image;

[0030] Wherein, the point cloud semantic segmentation model includes a feature embedding network, a classification network and an output network;

[0031] Wherein, the feature embedding network is configured as:

[0032] Dividing the first point cloud image into a plurality of slice images according to the value range of the depth information, wherein the slice images include the plane projections of the point cloud points within the corresponding depth value range;

[0033] The plurality of slice images are connected in depth order to form an embedded image;

[0034] Extracting an image feature sequence of the embedded image as the embedded feature;

[0035] The classification network is configured as:

[0036] Classify each point cloud point according to the embedded image features to determine the point cloud point category, wherein the point cloud point category includes a point track, a base track, and a background;

[0037] The input network is configured as:

[0038] Only point cloud points with the category of pointed rail or basic rail are retained in the first point cloud image, and a second point cloud image with category labels is output.

[0039] In some embodiments, modifying the initial switch state information according to the position information to determine the switch state of the target switch includes:

[0040] Determine a first parameter corresponding to the initial point rail state information, wherein the first parameter includes a first point rail gauge and / or first relative position information of the point rail in a close contact state with the base rail;

[0041] Determine a corresponding second parameter according to the position information, the second parameter including a second point rail gauge and / or second relative position information of the point rail in a close contact state with the base rail;

[0042] A weighted sum is performed on the first parameter and the second parameter to determine the switch rail state of the target switch.

[0043] In some embodiments, the first machine identifiable indicia and the second machine identifiable indicia are different indicia.

[0044] In some embodiments, different first machine identifiable marks are set on different point rails, and the relative positions of the two point rails are determined according to the spatial coordinates of the different first machine identifiable marks of the two point rails.

[0045] The embodiment of the present invention sets a binocular camera, obtains images of machine-readable marks set on the point rail and the base rail through the first camera and the second camera set side by side, obtains the spatial coordinate information of the machine-readable marks through images at different positions, and thus can monitor the displacement state of the point rail based on the spatial coordinate information. At the same time, under a specific state, a point cloud image is generated by the images obtained by the first binocular camera and the second binocular camera, and a point cloud image in which each pixel is marked with a point rail category or a base rail category is obtained through semantic segmentation of the point cloud image. The point cloud image is detected based on a deep learning model to determine the state of the point rail. In parallel, the image is detected based on another deep learning model to determine the state of the point rail. Through the outputs of two different models, they are calibrated with each other to obtain accurate information on the state of the point rail. Then, the state information of the point rail is calibrated based on the displacement state determined by the machine-readable mark. In this way, the state of the point rail can be monitored with high accuracy to ensure railway safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0047] Figure 1 is a schematic diagram of a point rail monitoring system according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of a base rail and a point rail with machine-readable markings on the sides;

[0049] Figure 3 It is a flow chart of a high-precision intelligent monitoring and detection method for the structure state of a turnout point rail based on binocular vision according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of the structure of a point cloud image semantic segmentation model according to an embodiment of the present invention;

[0051] Figure 5 It is a structural schematic diagram of an edge computing device or an industrial computer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0053] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0054] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0055] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0056] Figure 1 Schematic diagram of the point rail monitoring system according to an embodiment of the present invention. Figure 1As shown, the point rail monitoring system of the embodiment of the present invention includes a first binocular camera 11a and a second binocular camera 11b, an edge computing device 12 and an industrial computer 13. Among them, the first binocular camera 11a is arranged on the second side of the turnout, facing the first side of the turnout, and is used to detect the first point rail and the first basic rail on the first side. The second binocular camera 11b is arranged on the first side of the turnout, facing the second side of the turnout, and is used to detect the second point rail and the second basic rail on the second side. A binocular camera is an imaging system that imitates the principle of human binocular vision. It uses two cameras separated by a certain distance to simultaneously shoot the same scene and obtain two slightly different images. Due to the different positions of the two cameras, the images they capture will have disparity, that is, the position of the same object in the two pictures will be different. This disparity is related to the distance of the object: the closer the object is to the camera, the greater its disparity; otherwise, the smaller it is. The edge computing device 12 is communicatively connected to the first binocular camera 11a and the second binocular camera 11b, and is used to perform preliminary processing on the images obtained by the first binocular camera 11a and the second binocular camera 11b, and then transmit the images to the industrial computer 13, which further processes the images and monitors the status of the point rail.

[0057] The first binocular camera 11a is set at a position capable of photographing the first point rail, the first base rail, and the corresponding first machine-readable mark and the second machine-readable mark. The second binocular camera 11b is set at a position capable of photographing the second point rail, the second base rail, and the corresponding first machine-readable mark and the second machine-readable mark.

[0058] Among them, the first point rail and the second point rail are respectively provided with different first machine-recognizable marks in the direction facing the corresponding binocular camera. The first machine-recognizable marks can be easily recognized and distinguished by a computer in an image. The first basic rail and the second basic rail are respectively provided with different second machine-recognizable marks in the direction facing the corresponding binocular camera. The second machine-recognizable marks can be easily recognized by a computer in an image. Therefore, the corresponding machine-recognizable mark can be identified based on the image captured by the binocular camera, and it can be distinguished whether the point rail or the basic rail is observed in the current image, and which one of the point rails or the basic rails is observed. Figure 2 is a schematic diagram of a base rail and a point rail with machine-readable markings on the sides. Figure 2As shown, the first machine-readable mark and the second machine-readable mark 21 can be pasted and fixed on the side of the base rail or the point rail, at a position that is easily captured by the first binocular camera 11a or the second binocular camera 11b. The first binocular camera 11a is used to continuously acquire the first image and the second image. As described above, the first binocular camera has two cameras that respectively capture images with parallax (that is, the first image and the second image). The second binocular camera 11b is used to continuously acquire the third image and the fourth image. That is, two images with parallax.

[0059] Figure 3 1 is a flow chart of a high-precision intelligent monitoring and detection method for the structure state of a turnout point rail based on binocular vision according to an embodiment of the present invention. Figure 3 As shown, the method of this embodiment includes the following steps:

[0060] Step S100: continuously acquiring a first image and a second image from a first binocular camera, and continuously acquiring a third image and a fourth image from a second binocular camera.

[0061] Among them, the first point rail and the second point rail of the target turnout are provided with a first machine-recognizable mark on the side, the first basic rail and the second basic rail of the target turnout are provided with a second machine-recognizable mark on the side, the first binocular camera is set at a position capable of photographing the first point rail, the first basic rail and the corresponding first machine-recognizable mark and the second machine-recognizable mark, and the second binocular camera is set at a position capable of photographing the second point rail, the second basic rail and the corresponding first machine-recognizable mark and the second machine-recognizable mark.

[0062] Step S200, determining the first spatial coordinates of the first machine identifiable mark of the first point rail and the second spatial coordinates of the second machine identifiable mark of the first basic rail according to the positions of the first machine identifiable mark and the second machine identifiable mark in the first image and the second image.

[0063] Specifically, the above steps are implemented by the edge computing device 12 or the industrial computer 13. The coordinates of the center point of the first machine-recognizable mark and the coordinates of the center point of the second machine-recognizable mark are identified in the first image. Then, the coordinates of the center point of the first machine-recognizable mark and the coordinates of the center point of the second machine-recognizable mark are identified in the second image. Since the center point of the first machine-recognizable mark is the same point on the point rail. The horizontal displacement between them, i.e., the parallax d, can be calculated. Next, according to the geometric relationship, the distance Z from the point to the camera can be calculated using the triangulation principle, and the formula is Z=(B*f) / d, where B is the distance between the centers of the two cameras (baseline), and f is the focal length of the camera. Finally, combined with the internal parameter matrix of the camera (including information such as focal length, principal point, etc.) and the external parameters (the posture and position of the camera), the 2D coordinates on the image plane can be converted into 3D spatial coordinates in the world coordinate system, thereby obtaining the spatial coordinates relative to the first binocular camera, and then the spatial coordinates can be converted to a predetermined coordinate system to obtain the first spatial coordinates and the second spatial coordinates.

[0064] Step S300, determining the third spatial coordinates of the first machine identifiable mark of the second point rail and the fourth spatial coordinates of the second machine identifiable mark of the second basic rail according to the positions of the first machine identifiable mark and the second machine identifiable mark in the third image and the fourth image.

[0065] Step S400: determining the displacement state information of the target turnout according to the first space coordinate, the second space coordinate, the third space coordinate and the fourth space coordinate.

[0066] The displacement state information includes at least one of the relative displacement of the tip of the first and second point rails, the lowering value of the tip of the point rail, the gauge of the point rail, the displacement, speed and acceleration of the point rail when it is turned, and the gauge of the base rail. These displacement state information can be output as part of the state information of the point rail and presented to the user.

[0067] Step S500, in response to the displacement state information indicating that the turnout is in a point rail close contact state or a point rail repelling state, determining a first point cloud image of the target turnout according to the first image, the second image, the third image and the fourth image.

[0068] In this embodiment, it is expected to monitor the state of the switch rail of the target switch more accurately. Therefore, a point cloud generation technology can be used to generate a point cloud image.

[0069] Specifically, step S500 includes the following steps:

[0070] Step S510: determining a first initial point cloud in the coordinates of a first binocular camera according to the first image and the second image.

[0071] A typical process of generating point cloud data from the image of the binocular camera may be to first preprocess the first image and the second image, which may include graying, noise removal and other operations to improve the accuracy of subsequent steps. After ensuring that the binocular camera has been accurately calibrated, the original image is geometrically corrected using the obtained intrinsic and extrinsic parameters to make the scan lines of the two images parallel, simplifying the stereo matching process.

[0072] Then enter the feature matching or stereo matching stage, which is one of the key steps, the purpose is to find the corresponding real-world points in the left and right images. The first machine-recognizable marker can also be used as a base point to assist in the matching of real-world points. Common methods include region-based matching such as SAD SSD NCC and feature-based matching such as SIFT SURF. In some implementations, deep learning technology can also be used to improve matching accuracy. The result of this step is a disparity map that represents the horizontal displacement of each pixel between the left and right images.

[0073] With the disparity map, the depth value of each pixel can be calculated using the formula Z = (B * f) / d, where B is the baseline distance between the cameras, f is the focal length, and d is the disparity value of the point. Combined with the camera's intrinsic matrix, each pixel is converted from the image coordinate system to the three-dimensional coordinates in the world coordinate system. This step requires the application of the camera projection equation. For each pixel (u, v), the spatial coordinates in the world coordinate system are calculated through its corresponding depth value Z.

[0074] Finally, all the calculated three-dimensional coordinates are combined to form point cloud data. Each point represents the color and spatial information of a specific position in the original scene. The point cloud is usually post-processed by filtering and noise reduction to improve data quality and visual effects. The generated point cloud data can be saved in standard format files, such as PLY format or PCD format, or can be directly used for further analysis and processing.

[0075] Thus, a first initial point cloud can be generated according to the first image and the second image acquired by the first binocular camera.

[0076] Step S520: determining a second initial point cloud in the coordinates of a second binocular camera according to the third image and the fourth image.

[0077] The generation process and principle of the second initial point cloud are similar to those of the first initial point cloud, and will not be repeated here.

[0078] Step S530: performing coordinate transformation on the second initial point cloud to determine a second transformed point cloud corresponding to the coordinates of the first binocular camera.

[0079] Since the coordinate system of the second initial point cloud is different from that of the first initial point cloud, the first initial point cloud is transformed as a whole into the coordinate system corresponding to the first initial point cloud through the coordinate transformation matrix to obtain the second transformed point cloud.

[0080] Specifically, step S530 and step S540 perform point cloud fusion on the first initial point cloud and the second initial point cloud. Point cloud fusion refers to the process of merging multiple point cloud data sets from different perspectives or different sensors into a unified, more complete and accurate data set. First, it is necessary to ensure that each point cloud has undergone preliminary processing such as denoising filtering. The next step is one of the key steps, namely the registration between point clouds. It involves finding the best transformation relationship between two or more point clouds so that they can be aligned in the same coordinate system. Usually, the iterative closest point algorithm ICP or feature matching-based methods are used to estimate rotation and translation parameters to achieve high-precision registration. In some cases, global registration techniques such as random sampling consistency RANSAC combined with a global optimal search strategy may also be used.

[0081] Step S540: Fusing the first initial point cloud and the second transformed point cloud to determine the first point cloud image.

[0082] After the coordinate transformation is completed in step S530, the fusion stage begins. At this point, all point clouds are converted to the same coordinate system, and data integration can begin. In one implementation, all points in the point cloud are directly merged into one place. However, this may cause problems with duplicate points and uneven distribution. In other implementations, point cloud density information is considered to remove redundant points through voxel rasterization or statistical analysis to maintain appropriate resolution. At the same time, additional information such as color texture can be used for weighted averaging to generate smoother and more natural results. In some implementations, point cloud fusion can also be integrated into machine learning models, such as neural networks, to predict missing parts or enhance detail performance.

[0083] Finally, in order to ensure the quality of the fused point cloud, a series of post-processing operations are often required, including but not limited to boundary detection, hole repair, filling, and further noise suppression, etc. These steps help improve the visual effect and geometric accuracy of the final output.

[0084] Step S600: performing point cloud semantic segmentation on the first point cloud image, annotating the point cloud belonging to the point track and the point cloud belonging to the basic track to determine a second point cloud image.

[0085] After point cloud fusion, the first point cloud image can present the three-dimensional spatial status of the target turnout position. In order to better monitor its status in the future, the first point cloud image needs to be semantically segmented. Semantic segmentation of point cloud images refers to assigning each point in the point cloud data to a specific category in order to understand the content and structure of the scene. Semantic segmentation requires feature extraction first. During the feature extraction process, geometric or photometric features such as normal direction, curvature, color information, etc. are calculated for each point. These features help to distinguish different surface properties of objects. In some implementations, deep learning-based methods, such as convolutional neural networks (CNNs), can be applied to point cloud semantic segmentation tasks. For example, PointNet and its improved version PointNet++ can directly process unordered point sets and capture spatial relationships in local areas, thereby achieving efficient feature representation. In order to train the model, a labeled training data set needs to be prepared, in which each point is labeled with its category. Using this data, a classifier can be trained in a supervised learning manner so that it can predict the category label of each point from the input point cloud.

[0086] In an optional implementation, the following may be used: Figure 4 The model shown in FIG. 1 is used to perform semantic segmentation, that is, based on the point cloud semantic segmentation model, the first point cloud image is subjected to point cloud semantic segmentation to output the second point cloud image. Figure 4 As shown, the point cloud semantic segmentation model includes a feature embedding network 41, a classification network 42 and an output network 43.

[0087] Wherein, the feature embedding network 41 is configured as follows:

[0088] The first point cloud image is divided into a plurality of slice images according to the numerical range of depth information, wherein the slice images include the planar projections of the point cloud points within the corresponding depth numerical range; the plurality of slice images are connected into an embedded image in depth order; and the image feature sequence of the embedded image is extracted as the embedded feature.

[0089] The classification network 42 is configured as follows:

[0090] Each point cloud point is classified according to the embedded image features to determine the point cloud point category, wherein the point cloud point category includes a sharp track, a basic track and a background.

[0091] The output network 43 is configured as:

[0092] Only point cloud points with the category of pointed rail or basic rail are retained in the first point cloud image, and a second point cloud image with category labels is output.

[0093] In this way, the point cloud data can be semantically segmented effectively to obtain an accurate second point cloud image.

[0094] Step S700, input the second point cloud image into the first detection model to determine the corresponding first point rail state category. The first point rail state category belongs to a predetermined point rail state category set. Each point rail state category in the point rail state category set corresponds to a set of point rail state information; wherein the first detection model is a pre-trained classification model with point cloud data as input. The point rail state information includes the relative position parameters of the point rail and the base rail, and the morphological information of the point rail. The morphological information of the point rail and the relative position parameters of the point rail and the base rail can reflect whether the point rail is in a normal state and whether different types of faults occur.

[0095] The first detection model is obtained by training based on the pre-collected point cloud image and the point rail state information corresponding to the point cloud image. The first detection model can be a deep learning model based on a neural network, such as a convolutional neural network, a recurrent neural network or a residual network.

[0096] Step S800, input the first image and the third image into a second detection model to determine the corresponding second point rail state category, where the second point rail state category belongs to the point rail state category set; wherein the second detection model is a pre-trained classification model with image data as input.

[0097] As described above, the first image is an image of the first point rail and the first basic rail captured by one of the first binocular cameras. The third image is an image of the second point rail and the second basic rail captured by one of the second binocular cameras. The second detection model is a classification model that takes different image information as input and takes the state category of the point rail as output. Therefore, the state of the point rail can be classified again from the image level.

[0098] Step S900, in response to the first point rail state information and the second point rail state information being consistent, determining that the point rail state information corresponding to the second point rail state category is initial point rail state information.

[0099] In this step, a cross-validation is performed based on the detection results of the first detection model and the second detection model. Only when the two are consistent, the corresponding point rail state information is used as the initial point rail state information. If the two are inconsistent, the detection result of this time is discarded.

[0100] Step S1000, correcting the initial point rail state information according to the displacement state information to determine the point rail state of the target turnout.

[0101] Specifically, when the state of the point rail is the gauge of the point rail in different states or the relative position information with respect to the stock rail, correction can be performed in the following manner.

[0102] Determine a first parameter corresponding to the initial point rail state information, wherein the first parameter includes a first point rail gauge and / or first relative position information of the point rail in a close contact state with the base rail;

[0103] Determine a corresponding second parameter according to the position information, the second parameter including a second point rail gauge and / or second relative position information of the point rail in a close contact state with the base rail;

[0104] A weighted sum is performed on the first parameter and the second parameter to determine the switch rail state of the target switch.

[0105] In some implementations, the state of the point rail in a dynamic state can also be monitored based on the identification of the spatial position of the machine-readable mark. That is, after the method of this embodiment determines the spatial coordinates of the first machine-readable mark corresponding to the point rail based on the first image to the fourth image, it can obtain the first spatial coordinate sequence; obtain a subsequence of the changed part of the first spatial coordinate sequence; calculate the similarity between the subsequence and the standard subsequence; and determine the first defect information of the point rail based on the similarity, and the first defect information is used to characterize the defects of the point rail during movement. Thus, the changed part of the first spatial coordinate sequence of the first machine-readable mark is obtained. Since the change in the position of the point rail is actually a switch from a close contact state to a repulsive state, under normal circumstances, its change trajectory is relatively fixed. Therefore, the standard subsequence measured in advance can be compared with the detected subsequence representing the change process, so as to determine whether the point rail has defects during movement.

[0106] In some implementations, it is also possible to determine whether there are microscopic defects on the surface of the point rail based on the high-precision image. That is, the second image and the fourth image can be used as input to determine the second defect information of the first point rail and the second point rail based on the third detection model, and the second defect information includes at least one of: point rail crushing, point rail block loss, point rail fat edge, and point rail fracture.

[0107] The embodiment of the present invention sets a binocular camera, obtains images of machine-readable marks set on the point rail and the base rail through the first camera and the second camera set side by side, obtains the spatial coordinate information of the machine-readable marks through images at different positions, and thus can monitor the displacement state of the point rail based on the spatial coordinate information. At the same time, under a specific state, a point cloud image is generated by the images obtained by the first binocular camera and the second binocular camera, and a point cloud image in which each pixel is marked with a point rail category or a base rail category is obtained through semantic segmentation of the point cloud image. The point cloud image is detected based on a deep learning model to determine the state of the point rail. In parallel, the image is detected based on another deep learning model to determine the state of the point rail. Through the outputs of two different models, they are calibrated with each other to obtain accurate information on the state of the point rail. Then, the state information of the point rail is calibrated based on the displacement state determined by the machine-readable mark. In this way, the state of the point rail can be monitored with high accuracy to ensure railway safety.

[0108] Figure 5 Schematic diagram of the structure of the edge computing device or industrial computer in an embodiment of the present invention. Figure 5 As shown, the electronic device includes a general computer hardware structure, which includes at least a processor 51 and a memory 52, which is used to constitute an edge computing device or an industrial computer in an embodiment of the present invention.

[0109] The processor 51 and the memory 52 are connected via a bus 53. The memory 52 is suitable for storing instructions or programs executable by the processor 51. The processor 51 may be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 51 executes the instructions stored in the memory 52, thereby executing the method flow of the embodiment of the present invention as described above to process data and control other devices. The bus 53 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 54 and the display device and the input / output (I / O) device 55. The input / output (I / O) device 55 may be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 55 is connected to the system via an input / output (I / O) controller 56.

[0110] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0111] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0112] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0113] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0114] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0115] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high-precision intelligent monitoring and detection method for the structure status of turnout point rail based on binocular vision, characterized in that: The method comprises: Continuously acquiring a first image and a second image from a first binocular camera, and continuously acquiring a third image and a fourth image from a second binocular camera, wherein a first machine-identifiable mark is provided on the side of a first point rail and a second point rail of a target turnout, a second machine-identifiable mark is provided on the side of a first base rail and a second base rail of the target turnout, the first binocular camera is arranged at a position capable of photographing the first point rail, the first base rail, and the corresponding first machine-identifiable mark and the second machine-identifiable mark, and the second binocular camera is arranged at a position capable of photographing the second point rail, the second base rail, and the corresponding first machine-identifiable mark and the second machine-identifiable mark; determining first spatial coordinates of the first machine identifiable mark of the first point rail and second spatial coordinates of the second machine identifiable mark of the first base rail based on positions of the first machine identifiable mark and the second machine identifiable mark in the first image and the second image; determining third spatial coordinates of the first machine identifiable mark of the second point rail and fourth spatial coordinates of the second machine identifiable mark of the second base rail based on the positions of the first machine identifiable mark and the second machine identifiable mark in the third image and the fourth image; Determine the displacement state information of the target turnout according to the first spatial coordinate, the second spatial coordinate, the third spatial coordinate and the fourth spatial coordinate; In response to the displacement state information indicating that the turnout is in a point rail close contact state or a point rail repelling state, determining a first point cloud image of the target turnout according to the first image, the second image, the third image, and the fourth image; Performing point cloud semantic segmentation on the first point cloud image, marking the point cloud belonging to the point track and the point cloud belonging to the base track to determine a second point cloud image; Inputting the second point cloud image into a first detection model to determine a corresponding first point rail state category, wherein the first point rail state category belongs to a predetermined point rail state category set, and each point rail state category in the point rail state category set corresponds to a set of point rail state information; wherein the first detection model is a pre-trained classification model using point cloud data as input, and the point rail state information includes relative position parameters of the point rail and the base rail; Inputting the first image and the third image into a second detection model to determine a corresponding second point rail state category, the second point rail state category belonging to the point rail state category set; wherein the second detection model is a pre-trained classification model that takes image data as input; In response to the first point rail state information and the second point rail state information being consistent, determining that the point rail state information corresponding to the second point rail state category is initial point rail state information; The initial switch state information is corrected according to the displacement state information to determine the switch state of the target turnout.

2. The method according to claim 1, characterized in that The displacement state information includes at least one of the relative displacement of the tip of the first point rail and the second point rail, the lowering value of the tip of the point rail, the gauge of the point rail, the displacement, speed and acceleration of the point rail when it is turned, and the gauge of the basic rail.

3. The method according to claim 1, characterized in that The method further comprises: Acquire the first spatial coordinate sequence; Obtaining a subsequence of a changed portion of the first spatial coordinate sequence; Calculating the similarity between the subsequence and a standard subsequence; The first defect information of the point rail is determined according to the similarity, where the first defect information is used to characterize the defects of the point rail during the movement.

4. The method according to claim 1, characterized in that: The method further comprises: The second image and the fourth image are used as input, and second defect information of the first and second point rails is determined based on the third detection model, wherein the second defect information includes at least one of point rail crushing, point rail block falling, point rail fat edge, and point rail breakage.

5. The method according to claim 1, characterized in that Determining a first point cloud image of the target turnout according to the first image, the second image, the third image, and the fourth image includes: Determine a first initial point cloud in the coordinates of a first binocular camera according to the first image and the second image; Determine a second initial point cloud in the coordinates of a second binocular camera according to the third image and the fourth image; Performing coordinate transformation on the second initial point cloud to determine a second transformed point cloud corresponding to the coordinates of the first binocular camera; The first initial point cloud and the second transformed point cloud are fused to determine the first point cloud image.

6. The method according to claim 1, characterized in that Performing point cloud semantic segmentation on the first point cloud image, marking the point cloud belonging to the point track and the point cloud belonging to the base track to determine the second point cloud image includes: Based on the point cloud semantic segmentation model, performing point cloud semantic segmentation on the first point cloud image to output the second point cloud image; Wherein, the point cloud semantic segmentation model includes a feature embedding network, a classification network and an output network; Wherein, the feature embedding network is configured as: Dividing the first point cloud image into a plurality of slice images according to the value range of the depth information, wherein the slice images include the plane projections of the point cloud points within the corresponding depth value range; Connecting the plurality of slice images into an embedded image in depth order; Extracting an image feature sequence of the embedded image as an embedded image feature; The classification network is configured as: Classify each point cloud point according to the embedded image features to determine the point cloud point category, wherein the point cloud point category includes a point track, a base track, and a background; The output network is configured as: Only point cloud points with the category of pointed rail or basic rail are retained in the first point cloud image, and a second point cloud image with category labels is output.

7. The method according to claim 1, characterized in that Correcting the initial switch state information according to the displacement state information to determine the switch state of the target turnout includes: Determine a first parameter corresponding to the initial point rail state information, wherein the first parameter includes a first point rail gauge and / or first relative position information of the point rail in a close contact state with the base rail; Determine a corresponding second parameter according to the position information, the second parameter including a second point rail gauge and / or second relative position information of the point rail in a close contact state with the base rail; A weighted sum is performed on the first parameter and the second parameter to determine the switch rail state of the target switch.

8. The method according to claim 1, characterized in that The first machine-readable mark and the second machine-readable mark are different marks.

9. The method according to claim 1, characterized in that: Different first machine-identifiable marks are set on different point rails, and the relative positions of the two point rails are determined according to the spatial coordinates of the different first machine-identifiable marks of the two point rails.

Images