High-speed rail key point recognition method and medium based on deep neural network

Abstract

The invention discloses a method and medium for identifying key points of high-speed rail fortresses based on a deep neural network. Through a visible light camera, the visible light image of the target area is photographed and a visible light video stream is output. An intelligent edge computing device receives the visible light camera video, and through a high-speed target key point detection algorithm, the real-time detection of the target key points in the video stream is realized, and the pixel coordinates of the target key points are output, and the pixel length, pixel width and pixel position of the target rectangular bounding box are output simultaneously. Through a height inversion algorithm, the distance between the monocular visible light camera and the top surface of the rail fortress is calculated according to the pixel coordinates of the target key points. The key point detection algorithm of the invention can not only detect the position and size of the target in real time, but also detect the pixel coordinates of 5 key points, including 4 key points and the center point on the top surface of the target. Based on the key point coordinates, the distance to the target can be achieved with only a monocular visible light camera.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence image processing, and in particular to a high-speed rail key point recognition method and medium based on a deep neural network. Background Art

[0002] As a typical anti-landing obstacle, rail forts are usually deployed in clusters on the beach to hinder and delay landing. Therefore, "obstacle removal" has become an inevitable problem in landing operations. For the rail fort target cluster, the obstacle removal effect of the obstacle removal bomb is directly related to the target search success rate and key point aiming accuracy.

[0003] With the development of computer technology and the widespread application of computer vision, the research on real-time detection of targets using computer image and video processing technology has made great progress. This allows the obstacle-clearing platform to choose to use an optical camera to shoot images from the air to the ground and search for rail fort targets in the image. For detecting fixed targets in images, the most typical method is a feature-based detection algorithm. This method first extracts features from different candidate areas in the image to form a feature space; then a classifier is constructed in the feature space for classification. The image features selected by this method are very important. The optional features can be natural features such as the shape, size, texture, color, amplitude, etc. of the object, or artificially designed statistical features such as histograms, HOG, HAAR, SIFT, etc. The advantages of using the above traditional detection methods to detect rail forts are fast speed and good real-time performance; the disadvantage is poor robustness. Factors such as weather, sunshine, background changes, and even the manufacturing method or corrosion degree of the rail fort will cause the detection accuracy to decrease; in addition, it is impossible to give accurate key point coordinates and achieve height estimation.

[0004] In recent years, the accuracy of target detection methods based on deep learning has surpassed that of traditional methods. The essential reason is that based on a large amount of labeled data, as the number of layers of the neural network increases, it can extract better features than those designed manually, thereby greatly improving the performance of the algorithm. Therefore, it is feasible to design a deep neural network detection model to detect railroad fortification targets.

[0005] Domestic and foreign technical analysis

[0006] At present, there are limited public documents, patents and reports on rail bar detection and identification technology at home and abroad, and no relevant foreign literature has been retrieved. The domestic method for detecting and identifying rail bar is mainly to identify the target through image processing method.

[0007] The domestic patent "A Method and Device for Identifying Railroad Fortresses Based on Aerial Images" proposes to use histogram equalization processing, segment the color image after histogram equalization processing, convert the color image into a binary image, and obtain a set of pixel points of the railroad fortress in the color image; cluster the pixel points to obtain the image center point of the pixel point set of each railroad fortress in the color image; and determine the railroad fortress closest to the image center point as the target railroad fortress through a preset distance algorithm. This method identifies the target railroad fortress through a series of image transformations and clustering operations, and can only calculate the position of the railroad fortress, but cannot give the pixel coordinates of the four vertices (i.e., key points) on the upper surface of the railroad fortress.

[0008] A typical deep learning-based detection algorithm usually consists of two parts: a backbone network pre-trained on ImageNet; and a head for predicting object categories and bounding boxes. For the backbone part, the detector running on the GPU platform can have a high-performance network such as VGG, ResNet, ResNeXt, or DenseNet as the backbone; for the detector running on the CPU platform, the backbone can be a lightweight network such as SqueezeNet, MobileNet, or ShuffleNet as the backbone. The head part can be divided into two categories: one-stage detector and two-stage detector corresponding to different heads. The most representative two-stage object detector is the R-CNN series, including FastR-CNN, Faster R-CNN, R-FCN, and Libra R-CNN. The most representative first-stage object detector models are YOLO, SSD, and RetinaNet. In recent years, anchor-free first-stage object detectors have also made new developments, such as CenterNet, CornerNet, FCOS, etc. Object detectors that have emerged in recent years often insert some layers between Backbone and Head. These layers are used to collect feature maps at different stages, which can be called the neck of the object detector. Usually, Neck consists of several bottom-up paths and several top-down paths. Neck can be a feature pyramid network (FPN), a path aggregation network (PAN), BiFPN, and NAS-FPN. In summary, a typical object detector can be considered to consist of input, Backbone, Neck, and Head.

[0009] In practice, the deeper the backbone network, the better the detection effect. However, the deeper the network, the more parameters it brings, which seriously affects the forward reasoning speed of the model on the intelligent edge computing device. Therefore, conventional detection models need to consider the trade-off between detection accuracy and the number of parameters. Small backbone networks are conducive to improving computing speed, but often result in a loss of accuracy.

[0010] In the field of visual measurement, distance information is generally obtained through binocular camera calibration, binocular matching and other steps to calculate depth information. Summary of the invention

[0011] The invention proposes a high-speed rail barrier key point recognition method based on deep neural network, which can realize the rapid detection, recognition and height calculation of rail barrier targets by unmanned obstacle breaking or strike platforms, improve the aiming accuracy of target key points and calculate the distance between platform targets, and achieve optimal strike.

[0012] To achieve the above object, the present invention adopts the following technical solutions:

[0013] A high-speed rail fortification key point recognition method based on deep neural network is applied to an unmanned platform, including a monocular visible light camera and an intelligent edge computing device communication connection, wherein the monocular visible light camera is installed on the unmanned platform and is used to detect and identify rail fortification targets, including the following steps:

[0014] The visible light camera is used to capture visible light images of the target area and output visible light video streams;

[0015] The intelligent edge computing device receives the visible light camera video, runs a high-speed target key point detection algorithm and a height inversion algorithm, and connects to the UAV flight control system;

[0016] Among them, through the high-speed target key point detection algorithm, the real-time detection of the target key points in the video stream is realized, and the pixel coordinates of the target key points are output, and the pixel length, pixel width and pixel position of the target rectangular bounding box are output at the same time;

[0017] Through the height inversion algorithm, the distance between the monocular visible light camera and the top surface of the track fortress is calculated according to the pixel coordinates of the target key points, that is, the height data of the unmanned system from the top surface of the track fortress.

[0018] Furthermore, the high-speed target key point detection algorithm comprises the following steps:

[0019] Dataset construction: by marking the four corner points and one center point on the top surface of the rail fort target, a total of five key points combined with the location box of the rail fort cement pier as annotation information;

[0020] The model structure is bottle neck: feature pyramid FPN module;

[0021] The model includes an FPN module to extract four levels of features from the original image. First, the backbone network is used to perform bottom-up feature extraction to obtain a feature map {C 2 ,C 3 ,C 4 ,C 5}; then {C 2 ,C 3 ,C 4}Perform a lateral connection, that is, {C 2 ,C 3 ,C 4} respectively through a 1×1 convolutional layer to fuse information and reduce the channel dimension; finally, the channel fused {C 2 ,C 3 ,C 4} and the feature map obtained by top-down upsampling, the corresponding elements are added and fused to obtain the feature pyramid layer {P2,P 3 ,P 4}; The top feature map shares C with the top pyramid layer 5 =P 5 ;

[0022] Also includes head: context module;

[0023] The context module is used as the head, and independent context modules are applied on the four feature pyramid levels. Specifically, the original 3×3 convolution layer in the FPN structure is replaced by deformable convolution.

[0024] The loss function of the model Designed as a multi-task joint loss function:

[0025]

[0026] In the formula, and They are classification loss, bounding box regression loss, and key point deviation loss; λ 1 and λ 2 are the weight coefficients for bounding box regression and key point prediction losses respectively; They are category, bounding box, key point true value, and when there is a target at a certain anchor point otherwise c i , r i , p i They are the predicted values ​​of categories, bounding boxes, and key points respectively;

[0027] The output results of the model forward propagation include:

[0028] Classification results: 1 for target, 0 for background;

[0029] The upper left corner coordinates and width and height of the target minimum enclosing rectangle: (x, y, w, h);

[0030] The coordinates of the five key points of the target (x i ,y i ),i∈[1,5].

[0031] Furthermore, the height inversion algorithm comprises the following steps:

[0032] S21. Coordinate system

[0033] Pixel coordinate system: Pixel coordinates describe the specific position of each pixel in the image, with the upper left corner of the image as the origin of the pixel coordinate system. p , the horizontal rightward is the u-axis of the coordinate system, and the vertical downward is the v-axis of the coordinate system;

[0034] The unit of the pixel coordinate system is pixel. Any pixel in the pixel coordinate system is represented by (u i ,v i );

[0035] Image coordinate system: convert the pixel coordinates of the object in the image into image coordinates;

[0036] The specific method of coordinate system conversion is: the origin O of the image coordinate system is i is the center of the image, its x-axis is obtained by translating downward from the u-axis of the pixel coordinate system, and its y-axis is obtained by translating rightward from the v-axis of the pixel coordinate system;

[0037] Assume that the image center coordinates are expressed in the pixel coordinate system as (u 0 ,v 0 ), the pixel size of the camera imaging CCD is dx×dy; the relationship between any point (x, y) in the image coordinate system and the coordinates of the point in the pixel coordinate system (u, v) is expressed as:

[0038]

[0039] Coordinate system: Consider the drone and camera as a whole. The camera always looks downward relative to the drone. The optical center of the camera is taken as the origin of the camera coordinate system. c , x c Axis and y c The axes are parallel to the x-axis and y-axis of the image coordinate system, and the camera optical axis is used as the z-axis. c Axis, perpendicular to the imaging plane, establishes the camera coordinate system;

[0040] S22. Modeling

[0041] Conversion between image coordinate system and camera coordinate system:

[0042] The UAV collects ground information during operation, and the homogeneous coordinates of the identification point in the image coordinate system are P uv =(u,v,1) T , the coordinates of the recognition point in the camera coordinate system are P c =(x c ,y c ,z c ) T , according to the camera model formula:

[0043] z c ·P uv =K·P c

[0044] Where K is the camera intrinsic parameter matrix:

[0045]

[0046] Where f is the focal length of the camera, dx and dy are the horizontal and vertical dimensions of the pixel respectively;

[0047] u 0 and v 0 is the coordinate of the origin of the imaging plane in the pixel coordinate system, in pixels;

[0048] From the camera model formula:

[0049]

[0050] Among them, S c is the actual area of ​​the top surface of the rail fort, S uv is the area of ​​the top surface of the rail bar in the image coordinate system, S uv The calculation method is:

[0051]

[0052] (x 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 ) and (x 4 ,y 4 ) are the coordinates of the four vertices on the top surface of the rail fort captured in the image coordinate system.

[0053] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0054] It can be seen from the above technical scheme that the present invention proposes a high-speed rail fortress key point recognition and height inversion algorithm based on deep neural network. On the basis of existing unmanned obstacle removal or strike platforms such as drones, by adding a monocular visible light camera, an intelligent edge computing device, a high-speed target key point detection algorithm based on a deep neural network and a height inversion algorithm, the drone platform can quickly detect and identify the position and key points of the rail fortress target, complete the height inversion of the drone from the upper surface of the rail fortress, realize the monocular camera ranging, and lay the foundation for the unmanned platform to hover, aim and remove obstacles.

[0055] Specifically, the present invention sets a monocular camera for real-time acquisition of visible light images of the target and background; the intelligent edge computing device is used to receive the real-time image taken by the monocular camera through the CSI video interface, and runs the target key point detection algorithm and the height inversion algorithm, and is connected to the UAV flight control system; the high-speed target key point detection algorithm based on the deep neural network processes the real-time image, detects the rail fortress target from the image and calculates the pixel coordinates of the four key points on the top surface;

[0056] The height inversion algorithm based on the monocular camera is specifically as follows: when the actual area of ​​a surface and the pixel area are known, the distance between the camera sensor plane and the top surface of the target is calculated using the height inversion formula based on coordinate system conversion and area ratio. The actual area of ​​the top surface of the rail fort target can be calculated using the national standard and other conditions; the pixel coordinates of the four corner points of the top surface of the rail fort can be provided using the high-speed target key point detection algorithm based on the deep neural network; the height inversion algorithm uses the pixel coordinates of the four corner points of the top surface of the rail fort to calculate the pixel area of ​​the top surface of the target; the height inversion algorithm uses the actual area of ​​the top surface of the rail fort and the measured pixel area to calculate the distance between the camera sensor plane and the top surface of the target; the camera is installed on the unmanned system, and the distance between the camera sensor and the target can be used as the height of the unmanned system from the target for use by the flight control system.

[0057] The invention discloses a high-speed rail fort key point recognition method based on a deep neural network, and constructs a target key point detection algorithm and a height inversion algorithm based on a deep neural network. The key point detection algorithm can not only detect the position and size of the target in real time, but also detect the pixel coordinates of five key points, including four corner points and one center point on the top surface of the target; based on the key point coordinates, a monocular visible light camera is required to realize the real-time distance measurement of the rail fort target by the drone during flight through the height inversion algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the device composition and data processing of an embodiment of the present invention;

[0059] Figure 2It is a schematic diagram of high-speed rail key point recognition and height inversion of a deep neural network according to an embodiment of the present invention;

[0060] Figure 3 It is the FPN module adopted by the model of the embodiment of the present invention;

[0061] Figure 4 is a schematic diagram of a context module according to an embodiment of the present invention;

[0062] Figure 5 is a schematic diagram of the output result vector of an embodiment of the present invention;

[0063] Figure 6 Schematic diagram of key point detection results of rail barricades according to an embodiment of the present invention;

[0064] Figure 7 is a pixel coordinate system according to an embodiment of the present invention;

[0065] Figure 8 is the image coordinate system of the embodiment of the present invention;

[0066] Fig. 9 is the camera coordinate system of the embodiment of the present invention (UAV);

[0067] Fig.10 The drone of the embodiment of the present invention collects ground information;

[0068] Fig.11 This is a picture collected by the drone of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0070] like Figure 1 As shown, the high-speed rail fortress key point recognition method based on deep neural network described in this embodiment includes application to an unmanned platform, including a monocular visible light camera and an intelligent edge computing device communication connection, the monocular visible light camera is installed on the unmanned platform, and is used to detect and identify rail fortress targets, including the following steps:

[0071] The visible light camera is used to capture visible light images of the target area and output visible light video streams;

[0072] The intelligent edge computing device receives the visible light camera video, runs a high-speed target key point detection algorithm and a height inversion algorithm, and connects to the UAV flight control system;

[0073] Among them, through the high-speed target key point detection algorithm, the real-time detection of the target key points in the video stream is realized, and the pixel coordinates of the target key points are output, and the pixel length, pixel width and pixel position of the target rectangular bounding box are output at the same time;

[0074] Through the height inversion algorithm, the distance between the monocular visible light camera and the top surface of the track fortress is calculated according to the pixel coordinates of the target key points, that is, the height data of the unmanned system from the top surface of the track fortress.

[0075] Specifically, the present invention sets a monocular camera for real-time acquisition of visible light images of the target and background; the intelligent edge computing device is used to receive the real-time image taken by the monocular camera through the CSI video interface, and runs the target key point detection algorithm and the height inversion algorithm, and is connected to the UAV flight control system; the high-speed target key point detection algorithm based on the deep neural network processes the real-time image, detects the rail fortress target from the image and calculates the pixel coordinates of the four key points on the top surface;

[0076] The height inversion algorithm based on the monocular camera is specifically as follows: when the actual area of ​​a surface and the pixel area are known, the distance between the camera sensor plane and the top surface of the target is calculated using the height inversion formula based on coordinate system conversion and area ratio. The actual area of ​​the top surface of the rail fort target can be calculated using the national standard and other conditions; the pixel coordinates of the four corner points of the top surface of the rail fort can be provided using the high-speed target key point detection algorithm based on the deep neural network; the height inversion algorithm uses the pixel coordinates of the four corner points of the top surface of the rail fort to calculate the pixel area of ​​the top surface of the target; the height inversion algorithm uses the actual area of ​​the top surface of the rail fort and the measured pixel area to calculate the distance between the camera sensor plane and the top surface of the target; the camera is installed on the unmanned system, and the distance between the camera sensor and the target can be used as the height of the unmanned system from the target for use by the flight control system.

[0077] The following are respectively explained:

[0078] High-speed rail key point detection algorithm based on deep learning, such as Figure 2 As shown;

[0079] A deep learning based model for rail target keypoint detection is proposed using a multi-task loss with additional supervision and self-supervision signals.

[0080] 1.1 Dataset Construction

[0081] By marking the four corner points and one center point of the top surface of the rail fort target, a total of five key points are combined with the location box of the rail fort cement pier as the annotation information. Compared with the conventional deep learning-based detection algorithm, the five key points are used as external supervision signals.

[0082] 1.2 Model Structure

[0083] (1) Bottle neck: Feature pyramid FPN module, such as Figure 3 As shown;

[0084] The model contains an FPN module to extract four levels of features from the original image. First, the backbone network performs bottom-up feature extraction (downsampling ratio is 0.5) to obtain the feature map {C 2 ,C 3 ,C 4 ,C 5}; then {C 2 ,C 3 ,C 4}Perform a lateral connection, that is, {C 2 ,C 3 ,C 4} pass through a 1×1 convolutional layer to fuse (channel) information and reduce the channel dimension; finally, the channel fused {C 2 ,C 3 ,C 4 The corresponding elements of the feature map obtained by top-down upsampling (up sampling, sampling rate is 2) are added and fused to obtain the feature pyramid layer {P 2 ,P 3 ,P 4}. The top feature map shares C with the top pyramid layer 5 =P 5 The FPN module is shown in the figure. Each layer of PFN will have strong semantic information.

[0085] (2)head: context module, such as Figure 4 As shown;

[0086] This model uses a context module as the head. Independent context modules are applied at 4 feature pyramid levels. Specifically, the original 3×3 convolution layer in the FPN structure is replaced by a deformable convolution.

[0087] (3) Loss Function

[0088] The loss function of the model Designed as a multi-task joint loss function:

[0089]

[0090] In the formula, and They are classification (target / non-target binary classification) loss, bounding box regression loss, and key point deviation loss; λ 1 and λ 2 are the weight coefficients for bounding box regression and key point prediction losses respectively; They are category, bounding box, key point true value, and when there is a target at a certain anchor point otherwise c i , r i , p i They are the predicted values ​​of categories, bounding boxes, and key points respectively.

[0091] (4) Model output

[0092] like Figure 5 As shown in the figure, the output results of the model forward propagation include: ① classification result: 1 for target, 0 for background; ② the coordinates of the upper left corner and width and height of the target's minimum enclosing rectangle: (x, y, w, h); ③ the coordinates of the five key points of the target (x i ,y i ),i∈[1,5]; Figure 6 It is the key point detection result of the rail fortress.

[0093] 2 Height inversion algorithm

[0094] 2.1 Coordinate system

[0095] (1) Pixel coordinate system

[0096] Pixel coordinates describe the specific location of each pixel in the image. Generally, the upper left corner of the image is used as the origin of the pixel coordinate system. p , the horizontal right is the u axis of the coordinate system, and the vertical downward is the v axis of the coordinate system. Figure 7 shown.

[0097] The unit of the pixel coordinate system is pixel. Any pixel in the pixel coordinate system can be expressed as (u i ,v i ).

[0098] (2) Image coordinate system

[0099] The pixel coordinate system can only describe the coordinates of each pixel point, and cannot specifically describe the scale of the object in the image. Therefore, the image coordinate system must be introduced to convert the pixel coordinates of the object in the image into image coordinates.

[0100] like Figure 8 As shown, the specific method of coordinate system conversion is: the origin O of the image coordinate system iis the center of the image, its x-axis is obtained by translating downward from the u-axis of the pixel coordinate system, and its y-axis is obtained by translating rightward from the v-axis of the pixel coordinate system.

[0101] like Figure 2 As shown, assuming that the image center coordinates are expressed in the pixel coordinate system as (u 0 ,v 0 ), the pixel size of the camera imaging CCD is dx×dy. The relationship between any point (x, y) in the image coordinate system and the coordinates of the point in the pixel coordinate system (u, v) can be expressed as:

[0102]

[0103] (3) Coordinate system

[0104] The prerequisite is that the camera carried by the drone is fixed relative to the drone, so the drone and the camera can be regarded as a whole. Fig. 9 As shown, the camera is always looking downward relative to the drone. The optical center of the camera is taken as the origin of the camera coordinate system O c , x c Axis and y c The axes are parallel to the x-axis and y-axis of the image coordinate system, and the camera optical axis is used as the z-axis. c Axis, perpendicular to the imaging plane, establishes the camera coordinate system.

[0105] 2.2 Modeling

[0106] (1) Conversion between image coordinate system and camera coordinate system

[0107] The drone collects ground information during operation, such as Fig.10 As shown, the captured images are as follows Fig.11 shown.

[0108] The homogeneous coordinates of the identified point in the image coordinate system are P uv =(u,v,1) T , the coordinates of the recognition point in the camera coordinate system are P c =(x c ,y c ,z c ) T . According to the camera model formula:

[0109] z c ·P uv =K·P c

[0110] Where K is the camera intrinsic parameter matrix:

[0111]

[0112] Where f is the focal length of the camera, dx and dy are the horizontal and vertical dimensions of the pixel respectively. Generally, dx = dy. (All units are m)

[0113] u 0 and v 0 is the coordinate of the origin of the imaging plane in the pixel coordinate system, in pixels, and is generally taken as half of the image pixel width w and height h. 0 =w / 2,v 0 =h / 2.

[0114] From the camera model formula we can get:

[0115]

[0116] Among them, S c is the actual area of ​​the top surface of the rail fort, S uv is the area of ​​the top surface of the rail bar in the image coordinate system, S uv The calculation method is:

[0117]

[0118] (x 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 ) and (x 4 ,y 4 ) are the coordinates of the four vertices of the top surface of the rail fort captured in the image coordinate system.

[0119] The key point recognition algorithm described in the embodiment of the present invention adopts a deep neural network structure; the key point recognition algorithm can simultaneously perform multiple tasks of target position detection and target internal key point detection; the key point recognition algorithm can run in real time on the selected intelligent edge computing device, with a frame rate ≥ 25PFS (frames / s). The height inversion algorithm does not require binocular cameras, lidar, ultrasonic and other ranging hardware devices to provide additional information; the height inversion algorithm formula obtains the inversion formula through coordinate transformation through the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system; the height inversion algorithm can calculate the height of the camera from the top surface of the rail fortress target in real time.

[0120] In summary, the embodiment of the present invention proposes a high-speed rail fortress key point recognition and height inversion algorithm and device based on deep neural network. On the basis of existing unmanned obstacle removal or strike platforms such as drones, by adding a monocular visible light camera, an intelligent edge computing device, a high-speed target key point detection algorithm and a height inversion algorithm based on a deep neural network, the drone platform can quickly detect and identify the position and key points of the rail fortress target, complete the height inversion of the drone from the upper surface of the rail fortress, realize the monocular camera ranging, and lay the foundation for the unmanned platform to hover, aim and remove obstacles.

[0121] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of any of the above methods.

[0122] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any of the above methods.

[0123] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any method in the above embodiments.

[0124] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-speed rail key point recognition method based on deep neural network, characterized in that: The invention is applied to an unmanned platform, including a monocular visible light camera and an intelligent edge computing device communication connection. The monocular visible light camera is installed on the unmanned platform and used to detect and identify railroad fortification targets, including the following steps: The visible light camera is used to capture visible light images of the target area and output visible light video streams; The intelligent edge computing device receives the visible light camera video, runs a high-speed target key point detection algorithm and a height inversion algorithm, and connects to the UAV flight control system; Among them, through the high-speed target key point detection algorithm, the real-time detection of the target key points in the video stream is realized, and the pixel coordinates of the target key points are output, and the pixel length, pixel width and pixel position of the target rectangular bounding box are output at the same time; Through the height inversion algorithm, according to the pixel coordinates of the target key points, the distance between the monocular visible light camera and the top surface of the rail fort is calculated, that is, the height data of the unmanned system from the top surface of the rail fort; The height inversion algorithm comprises the following steps, S21. Coordinate system Pixel coordinate system: Pixel coordinates describe the specific position of each pixel in the image, with the upper left corner of the image as the origin of the pixel coordinate system. , horizontally to the right is the coordinate system Axis, vertically downward is the coordinate system axis; The unit of the pixel coordinate system is pixel. Any pixel in the pixel coordinate system is represented as ; Image coordinate system: convert the pixel coordinates of the object in the image into image coordinates; The specific method of coordinate system conversion is: the origin of the image coordinate system is the center of the image, The axis is the pixel coordinate system. The axis is translated downward to obtain The axis is the pixel coordinate system. The axis is translated to the right to obtain; Assume that the image center coordinates are expressed in the pixel coordinate system as , camera imaging The pixel size is ; Any point in the image coordinate system The coordinates of the point in the pixel coordinate system The relationship between them is expressed as: Coordinate system: Consider the drone and camera as a whole. The camera is always looking downward relative to the drone. The optical center of the camera is taken as the origin of the camera coordinate system. , Axis and The axes are parallel to the x-axis and y-axis of the image coordinate system, and the camera optical axis is used as Axis, perpendicular to the imaging plane, establishes the camera coordinate system; S22. Modeling Conversion between image coordinate system and camera coordinate system: The UAV collects ground information during operation, and the homogeneous coordinates of the identification points in the image coordinate system are: , the coordinates of the identified point in the camera coordinate system are , according to the camera model formula: Where K is the camera intrinsic parameter matrix: in, is the camera focal length, and are the horizontal and vertical dimensions of the pixel respectively; and is the coordinate of the origin of the imaging plane in the pixel coordinate system, in pixels; From the camera model formula: in, is the actual area of ​​the top surface of the rail fort, is the area of ​​the top surface of the rail bar in the image coordinate system, The calculation method is: , , and They are the coordinates of the four vertices of the top surface of the rail fort captured in the image coordinate system.

2. The high-speed rail key point identification method based on deep neural network according to claim 1 is characterized in that: The high-speed target key point detection algorithm comprises the following steps: Dataset construction: by marking the four corner points and one center point on the top surface of the rail fort target, a total of five key points combined with the location box of the rail fort cement pier as annotation information; The model structure is bottle neck: feature pyramid FPN module; The model includes an FPN module to extract four levels of features from the original image. First, the backbone network is used to perform bottom-up feature extraction to obtain a feature map. ; then Perform a lateral connection, that is, Through one The convolutional layer is used to fuse information and reduce the channel dimension. Finally, the channel fused The corresponding elements of the feature map obtained by top-down upsampling are added and fused to obtain the feature pyramid layer ; The top feature map is shared with the top pyramid layer ; Also includes head: context module; The context module is used as the head, and independent context modules are applied at the four feature pyramid levels. Specifically, deformable convolution is used to replace the original FPN structure. Convolutional layers; The loss function of the model Designed as a multi-task joint loss function: In the formula, , and They are classification loss, bounding box regression loss, and key point deviation loss; and are the weight coefficients for bounding box regression and key point prediction losses respectively; , , They are category, bounding box, key point true value, and when there is a target at a certain anchor point ,otherwise ; , , They are the predicted values ​​of categories, bounding boxes, and key points respectively; The output results of the model forward propagation include: Classification results: 1 for target, 0 for background; The upper left corner coordinates, width and height of the target minimum enclosing rectangle: ; Coordinates of the five key points of the target .

3. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the method according to claim 1 or 2.

Images