Railway inspection scene modeling method based on image recognition

Abstract

The invention discloses a railway inspection scene modeling method based on image recognition. The method comprises the following steps: acquiring a visible light image set and an infrared image set; preprocessing the image; obtaining a first positive example feature map; obtaining a second positive example feature map; fusing the image corresponding to the first positive example feature map under the same semantics and the image corresponding to the second positive example feature map under the same semantics to obtain a fused image; acquiring an image target frame of the fused image; acquiring a corresponding relation between the image target frame and the GPS coordinate position; and constructing a three-dimensional map. According to the invention, the visible light image and the infrared image are fused for use, so that all-time image detection along the railway can be realized; the first positive example feature map and the second positive example feature map are screened, a target frame is positioned for the image with the sensitive event label, and the calculation amount is reduced; the three-dimensional scene is obtained according to the image target frame, and meanwhile, the position of the three-dimensional scene is accurately positioned, so that a worker is assisted to quickly go to an abnormal place, and problem tracking and closed-loop management are realized.

Description

Technical Field

[0001] The invention belongs to the technical field of image data processing and image recognition, and in particular relates to a railway inspection scene modeling method based on image recognition. Background Art

[0002] The workflow of railway inspection is a systematic and meticulous process aimed at ensuring the safe, stable and reliable operation of railway lines and their related facilities.

[0003] Comprehensive inspection: The inspectors start from the station where they are located and check the operation of each signal machine, station sign, crossing and other facilities one by one according to the line inspection map, and check whether the signal board, station name board, kilometer sign and other signboards are intact. At the same time, a comprehensive inspection is carried out on all aspects of the train running line, signal equipment, electrical facilities, tracks and switches.

[0004] During the inspection, inspectors need to conduct a comprehensive inspection of the train lines, train operating environment, signal equipment, electrical facilities, tracks and switches. Signal equipment detection includes the normal operation of signal lights, traffic lights, cables, contact networks and other equipment. For train lines, it is necessary to check whether the switch, gauge, track and other equipment are intact, without abnormal ferromagnetic damage and corrosion, bolts, nails and other conditions; for the train operating environment, it is necessary to check whether there are natural disasters such as landslides and mudslides in tunnels, culverts, bridges, ditches, etc., and whether there are animals, plants or human occupation. For signal machines, it is necessary to check whether the line indicator lights on them are normal and whether various displays accurately show the location of the train.

[0005] If any equipment failure or safety hazard is found, the inspectors shall deal with it in time. For problems that can be solved on site, such as clearing tree branches, weeds and other obstacles, the inspectors can handle them by themselves; for problems that cannot be solved on site, such as equipment failure, track damage, etc., they shall report them to the superior department in time.

[0006] Railway inspections face many challenges, including:

[0007] 1. Data management is difficult. There are barriers to sharing inspection data obtained by different inspectors, inspection data of railway lines in different sections, and inspection data between inspectors and managers, which leads to poor information flow and affects the overall efficiency of inspection work.

[0008] 2. Environmental challenges. Under extreme weather conditions such as heavy rain, heavy snow, high temperature, and haze, the difficulty of both drone inspections and manual inspections increases significantly. These conditions not only affect the sight of inspectors and machine vision, but may also cause the performance of inspection equipment to decline.

[0009] Therefore, we hope to have an inspection method that can transmit railway line inspection data back to the system background, establish global and full-scenario inspection data sharing, and improve inspection efficiency even under low visibility conditions, so as to assist staff to grasp the on-site conditions of railway lines in a timely manner and promote the railway line operation and maintenance from "fault repair" and "planned repair" to a new model of "condition repair" and "predictive repair". Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a railway inspection scene modeling method based on image recognition in response to the deficiencies in the above-mentioned prior art. The method has a simple structure and a reasonable design. By fusing visible light images and infrared images, it can realize full-time image detection along the railway. It can locate the target frame of images with sensitive event labels to reduce the amount of calculation. It can obtain a three-dimensional scene based on the image target frame, and accurately locate the position of the three-dimensional scene, so as to assist staff to quickly go to abnormal locations and realize problem tracking and closed-loop management.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is: a railway inspection scene modeling method based on image recognition, characterized in that it includes the following steps:

[0012] Step 1: obtain real-time images along the railway from the image acquisition unit carried by the UAV to obtain a visible light image set and an infrared image set; pre-process the visible light image set and the infrared image set respectively;

[0013] Step 2: Input each visible light image frame in the visible light image set into a trained first target detection and recognition model, and the first target detection and recognition model outputs a first feature map, wherein the first feature map includes a first positive feature map with a sensitive event label and a first negative feature map without a sensitive event label;

[0014] Step 3: Input each infrared image in the infrared image set into the trained second target detection and recognition model, and the second target detection and recognition model outputs a second feature map, which includes a second positive feature map with a sensitive event label and a second negative feature map without a sensitive event label;

[0015] Step 4: fuse the images of the first positive feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a first fused image set; fuse the images of the second positive feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a second fused image set; the first fused image set and the second fused image set constitute a fused image set;

[0016] Step 5: The images in the fusion image set are input into the target frame recognition model to obtain the image target frame;

[0017] Step 6: Establish the corresponding relationship between the pixel points in the image target frame and the actual GPS coordinate position;

[0018] Step 7: Generate orthophoto and DEM data based on the image target frame;

[0019] Step 8: Build a three-dimensional map.

[0020] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the training method of the "first target detection and recognition model" is:

[0021] Step 101, obtaining a first sample image based on visible light;

[0022] Step 102, manually classify and annotate the first sample images, obtain the target category of each first sample image, and form a first data set, where the target category includes a normal label and a sensitive event label;

[0023] Step 103, based on the CNN network structure, construct a first target detection and recognition model, the first target detection and recognition model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer;

[0024] Step 104: Input the first sample image into the first target detection and recognition model for machine learning. The pooling layer of the first target detection and recognition model outputs a feature map of the first sample image. The output layer of the first target detection and recognition model outputs a predicted probability that the first sample image belongs to each target category. After the learning is completed, a trained first target detection and recognition model is obtained.

[0025] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the training method of the "second target detection and recognition model" is:

[0026] Step 201, obtaining a second sample image based on infrared thermal imaging;

[0027] Step 202, manually classify and annotate the second sample images to obtain the target category of each second sample image to form a second data set;

[0028] Step 203, based on the ISTDU-Net network structure, construct a second target detection and recognition model, the second target detection and recognition model includes an encoder, a skip connection, a decoder and a classification layer;

[0029] Step 204: input the second sample image into the second target detection and recognition model for machine learning. The decoder of the second target detection and recognition model outputs a feature map of the second sample image. The classification layer of the second target detection and recognition model outputs the predicted probability that the second sample image belongs to each target category. After the learning is completed, a trained second target detection and recognition model is obtained.

[0030] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the specific step of "fusing the images corresponding to the first positive example feature map under the same semantics in the visible light image set and the infrared image set respectively" is to regard each pixel of the image corresponding to the first positive example feature map in the visible light image set and the infrared image set respectively as an independent sample, learn the relationship between each pixel through a deep neural network, and then fuse the pixels of the two images according to the fusion strategy.

[0031] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the training method of the target frame recognition model is:

[0032] Step 301: using a camera and an infrared thermal imager carried by a drone to respectively acquire a first sample image and a second sample image along the railway, and fusing the first sample image and the second sample image under the same semantics to obtain a third sample image;

[0033] Step 302: manually marking the upper left corner point and the lower right corner point of the third sample image, wherein the combination of the upper left corner point and the lower right corner point constitutes a target frame;

[0034] Step 303: Building a target box recognition model based on the CornerNet network structure;

[0035] Step 304: Send the third sample image to the target frame recognition model for machine learning, and obtain a trained target frame recognition model after the learning is completed.

[0036] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the specific method of "establishing the correspondence between the pixel points in the image target frame and the actual GPS coordinate positions" is:

[0037] Step 401, obtaining the intrinsic parameters and extrinsic parameters of the camera; the intrinsic parameters include the focal length of the camera and the size of the imaging sensor, and the extrinsic parameters include the latitude and longitude, altitude, orientation and tilt angle of the camera during shooting;

[0038] Step 402: Convert the pixel coordinates in the image target frame into three-dimensional coordinates in the camera coordinate system;

[0039] Step 403: convert the three-dimensional coordinates in the camera coordinate system into coordinates in the world coordinate system;

[0040] Step 404, converting the coordinates in the world coordinate system into geodetic coordinates;

[0041] Step 405, converting the geodetic coordinates into a plane coordinate system; the plane coordinate system adopts the UTM projection coordinate system, which is the coordinate system adopted by most national public map websites on the Internet.

[0042] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the external parameters of the camera are obtained by the sensor module carried by the drone.

[0043] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the specific method of "constructing a three-dimensional map" is:

[0044] Step 601, creating a new three-dimensional scene in GIS software;

[0045] Step 602: import the DEM data corresponding to each image target frame into the three-dimensional scene, and the GIS software generates a terrain surface according to the DEM data;

[0046] Step 603: superimpose the orthophoto corresponding to each image target frame as a texture on the terrain surface;

[0047] Step 604: Integrate other geospatial data of each actual GPS coordinate position corresponding to the image target frame into the three-dimensional scene; other geospatial data include roads, buildings, vegetation, etc., which can be imported in vector or raster form and adjusted in position, resized and styled as needed;

[0048] Step 605: Optimize and render the three-dimensional scene.

[0049] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that: the three-dimensional scene is published on the platform, and the terminal accesses the platform through a web page, an application or scanning a QR code to view the three-dimensional scene.

[0050] The above-mentioned railway inspection scene modeling method based on image recognition is characterized in that the actual GPS coordinate position corresponding to the image target frame is added to the re-inspection position database.

[0051] Compared with the prior art, the present invention has the following advantages:

[0052] 1. The invention has a simple structure, reasonable design, and is easy to implement and operate.

[0053] 2. The present invention first screens images with sensitive event labels, then locates the target frame for the images with sensitive event labels, and screens them layer by layer to reduce the amount of calculation.

[0054] 3. The present invention combines visible light images and infrared images. Infrared images are not affected by visibility and can produce clear images even in darkness or inclement weather conditions. They can display sensitive events such as biological invasions and fires. When used in conjunction with visible light images, they can achieve full-time image detection along the railway and provide early warning information for staff.

[0055] 4. Based on the ISTDU-Net network structure, the present invention adds a classification layer to expand the ISTDU-Net network structure to output target classification labels. The ISTDU-Net network effectively improves the detection performance of small infrared targets through a unique feature extraction and enhancement mechanism.

[0056] 5. The present invention obtains a three-dimensional scene based on an image target frame, and simultaneously accurately locates the position of the three-dimensional scene, and pushes it to the platform, thereby assisting staff to quickly go to abnormal locations and achieve problem tracking and closed-loop management.

[0057] In summary, the present invention has a simple structure and a reasonable design. By fusing visible light images and infrared images, it can realize full-time image detection along the railway; locate the target frame of images with sensitive event labels to reduce the amount of calculation; obtain a three-dimensional scene based on the image target frame, and accurately locate the position of the three-dimensional scene, assisting staff to quickly go to abnormal locations and realize problem tracking and closed-loop management.

[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The figure is a flow chart of the method of the present invention.

[0060] Figure 2 This is a comparison diagram of the visible light image and the infrared image under the same semantics of the present invention.

[0061] Figure 3 It is a three-dimensional modeling diagram of a local railway segment of the present invention.

[0062] Figure 4 It is the orthophoto of the local scene of the present invention.

[0063] Figure 5 This is a three-dimensional modeling diagram of a local railway station according to the present invention. DETAILED DESCRIPTION

[0064] The method of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0065] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0066] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0067] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0069] like Figure 1 As shown, a railway inspection scene modeling method based on image recognition of the present invention comprises the following steps:

[0070] Step 1: Acquire real-time images along the railway from the image acquisition unit carried by the UAV to obtain a visible light image set and an infrared image set; pre-process the visible light image set and the infrared image set respectively.

[0071] In actual use, the real-time images include visible light images captured by the camera and infrared images captured by the infrared thermal imager. The visible light images constitute a visible light image set, and the infrared images constitute an infrared image set.

[0072] For example, in the railway inspection area, animals such as sheep, birds, snakes, and rodents may enter the railway track area and be blocked by the railway track, posing a threat to the safety of train operation. In the case of low visibility or when the animals are blocked, the visible light image collected by the camera cannot reflect this information. However, the infrared thermal imager receives the temperature emitted by the target object and displays it as a visible thermal image after processing. This technology is not affected by visible light and can penetrate obstructions such as haze and smoke. It can also clearly image even in darkness or bad weather conditions, and can show the presence of these small animals. It is very suitable for foreign body intrusion detection during railway inspections. When used in conjunction with a camera, it can realize full-time image detection along the railway and provide early warning information to staff.

[0073] Step 2: First, train the first target detection and recognition model. The training method of the "first target detection and recognition model" is:

[0074] Step 101, obtaining a first sample image based on visible light;

[0075] Step 102, manually classify and annotate the first sample images to obtain the target category of each first sample image to form a first data set;

[0076] Step 103, based on the CNN network structure, construct a first target detection and recognition model, the first target detection and recognition model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer;

[0077] Step 104: input the first sample image into the first target detection and recognition model for machine learning. The pooling layer of the first target detection and recognition model outputs a feature map of the first sample image, and the output layer of the first target detection and recognition model outputs a predicted probability that the first sample image belongs to each target category.

[0078] In this embodiment, the CNN network structure adopts a cross entropy loss function, which can well characterize the probability distribution and is highly robust to noise. The loss is calculated based on the difference between the predicted probability and the true label probability distribution to measure the performance of the model, and the parameters in the first target detection and recognition model are updated through optimization algorithms such as gradient descent to minimize the loss.

[0079] After the learning is completed, the trained first target detection and recognition model is obtained.

[0080] Then, each frame of visible light image in the visible light image set is input into the trained first target detection and recognition model respectively, the pooling layer of the first target detection and recognition model outputs the first feature map, and the output layer of the first target detection and recognition model outputs the predicted probability that the visible light image belongs to each target category.

[0081] In one possible embodiment, the first sample image based on visible light includes the RailSem19 dataset and a sensitive event set based on drone photography. The RailSem19 dataset is a dataset focused on railway scene understanding, providing 8,500 unique images taken from the train's own perspective. These images cover a wide range of railway scenes, including train operation, track crossings, stations, etc.

[0082] In a possible embodiment, the target categories include normal labels and sensitive event labels. 1,000 images from the RailSem19 dataset are selected and labeled with normal labels. Sensitive event labels include unauthorized personnel intruding into the protective net, livestock intrusion, and fire. For the sensitive event set based on drone photography, sensitive event labels are manually labeled.

[0083] In another possible embodiment, the target categories include normal labels and sensitive event labels, and the sensitive event labels include railway cracks, foreign objects intruding on the track, strangers intruding into the protective net, livestock intrusion, and fire.

[0084] If the target category corresponding to the largest predicted probability belongs to the sensitive event label, the visible light image belongs to the first positive feature map.

[0085] Visible light images with sensitive event labels are screened through the first target detection and recognition model to reduce the subsequent calculation amount.

[0086] Step 3: First, train the second target detection and recognition model. The training method of the “second target detection and recognition model” is:

[0087] Step 201: Acquire a second sample image based on infrared thermal imaging.

[0088] In one possible embodiment, the second sample image based on infrared thermal imaging includes a data set of electrical equipment detection beside the railway, a data set of foreign object intrusion detection on the rails, and a sensitive event set based on drone photography. 600 images out of 4,000 images in the data set of electrical equipment detection beside the railway and 600 images out of 14,000 images in the data set of foreign object intrusion detection on the rails are selected and annotated with normal labels. The sensitive event labels include intrusion of strangers into the protective net, intrusion of livestock, and fire. For the sensitive event set based on drone photography, the sensitive event labels are manually annotated.

[0089] Step 202, manually classify and annotate the second sample images to obtain the target category of each second sample image to form a second data set;

[0090] Step 203: Based on the ISTDU-Net network structure, a second target detection and recognition model is constructed, where the second target detection and recognition model includes an encoder, a skip connection, a decoder, and a classification layer.

[0091] It should be noted that the first target detection and recognition model is based on the CNN network structure, and the second target detection and recognition model is based on the ISTDU-Net network structure. ISTDU-Net is a deep learning network specifically used for infrared small target detection. The network is based on a U-shaped structure and effectively improves the detection performance of infrared small targets through a unique feature extraction and enhancement mechanism.

[0092] In this embodiment, on the basis of the ISTDU-Net network structure, a classification layer is added to expand the ISTDU-Net network structure into an output target classification label. The classification layer includes a fully connected layer, which is used to map the extracted features to the target category. The number of output nodes of the fully connected layer should match the number of target categories, and the output of each node represents the probability that the input image belongs to the category. In a possible embodiment, the target categories include normal labels, idle personnel intrusion protection nets, livestock intrusions, and fires, so the number of output nodes of the fully connected layer is 4.

[0093] For this embodiment, the prediction probability is a multi-category classification task, so the loss function adopts a multi-classification cross entropy loss function, which performs well when dealing with multi-classification problems and can effectively promote the second target detection and recognition model to converge to the optimal solution during the training process.

[0094] Step 204: input the second sample image into the second target detection and recognition model for machine learning. The decoder of the second target detection and recognition model outputs a feature map of the second sample image. The classification layer of the second target detection and recognition model outputs the predicted probability that the second sample image belongs to each target category. After the learning is completed, a trained second target detection and recognition model is obtained.

[0095] Specifically, the softmax function is used in the classification layer of the ISTDU-Net network structure. The softmax function converts the output of the decoder into a probability distribution, and then the difference between the predicted probability distribution and the true label distribution is calculated through the multi-classification cross entropy loss function. The detection and classification performance is optimized by adjusting the network parameters and the weights of the loss function, and the performance of the second target detection and recognition model is evaluated using indicators such as ROC curves, AUC values, and classification accuracy. The parameters in the second target detection and recognition model are updated through optimization algorithms such as gradient descent to minimize the loss.

[0096] Then, each frame of infrared image in the infrared image set is input into the trained second target detection and recognition model respectively, the pooling layer of the second target detection and recognition model outputs the second feature map, and the classification layer of the second target detection and recognition model outputs the predicted probability that the infrared image belongs to each target category.

[0097] In a possible embodiment, the target categories include normal labels and sensitive event labels, and the sensitive event labels include strangers intruding into the protective net, livestock intrusion, and fire.

[0098] In another possible embodiment, the target categories include normal labels and sensitive event labels, and the sensitive event labels include railway cracks, foreign objects intruding on the track, strangers intruding into the protective net, livestock intrusion, and fire.

[0099] If the target category corresponding to the predicted probability with the largest value belongs to the sensitive event label, the infrared image corresponding to the predicted probability belongs to the second positive example feature map.

[0100] Visible light images with sensitive event labels are screened through the second target detection and recognition model to reduce the subsequent calculation amount.

[0101] In one possible embodiment, Figure 2 As shown in the figure, the infrared thermal imager and the camera respectively take pictures of the same section of railway. After grayscale processing, Figure 2 a is the grayscale image of infrared image, Figure 2 b is the grayscale image of the visible light image. It can be clearly seen that in an environment with low visibility, the infrared image has a very good imaging effect.

[0102] In actual use, the drone conducts automated aerial inspections along both sides of the railway line. The infrared thermal imager and camera mounted on the drone respectively take pictures of scenes with the same semantics. The drone background recognizes, classifies and locates the images, identifies images with sensitive event labels, divides the images into positive feature maps and negative feature maps based on the labels, and locates the GPS coordinates of the image target box.

[0103] Step 4: fuse the images of the first positive example feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a first fused image set; fuse the images of the second positive example feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a second fused image set; the first fused image set and the second fused image set constitute a fused image set.

[0104] In a possible embodiment, each pixel of the image corresponding to the first positive feature map in the visible light image set and the infrared image set is regarded as an independent sample, the relationship between each pixel is learned through a deep neural network, and then the pixels of the two images are fused according to a fusion strategy, which can use an autoencoder or a residual network.

[0105] In a possible embodiment, the images of the first positive feature map under the same semantics in the visible light image set and the infrared image set are fused respectively. First, the images of the first positive feature map under the same semantics in the visible light image set and the infrared image set are normalized and then spliced.

[0106] By stitching together infrared images and visible light images, we can combine the advantages of both and generate a fused image that contains both the target detection capability of infrared images and the color and texture information of visible light images. The fused image can provide more comprehensive and accurate scene information, which helps to improve the accuracy of target detection and recognition.

[0107] Step 5: First, train the target frame recognition model. The training method of the target frame recognition model is:

[0108] Step 301: using a camera and an infrared thermal imager carried by a drone to respectively acquire a first sample image and a second sample image along the railway, and fusing the first sample image and the second sample image under the same semantics to obtain a third sample image;

[0109] Step 302: manually marking the upper left corner point and the lower right corner point of the third sample image, wherein the combination of the upper left corner point and the lower right corner point constitutes a target frame;

[0110] Step 303: Based on the CornerNet network structure, a target box recognition model is constructed; CornerNet is a target detection model that predicts the bounding box of a target, and its network structure mainly consists of two parts: a backbone network and a prediction module. The loss functions of the CornerNet network structure include the loss of Heatmaps, the loss of Embeddings, and the loss of Offsets. These loss functions work together in the training process of the network, allowing the network to learn more accurate corner detection capabilities. The joint action of the loss function belongs to the prior art and will not be repeated here.

[0111] Step 304: Send the third sample image to the target frame recognition model for machine learning, and obtain a trained target frame recognition model after the learning is completed.

[0112] Then, the images in the fused image set in step 4 are respectively input into the target frame recognition model to obtain the image target frame.

[0113] The target box recognition model uses the upper left corner and the lower right corner to determine the target box, eliminating the need for four corner points to determine the target box.

[0114] Step 6: Establish the corresponding relationship between the pixel points in the image target frame and the actual GPS coordinate position; the specific method of "establishing the corresponding relationship between the pixel points in the image target frame and the actual GPS coordinate position" is:

[0115] Step 401, obtaining the intrinsic parameters and extrinsic parameters of the camera; the intrinsic parameters include the focal length of the camera and the size of the imaging sensor, and the extrinsic parameters include the latitude and longitude, altitude, orientation and tilt angle of the camera during shooting; the extrinsic parameters of the camera are obtained by the sensor module carried by the drone.

[0116] The sensor modules carried by the drone include inertial sensors, positioning sensors and magnetometers. These sensors measure the attitude and position of the drone, and then through camera calibration, the position and orientation of the camera can be obtained, thereby obtaining external parameters.

[0117] It should be noted that before the drone flies, the internal parameters of the camera information carried by the drone should be calibrated in advance.

[0118] Step 402: Convert the pixel coordinates in the image target frame into three-dimensional coordinates in the camera coordinate system;

[0119] Step 403: convert the three-dimensional coordinates in the camera coordinate system into coordinates in the world coordinate system;

[0120] Step 404, converting the coordinates in the world coordinate system into geodetic coordinates;

[0121] Step 405, converting the geodetic coordinates into a plane coordinate system; the plane coordinate system adopts the UTM projection coordinate system, which is the coordinate system adopted by most national public map websites on the Internet.

[0122] Step 7: Generate orthophoto and DEM data based on the image target frame. Figure 3 It is an orthophoto of the local scene.

[0123] Step 8: Build a 3D map. The specific method of "building a 3D map" is:

[0124] Step 601, creating a new three-dimensional scene in GIS software;

[0125] Step 602: import the DEM data corresponding to each image target frame into the three-dimensional scene, and the GIS software generates a terrain surface according to the DEM data;

[0126] Step 603: superimpose the orthophoto corresponding to each image target frame as a texture on the terrain surface;

[0127] Step 604: Integrate other geospatial data of each actual GPS coordinate position corresponding to the image target frame into the three-dimensional scene; other geospatial data include roads, buildings, vegetation, etc., which can be imported in vector or raster form and adjusted in position, resized and styled as needed;

[0128] Step 605: Optimizing and rendering the three-dimensional scene. Optimizing and rendering the three-dimensional scene refers to using the three-dimensional scene of the public map to fill in the blank areas of the three-dimensional scene constructed in the GIS software.

[0129] like Figure 4 and Figure 5 As shown, the three-dimensional scene constructed in the GIS software is only for the target frame image, and the blank scenes in the map are filled with the three-dimensional scenes of the public map. Figure 4 It is the orthophoto of the local scene of the present invention. Figure 5 This is a three-dimensional modeling diagram of a local railway station according to the present invention.

[0130] There are many public maps with three-dimensional maps. In one possible embodiment, the three-dimensional scenes of the public maps use the three-dimensional map data of TiandiTu. TiandiTu is a geographic information comprehensive service website led by the National Administration of Surveying, Mapping and Geoinformation of China. It not only provides two-dimensional map services, but also covers the display of three-dimensional maps. The three-dimensional map data of TiandiTu can exchange data with GIS software and support the import and export of multiple data formats, including common Shapefiles and CAD files.

[0131] The backend of the drone has an image recognition module, an image positioning module, a 3D modeling module and a storage module. The backend of the drone is used to locate the image target frame and build a 3D scene for the pixels of the image target frame. After the 3D scene is built, it is stored in the storage module and published on the platform. The terminal that the staff can view can access the platform through web pages, applications or by scanning QR codes to view the 3D scene, helping the staff to grasp the on-site situation of the railway line in a timely manner.

[0132] In one possible embodiment, the actual GPS coordinate position corresponding to the image target frame is added to the recheck position database. The coordinate position in the recheck position database is added to the flight path, and the drone rechecks the coordinate position in the recheck position database according to the pre-planned flight path during a patrol.

[0133] The above description is only an embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A railway inspection scene modeling method based on image recognition, characterized in that: The following steps are involved: Step 1: obtain real-time images along the railway from the image acquisition unit carried by the UAV to obtain a visible light image set and an infrared image set; pre-process the visible light image set and the infrared image set respectively; Step 2: Input each visible light image frame in the visible light image set into a trained first target detection and recognition model, and the first target detection and recognition model outputs a first feature map, wherein the first feature map includes a first positive feature map with a sensitive event label and a first negative feature map without a sensitive event label; Step 3: Input each infrared image in the infrared image set into the trained second target detection and recognition model, and the second target detection and recognition model outputs a second feature map, which includes a second positive feature map with a sensitive event label and a second negative feature map without a sensitive event label; Step 4: fuse the images of the first positive feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a first fused image set; fuse the images of the second positive feature map under the same semantics in the visible light image set and the infrared image set respectively to obtain a second fused image set; the first fused image set and the second fused image set constitute a fused image set; Step 5: The images in the fusion image set are input into the target frame recognition model to obtain the image target frame; Step 6: Establish the corresponding relationship between the pixel points in the image target frame and the actual GPS coordinate position; Step 7: Generate orthophoto and DEM data based on the image target frame; Step 8: Build a three-dimensional map.

2. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The training method of the "first target detection and recognition model" is: Step 101, obtaining a first sample image based on visible light; Step 102, manually classify and annotate the first sample images, obtain the target category of each first sample image, and form a first data set, where the target category includes a normal label and a sensitive event label; Step 103, based on the CNN network structure, construct a first target detection and recognition model, the first target detection and recognition model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer and an output layer; Step 104: Input the first sample image into the first target detection and recognition model for machine learning. The pooling layer of the first target detection and recognition model outputs a feature map of the first sample image. The output layer of the first target detection and recognition model outputs a predicted probability that the first sample image belongs to each target category. After the learning is completed, a trained first target detection and recognition model is obtained.

3. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The training method of the "second target detection and recognition model" is: Step 201, obtaining a second sample image based on infrared thermal imaging; Step 202, manually classify and annotate the second sample images to obtain the target category of each second sample image to form a second data set; Step 203, based on the ISTDU-Net network structure, construct a second target detection and recognition model, the second target detection and recognition model includes an encoder, a skip connection, a decoder and a classification layer; Step 204: input the second sample image into the second target detection and recognition model for machine learning. The decoder of the second target detection and recognition model outputs a feature map of the second sample image. The classification layer of the second target detection and recognition model outputs the predicted probability that the second sample image belongs to each target category. After the learning is completed, a trained second target detection and recognition model is obtained.

4. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The specific steps of "fusing the images corresponding to the first positive example feature map under the same semantics in the visible light image set and the infrared image set respectively" are: treating each pixel of the image corresponding to the first positive example feature map in the visible light image set and the infrared image set respectively as an independent sample, learning the relationship between each pixel through a deep neural network, and then fusing the pixels of the two images according to the fusion strategy.

5. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The training method of the target box recognition model is: Step 301: using a camera and an infrared thermal imager carried by a drone to respectively acquire a first sample image and a second sample image along the railway, and fusing the first sample image and the second sample image under the same semantics to obtain a third sample image; Step 302: manually marking the upper left corner point and the lower right corner point of the third sample image, wherein the combination of the upper left corner point and the lower right corner point constitutes a target frame; Step 303: Building a target box recognition model based on the CornerNet network structure; Step 304: Send the third sample image to the target frame recognition model for machine learning, and obtain a trained target frame recognition model after the learning is completed.

6. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The specific method of "establishing the correspondence between the pixel points in the image target frame and the actual GPS coordinate position" is: Step 401, obtaining the intrinsic parameters and extrinsic parameters of the camera; the intrinsic parameters include the focal length of the camera and the size of the imaging sensor, and the extrinsic parameters include the latitude and longitude, altitude, orientation and tilt angle of the camera during shooting; Step 402: Convert the pixel coordinates in the image target frame into three-dimensional coordinates in the camera coordinate system; Step 403: convert the three-dimensional coordinates in the camera coordinate system into coordinates in the world coordinate system; Step 404, converting the coordinates in the world coordinate system into geodetic coordinates; Step 405, converting the geodetic coordinates into a plane coordinate system; the plane coordinate system adopts the UTM projection coordinate system, which is the coordinate system adopted by most national public map websites on the Internet.

7. A railway inspection scene modeling method based on image recognition according to claim 6, characterized in that: The external parameters of the camera are obtained by the sensor module carried by the drone.

8. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The specific method of "building a three-dimensional map" is: Step 601, creating a new three-dimensional scene in GIS software; Step 602: import the DEM data corresponding to each image target frame into the three-dimensional scene, and the GIS software generates a terrain surface according to the DEM data; Step 603: superimpose the orthophoto corresponding to each image target frame as a texture on the terrain surface; Step 604: Integrate other geospatial data of each actual GPS coordinate position corresponding to the image target frame into the three-dimensional scene; Other geospatial data includes roads, buildings, vegetation, etc., which can be imported in vector or raster form and positioned, resized, and styled as needed; Step 605: Optimize and render the three-dimensional scene.

9. A railway inspection scene modeling method based on image recognition according to claim 8, characterized in that: Publish the 3D scene to the platform, and the terminal accesses the platform through a web page, application or by scanning a QR code to view the 3D scene.

10. A railway inspection scene modeling method based on image recognition according to claim 1, characterized in that: The actual GPS coordinate position corresponding to the image target frame is added to the re-inspection position database.

Images