Multi-sensor fused high-speed railway perimeter intrusion alarm method, system, equipment and medium

Abstract

The invention provides a multi-sensor fused high-speed railway perimeter intrusion alarm method, system and device and a medium. The method comprises the following steps: acquiring a to-be-detected image of a high-speed railway perimeter; under the condition that the high-speed railway is in the first scene, detecting the to-be-detected image by using a first model to obtain a detection result; wherein the first model is obtained by training based on a first training image acquired by a laser camera and a second training image acquired by an infrared camera; under the condition that the high-speed railway is in the second scene, detecting the to-be-detected image by using a second model to obtain a detection result; wherein the second model is obtained based on a three-dimensional point cloud training set and two-dimensional color image training, and the protection area range corresponding to the first scene is larger than that of the second scene. The method can make full use of the advantages of different sensors, is suitable for various typical scenes of high-speed rails and various weather conditions, and improves the accuracy of perimeter intrusion monitoring.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-sensor fusion high-speed railway perimeter intrusion alarm method, system, equipment and medium. Background Art

[0002] The operational safety of high-speed railways has attracted much attention. However, people and foreign objects that invade the perimeter of high-speed railways seriously threaten operational safety. At present, the commonly used detection methods for perimeter intrusion include pulse electronic fences, vibrating optical fibers, infrared, millimeter-wave radars, lidars, intelligent video analysis, etc. These methods have their own advantages and disadvantages, but any single sensing method cannot meet the perimeter intrusion alarm requirements of zero missed reports and low false alarms in all kinds of scenarios mentioned above under all-weather conditions. It is urgent to study the applicable multi-sensor fusion strategy for different environments and scenarios of high-speed railways and develop related products, which is also a very challenging task.

[0003] In view of the shortcomings of the above-mentioned single sensors in intrusion monitoring, high-speed railway perimeter intrusion monitoring based on multi-sensor technology fusion has become a new direction. Common multi-sensor fusion and linkage solutions include:

[0004] (1) The linkage between vibrating optical fiber and visible light video intelligent analysis, which uses vibrating optical fiber to achieve large-scale intrusion monitoring, and links video analysis to conduct local confirmation in the area where the intrusion is initially confirmed. The advantages of this solution are large coverage and low false alarm rate, but it cannot detect targets that cross the guardrail and enter from the air.

[0005] (2) The linkage between infrared thermal imaging and visible light video intelligent analysis. It uses infrared thermal imaging to realize large-scale intrusion monitoring. In the area where the intrusion is initially confirmed, it is linked to video analysis for local confirmation. Its advantages are large coverage and low false alarm rate. However, due to the high complexity of the scenes along the high-speed railway, infrared thermal imaging technology cannot completely capture the intrusion targets in the obstructed areas. At the same time, the contrast is not obvious enough for the targets with small temperature difference from the background.

[0006] In summary, there is an urgent need for a high-speed railway perimeter intrusion alarm method with multi-sensor fusion that can be applied to various scenarios and has high detection accuracy. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a multi-sensor fusion high-speed railway perimeter intrusion alarm method, system, equipment and medium.

[0008] In order to achieve the above object, the present invention adopts the following technical scheme.

[0009] In a first aspect, the present invention provides a multi-sensor fusion high-speed railway perimeter intrusion alarm method, comprising:

[0010] Acquire the image to be inspected of the perimeter of the high-speed railway;

[0011] In the case where the high-speed railway is the first scene, the image to be detected is detected using the first model to obtain a detection result; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera;

[0012] When the high-speed railway is the second scene, the image to be detected is detected using a second model to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

[0013] Furthermore, when there is no obstruction in the protection area of ​​the image to be detected, the first model includes a first backbone network, a second backbone network and a cross-modal fusion module, and the image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera;

[0014] The detecting the image to be detected by using the first model includes:

[0015] Inputting the first image to be detected into the first backbone network to obtain a first feature; inputting the second image to be detected into the second backbone network to obtain a second feature;

[0016] The first feature and the second feature are cross-modally fused through the cross-modal fusion module to obtain a fused feature, and a detection result is obtained according to the fused feature.

[0017] Further, the cross-modal fusion of the first feature and the second feature by the cross-modal fusion module to obtain a fusion feature includes:

[0018] splicing the first feature and the second feature, obtaining local information from the spliced ​​features as position coding information, and obtaining a coding feature based on the spliced ​​features and the position coding information;

[0019] Processing the coding feature by using an attention mechanism to capture the interaction and dependency between the first image to be detected and the second image to be detected, obtaining an attention feature, and obtaining a third feature according to the attention feature and the coding feature;

[0020] A fusion feature is obtained according to the third feature and the spliced ​​feature.

[0021] Furthermore, when there is an obstruction in the protection area of ​​the image to be detected, the first model includes a first sub-model and a second sub-model, the first sub-model is a laser image detection model, and the second sub-model is an infrared image detection model; the image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera;

[0022] The detecting the image to be detected by using the first model to obtain a detection result includes:

[0023] Determine the warning area information through vibrating optical fiber;

[0024] The warning area information and the first image to be detected are input into the first sub-model, and the warning area information and the second image to be detected are input into the second sub-model, and the first detection result and the second detection result are obtained accordingly, and the first detection result and the second detection result are fused to obtain the final detection result.

[0025] Furthermore, the second model includes a third sub-model and a fourth sub-model, the third sub-model is a three-dimensional target detection model, and the fourth sub-model is a two-dimensional target detection model; the image to be detected includes a three-dimensional point cloud and a two-dimensional image;

[0026] The detecting the image to be detected by using the second model includes:

[0027] Extracting point cloud features from the three-dimensional point cloud and extracting image features from the two-dimensional image; obtaining calibration information based on the point cloud features and the image features;

[0028] Input the point cloud features into the third sub-model to obtain a three-dimensional point cloud detection result; input the image features into the fourth sub-model to obtain a two-dimensional image detection result;

[0029] The three-dimensional point cloud detection result is projected onto the two-dimensional image detection result based on the calibration information to obtain a final detection result.

[0030] Further, the point cloud feature is a three-dimensional straight line feature, and the image feature is a two-dimensional straight line feature;

[0031] The obtaining of calibration information based on the point cloud features and the image features includes:

[0032] Matching the three-dimensional straight line features with the two-dimensional straight line features by a nearest neighbor search algorithm to obtain a plurality of pairs of straight line features;

[0033] The plurality of pairs of straight line feature pairs are optimized to obtain an optimal straight line feature pair, and corresponding calibration information is obtained based on the optimal straight line feature pair.

[0034] Furthermore, the method also includes:

[0035] The protection area indicated by the current image to be detected is compared with the area threshold, and when the protection area exceeds the area threshold, it is determined to be the first scene; when the protection area does not exceed the area threshold, it is determined to be the second scene.

[0036] In a second aspect, the present invention further provides a multi-sensor fusion high-speed railway perimeter intrusion alarm system, comprising:

[0037] An image acquisition unit, used to acquire an image to be detected of the perimeter of the high-speed railway;

[0038] A first detection unit is used to detect the image to be detected using a first model to obtain a detection result when the high-speed railway is the first scene; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera;

[0039] The second detection unit, when the high-speed railway is the second scene, uses the second model to detect the image to be detected to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

[0040] In a third aspect, the present invention also provides an electronic device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned multi-sensor fusion high-speed railway perimeter intrusion alarm method.

[0041] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned multi-sensor fusion high-speed railway perimeter intrusion alarm method.

[0042] Beneficial effects of the present invention: The present invention provides a multi-sensor fusion high-speed railway perimeter intrusion alarm method, system, equipment and medium, wherein the method uses the first model for detection when the high-speed railway is the first scene; and uses the second model for detection when the high-speed railway is the second scene; wherein the first model is trained based on the first training image collected by the laser camera and the second training image collected by the infrared camera; the second model is trained based on the three-dimensional point cloud training set and the two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene. It formulates different multi-sensor fusion strategies for different railway scenes, fully utilizes the advantages of different sensors, and comprehensively utilizes the data by designing a multi-level fusion algorithm, which is suitable for various high-speed railway typical scenes and various weather conditions, and improves the accuracy of perimeter intrusion monitoring.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0045] Figure 1 A schematic flow chart of a multi-sensor fusion high-speed railway perimeter intrusion alarm method provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a first scenario provided by an embodiment of the present invention;

[0047] Figure 3 One of the flowcharts of the first model detection provided by the embodiment of the present invention;

[0048] Figure 4 for Figure 3 Schematic diagram of the structure of the mid-cross modality fusion module;

[0049] Figure 5 A second schematic diagram of a flow chart of the first model detection provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a second model detection process provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of a process for obtaining calibration information provided by an embodiment of the present invention;

[0052] Figure 8 for Figure 6 Schematic diagram of the process of fusion of detection results. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0054] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "an" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0056] Although the fusion of multiple sensing technologies can significantly reduce the false alarm rate of perimeter intrusion systems, the fixed fusion mode is not suitable for all scenarios. By integrating the advantages of sensors with different detection principles, the accuracy of perimeter intrusion monitoring can be improved, but the fixed combination mode is still not fully applicable to various high-speed rail typical scenarios and various weather conditions. For example, the vibration fiber + video linkage solution can reduce false alarms, but it cannot monitor targets that cross the guardrail and invade from above. The infrared thermal imaging + video linkage solution cannot monitor intrusions under tree cover conditions.

[0057] Based on this, the present invention designs different multi-sensor fusion strategies and algorithms and develops integrated equipment for different typical scenarios of high-speed railways. The following will further explain the high-speed railway perimeter intrusion alarm method, system, equipment and medium with multi-sensor fusion provided by the present invention by taking several specific embodiments as examples in combination with the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0058] Example 1

[0059] See also Figure 1 , a multi-sensor fusion high-speed railway perimeter intrusion alarm method, comprising the following steps:

[0060] S101, obtaining an image to be detected of the perimeter of a high-speed railway.

[0061] In this step, the image to be detected includes multiple images captured by different sensors or cameras, such as images obtained by infrared thermal imaging, images obtained by a laser camera, and the like.

[0062] S102: When the high-speed railway is the first scene, the image to be detected is detected using the first model to obtain a detection result.

[0063] The first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera.

[0064] S103: When the high-speed railway is the second scene, the image to be detected is detected using the second model to obtain a detection result.

[0065] Among them, the second model is obtained based on a three-dimensional point cloud training set and a two-dimensional color image training, and the range corresponding to the first scene is larger than that of the second scene.

[0066] The first scene and the second scene are determined according to the size of the protection area in the current image to be detected. If the current protection area exceeds the area threshold, it is determined that the current image to be detected is the first scene. The scope of the first scene is larger. In the first scene, the multi-sensor fusion high-speed railway perimeter intrusion alarm method is mainly used to monitor personnel and large animals.

[0067] If the current protection area does not exceed the area threshold, it is determined that the current image to be detected is the second scene, which is smaller than the first scene. In the second scene, the multi-sensor fusion high-speed railway perimeter intrusion alarm method is mainly used to monitor rockfall, mudslides, etc. According to different monitoring targets, accurate detection is achieved through images sensed by different sensors and different image processing algorithms.

[0068] It should be noted that both the first model and the second model are models based on deep learning. In this embodiment, the regional threshold is 70m, and in other embodiments of the present invention, it can be adjusted according to actual scenarios or requirements, and the present invention does not limit this.

[0069] The multi-sensor fusion high-speed railway perimeter intrusion alarm method provided by the embodiment of the present invention uses the first model for detection when the high-speed railway is the first scene; and uses the second model for detection when the high-speed railway is the second scene; wherein the first model is trained based on the first training image collected by the laser camera and the second training image collected by the infrared camera; the second model is trained based on the three-dimensional point cloud training set and the two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene. It formulates different multi-sensor fusion strategies for different railway scenes, fully utilizes the advantages of different sensors, and comprehensively utilizes the data by designing a multi-level fusion algorithm. It is applicable to various typical high-speed railway scenes and various weather conditions, and improves the accuracy of perimeter intrusion monitoring.

[0070] In some embodiments of this embodiment, when it is determined that the current image to be detected is in the first scene, it is possible to continue to determine which sensors are used to complete the high-speed railway perimeter intrusion alarm based on whether the protection area in the image to be detected is blocked. Figure 2 As shown, when the guardrail area is open and unobstructed, infrared thermal imaging warning can be used in combination with laser camera linkage confirmation.

[0071] Specifically, when there is no obstruction in the protection area of ​​the image to be detected, the first model includes a first backbone network, a second backbone network, a cross-modal fusion module, and a detector, and the image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera. It should be noted that the first image to be detected and the second image to be detected are images of the same protection area.

[0072] like Figure 3 As shown, the first model is used to detect the image to be detected, including:

[0073] The first image to be detected is input into the first backbone network to obtain the first feature; the second image to be detected is input into the second backbone network to obtain the second feature.

[0074] The first backbone network is actually a laser camera image feature extraction backbone network, and the second backbone network is actually an infrared camera image feature extraction backbone network. The first backbone network can be PointPillars, PointNet, etc., and the second backbone network can be ResNet, DenseNet, etc.

[0075] The first feature and the second feature are cross-modally fused through a cross-modal fusion module to obtain a fused feature, and the fused feature is input into a detector to obtain a detection result.

[0076] It should be noted that both the first backbone network and the second backbone network include multiple layers of convolutional layers. The features output by each convolutional layer in the corresponding backbone network can be collectively referred to as first features and second features. There can be multiple cross-modal fusion modules, which fuse the first features and second features output by different convolutional layers to obtain multiple fused features.

[0077] Furthermore, the fusion features are obtained as follows:

[0078] The first feature and the second feature are concatenated, and local information is obtained from the concatenated features as position coding information, and a coding feature is obtained based on the concatenated features and the position coding information;

[0079] The coding feature is processed by using an attention mechanism to capture the interaction and dependency between the first image to be detected and the second image to be detected, to obtain an attention feature, and a third feature is obtained according to the attention feature and the coding feature;

[0080] The fusion feature is obtained according to the third feature and the spliced ​​feature.

[0081] More specifically, if Figure 4 As shown in Figure 2, the cross-modal fusion module includes a normalization layer, an attention mechanism, a perception layer, etc.

[0082] In the process of splicing the first feature and the second feature, it is known that the shapes of the first feature and the second feature are Then the two are concatenated according to the channel dimension to obtain the concatenated feature

[0083] Through the convolution operation, the concatenated feature F 拼接 The local information is extracted from the local information and used as the position encoding information W pos .

[0084] According to the position encoding information W pos And the concatenated feature F 拼接 Get the encoding feature I fuse , I fuse =F 拼接 +W pos . Through the normalization layer, the encoding feature I fuse Perform standardization to obtain the normalized encoding feature I' fuse . The normalization process makes the first model more stable during the training process.

[0085] Normalized encoded feature I'fuse Under the attention mechanism, Q ,W K ,W V The three different weight matrices are multiplied to obtain the three variables Q, K, and V. The specific calculation process is as follows:

[0086] Q=I' fuse ·W Q

[0087] K=I' fuse ·W K

[0088] V=I' fuse ·W V

[0089] In the formula, W represents the weight, Q, K, and V represent the query vector, key vector, and value vector respectively. Q , W K , W V Represent the weight moments of query vector, key vector, and value vector respectively.

[0090] According to the above query vector Q, key vector K, and value vector V, the interaction and dependency between the laser image and the infrared image are captured to obtain the attention feature Z. The specific calculation process is as follows:

[0091]

[0092] In the formula, Attention represents the attention operation, Softmax represents the activation function, and d k represents the dimension of the key vector K, represents the scaling factor, the The purpose is to avoid the problem of gradient disappearance or gradient explosion when the range of dot product values ​​becomes too large or too small as the dimension of the input vector increases when calculating the dot product.

[0093] Preferably, in order to capture information at different levels and enhance the ability to learn various aspects of input data, this embodiment adopts a two-head attention mechanism. This method enables each attention head to focus on different features, so that the model can obtain richer and more diverse representations. The calculation process of the two-head attention mechanism is as follows: Z′=TwoHead(Q,K,V)=Concat(Z1,Z2)W o , W o It represents the projection matrix after the features Z1 and Z2 are concatenated. TwoHead represents the dual-head attention mechanism. Concat represents feature concatenation. Z1 and Z2 represent the concatenation of features twice. This formula calculates the attention features Z1 and Z2.

[0094] After obtaining Z′ using the dual-head attention mechanism, it is combined with the encoded feature I fuse Add them together to form a third feature F3, and then use the normalization layer to normalize F3, and finally use a single-layer perception layer to make the dimension of F3 consistent with the concatenated feature dimension. The final fusion feature F 融合 It can be expressed as: F 融合 =F3'+F 拼接 =F3'+F 激光 +F 红外 , where F3' is the feature of the third feature F3 after being processed by the normalization layer and the perception layer.

[0095] The features F are fused through multiple fully connected layers in the detector 融合 Perform intrusion target detection to obtain the final detection result. When the detection result shows that there is a target invading the current protection area, an alarm is issued. When the detection result shows that there is no target invading the current protection area, no alarm is issued.

[0096] In other embodiments of the present invention, for shielded protection areas, monitoring can be achieved through the fusion of vibrating optical fibers, laser cameras, and infrared cameras.

[0097] Specifically, when there is an obstruction in the protection area of ​​the image to be detected, the first model includes a first sub-model and a second sub-model, the first sub-model is a laser image detection model, and the second sub-model is an infrared image detection model. Schematically, the laser image detection model can be VoxelNet, Complex-YOLO, DeepFusion, etc., and the infrared image detection model can be CBAM-YOLOv5, DCS-YOLOv8, GPNet, etc. The image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera. It should be noted that the first image to be detected and the second image to be detected here are images of the same protection area.

[0098] like Figure 5 As shown, the first model is used to detect the image to be detected to obtain a detection result, including the following sub-steps:

[0099] The warning area information is determined by the vibrating optical fiber. That is, through the linkage between the vibrating optical fiber and the laser camera and the infrared camera, the warning area location information determined by the vibrating optical fiber is synchronized to the laser camera and the infrared camera.

[0100] The warning area information and the first image to be detected are input into the first sub-model, and the warning area information and the second image to be detected are input into the second sub-model, and the first detection result and the second detection result are obtained accordingly. The first detection result and the second detection result are fused at the result level to obtain the final detection result and output it.

[0101] Among them, the first sub-model and the second sub-model will respectively amplify the warning areas in the first image to be detected and the second image to be detected and lower the judgment threshold according to the warning area information, thereby improving the detection rate of the first sub-model and the second sub-model.

[0102] The multi-sensor fusion high-speed railway perimeter intrusion alarm method provided by the embodiment of the present invention adopts a vibrating optical fiber warning for the situation where the protection area is blocked in the first scene. It synchronizes the area location information to the laser camera and the infrared camera, so that the first sub-model and the second sub-model can determine the area to be detected from the image according to the warning area location information during image detection, and then further process the image of the area after amplification, thereby improving the detection accuracy. In addition, since the vibrating optical fiber is not affected by blockage, infrared thermal imaging is good at detecting intruders and can avoid missing reports; combined with the laser camera image intelligent analysis confirmation to reduce false alarms, the monitoring range is large and the overall cost is low.

[0103] In some embodiments of the present invention, if the protection area in the current image to be detected is in the second scene, the second model includes a third sub-model and a fourth sub-model, the third sub-model is a three-dimensional target detection model, and the fourth sub-model is a two-dimensional target detection model. Schematically, the two-dimensional target detection model can be R-CNN series, YOLO series, SSD, etc., and the three-dimensional target detection model can be PointNet, VoxelNet, PillarNet, Graph CNN, etc. The image to be detected includes a three-dimensional point cloud and a two-dimensional image, wherein the three-dimensional point cloud and the two-dimensional image are obtained based on the same protection area.

[0104] like Figure 6 As shown, using the second model to detect the image to be detected includes the following sub-steps:

[0105] Extract point cloud features from 3D point clouds and extract image features from 2D images; obtain calibration information based on point cloud features and image features;

[0106] Input the point cloud features into the third sub-model to obtain a three-dimensional point cloud detection result; input the image features into the fourth sub-model to obtain a two-dimensional image detection result;

[0107] Based on the calibration information and the three-dimensional point cloud detection result, the three-dimensional point cloud detection result is projected onto the two-dimensional image detection result to obtain the final detection result and output it.

[0108] Furthermore, the point cloud features are three-dimensional straight line features, and the image features are two-dimensional straight line features.

[0109] like Figure 7 As shown, obtaining calibration information based on point cloud features and image features includes the following sub-steps:

[0110] The three-dimensional straight line features are matched with the two-dimensional straight line features by using the nearest neighbor search algorithm to obtain multiple pairs of straight lines;

[0111] The optimization solution is performed on multiple pairs of straight lines to obtain the optimal straight line pair, and the corresponding calibration information is obtained based on the optimal straight line pair.

[0112] In this embodiment, the calibration information is determined by the straight line features such as rails and contact network columns in the high-speed railway. Specifically, the straight line features such as rails and contact network columns are extracted from the three-dimensional point cloud and the two-dimensional image, respectively, as three-dimensional straight line features line 3D and two-dimensional line feature line 2D .

[0113] In the three-dimensional line feature line 3D After abnormal filtering, the two-dimensional line feature line 2D Pairing is performed by using the nearest neighbor search algorithm to filter out the three-dimensional straight line features line 3D 'With the two-dimensional straight line feature line 2D During the pairing process, as the distance of the LiDAR point cloud increases, the point cloud density will gradually decrease, and the number of straight line features that can be extracted will also decrease. 3D The feature data contained in is less than that of line 2D The number of features, so the pairing process is line 3D ' for reference.

[0114] In the above-mentioned abnormal filtering process of three-dimensional straight line features, for the straight line features of the contact network column, since the contact network column is perpendicular to the ground, the straight line of the contact network is also perpendicular to the ground, so it is necessary to filter out the straight line features that are not perpendicular to the ground. For the straight line features extracted from the rails, the rails on a line exist in pairs, and the line spacing of the rail straight lines is fixed, so the straight lines that are not in pairs and have a fixed distance are filtered out.

[0115] During the pairing process, specifically on line 2D Find the line 3D ′The closest two-dimensional straight line feature is used to form a straight line feature pair. Finally, by constructing an optimization function, the straight line coincidence degree in the straight line feature pair is maximized, thereby obtaining the final calibration parameters. The optimization process is shown in the following formula:

[0116]

[0117] In the formula, R, t represents the calibration external parameters, which are the rotation matrix and the translation vector respectively, arg min represents the optimization solution, and dis(.) represents the distance metric function, which is used to calculate the distance between the three-dimensional line feature and the two-dimensional line feature. 3D ′ represents the three-dimensional straight line feature after filtering, line 2D ′ indicates that the 3D ’Matching two-dimensional straight line features.

[0118] In the process of using the 3D target detection model to detect the point cloud features, the point cloud P for detection is obtained through methods such as region cropping and ground removal, and then the spatial clustering algorithm DBSCAN is used for clustering. For the clustering results, the confidence is judged by the target position and the number of target points, and the targets with too low confidence are filtered out, and finally the 3D point cloud detection result object is obtained. 3D .

[0119] In the process of using the two-dimensional target detection model to detect the image features, a method based on deep learning and background difference is used to perform two-dimensional target detection and obtain the two-dimensional image detection result object 2D .

[0120] like Figure 8 As shown, based on the calibration information, the 3D point cloud detection result is projected onto the 2D image detection result to obtain the final detection result, which specifically includes the following sub-steps:

[0121] The three-dimensional point cloud detection result object 3D The calibration information is projected into the two-dimensional image space to obtain the projected result. Then calculate the projection result And the two-dimensional image detection result object 2D The intersection-over-union (IOU) scores between the corresponding targets are used to make a comprehensive judgment. If the score exceeds the set threshold, the detected target is considered to be a foreign object and an alarm is issued. If the score does not exceed the threshold, it is considered not to be a foreign object and no alarm is issued.

[0122] Example 2

[0123] On the basis of Example 1, this Example 2 provides a multi-sensor fusion high-speed railway perimeter intrusion alarm system, which corresponds to the multi-sensor fusion high-speed railway perimeter intrusion alarm described above, and specifically includes:

[0124] An image acquisition unit, used to acquire an image to be detected of the perimeter of the high-speed railway;

[0125] A first detection unit is used to detect the image to be detected using a first model to obtain a detection result when the high-speed railway is the first scene; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera;

[0126] The second detection unit, when the high-speed railway is the second scene, uses the second model to detect the image to be detected to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

[0127] For specific details, please refer to the description of the high-speed railway perimeter intrusion alarm method based on multi-sensor fusion, which will not be repeated here.

[0128] Example 3

[0129] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, the processor calls the program instructions to execute a multi-sensor fusion high-speed railway perimeter intrusion alarm method, the method includes the following process steps:

[0130] Acquire the image to be inspected of the perimeter of the high-speed railway;

[0131] In the case where the high-speed railway is the first scene, the image to be detected is detected using the first model to obtain a detection result; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera;

[0132] When the high-speed railway is the second scene, the image to be detected is detected using a second model to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

[0133] Example 4

[0134] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, a high-speed railway perimeter intrusion alarm method using multi-sensor fusion is implemented. The method includes the following process steps:

[0135] Acquire the image to be inspected of the perimeter of the high-speed railway;

[0136] In the case where the high-speed railway is the first scene, the image to be detected is detected using the first model to obtain a detection result; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera;

[0137] When the high-speed railway is the second scene, the image to be detected is detected using a second model to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

[0138] In summary, the multi-sensor fusion high-speed railway perimeter intrusion alarm method, system, device and medium provided by the embodiment of the present invention use the first model for detection when the high-speed railway is the first scene; and use the second model for detection when the high-speed railway is the second scene; wherein the first model is trained based on the first training image collected by the laser camera and the second training image collected by the infrared camera; the second model is trained based on the three-dimensional point cloud training set and the two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene. It formulates different multi-sensor fusion strategies for different railway scenes, fully utilizes the advantages of different sensors, and comprehensively utilizes the data by designing a multi-level fusion algorithm, which is applicable to various typical high-speed railway scenes and various weather conditions, and improves the accuracy of perimeter intrusion monitoring.

[0139] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0140] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The method and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0141] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-sensor fusion high-speed railway perimeter intrusion alarm method, characterized in that: include: Acquire the image to be inspected of the perimeter of the high-speed railway; In the case where the high-speed railway is the first scene, the image to be detected is detected using the first model to obtain a detection result; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera; When the high-speed railway is the second scene, the image to be detected is detected using a second model to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

2. The method according to claim 1, characterized in that: When there is no obstruction in the protection area of ​​the image to be detected, the first model includes a first backbone network, a second backbone network and a cross-modal fusion module, and the image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera; The detecting the image to be detected by using the first model includes: Inputting the first image to be detected into the first backbone network to obtain a first feature; inputting the second image to be detected into the second backbone network to obtain a second feature; The first feature and the second feature are cross-modally fused through the cross-modal fusion module to obtain a fused feature, and a detection result is obtained according to the fused feature.

3. The method according to claim 2, characterized in that The cross-modal fusion of the first feature and the second feature by the cross-modal fusion module to obtain a fusion feature includes: splicing the first feature and the second feature, obtaining local information from the spliced ​​features as position coding information, and obtaining a coding feature based on the spliced ​​features and the position coding information; Processing the coding feature by using an attention mechanism to capture the interaction and dependency between the first image to be detected and the second image to be detected, obtaining an attention feature, and obtaining a third feature according to the attention feature and the coding feature; A fusion feature is obtained according to the third feature and the spliced ​​feature.

4. The method according to claim 1, characterized in that When there is an obstruction in the protection area of ​​the image to be detected, the first model includes a first sub-model and a second sub-model, the first sub-model is a laser image detection model, and the second sub-model is an infrared image detection model; the image to be detected includes a first image to be detected acquired by a laser camera and a second image to be detected acquired by an infrared camera; The detecting the image to be detected by using the first model to obtain a detection result includes: Determine the warning area information through vibrating optical fiber; The warning area information and the first image to be detected are input into the first sub-model, and the warning area information and the second image to be detected are input into the second sub-model, and the first detection result and the second detection result are obtained accordingly, and the first detection result and the second detection result are fused to obtain the final detection result.

5. The method according to claim 1, characterized in that The second model includes a third sub-model and a fourth sub-model, the third sub-model is a three-dimensional target detection model, and the fourth sub-model is a two-dimensional target detection model; the image to be detected includes a three-dimensional point cloud and a two-dimensional image; The detecting the image to be detected by using the second model includes: Extracting point cloud features from the three-dimensional point cloud and extracting image features from the two-dimensional image; obtaining calibration information based on the point cloud features and the image features; Input the point cloud features into the third sub-model to obtain a three-dimensional point cloud detection result; input the image features into the fourth sub-model to obtain a two-dimensional image detection result; The three-dimensional point cloud detection result is projected onto the two-dimensional image detection result based on the calibration information to obtain a final detection result.

6. The method according to claim 5, characterized in that The point cloud feature is a three-dimensional straight line feature, and the image feature is a two-dimensional straight line feature; The obtaining of calibration information based on the point cloud features and the image features includes: Matching the three-dimensional straight line features with the two-dimensional straight line features by a nearest neighbor search algorithm to obtain a plurality of pairs of straight line features; The plurality of pairs of straight line feature pairs are optimized to obtain an optimal straight line feature pair, and corresponding calibration information is obtained based on the optimal straight line feature pair.

7. The method according to claim 1, characterized in that The method also includes: The protection area indicated by the current image to be detected is compared with the area threshold, and when the protection area exceeds the area threshold, it is determined to be the first scene; when the protection area does not exceed the area threshold, it is determined to be the second scene.

8. A multi-sensor fusion high-speed railway perimeter intrusion alarm system, characterized in that: include: An image acquisition unit, used to acquire an image to be detected of the perimeter of the high-speed railway; A first detection unit is used to detect the image to be detected using a first model to obtain a detection result when the high-speed railway is the first scene; wherein the first model is trained based on a first training image captured by a laser camera and a second training image captured by an infrared camera; The second detection unit, when the high-speed railway is the second scene, uses the second model to detect the image to be detected to obtain a detection result; wherein the second model is trained based on a three-dimensional point cloud training set and a two-dimensional color image, and the protection area corresponding to the first scene is larger than the second scene.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The device stores a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images