Rail transit platform door and train foreign matter detection and area perception method based on multi-task model

Abstract

The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, and relates to the technical field of rail transit platform door and train safety. The application discloses a

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety monitoring, more particularly, to a rail transit platform door and train foreign matter detection and area perception method based on a multi-task model. BACKGROUND

[0002] Currently, rail transit such as intercity railways, high-speed rails, light rails, etc. have generally or begun to set platform doors, however, some foreign matter intrusion events affecting safety often occur in the gap (risk space) between the platform door and the train. The foreign matter intrusion events occurring in the risk space include the gap foreign matters on the platform tread board, the anti-tread-off rubber strip, and the gap between the anti-tread-off rubber strip and the train, the foreign matters clamped by the platform sliding door gap and the train door gap. The foreign matter detection in the above areas mainly has traditional machine vision methods and deep learning methods. The traditional machine vision algorithm has a large number of hyperparameters, and when the light changes significantly, the foreign matter detection effect is unstable. Moreover, because the traditional machine vision algorithm uses SVM and other methods for foreign matter classification, the expression ability of the model itself is not strong, and it is difficult to classify the foreign matter. However, it is very important to inform the staff of the type of foreign matter in rail transit operation. If a small foreign matter such as a mobile phone is left in the gap between the train door and the platform door, and with the vibration of the train starting, the foreign matter is likely to fall into the track area. If the foreign matter is a mobile phone and other electronic products, the driver can be notified to stop the train immediately and take out the mobile phone with tools, avoiding the train crushing the mobile phone to produce sparks and causing a major traffic accident. The deep learning method learns the labeled data set with the help of high-performance equipment, compared with the traditional machine vision algorithm, it does not need to perform complex preprocessing on the image and has stronger model expression ability, and it is easy to realize foreign matter positioning and classification and achieve higher detection accuracy. However, the mainstream deep learning method detects foreign matter in the whole area of the image, and external disturbances such as light are easy to produce foreign matter reflections in non-key areas such as doors and windows, thus causing certain false detections. In order to exclude the interference of foreign matter reflections, a multi-task deep learning model is proposed to complete the three tasks of foreign matter detection, area segmentation, and area line detection, and by means of the scene characteristics that the foreign matter exists in a specific area and will break the area line, the segmentation result and the foreign matter detection task are fused to exclude the interference of non-key areas such as doors and windows, and further improve the accuracy of foreign matter detection. SUMMARY

[0003] The present application is to overcome the problems of the prior art rail transit platform door and train foreign matter automatic detection method that it is difficult to predict the type and position of the foreign matter, and proposes a rail transit platform door and train foreign matter detection and area perception method based on a multi-task model.

[0004] To solve the above technical problems, the technical scheme adopted by the present application is: a foreign matter detection and area perception method between a rail transit platform door and a train based on a multi-task model, comprising the following steps:

[0005] S1: Collect multiple pictures of the rail transit platform door and the train with foreign matters, label the foreign matter detection true label, the area segmentation true label, and the area line detection true label, and provide the true labels for subsequent training;

[0006] S2: Preprocess the pictures collected in step S1;

[0007] S3: Construct an encoder of the model, the encoder comprising a Backbone module and a Neck module, the Backbone module being used to extract features of the pictures preprocessed in step S2 for use by the Neck module, and the Neck module being used to fuse feature information of different stages of the pictures to extract more in-depth information for use by a subsequent decoder;

[0008] S4: Construct a decoder of the model, the decoder corresponding to the tasks of foreign matter detection, area segmentation, and area line detection respectively, the decoder obtaining preliminary foreign matter detection prediction results, area segmentation prediction results, and area line detection prediction results respectively according to the more in-depth information in step S3;

[0009] S5: Calculate a foreign matter detection loss according to the foreign matter detection true label in step S1 and the preliminary foreign matter detection prediction results in step S4, calculate an area segmentation loss according to the area segmentation true label in step S1 and the preliminary area segmentation prediction results in step S4, calculate an area line detection loss according to the area line detection true label in step S1 and the preliminary area line detection prediction results in step S4, construct a total loss function to obtain a total loss, the total loss being a weighted sum of the foreign matter detection loss, the area segmentation loss, and the area line detection loss;

[0010] S6: Perform backpropagation training on the model according to the total loss calculated in step S5 to obtain specific values of the model weights and biases, so as to infer more accurate prediction box results containing foreign matters, foreign matter detection prediction results, area segmentation prediction results, and area line detection prediction results output by the detection head Detection on the model with specific weights and biases;

[0011] S7: Adjust the prediction results of the foreign matter detection according to the area segmentation prediction results and the area line detection prediction results obtained in step S6, which is referred to as result fusion.

[0012] Preferably, in the step S3, the processing of the Backbone module comprises the following steps:

[0013] S31: Convert the image preprocessed in step S2 and with a size of W×H×3 into a feature map F1 with a size of W×H×32 using the Focus module;

[0014] S32: Pass feature map F1 through the CSP module to obtain high-dimensional feature map F. 11 Feature map F 11 After passing through two layers of CSP modules, the high-dimensional feature map F is obtained sequentially. 12 F 13 Simultaneously, a Convolutional BlockAttention Module is added to the last standard convolutional layer, and the feature map F 13 This convolution process yields a new feature map F3, allowing the Neck module to focus more on the location information of foreign objects in the image.

[0015] Preferably, in step S3, the processing of the Neck module includes the following steps:

[0016] S33: Using the Spatial Pyramid Pooling module in YOLOV3-SPP, feature map F4, which incorporates information from multiple receptive fields, is extracted from feature map F3 using various pooling methods.

[0017] S34: Then, using FeaturePyramidNetworks, the feature map F4 extracted in step S33 is upsampled twice and successively combined with the high-dimensional feature map F obtained in step S32. 12 and F 13 The fusion yields a feature map F5 that can convey strong semantic information.

[0018] Preferably, in step S33, the multiple pooling methods include four types of pooling: 1×1, 5×5, 9×9, and 13×13.

[0019] Preferably, in step S4, the decoder has three parts, corresponding to the three tasks of foreign object detection, region segmentation, and region line detection, respectively. The foreign object detection task includes the following steps:

[0020] S41: Obtain feature maps of three sizes using the Path Aggregation Network structure. X Each feature map F X It contains strong semantic and strong location information;

[0021] S42: Combine feature maps of three sizes F X The input is fed into the detection head for prediction, and the feature map F X Each grid has three prior boxes of three sizes, and each prior box will generate three predicted boxes;

[0022] S43: remove a large number of redundant prediction boxes by using Non-Maximum Suppression, and the remaining prediction boxes are foreign object detection prediction results output by the model, which include confidence probability values, category probability values and bounding box position information of foreign objects.

[0023] Preferably, in the step S4, the two tasks of region segmentation and region line detection use the same model, and the feature map F5 is twice up-sampled to obtain a WxHx2 feature map, which is the region segmentation prediction result and the region line detection prediction result, and the two channels in the feature map represent the probability of each pixel being a region or background.

[0024] Preferably, in the step S5, the total loss function is:

[0025]

[0026] The foreign object detection loss includes a foreign object classification loss a foreign object confidence loss and a foreign object prediction box position loss The foreign object detection classification loss and the confidence loss The cross-entropy loss is used, and the prediction box position loss The GIoU Loss is used; in the step S1, the foreign object detection real label includes a foreign object classification real label and a foreign object position real label, and in the step S4, the preliminary foreign object detection prediction result includes a foreign object classification prediction result and a foreign object position prediction result, the foreign object classification loss is calculated from the foreign object classification real label and the foreign object classification prediction result, and the foreign object prediction box position loss is calculated from the foreign object position real label and the foreign object position prediction result; the confidence loss is calculated from the number 1 minus the intersection over union (IoU) of the foreign object position prediction result and the foreign object position real label; the region segmentation loss and the region line detection loss also use the cross-entropy loss; α1, α2, α3 and β1, β2, β3 are constants obtained by manual fine-tuning to balance the importance of the three tasks of foreign object detection, region segmentation and region line detection.

[0027] Preferably, the step S7 specifically includes the following steps:

[0028] S71: The result fusion process first determines whether the number of intersecting pixels between the prediction box containing the foreign matter output by the detection head Detection in step S6 and the WxHx2 feature map obtained by the region segmentation task exceeds a specified threshold according to the fact that the foreign matter is usually contained in the region segmentation prediction result. If yes, step S72 is performed. If no, step S73 is performed.

[0029] S72: If the number of intersecting pixels exceeds the specified threshold, it is considered that the position of the prediction box is consistent with the actual situation, and the confidence probability value of the foreign matter in the prediction box is not adjusted. Then, step S76 is performed.

[0030] S73: If the number of intersecting pixels is not enough for the specified threshold, it is determined whether the pixels inside the prediction box are on the straight line of the WxHx2 feature map obtained by the region line detection task according to the phenomenon of foreign matter occlusion region line. If no, step S74 is performed. If yes, step S75 is performed.

[0031] S74: If none of the pixels is on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is reduced. Then, step S76 is performed.

[0032] S75: If there are pixels on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is unchanged. Then, step S76 is performed.

[0033] S76: Then, the NMS is performed to screen out redundant prediction boxes, and the fused foreign matter detection prediction result is obtained.

[0034] Preferably, the specified threshold is in the range of 45-50.

[0035] Preferably, step S2 specifically comprises the following steps:

[0036] S21: The collected picture is converted from RGB format to HSV format, so that the hue, saturation and brightness of the picture are numerically adjusted. After the adjustment, the picture is converted to RGB format again.

[0037] S22: The adjusted picture is randomly rotated, scaled, translated, sheared and left-right flipped. The above data enhancement means can make the model more robust.

[0038] Compared with the prior art, the method can reduce the interference of doors and windows and the external environment through regional perception, has high foreign matter detection accuracy, and is suitable for application in regional perception and foreign matter identification between a rail transit platform door and a train. In addition, the method can visualize the detection result, inform the type of foreign matter and the position of foreign matter, so that rail transit staff can timely process the foreign matter and solve the safety hazard. Moreover, the model used in the present application simultaneously completes three tasks of foreign matter detection, regional segmentation and regional line detection, can reduce the inference time and meet the real-time requirement, in addition to improving the accuracy of foreign matter detection and reducing the inference time on the model, the segmentation region contains foreign matter and the foreign matter will break the regional line, so that the accuracy of foreign matter detection is further improved through result fusion. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of the rail transit platform door and train foreign matter detection and regional perception method based on the multi-task model of the present application;

[0040] Figure 2 is a display diagram of three regions in which foreign matter may exist between the rail transit platform door and the train;

[0041] Figure 3 is a display diagram of a segmented region containing foreign matter;

[0042] Figure 4 is a display diagram of foreign matter breaking a regional line;

[0043] Figure 5 is a display diagram of foreign matter detection, regional segmentation and regional line detection results under normal illumination;

[0044] Figure 6 is a display diagram of foreign matter detection, regional segmentation and regional line detection results under light interference;

[0045] Figure 7 is an architecture diagram of the multi-task model of the present application;

[0046] Figure 8 is a WxHx2 feature map obtained by the regional line detection task;

[0047] Figure 9 is a Canny edge detection result;

[0048] Figure 10 is a straight line found by the Hough line transformation of a large threshold value;

[0049] Figure 11 is a straight line found by the Hough line transformation of a small threshold value;

[0050] Figure 12 is a remaining straight line after screening;

[0051] Figure 13 is a fusion process structure diagram on a multi-task model. DETAILED DESCRIPTION

[0052] The accompanying drawings are only intended to illustrate, and cannot be understood as a limitation to the patent; in order to better illustrate the embodiments, some components in the drawings can be omitted, enlarged or reduced, and do not represent the actual size of the product; it can be understood by those skilled in the art that some well-known structures and their descriptions in the drawings can be omitted. The positional relationship described in the drawings is only intended to illustrate, and cannot be understood as a limitation to the patent.

[0053] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms “upper”, “lower”, “left”, “right”, “long”, “short” and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only intended to illustrate, and cannot be understood as a limitation to the patent, and for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0054] The technical solutions of the present application will be further described in detail below through specific embodiments, and in conjunction with the drawings:

[0055] Embodiment 1

[0056] As shown in the figure, the foreign matter detection and area perception method between the rail transit platform door and the train based on the multi-task model includes the following steps: Figure 1

[0057] S1: Collect multiple pictures of the rail transit platform door and the train with foreign matters, label the foreign matter detection true label, the area segmentation true label and the area line detection true label, and provide each true label for subsequent training;

[0058] S2: Preprocess the pictures collected in step S1;

[0059] S3: Construct an encoder of the model, the encoder includes a Backbone module and a Neck module, the Backbone module is used to extract the features of the pictures preprocessed in step S2 for use by the Neck module, and the Neck module is used to fuse the feature information of the pictures at different stages to extract more in-depth information for use by the subsequent decoder;

[0060] ​S4: constructing a decoder corresponding to the tasks of foreign object detection, region segmentation and region line detection respectively, the decoder obtaining preliminary foreign object detection prediction results, region segmentation prediction results and region line detection prediction results respectively according to the deeper information in step S3;

[0061] S5: calculating a foreign object detection loss according to the foreign object detection real label in step S1 and the preliminary foreign object detection prediction results in step S4, calculating a region segmentation loss according to the region segmentation real label in step S1 and the preliminary region segmentation prediction results in step S4, calculating a region line detection loss according to the region line detection real label in step S1 and the preliminary region line detection prediction results in step S4, and constructing a total loss function to obtain a total loss which is a weighted sum of the foreign object detection loss, the region segmentation loss and the region line detection loss;

[0062] S6: performing back propagation on the trained model according to the total loss calculated in step S5 to obtain specific values of the model weights and biases, so as to infer more accurate prediction box results containing foreign objects, foreign object detection prediction results, region segmentation prediction results and region line detection prediction results from the Detection output of the model with specific weights and biases;

[0063] S7: adjusting the prediction results of foreign object detection according to the region segmentation prediction results and the region line detection prediction results obtained in step S6, which is called result fusion.

[0064] It should be noted that in step S1, the positions of foreign objects in the pictures and the types of the foreign objects are manually labeled to obtain the foreign object detection real label, and the binary images of the foreign object regions and the region lines are labeled to obtain the region segmentation real label and the region line detection real label, which are used for subsequent training. In addition, it should be noted that, as shown in Figure 2 The region division of foreign objects between the platform door and the train of the rail transit station is divided into three, i.e. the platform door region, the gap and the train door region, which represent the regions where foreign objects may exist. As shown in Figure 3 There is a foreign object in the gap region, as shown in Figure 4 The foreign object breaks the region line on the side of the gap region close to the platform door region.

[0065] In addition, in the step S3, the processing of the Backbone module includes the following steps:

[0066] S31: converting the picture preprocessed in step S2 and having a size of WxHx3 into a feature map F1 having a size of WxHx32 through the Focus module;

[0067] S32: Pass feature map F1 through the CSP module to obtain high-dimensional feature map F. 11 Feature map F 11 After passing through two layers of CSP modules, the high-dimensional feature map F is obtained sequentially. 12 F 13 Simultaneously, a Convolutional Block Attention Module (CBAM) is added to the last standard convolutional layer, and the feature map F 13 This convolution yields a new feature map F3, allowing the Neck module to focus more on the location of the foreign object in the image. In this embodiment, the CSP module is used because it divides the input feature map into two parts and then merges them through a cross-stage hierarchical structure. This reduces computation while maintaining accuracy. The specific structure of the CSP module is as follows: Figure 7 The module in the lower right corner is shown. The CBAM module can enhance the fusion of channel and spatial information, allowing the network to pay more attention to the location information of foreign objects in the image.

[0068] In step S3, the processing of the Neck module includes the following steps:

[0069] S33: Using the Spatial Pyramid Pooling module in YOLOV3-SPP, feature map F4, which incorporates information from multiple receptive fields, is extracted from feature map F3 using various pooling methods.

[0070] S34: Then, using FeaturePyramidNetworks, the feature map F4 extracted in step S33 is upsampled twice and successively combined with the high-dimensional feature map F obtained in step S32. 12 and F 13 The fusion yields a feature map F5 that can convey strong semantic information.

[0071] In addition, in step S33, the multiple pooling methods include four types of pooling: 1×1, 5×5, 9×9, and 13×13.

[0072] In step S4, the decoder has three parts, corresponding to the tasks of foreign object detection, region segmentation, and region line detection, respectively. The foreign object detection task includes the following steps:

[0073] S41: Obtain feature maps of three sizes using the Path Aggregation Network structure. X Each feature map F X It contains strong semantic and strong location information;

[0074] S42: Combine feature maps of three sizes F XInput to the detection head Detection for prediction, feature map F X Each grid has three prior boxes of different scales, and each prior box gets three predicted boxes.

[0075] S43: Use Non-Maximum Suppression to remove a large number of redundant predicted boxes, and the remaining predicted boxes are the foreign object detection prediction results output by the model, which include confidence probability values, category probability values, and bounding box position information.

[0076] In addition, in the step S4, the two tasks of region segmentation and region line detection use the same model to perform two upsampling operations on the feature map F5 to obtain a WxHx2 feature map, which is the region segmentation prediction result and the region line detection prediction result. The two channels in the feature map represent the probability of each pixel being a region or background. Information sharing between multiple tasks improves the accuracy of segmentation, and only two upsampling operations make the model less complex and faster.

[0077] The total loss function is:

[0078]

[0079] The foreign object detection loss includes the foreign object classification loss The foreign object confidence loss And the foreign object prediction box position loss The classification loss of foreign object detection And the confidence loss The cross-entropy loss is used, and the prediction box position loss The GIoU Loss is used; in step S1, the foreign object detection real label includes the foreign object classification real label and the foreign object position real label, and in step S4, the preliminary foreign object detection prediction result includes the foreign object classification prediction result and the foreign object position prediction result. The foreign object classification loss Is calculated from the foreign object classification real label and the foreign object classification prediction result, and the foreign object prediction box position loss Is calculated from the foreign object position real label and the foreign object position prediction result; the confidence loss Is calculated by subtracting the intersection over union (IoU) of the foreign object position prediction result and the foreign object position real label from 1; the region segmentation loss And the region line detection loss Also use the cross-entropy loss; α1, α2, α3 and β1, β2, β3 are constants obtained by manual fine-tuning to balance the importance of the three tasks of foreign object detection, region segmentation, and region line detection.

[0080] In addition, step S7 specifically includes the following steps:

[0081] S71: The result fusion process first performs pixel intersection statistics on the prediction box containing the foreign matter output by the detection head Detection in step S6 and the WxHx2 feature map obtained by the region segmentation task according to the fact that the foreign matter is usually contained in the region segmentation prediction result, and determines whether the number of intersected pixel points is greater than a specified threshold. If yes, step S72 is performed; if no, step S73 is performed.

[0082] S72: If the number of intersected pixel points is greater than the specified threshold, it is considered that the position of the prediction box is consistent with the actual situation, and the confidence probability value of the foreign matter in the prediction box is not adjusted, and step S76 is directly performed.

[0083] S73: If the number of intersected pixel points is not greater than the specified threshold, it is determined whether the pixels inside the prediction box are on the straight line of the WxHx2 feature map obtained by the region line detection task according to the fact that the foreign matter occludes the region line. If no, step S74 is performed; if yes, step S75 is performed.

[0084] S74: If no pixel is on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is reduced, and step S76 is directly performed.

[0085] S75: If there is a pixel on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is not changed, and step S76 is directly performed.

[0086] S76: Then, the NMS is performed to filter out the redundant prediction boxes, and the fused foreign matter detection prediction result is obtained.

[0087] In the embodiment, if the number of intersected pixel points is not greater than the specified threshold, it is determined whether the pixels inside the prediction box are on the straight line of the WxHx2 feature map obtained by the region line detection task according to the fact that the foreign matter occludes the region line. The feature map is as shown in FIG. 8. Figure 8 The WxHx2 feature map obtained by the region line detection task is first subjected to Canny edge detection to find the edges in the feature map, and the result is as shown in FIG. 9. Figure 9 Then, the Hough line transformation is used to find the straight lines in the feature map. Since the Hough line transformation needs a threshold Threshold to determine how long the edge is to be recognized as a straight line, a smaller Threshold will result in too many straight lines found, and a larger Threshold will result in missing important straight lines. According to the characteristics of the region line, a too small threshold will result in too many horizontal lines, as shown in FIG. 10, and a too large threshold will result in missing two door gap lines, as shown in FIG. 11. Figure 10 Figure 11 ​The Hough line transform results of the two groups of thresholds need to be combined, that is, all straight lines found by the Hough line transform when the large threshold is retained, and the large number of nearly horizontal straight lines appearing when the small threshold is deleted, and the inclined straight lines found by the small threshold are retained, to obtain more reasonable straight lines. Finally, the straight lines with similar slopes and the distance between the center points of the straight lines less than 8 pixels are deleted to obtain the final remaining straight line examples as follows Figure 12 The pixels inside the prediction box are traversed to find whether there is a pixel on the final remaining straight line. If there is no pixel on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is reduced. If there is a pixel on the final remaining straight line, the confidence probability value of the foreign matter in the prediction box is unchanged. The NMS algorithm is used to screen all prediction boxes after the confidence is adjusted to obtain the fused foreign matter detection result, and the entire fusion process of step S7 is as shown in the following figure Figure 13

[0088] The specified threshold range is 45-50.

[0089] Embodiment 2

[0090] The difference from embodiment 1 is that step S2 specifically comprises the following steps:

[0091] S21: The collected picture is converted from RGB format to HSV format, so that the hue, saturation and brightness of the picture are numerically adjusted, and then converted to RGB format after adjustment;

[0092] S22: The picture after adjustment is randomly rotated, scaled, translated, sheared and left-right flipped, and the above data enhancement means can make the model more robust. It should be noted that before inputting the model, the resolution of all pictures needs to be uniformly changed from 640*480 to 640*640 to facilitate model training.

[0093] Embodiment 3

[0094] ​Different from embodiments 1 and 2, the embodiment also provides experimental result analysis, the method of the embodiment is a multi-task model and a multi-task model after result fusion, and the model of the embodiment further has a region segmentation and a region line detection task on the basis of the foreign matter detection task. The foreign matter detection result is shown in Table 1, and the mAP@0.5 of the foreign matter detection task is 3.7% higher than that of YOLOv4. The reason why the FPS is lower than that of YOLOv4 is that the model simultaneously completes three tasks, and the other two tasks not only help to improve the foreign matter detection accuracy, but also inform the position of the foreign matter, so that the rail transit staff can more properly and quickly handle the foreign matter. The results of the region segmentation and the region line detection are shown in Table 2 and Table 3 respectively. Although the region segmentation and the region line detection task are only used twice, the effect and speed are very excellent. Generally, if the FPS is more than 30, the model can meet the real-time requirement, so the model meets the real-time requirement. The actual effects of the three tasks of target detection, region segmentation and region line detection are shown in Figure 5 and Figure 6 It can be seen that even if some images have light interference, the actual performance of the three tasks is very good. It should be noted that, as shown in Figure 5 , Figure 6 It can be seen that the foreign matter detection, region segmentation and region line detection results by applying the method are very accurate.

[0095]

[0096] Table 1 foreign matter detection result

[0097]

[0098] Table 2 region segmentation result

[0099]

[0100] Table 3 region line detection result

[0101] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the implementation modes are not required or can not be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the application should be included in the protection scope of the claims of the application.

Claims

1. A method for foreign object detection and area perception between a platform door and a train in rail transit based on a multi-task model, characterized in that, The method comprises the following steps: S1: Collect multiple pictures of foreign matters between a rail transit platform door and a train, label foreign matter detection real labels, region segmentation real labels, and region line detection real labels, and provide the real labels for subsequent training; S2: Preprocess the pictures collected in step S1; S3: Build an encoder of the model, wherein the encoder comprises a Backbone module and a Neck module, the Backbone module is used to extract features of the pictures preprocessed in step S2 for use by the Neck module, and the Neck module is used to fuse feature information of the pictures at different stages to extract more in-depth information for use by a subsequent decoder; S4: Build a decoder of the model, wherein the decoder corresponds to the tasks of foreign matter detection, region segmentation, and region line detection respectively, and the decoder obtains preliminary foreign matter detection prediction results, region segmentation prediction results, and region line detection prediction results respectively according to the more in-depth information in step S3; S5: Calculate a foreign matter detection loss according to the foreign matter detection real labels in step S1 and the preliminary foreign matter detection prediction results in step S4, calculate a region segmentation loss according to the region segmentation real labels in step S1 and the preliminary region segmentation prediction results in step S4, calculate a region line detection loss according to the region line detection real labels in step S1 and the preliminary region line detection prediction results in step S4, build a total loss function to obtain a total loss, and the total loss is a weighted sum of the foreign matter detection loss, the region segmentation loss, and the region line detection loss; S6: Perform back propagation on the model according to the total loss calculated in step S5 to obtain specific values of model weights and biases, so as to infer more accurate prediction box results containing foreign matters, foreign matter detection prediction results, region segmentation prediction results, and region line detection prediction results output by a detection head Detection on the model with specific weights and biases; S7: Adjust the prediction results of foreign matter detection according to the region segmentation prediction results and the region line detection prediction results obtained in step S6, and this process is called result fusion; The step S7 specifically comprises the following steps: S71: In the result fusion process, first, according to the fact that foreign matters are usually contained in region segmentation prediction results, perform pixel intersection statistics on the prediction box containing foreign matters output by the detection head Detection in step S6 and the W×H×2 feature map obtained by the region segmentation task, and determine whether the number of intersected pixel points exceeds a specified threshold, if yes, execute step S72, and if no, execute step S73; S72: If the number of intersected pixel points exceeds the specified threshold, it is considered that the position of the prediction box is consistent with the actual situation, and the confidence probability value adjustment of the foreign matter in the prediction box is not performed, and the process jumps to step S76; S73: If the number of intersected pixel points is not enough for the specified threshold, then according to the phenomenon that foreign matters are shielded by region lines, determine whether the pixels inside the prediction box are on the straight lines of the W×H×2 feature map obtained by the region line detection task, if no, execute step S74, and if yes, execute step S75; S74: If the pixels inside the prediction box are not on the straight lines of the W×H×2 feature map obtained by the region line detection task, it is considered that the position of the prediction box is not consistent with the actual situation, and the confidence probability value adjustment of the foreign matter in the prediction box is performed, and the process jumps to step S75; S75: If the pixels inside the prediction box are on the straight lines of the W×H×2 feature map obtained by the region line detection task, it is considered that the position of the prediction box is consistent with the actual situation, and the confidence probability value adjustment of the foreign matter in the prediction box is not performed, and the process jumps to step S76; S76: Perform the confidence probability value adjustment of the foreign matter in the prediction box according to the region line detection prediction results obtained in step S6, and the process ends. S74: If there is no pixel on the final remaining straight line, the confidence probability value of the prediction box foreign matter is reduced, and the step S76 is jumped to; S75: If there is a pixel on the final remaining straight line, the confidence probability value of the prediction box foreign matter is unchanged, and the step S76 is jumped to; S76: Then, the NMS is performed to screen out redundant prediction boxes, and a fused foreign matter detection prediction result is obtained.

2. The method for object detection and area perception between platform screen door and train based on multi-task model according to claim 1, characterized in that, In the step S3, the processing of the Backbone module includes the following steps: S31: The picture of WxHx3 size after the preprocessing in the step S2 is converted into a feature map F1 of WxHx32 size by the Focus module; S32: passing the feature map F1 through the CSP module to obtain a high-dimensional feature map F 11 , and then passing the feature map F 11 through two layers of CSP modules to sequentially obtain high-dimensional feature maps F 12 , F 13 , and adding a Convolutional Block Attention Module to the last layer of standard convolution, so that the feature map F 13 passes through the convolution to obtain a new feature map F3, so that the Neck module can pay more attention to the position information of the foreign matter in the picture.

3. The method of claim 2, wherein the method further comprises: In the step S3, the processing of the Neck module includes the following steps: S33: The Spatial Pyramid Pooling module in the YOLOV3-SPP is used to extract a feature map F4 that fuses information brought by multiple receptive fields from the feature map F3 by using multiple pooling methods; S34: Reuse FeaturePyramidNetworks to top-downly pass the feature map F4 extracted in step S33 through two upsampling operations in sequence with the high-dimensional feature map F 12 and F 13 fusion to obtain a feature map F5 capable of conveying strong semantic information.

4. The method of claim 3, wherein the method further comprises: In the step S33, the multiple pooling methods include 1x1, 5x5, 9x9 and 13x13 four kinds of pooling.

5. The method for object detection and area perception between platform screen door and train based on multi-task model according to claim 3, characterized in that, In the step S4, the decoder has three, which correspond to the three tasks of foreign matter detection, region segmentation and region line detection, wherein the foreign matter detection task includes the following steps: S41: Obtain three size feature maps F using the Path Aggregation Network structure X Each feature map F X contains strong semantic and strong positioning information; S42: input the three size feature maps F X to the detection head Detection for prediction, and each grid of the feature map F X has three scale size prior boxes, and each prior box will obtain three prediction boxes; S43: The Non-Maximum Suppression is used to remove a large number of redundant prediction boxes, and the remaining prediction boxes are the foreign matter detection prediction result output by the model, which contains the confidence probability value, the category probability value and the box position information of the foreign matter.

6. The method of foreign object detection and area perception between platform screen doors and trains for rail transit stations based on a multi-task model according to claim 5, wherein, In the step S4, the same model is used for the region segmentation and region line detection tasks, and the feature map F5 is twice up-sampled to obtain a WxHx2 feature map, which is the region segmentation prediction result and the region line detection prediction result, and the two channels in the feature map represent the probability that each pixel is a region or a background.

7. The multi-task model based method for foreign object detection and area perception between platform screen door and train of rail transit station according to claim 1, characterized in that, In the step S5, the total loss function is: The foreign matter detection loss includes a foreign matter classification loss , a foreign matter confidence loss , and a foreign matter bounding box position loss The foreign matter classification loss and the confidence loss use a cross-entropy loss, and the bounding box position loss uses a GIoU Loss; in step S1, the foreign matter detection ground truth includes a foreign matter classification ground truth and a foreign matter position ground truth, and in step S4, the preliminary foreign matter detection prediction result includes a foreign matter classification prediction result and a foreign matter position prediction result, the foreign matter classification loss is calculated from the foreign matter classification ground truth and the foreign matter classification prediction result, and the foreign matter bounding box position loss is calculated from the foreign matter position ground truth and the foreign matter position prediction result; the confidence loss is calculated from the intersection over union (IoU) of the foreign matter position prediction result and the foreign matter position ground truth, and the confidence loss is calculated from the number 1 minus the IoU; the region segmentation loss and the region line detection loss also use a cross-entropy loss; and are constants used to balance the importance of the three tasks of foreign matter detection, region segmentation, and region line detection.

8. The multi-task model based method for foreign object detection and area perception between platform screen door and train of rail transit station according to claim 1, characterized in that, The specified threshold value ranges from 45 to 50.

9. The multi-task model based method for foreign object detection and area perception between platform screen door and train of rail transit station according to claim 1, characterized in that, The step S2 specifically includes the following steps: S21: The collected picture is converted from the RGB format to the HSV format, so that the hue, saturation and brightness of the picture are numerically adjusted, and then converted to the RGB format after the adjustment; S22: The adjusted picture is randomly rotated, scaled, translated, sheared and left-right flipped.

Images