A foreign body detection method and device for steel support in subway work area

Abstract

The present application provides a method and device for detecting foreign objects in steel supports in a subway work area. The method obtains a real-time steel support image taken by a camera in the subway work area and inputs it into an image depth detection model to determine a real-time depth image; inputs the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, segments the target objects in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image; classifies and identifies the target objects to determine the steel supports and foreign objects in the target objects; verifies whether the foreign objects are target foreign objects on the steel supports based on verification rules in multiple dimensions; if so, determines that the foreign object detection result is that the target foreign objects exist on the steel supports, determines the foreign object detection result, and thus continuously and in real time performs foreign object detection on the steel supports.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting foreign matter in steel supports in a subway work area. Background Art

[0002] The steel support in the subway work area is applied to the subway project. After determining the construction area (referred to as the work area) corresponding to the subway project, steel support (i.e. steel purlin) is usually used in the entire work area. The steel purlin can be 1 layer or multiple layers (usually 2-3 layers). The steel support in the subway work area is an indispensable part of subway construction. It is mainly used to support and stabilize the structure of subway tunnels or stations to ensure safety during construction. The stability and safety of the steel support are directly related to the overall quality of the subway project and the life safety of construction workers.

[0003] If foreign matter is retained on the steel support, it may not only directly affect the supporting force of the steel support itself, weaken its original design efficiency, and threaten the stability of the supported object, but may also become a potential safety hazard and increase risks.

[0004] For the problem of foreign matter trapped on steel supports, existing technologies mainly rely on manual field inspections. Specifically, construction sites will arrange special personnel to conduct regular rotations to check whether there are foreign objects attached to the surface of the steel supports, and regularly summarize and analyze the inspection results. However, this traditional manual inspection method has many shortcomings: on the one hand, manual inspections are limited by personnel's experience and visual sensitivity, and it may be difficult to find all small or hidden foreign objects, resulting in low inspection accuracy; on the other hand, manual inspections require a lot of manpower and time, especially in large and complex construction sites, where inspection efficiency is low and it is difficult to respond and deal with the problem of foreign matter trapped in a timely manner. Summary of the invention

[0005] In view of this, the purpose of the present application is to provide a method and device for detecting foreign objects on steel supports in subway work areas, which can automatically and timely detect the phenomenon of foreign objects being retained on steel supports in subway work areas.

[0006] The present application provides a method for detecting foreign matter in a steel support of a subway work area, the method comprising the following steps:

[0007] Obtain real-time steel support images taken by cameras in subway work areas;

[0008] Inputting the real-time steel support image into an image depth detection model, determining the depth information of the real-time steel support image, and obtaining a real-time depth image corresponding to the real-time steel support image;

[0009] Inputting the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, the target segmentation model segments the target objects in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputting a target segmentation map of the real-time steel support image;

[0010] Classifying and identifying the target object to determine the steel support and foreign matter in the target object;

[0011] Verifying whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions;

[0012] If so, the foreign body detection result is determined to be the target foreign body present on the steel support, and the foreign body detection result is determined.

[0013] In some embodiments, in the foreign body detection method for steel support in a subway work area, verifying whether the foreign body is a target foreign body on the steel support based on verification rules in multiple dimensions includes:

[0014] For the foreign matter that has passed the verification, verify the foreign matter based on the properties of the foreign matter, and select the target foreign matter that has passed the verification;

[0015] Based on the depth information matching result between the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and foreign objects that pass the secondary check are screened out.

[0016] In some embodiments, in the foreign body detection method for steel support in a subway work area, based on the depth information matching result between the foreign body and the steel support, a secondary check is performed to determine whether the foreign body is an obstruction, and foreign bodies that pass the secondary check are screened out, including:

[0017] Determining a relative distance between the foreign object and the steel support based on depth information of the foreign object and the steel support;

[0018] If the relative distance between the foreign object and the steel support is greater than a first preset threshold, it is determined that the foreign object is an obstruction.

[0019] In some embodiments, in the foreign body detection method for steel support in subway work area, for the foreign body that has passed the verification, based on the properties of the foreign body, verifying the foreign body, and screening out the target foreign body that has passed the verification, includes:

[0020] When the size of the foreign object exceeds a first preset size threshold and is smaller than a second preset size threshold, determining that the foreign object has passed the verification;

[0021] The first preset size threshold and the second preset size threshold are determined based on the size and arrangement of the steel supports in the subway work area.

[0022] In some embodiments, in the foreign body detection method for steel supports in subway work areas, after the target segmentation model segments the target objects in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, the method further includes:

[0023] Recognize the real-time steel support image, and determine the pedestrian recognition result and the optical flow recognition result of the real-time steel support image;

[0024] Storing pedestrian recognition results and optical flow recognition results corresponding to the real-time steel support image;

[0025] After determining that the foreign matter detection result is that the target foreign matter exists on the steel support, the method further includes:

[0026] For the first detected target foreign object, obtain the pedestrian recognition result and optical flow recognition result of the real-time steel support image within a preset time period;

[0027] Determine whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period;

[0028] If it is unreasonable, the target foreign matter is screened out.

[0029] In some embodiments, in the foreign body detection method for steel supports in subway work areas, judging whether the source of the target foreign body is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period includes:

[0030] When the pedestrian recognition result of the real-time steel support image within a preset time period is that a pedestrian exists, and / or the optical flow recognition result is that there is an optical flow within a preset range near the target foreign object, and the range of the optical flow is greater than a preset range threshold, it is determined that the source of the target foreign object is reasonable.

[0031] In some embodiments, in the foreign body detection method for steel support in a subway work area, determining the foreign body detection result as the presence of a target foreign body on the steel support, and determining the foreign body detection result includes:

[0032] Determine the target segmentation area where the foreign body detection result is the target foreign body present on the steel support;

[0033] Merge different target segmentation regions that meet the preset connectivity conditions to obtain an updated target segmentation region of the target foreign object;

[0034] The target segmentation region of the target foreign object is identified, and the target segmentation map is updated.

[0035] In some embodiments, in the foreign body detection method for steel support in a subway work area, after identifying the target segmentation area of ​​the target foreign body, the method further includes:

[0036] Determine target geographic location prompt information of the target segmentation area of ​​the target foreign object based on the pre-configured geographic location information of the subway work area and the camera;

[0037] The updated target segmentation map and target geographic location prompt information are sent to the prompt terminal.

[0038] In some embodiments, in the foreign body detection method for steel support in a subway work area, the method further comprises:

[0039] Through the video data captured by different cameras in different subway work areas, the sample steel support images at different times, angles and distances are obtained;

[0040] Cleaning the sample steel support image, and labeling the cleaned sample steel support image to construct a training data set;

[0041] The target segmentation model is trained based on the training data set to obtain a trained target segmentation model.

[0042] In some embodiments, a foreign body detection device for steel support in a subway work area is also provided, the device comprising:

[0043] An acquisition module is used to obtain real-time steel support images taken by cameras in the subway work area;

[0044] A first determination module is used to input the real-time steel support image into an image depth detection model, determine the depth information of the real-time steel support image, and obtain a real-time depth image corresponding to the real-time steel support image;

[0045] A segmentation module, used for inputting the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, wherein the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image;

[0046] A second determination module is used to classify and identify the target object, and determine the steel support and foreign matter in the target object;

[0047] A verification module, used to verify whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions;

[0048] The third determination module is used to determine the foreign object detection result as the presence of the target foreign object on the steel support when the foreign object is the target foreign object on the steel support, and determine the foreign object detection result.

[0049] The present application provides a method and device for detecting foreign matter in steel supports in a subway work area. The method obtains a real-time steel support image taken by a camera in the subway work area; inputs the real-time steel support image into an image depth detection model, determines the depth information of the real-time steel support image, and obtains a real-time depth image corresponding to the real-time steel support image; inputs the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, and the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image; and The object is classified and identified to determine the steel support and foreign matter in the target object; based on the verification rules of multiple dimensions, it is verified whether the foreign matter is the target foreign matter on the steel support; if so, the foreign matter detection result is determined to be the presence of the target foreign matter on the steel support, and the foreign matter detection result is determined, and the steel support is continuously monitored for foreign matters in real time. Compared with manual inspection, the time interval is greatly reduced, so that the situation of foreign matter retention can be quickly discovered; the camera can cover a wider area and observe multiple steel support points at the same time, thereby improving the overall monitoring efficiency; and the foreign matter on the steel support is identified by referring to the depth information of different target objects, and verification is performed based on the verification rules of multiple dimensions, so as to ensure the accuracy of foreign matter detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flow chart showing a method for detecting foreign matter in a steel support of a subway work area according to an embodiment of the present application is shown;

[0052] Figure 2 The corresponding depth map of the subway work area steel support picture described in the embodiment of the present application is shown;

[0053] Figure 3 The target segmentation diagram of the real-time steel support image outputted in the embodiment of the present application is shown;

[0054] Figure 4 A schematic diagram of the structure of a foreign body detection device for steel support in a subway work area according to an embodiment of the present application is shown;

[0055] Figure 5 A schematic structural diagram of an electronic device described in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0057] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0058] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0059] The steel support in the subway work area is applied to the subway project. After determining the construction area (referred to as the work area) corresponding to the subway project, steel support (i.e. steel purlin) is usually used in the entire work area. The steel purlin can be 1 layer or multiple layers (usually 2-3 layers). The steel support in the subway work area is an indispensable part of subway construction. It is mainly used to support and stabilize the structure of subway tunnels or stations to ensure safety during construction. The stability and safety of the steel support are directly related to the overall quality of the subway project and the life safety of construction workers.

[0060] If foreign matter is retained on the steel support, it may not only directly affect the supporting force of the steel support itself, weaken its original design efficiency, and threaten the stability of the supported object, but may also become a potential safety hazard and increase risks, such as causing accidents such as workers tripping and objects falling during construction.

[0061] For the problem of foreign matter trapped on steel supports, existing technologies mainly rely on manual field inspections. Specifically, construction sites will arrange special personnel to conduct regular rotations to check whether there are foreign objects attached to the surface of the steel supports, and regularly summarize and analyze the inspection results. However, this traditional manual inspection method has many shortcomings: on the one hand, manual inspections are limited by personnel's experience and visual sensitivity, and it may be difficult to find all small or hidden foreign objects, resulting in low inspection accuracy; on the other hand, manual inspections require a lot of manpower and time, especially in large and complex construction sites, where inspection efficiency is low and it is difficult to respond and deal with the problem of foreign matter trapped in a timely manner.

[0062] Based on this, a method and device for detecting foreign matter in steel supports in a subway work area are provided in an embodiment of the present application. The method obtains a real-time steel support image taken by a camera in the subway work area; inputs the real-time steel support image into an image depth detection model, determines the depth information of the real-time steel support image, and obtains a real-time depth image corresponding to the real-time steel support image; inputs the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, and the target segmentation model segments the target objects in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image; and performs a target segmentation on the target objects. Classification and identification are performed to determine the steel support and foreign matter in the target object; based on verification rules of multiple dimensions, it is verified whether the foreign matter is the target foreign matter on the steel support; if so, the foreign matter detection result is determined to be the presence of the target foreign matter on the steel support, and the foreign matter detection result is determined. The steel support is continuously monitored for foreign matters in real time. Compared with manual inspections, the time interval is greatly reduced, so that foreign matter retention can be quickly discovered; the camera can cover a wider area and observe multiple steel support points at the same time, thereby improving the overall monitoring efficiency; and the foreign matter on the steel support is identified by referring to the depth information of different target objects, and verification is performed based on verification rules of multiple dimensions, thereby ensuring the accuracy of foreign matter detection.

[0063] Please refer to Figure 1 , Figure 1 The flowchart of the foreign body detection method of the steel support in the subway work area described in the embodiment of the present application is shown; Figure 1 As shown, the foreign body detection method of the steel support in the subway work area includes the following steps S101-S106:

[0064] S101, obtaining a real-time steel support image captured by a camera in a subway work area;

[0065] S102, inputting the real-time steel support image into an image depth detection model, determining depth information of the real-time steel support image, and obtaining a real-time depth image corresponding to the real-time steel support image;

[0066] S103, inputting the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, wherein the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image;

[0067] S104, classifying and identifying the target object to determine the steel support and foreign matter in the target object;

[0068] S105, verifying whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions;

[0069] S106. If yes, determine that the foreign matter detection result is that the target foreign matter exists on the steel support, and determine the foreign matter detection result.

[0070] In the embodiment of the present application, it is necessary to prepare a training data set to train the target segmentation model.

[0071] Specifically, the foreign body detection method for steel support in a subway work area described in the embodiment of the present application also includes:

[0072] Through the video data captured by different cameras in different subway work areas, the sample steel support images at different times, angles and distances are obtained;

[0073] Cleaning the sample steel support image, and labeling the cleaned sample steel support image to construct a training data set;

[0074] The target segmentation model is trained based on the training data set to obtain a trained target segmentation model.

[0075] Specifically, steel support data at different times, angles, and distances are obtained through on-site videos to ensure data diversity and ensure that the training data set contains diverse samples to improve the generalization ability of the model.

[0076] The data was cleaned by a python automated processing program to remove blurry, poorly lit, and angled data, and the labelme annotation tool was used to annotate the steel supports for subsequent training.

[0077] The cleaned sample steel support image is marked. Specifically, the steel support body, the location of foreign matter on the steel support and its type (such as metal fragments, dirt, plastic bags, clothes, etc.) are clearly marked.

[0078] In step S101, a real-time steel support image captured by a camera in a subway work area is obtained.

[0079] The camera may be a high-definition camera, an infrared camera, a camera with a night vision function, etc.

[0080] The real-time video data captured by the camera is transmitted to a monitoring center or a cloud server, and image frames of the real-time video data are extracted to obtain a real-time steel support image.

[0081] In the step S102, the real-time steel support image is input into an image depth detection model to determine the depth information of the real-time steel support image, and obtain a real-time depth image corresponding to the real-time steel support image.

[0082] There are usually shadows, rust and other information on the steel supports in subway work areas. If conventional image processing technologies such as edge detection and morphological processing are used, shadows, rust and other information can be easily identified as foreign objects.

[0083] Based on this, in an embodiment of the present application, the real-time steel support image is input into an image depth detection model to determine the depth information of the real-time steel support image, and obtain a real-time depth image corresponding to the real-time steel support image.

[0084] The real-time depth image is an image representing the depth information of each pixel in the steel support image, which provides three-dimensional information of the image; specifically, each pixel value in the depth image represents the depth (or distance) of the pixel in the image, usually expressed in grayscale value or pseudo color; therefore, the real-time depth image represents the spatial position and distance relationship of different objects.

[0085] Please refer to Figure 2 , Figure 2 The corresponding depth map of the subway work area steel support image described in the embodiment of the present application is shown. Figure 2 As shown, the targets at the same depth have similar colors, so that the steel supports and obstructions can be distinguished, while the rust, shadows and other information on the steel supports are concealed.

[0086] In the embodiment of the present application, the image depth detection model adopts the depth-anything model. The depth-anything model is an advanced model for monocular depth estimation. The core of the model is to use large-scale unlabeled data to improve the accuracy and robustness of depth estimation; the goal is to train the MDE model with a diverse training set so that the depth can be predicted for any given image.

[0087] In step S103, the real-time depth image and the real-time steel support image are input into a pre-trained target segmentation model, and the target segmentation model segments the target objects in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image.

[0088] Since the real-time depth image can show that the invisible, rusty and other areas on the steel support belong to the same depth as the steel support, the target object in the segmented real-time steel support image does not include non-object false detections caused by the shadows and rust, thereby improving the detection accuracy.

[0089] At the same time, since the annotated steel support image includes steel supports with shadows and rust when training the target segmentation model, the target segmentation model can accurately segment the steel supports with shadows and rust without identifying the shadows and rust on the steel supports.

[0090] That is to say, the embodiment of the present application prevents the misidentification of shadows and rust on the steel support from two aspects.

[0091] Segment the target object in the real-time steel support image and output the target segmentation map of the real-time steel support image, that is, you can frame the different identified target objects, please refer to Figure 3 , Figure 3 The target segmentation diagram of the real-time steel support image outputted in the embodiment of the present application is shown, wherein different objects are distinguished by different colors or labels.

[0092] In the step S103, the target object is classified and identified to determine the steel support and foreign matter in the target object.

[0093] Each segmented area is extracted from the target segmentation map. These areas correspond to different objects in the image. Features are extracted from each segmented area, such as shape, size, color, texture, depth, etc., to determine whether each segmented area is a steel support or a foreign object.

[0094] In the embodiment of the present application, the target segmentation model adopts the segmentation model segmentAnything model SAM; its main function is to be able to accurately segment any object in any image.

[0095] The SAM model architecture consists of three parts:

[0096] Encoder: SAM uses a pre-trained image encoder, such as ViT (Vision Transformer), to extract image features; this encoder can understand the global and local information in the image.

[0097] Prompt Encoder: This component is responsible for processing user-provided prompts (such as points, boxes, or text). It converts these prompts into fixed-size embedding vectors, which are subsequently used to guide the segmentation process.

[0098] Decoder: The decoder is the core part of the model, which receives information from the encoder and the hint encoder and generates a segmentation mask; this decoder is usually a lightweight network that can predict the probability of each pixel belonging to the foreground or background.

[0099] In actual use, the same picture only needs to be encoded once by the encoder, and then the prompt encoder and decoder input different prompt contents for prediction.

[0100] In the step S105, it is verified whether the foreign object is the target foreign object on the steel support based on verification rules of multiple dimensions.

[0101] Specifically, the verification rules based on multiple dimensions verify whether the foreign object is a target foreign object on the steel support, including:

[0102] For the foreign matter that has passed the verification, verify the foreign matter based on the properties of the foreign matter, and select the target foreign matter that has passed the verification;

[0103] Based on the depth information matching result between the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and foreign objects that pass the secondary check are screened out.

[0104] Based on the depth information matching result of the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and the foreign objects that pass the secondary check are screened out, including:

[0105] Determining a relative distance between the foreign object and the steel support based on depth information of the foreign object and the steel support;

[0106] If the relative distance between the foreign object and the steel support is greater than a first preset threshold, it is determined that the foreign object is an obstruction.

[0107] When the relative distance between the foreign object and the steel support is greater than the first preset threshold, it means that the foreign object is far away from the steel support and is not retained on the steel support, and is not a target foreign object to be detected in the embodiment of the present application.

[0108] As an example only, the obstruction is an object such as a utility pole, a building, or equipment between the steel support and the camera.

[0109] In this way, the foreign object detection method described in the embodiment of the present application detects the obstruction through the depth information of the foreign object and the steel support, which further reduces the probability of false detection and improves the accuracy of foreign object detection.

[0110] Verifying the foreign matter based on the property of the foreign matter and selecting the target foreign matter that passes the verification includes:

[0111] When the size of the foreign object exceeds a first preset size threshold and is smaller than a second preset size threshold, determining that the foreign object has passed the verification;

[0112] The first preset size threshold and the second preset size threshold are determined based on the size and arrangement of the steel supports in the subway work area.

[0113] The size of the foreign body should be in a certain proportion to the steel support, and foreign bodies that exceed the proportion are less likely to be contaminated.

[0114] For example, there may be tiny burrs, bumps and other traces on the steel support, which are unlikely to be foreign matter. Therefore, when the size of the foreign matter is too small, it is filtered out and no longer segmented.

[0115] However, due to the size, shape and arrangement of the steel supports, it is impossible for oversized objects, such as objects that occupy half of the image, to remain on the steel supports. These objects may be caused by the camera shooting angle, camera shake, etc. Therefore, in the embodiments of the present application, these oversized or undersized objects are directly filtered out.

[0116] Based on this, in the embodiment of the present application, objects that are too large or too low are first filtered out, and then obstructions are screened out based on the depth information matching results between the foreign objects and the steel support, and the detection accuracy of foreign objects is improved through secondary inspection.

[0117] In addition, foreign objects usually do not appear suddenly. Foreign objects are usually placed by humans or dropped from the upper steel supports. Based on these two points, we can determine whether someone passed by some time before the object appeared. If someone passed by, the possibility that it was the object was high. We can also determine whether anything moved from the upper part to the steel support some time before the object appeared. If so, the possibility that the object was a foreign object was high.

[0118] Based on this, in the embodiment of the present application, after the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, the method further includes:

[0119] Recognize the real-time steel support image, and determine the pedestrian recognition result and the optical flow recognition result of the real-time steel support image;

[0120] The pedestrian recognition result and the optical flow recognition result corresponding to the real-time steel support image are stored.

[0121] After determining that the foreign matter detection result is that the target foreign matter exists on the steel support, the method further includes:

[0122] For the first detected target foreign object, obtain the pedestrian recognition result and optical flow recognition result of the real-time steel support image within a preset time period;

[0123] Determine whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period;

[0124] If it is unreasonable, the target foreign matter is screened out.

[0125] Specifically, judging whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period includes:

[0126] When the pedestrian recognition result of the real-time steel support image within a preset time period is that a pedestrian exists, and / or the optical flow recognition result is that there is an optical flow within a preset range near the target foreign object, and the range of the optical flow is greater than a preset range threshold, it is determined that the source of the target foreign object is reasonable.

[0127] The detection accuracy of pedestrian recognition results and optical flow recognition results is very high because their characteristics are specific and clear; the foreign object detection method described in the embodiment of the present application verifies the rationality of foreign objects through high-precision pedestrian recognition results and optical flow recognition results, thereby further improving the foreign object detection accuracy.

[0128] In step S106, if yes, the foreign matter detection result is determined to be the presence of target foreign matter on the steel support, and the foreign matter detection result is determined.

[0129] Determine the foreign body detection result as the target foreign body on the steel support, and determine the foreign body detection result, including:

[0130] Determine the target segmentation area where the foreign body detection result is the target foreign body present on the steel support;

[0131] Merge different target segmentation regions that meet the preset connectivity conditions to obtain an updated target segmentation region of the target foreign object;

[0132] The target segmentation region of the target foreign object is identified, and the target segmentation map is updated.

[0133] Here, the target foreign matter is a foreign matter that has passed multiple verifications, and the accuracy is guaranteed.

[0134] The preset connectivity condition is only used as an example, and the distance between different target segmentation areas is less than a preset distance threshold.

[0135] Since the ultimate goal of the foreign body detection method for steel supports in subway work areas is to determine whether there are foreign bodies on the steel supports rather than to detect the types of foreign bodies, target segmentation areas with similar positions are merged to simplify the target segmentation map.

[0136] In the foreign body detection method described in the embodiment of the present application, after identifying the target segmentation area of ​​the target foreign body, the method further includes:

[0137] Determine target geographic location prompt information of the target segmentation area of ​​the target foreign object based on the pre-configured geographic location information of the subway work area and the camera;

[0138] The updated target segmentation map and target geographic location prompt information are sent to the prompt terminal.

[0139] Based on the pre-configured geographic location information of the subway work area and the camera, the target geographic location prompt information of the target segmentation area of ​​the target foreign object is determined. As an example only, the target geographic location prompt information can be: in the subway work area at the south end, middle, left side, 10m away from the south end, etc., so as to prompt the staff the location of the foreign object.

[0140] Since the target segmentation map is simplified, foreign objects with close locations correspond to the same target geographic location prompt information, and the target segmentation map and target geographic location prompt information sent to the prompt terminal are more concise and more user-friendly.

[0141] Based on the same inventive concept, the embodiment of the present application also provides a foreign object detection device for steel supports in subway work areas corresponding to the foreign object detection method for steel supports in subway work areas. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the foreign object detection method for steel supports in subway work areas in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0142] Please refer to Figure 4 , Figure 4 The schematic diagram of the structure of the foreign body detection device for the steel support in the subway work area according to the embodiment of the present application is shown; the foreign body detection device for the steel support in the subway work area comprises:

[0143] An acquisition module 401 is used to acquire a real-time steel support image captured by a camera in a subway work area;

[0144] A first determination module 402 is used to input the real-time steel support image into an image depth detection model, determine the depth information of the real-time steel support image, and obtain a real-time depth image corresponding to the real-time steel support image;

[0145] A segmentation module 403 is used to input the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, the target segmentation model refers to the depth information of different target objects in the real-time steel support image to segment the target object in the real-time steel support image, and output a target segmentation map of the real-time steel support image;

[0146] The second determination module 404 is used to classify and identify the target object, and determine the steel support and foreign matter in the target object;

[0147] A verification module 405 is used to verify whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions;

[0148] The third determination module 406 is used to determine the foreign object detection result as the presence of the target foreign object on the steel support when the foreign object is the target foreign object on the steel support, and determine the foreign object detection result.

[0149] In some embodiments, in the foreign body detection device for steel support in a subway work area, the verification module, when verifying whether the foreign body is a target foreign body on the steel support based on verification rules of multiple dimensions, is specifically used to:

[0150] For the foreign matter that has passed the verification, verify the foreign matter based on the properties of the foreign matter, and select the target foreign matter that has passed the verification;

[0151] Based on the depth information matching result between the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and foreign objects that pass the secondary check are screened out.

[0152] In some embodiments, in the foreign body detection device for steel support in subway work area, the verification module, based on the depth information matching result between the foreign body and the steel support, performs a secondary verification to determine whether the foreign body is an obstruction, and selects the foreign body that passes the secondary verification, is specifically used to:

[0153] Determining a relative distance between the foreign object and the steel support based on depth information of the foreign object and the steel support;

[0154] If the relative distance between the foreign object and the steel support is greater than a first preset threshold, it is determined that the foreign object is an obstruction.

[0155] In some embodiments, in the foreign body detection device for steel support in subway work area, the verification module, when verifying the foreign body that has passed the verification based on the properties of the foreign body and screening out the target foreign body that has passed the verification, is specifically used to:

[0156] When the size of the foreign object exceeds a first preset size threshold and is smaller than a second preset size threshold, determining that the foreign object has passed the verification;

[0157] The first preset size threshold and the second preset size threshold are determined based on the size and arrangement of the steel supports in the subway work area.

[0158] In some embodiments, in the foreign body detection device for steel support in a subway work area, the segmentation module, after the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, is further used to:

[0159] Recognize the real-time steel support image, and determine the pedestrian recognition result and the optical flow recognition result of the real-time steel support image;

[0160] Storing pedestrian recognition results and optical flow recognition results corresponding to the real-time steel support image;

[0161] The third determination module, after determining that the foreign matter detection result is that the target foreign matter exists on the steel support, is further used to:

[0162] For the first detected target foreign object, obtain the pedestrian recognition result and optical flow recognition result of the real-time steel support image within a preset time period;

[0163] Determine whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period;

[0164] If it is unreasonable, the target foreign matter is screened out.

[0165] In some embodiments, in the foreign body detection device for steel support in subway work area, the third determination module, when judging whether the source of the target foreign body is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period, is specifically used to:

[0166] When the pedestrian recognition result of the real-time steel support image within a preset time period is that a pedestrian exists, and / or the optical flow recognition result is that there is an optical flow within a preset range near the target foreign object, and the range of the optical flow is greater than a preset range threshold, it is determined that the source of the target foreign object is reasonable.

[0167] In some embodiments, in the foreign body detection device for steel support in a subway work area, the third determination module, when determining that the foreign body detection result is that there is a target foreign body on the steel support, is specifically used to:

[0168] Determine the target segmentation area where the foreign body detection result is the target foreign body present on the steel support;

[0169] Merge different target segmentation regions that meet the preset connectivity conditions to obtain an updated target segmentation region of the target foreign object;

[0170] The target segmentation region of the target foreign object is identified, and the target segmentation map is updated.

[0171] In some embodiments, in the foreign body detection device for steel support in a subway work area, the third determination module is further used to determine target geographical location prompt information of the target segmented area of ​​the target foreign body based on pre-configured geographical location information of the subway work area and the camera after identifying the target segmented area of ​​the target foreign body;

[0172] The updated target segmentation map and target geographic location prompt information are sent to the prompt terminal.

[0173] In some embodiments, the foreign body detection device for steel support in subway work areas further includes a training module, the training module being used to obtain sample steel support images at different times, angles and distances through video data captured by different cameras in different subway work areas;

[0174] Cleaning the sample steel support image, and labeling the cleaned sample steel support image to construct a training data set;

[0175] The target segmentation model is trained based on the training data set to obtain a trained target segmentation model.

[0176] Based on the same inventive concept, an electronic device corresponding to the foreign object detection method for steel supports in subway work areas is also provided in the embodiments of the present application. Since the principle of solving the problem by the electronic device in the embodiments of the present application is similar to the foreign object detection method for steel supports in subway work areas in the embodiments of the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0177] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of the electronic device described in an embodiment of the present application is shown; the electronic device 500 includes: a processor 502, a memory 501 and a bus, the memory 501 stores machine-readable instructions executable by the processor 502, and when the electronic device 500 is running, the processor 502 communicates with the memory 501 through the bus, and when the machine-readable instructions are executed by the processor 502, the steps of the foreign body detection method for steel supports in subway work areas are executed.

[0178] Based on the same inventive concept, an electronic device corresponding to the foreign object detection method for steel supports in subway work areas is also provided in the embodiments of the present application. Since the principle of solving the problem by the electronic device in the embodiments of the present application is similar to the foreign object detection method for steel supports in subway work areas in the embodiments of the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0179] A computer-readable storage medium stores a computer program, which executes the steps of the foreign body detection method for steel supports in subway work areas when the computer program is run by a processor.

[0180] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0181] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0184] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting foreign matter in steel supports in subway work areas, characterized in that: The method comprises the following steps: Obtain real-time steel support images taken by cameras in subway work areas; Inputting the real-time steel support image into an image depth detection model, determining the depth information of the real-time steel support image, and obtaining a real-time depth image corresponding to the real-time steel support image; Inputting the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputting a target segmentation map of the real-time steel support image; Classifying and identifying the target object to determine the steel support and foreign matter in the target object; Verifying whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions; If yes, the foreign body detection result is determined to be the target foreign body on the steel support, and the foreign body detection result is determined; Verify whether the foreign object is the target foreign object on the steel support based on verification rules in multiple dimensions, including: For the foreign objects that have passed the verification, verify the foreign objects based on the properties of the foreign objects, and select the target foreign objects that have passed the verification; Based on the depth information matching result between the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and the foreign objects that pass the secondary check are selected; Based on the depth information matching result of the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and the foreign objects that pass the secondary check are screened out, including: Determining a relative distance between the foreign object and the steel support based on depth information of the foreign object and the steel support; If the relative distance between the foreign object and the steel support is greater than a first preset threshold, the foreign object is determined to be an obstruction; for the foreign object that has passed the verification, the foreign object is verified based on the properties of the foreign object, and the target foreign object that has passed the verification is screened out, including: When the size of the foreign object exceeds a first preset size threshold and is smaller than a second preset size threshold, determining that the foreign object has passed the verification; The first preset size threshold and the second preset size threshold are determined based on the size and arrangement of the steel supports in the subway work area.

2. The foreign body detection method for steel support in subway work area according to claim 1 is characterized in that: After the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, the method further includes: Recognize the real-time steel support image, and determine the pedestrian recognition result and the optical flow recognition result of the real-time steel support image; Storing pedestrian recognition results and optical flow recognition results corresponding to the real-time steel support image; After determining that the foreign matter detection result is that the target foreign matter exists on the steel support, the method further includes: For the first detected target foreign object, obtain the pedestrian recognition result and optical flow recognition result of the real-time steel support image within a preset time period; Determine whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period; If it is unreasonable, the target foreign matter is screened out.

3. The foreign body detection method for steel support in subway work area according to claim 2 is characterized in that: Judging whether the source of the target foreign object is reasonable based on the pedestrian recognition result and the optical flow recognition result of the real-time steel support image within the preset time period includes: When the pedestrian recognition result of the real-time steel support image within a preset time period is that a pedestrian exists, and / or the optical flow recognition result is that there is an optical flow within a preset range near the target foreign object, and the range of the optical flow is greater than a preset range threshold, it is determined that the source of the target foreign object is reasonable.

4. The foreign body detection method for steel support in subway work area according to claim 1 or 3 is characterized in that: Determine the foreign body detection result as the target foreign body on the steel support, and determine the foreign body detection result, including: Determine the target segmentation area where the foreign matter detection result is the target foreign matter existing on the steel support; Merge different target segmentation regions that meet the preset connectivity conditions to obtain an updated target segmentation region of the target foreign object; A target segmentation region of the target foreign object is identified, and the target segmentation map is updated.

5. The foreign body detection method for steel support in subway work area according to claim 4 is characterized in that: After identifying the target segmentation area of ​​the target foreign object, the method further includes: Based on the pre-configured geographic location information of the subway work area and the camera, determine the target geographic location prompt information of the target segmentation area of ​​the target foreign object; The updated target segmentation map and target geographic location prompt information are sent to the prompt terminal.

6. The foreign body detection method for steel support in subway work area according to claim 1 is characterized in that: The method further comprises: Through the video data captured by different cameras in different subway work areas, the sample steel support images at different times, angles and distances are obtained; Cleaning the sample steel support image, and labeling the cleaned sample steel support image to construct a training data set; The target segmentation model is trained based on the training data set to obtain a trained target segmentation model.

7. A foreign body detection device for steel support in subway work area, characterized in that: The device comprises: An acquisition module is used to obtain real-time steel support images taken by cameras in the subway work area; A first determination module is used to input the real-time steel support image into an image depth detection model, determine the depth information of the real-time steel support image, and obtain a real-time depth image corresponding to the real-time steel support image; A segmentation module, used for inputting the real-time depth image and the real-time steel support image into a pre-trained target segmentation model, wherein the target segmentation model segments the target object in the real-time steel support image with reference to the depth information of different target objects in the real-time steel support image, and outputs a target segmentation map of the real-time steel support image; A second determination module is used to classify and identify the target object, and determine the steel support and foreign matter in the target object; A verification module, used to verify whether the foreign object is a target foreign object on the steel support based on verification rules of multiple dimensions; A third determination module is used to determine, when the foreign object is a target foreign object on the steel support, that the foreign object detection result is that the target foreign object exists on the steel support, and determine the foreign object detection result; Verify whether the foreign object is the target foreign object on the steel support based on verification rules in multiple dimensions, including: For the foreign objects that have passed the verification, verify the foreign objects based on the properties of the foreign objects, and select the target foreign objects that have passed the verification; Based on the depth information matching result between the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and the foreign objects that pass the secondary check are selected; Based on the depth information matching result of the foreign object and the steel support, a secondary check is performed to determine whether the foreign object is an obstruction, and the foreign objects that pass the secondary check are screened out, including: Determining a relative distance between the foreign object and the steel support based on depth information of the foreign object and the steel support; If the relative distance between the foreign object and the steel support is greater than a first preset threshold, the foreign object is determined to be an obstruction; for the foreign object that has passed the verification, the foreign object is verified based on the properties of the foreign object, and the target foreign object that has passed the verification is screened out, including: When the size of the foreign object exceeds a first preset size threshold and is smaller than a second preset size threshold, determining that the foreign object has passed the verification; The first preset size threshold and the second preset size threshold are determined based on the size and arrangement of the steel supports in the subway work area.

Images