Road foreign matter detection method based on vision of tunnel inspection robot and related equipment

Abstract

The invention provides a road foreign matter detection method based on tunnel inspection robot vision and related equipment, and relates to the technical field of intelligent traffic tunnel inspection robots. The method comprises the following steps: training by using an existing tunnel robot video data set to obtain an ROI region and a road object instance segmentation model; acquiring a video stream data frame of the inspection robot, generating an edge binary image through edge detection, obtaining related mask data through model reasoning, and recording classified foreign matter surrounding frame data; processing and combining various data, judging residual objects through morphological operation and contour detection, and determining unclassified foreign matter bounding box data; then reporting two types of foreign matter bounding box data according to rules; and finally, generating an annotation file according to the unclassified foreign matters, and updating the road object instance segmentation model. By implementing the method, tunnel road foreign matters can be timely and accurately detected, potential safety hazards are reduced, smooth tunnel traffic is guaranteed, and the intelligent and automatic level of tunnel inspection is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent traffic tunnel inspection technology, and in particular to a road foreign object detection method and related equipment based on the vision of a tunnel inspection robot. Background Art

[0002] In modern transportation systems, tunnels are a key component of the transportation network, and their safe operation is crucial. Tunnels are characterized by high vehicle speeds and relatively enclosed spaces. The presence of foreign objects in tunnels can easily lead to traffic accidents, severely disrupting traffic order and even endangering lives. Therefore, timely and accurate detection of foreign objects in tunnels is crucial for ensuring tunnel traffic safety.

[0003] Currently, some existing technologies for detecting foreign objects in tunnels rely primarily on manual inspections. Workers regularly enter the tunnel and inspect the roadway, visually identifying any foreign objects. Other technologies employ fixed cameras for monitoring, using simple image comparison algorithms to determine the presence of foreign objects. For example, they compare the current image with a pre-stored background image; any discrepancies indicate the presence of a foreign object.

[0004] However, although the smart cameras fixedly installed above the tunnel can detect objects to a certain extent, the installation layout of the cameras is limited by the tunnel structure, and there are detection fuzzy areas. They can only accurately detect areas within about 30 meters of the camera and cannot fully and accurately cover the entire tunnel road area. Summary of the Invention

[0005] The present application provides a method and related equipment for detecting foreign objects on the road based on the vision of a tunnel inspection robot, which is used to improve the accuracy and timeliness of foreign object detection on tunnel roads, thereby effectively ensuring traffic safety and smooth flow in the tunnel.

[0006] In the first aspect, the present application provides a road foreign object detection method based on the vision of a tunnel inspection robot, which is applied to a server. The method comprises: training an ROI region instance segmentation model and a road object instance segmentation model based on an existing tunnel robot video data set; obtaining a video stream data frame sent by the inspection robot and performing edge detection on the video stream data frame to generate a corresponding edge binary map; inferring the original data frame obtained by the camera based on the ROI region instance segmentation model and the road object instance segmentation model to obtain an inference result, which includes ROI region mask data and road object mask data. If the classified foreign object segmentation data appears in the inference result, and the foreign object segmentation data If the confidence level of the data exceeds a preset confidence threshold, the corresponding bounding box data is recorded as BBOX_ABNORMAL_1st; the edge binary image, the ROI area mask data, and the road object mask data are processed and merged to obtain a merged data image, and a morphological dilation operation is performed on the merged data image, and the remaining object contour map data is obtained through a contour detection algorithm; based on the area characteristics, speed characteristics, and continuity characteristics of the remaining object in the remaining object contour map data, it is again determined whether the remaining object is noise data. The remaining object that is not noise data is determined as an unclassified road foreign object, and the corresponding bounding box data is determined as BBOX_ABNORMAL_2nd; The BBOX_ABNORMAL_1st and the BBOX_ABNORMAL_2nd are reported to the preset management terminal according to the preset warning rules; a labeling file is automatically generated according to the unclassified road foreign object, and the corresponding image frame is loaded into the training database for model training to obtain a reloaded and trained road object instance segmentation model.

[0007] By employing the above technical solution, a model trained using an existing tunnel robot video data set performs edge detection and instance segmentation reasoning on the inspection robot's video stream data frames, combining multi-faceted data processing and feature analysis to identify foreign objects. This multi-model combination improves foreign object recognition accuracy, enabling the detection of both classified and unclassified foreign objects. Annotation files are generated for unclassified foreign objects, updating the model so that the model continuously learns and adapts to new foreign objects. This enables timely and accurate detection of foreign objects in tunnels, reducing safety risks, ensuring smooth tunnel traffic, and enhancing the intelligent and automated level of tunnel inspection.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of training an ROI region instance segmentation model and a road object instance segmentation model based on an existing tunnel robot video data set specifically includes: collecting video data collected by the tunnel robot at different time periods, different tunnel positions, and different camera angles, and extracting frames to make a picture training set; copying the picture training set to obtain an ROI picture training set and a road object image set, and performing instance segmentation and annotation on each of them. The ROI picture training set is a road monitoring foreign object area, that is, excluding the curbs or maintenance road parts on both sides, excluding the road part beyond a set distance, and only retaining the road area within the set distance for annotation. The road object image set includes at least lane lines, vehicles, pedestrians, non-motor vehicles and foreign objects. If the foreign object has confirmed foreign object data recorded in the traffic history video, it is a classified foreign object; using a preset instance segmentation model, the ROI picture training set and the road object image set are trained to obtain a ROI region instance segmentation model and a road object instance segmentation model.

[0009] By employing the above technical solution, tunnel robot video footage under various conditions was collected to create an image training set. This set of images was then divided into a ROI training set and a road object image set, each of which was annotated separately. This comprehensive data covers various scenarios, while ROI annotation focuses on key monitoring areas and road object annotation provides a richer classification. Using pre-set training models, the ROI instance segmentation model and the road object instance segmentation model learn distinct features, improving their recognition capabilities for diverse scenarios and objects, laying the foundation for accurate foreign object detection in tunnels.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the step of obtaining the video stream data frame sent by the inspection robot and performing edge detection on the video stream data frame to generate the corresponding edge binary map specifically includes: after obtaining the video stream data frame in the tunnel in real time through the high-definition camera carried by the inspection robot in the tunnel, the video stream data frame is converted into a tensor in the GPU, the number of channels of the tensor is 3, the width is WIDTH_org, and the height is HEIGHT_org; Gaussian filtering and denoising are performed on each of the obtained video stream data frames, and then grayscale processing is performed to obtain a grayscale map; an edge detection algorithm is applied to the grayscale map to obtain an edge binary map.

[0011] By employing this technical solution, the inspection robot's video stream data frames are converted into GPU tensors, subjected to Gaussian filtering for denoising and grayscale conversion, and then an edge detection algorithm is used to generate a binary edge map. GPU acceleration improves efficiency, Gaussian filtering for denoising reduces interference, grayscale conversion simplifies the data, and edge detection highlights object contours. The clear binary edge map facilitates subsequent integration with instance segmentation data for more accurate identification of foreign objects on the road, improving the accuracy and reliability of foreign object detection.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the original data frame acquired by the camera is inferred based on the ROI region instance segmentation model and the road object instance segmentation model to obtain an inference result, which includes ROI region mask data and road object mask data. If classified foreign body segmentation data appears in the inference result and the confidence of the foreign body segmentation data exceeds a preset confidence threshold, the corresponding bounding box data is recorded as BBOX_ABNORMAL_1st, which specifically includes: performing instance segmentation inference on the video stream data frame The ROI region instance segmentation model is used to infer the ROI region mask data tensor MSK_ROI. The MSK_ROI tensor is a binary tensor with a value of 0 or 1, corresponding to a width of WIDTH_msk, a height of HEIGHT_msk, and a dimension of [1, WIDTH_msk, HEIGHT_msk]. The road object instance segmentation model is used to infer the road object mask data tensor MSK_STUFF and bounding box data BBOX_STUFF. The dimension of the MSK_STUFF tensor is [k, WIDTH_msk, HEIGHT_msk], where k is the number of identified segmentation instances. If the MSK_STUFF tensor contains a foreign object mask and its confidence is greater than the preset confidence threshold, it means that the object represented by the foreign object mask is a classified road foreign object. In this case, the bounding box data of the road foreign object is extracted from the bounding box data BBOX_STUFF and recorded as BBOX_ABNORMAL_1st.

[0013] By employing this technical solution, instance segmentation reasoning is performed on video stream data frames, generating ROI (Region of Interest) and road object mask data. If classified foreign object segmentation data exceeds a confidence threshold, bounding box data is recorded. Model reasoning accurately identifies objects, and confidence threshold screening ensures identification reliability. Timely recording of classified foreign object bounding box data enables rapid object location, providing a basis for subsequent warning and response, and improving the efficiency of tunnel foreign object detection and response.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after processing the edge binary map, the ROI region mask data, and the road object mask data, they are merged to obtain a merged data map, and then a morphological dilation operation is performed on the merged data map, and the remaining object contour map data is obtained by a contour detection algorithm. The steps specifically include: removing the padding operation on the ROI region mask data and the road object mask data, and then interpolating and scaling them to the size of the original frame image, where the width of the original frame image is WIDTH_org and the height is HEIGHT_org; performing a morphological dilation operation on the mask data in the tensor MSK_STUFF, where the morphological dilation operation is used to expand the contour area of the identified data including at least lane lines, vehicles, people, and foreign objects; performing a bitwise logical inversion operation on the mask data in the tensor MSK_STUFF to obtain MSK_STUFF_INV; splicing the MSK_ STUFF_INV and the tensor MSK_ROI are used to obtain the tensor MSK_CAT, the dimension of which is [1+k, WIDTH_org, HEIGHT_org]. The tensor MSK_CAT is multiplied and merged by positional element values in the direction of the first dimension to obtain a result tensor, and the result tensor is byte-ized and all elements of the result tensor are multiplied by 255 to obtain the final mask data tensor MSK_DONE, the dimension of which is [1, WIDTH_org, HEIGHT_org]. The edge binary map tensor EDGE_ORG is bitwise ANDed with the mask data tensor MSK_DONE, and then a morphological dilation operation is performed. The remaining object contour map data is obtained according to the contour detection algorithm, and the remaining object contour map data includes the contour map data list LEFT_CONTOUR_LIST.

[0015] By employing the above technical solution, the ROI mask data, road object mask data, and binary edge maps are processed, merged, expanded, and contour detected. Padding is removed, interpolation and scaling are used to unify the data size, dilation operations are used to expand object contours, and bitwise logical operations and merging are used to integrate multi-source data. The resulting residual object contour map provides a clearer representation of possible foreign object outlines and, combined with edge information, improves the accuracy of unclassified foreign object recognition.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the step of determining whether the remaining object is noise data again based on the area features, speed features, and continuity features of the remaining objects in the remaining object contour map data, determining the remaining object that is not noise data as an unclassified road foreign object, and determining the corresponding bounding box data as BBOX_ABNORMAL_2nd specifically includes: calculating the area of each contour in the LEFT_CONTOUR_LIST in the remaining object contour map data, filtering contours with areas greater than a preset upper threshold, and filtering out contours with areas less than a preset lower threshold; tracking the remaining contours frame by frame using a tracking algorithm to obtain the tracking ID and the coordinates [center_x, center_y] of the center point of the lower edge of the contour, where the remaining contour refers to a contour that is less than the preset upper threshold and greater than the preset lower threshold; recording the existence time of the tracking ID and filtering out contours that are less than the set time; converting the coordinates of the center point of the lower edge of the contour into tunnel space coordinates using a coordinate conversion algorithm; calculating the relative moving speed V_tunnel of the coordinates of the center point of the lower edge of the contour in the tunnel coordinate system during a specified time interval, and calculating the actual moving speed of the corresponding contour: , where V_robot is the absolute moving speed of the robot; the contour whose actual moving speed is less than the set speed threshold is retained, and the contour of the teammate is determined to be a real unclassified road foreign object. The bounding box data of the road foreign object is BBOX_ABNORMAL_2nd, and the contour data of the road foreign object is ABNORMAL_CONTOUR.

[0017] By employing this technical solution, the remaining object outline data is judged as noise based on its area, speed, and continuity characteristics, and unclassified road foreign objects and their bounding box data are identified. Area threshold filtering eliminates excessive or small interference, tracking algorithms and duration judgment ensure the object's presence, and speed characteristics filter out stationary or slow-moving foreign objects. This allows for accurate identification of unclassified road foreign objects, avoiding misjudgments and ensuring road safety in tunnels.

[0018] In combination with some embodiments of the first aspect, in some embodiments, a labeling file is automatically generated based on the unclassified road foreign object, and the corresponding image frame is loaded into a training database for model training to obtain a reloaded trained road object instance segmentation model. The steps specifically include: automatically saving the image frame where the unclassified road foreign object appears while reporting information to a preset management terminal; automatically creating a labeling file using contour data ABNORMAL_CONTOUR; re-adjusting the original road object instance segmentation model using the image frame and the labeling file; and inferring the real-time frame by loading the trained road object instance segmentation model.

[0019] By implementing this technical solution, unclassified road object image frames are automatically saved, and annotation files are created using the contour data. The original road object instance segmentation model is then re-adjusted and used for real-time frame inference. Automatically saving and creating annotation files improves efficiency, and the model continuously updates and learns new object features. This allows the model to adapt to the ever-changing tunnel environment and object types, improving the accuracy and adaptability of object detection and ensuring long-term, stable and effective tunnel inspections.

[0020] In a second aspect, the present application provides a server comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to cause the server to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a server, cause the server to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product, which, when executed on a server, enables the server to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By using the existing tunnel robot video data set to train the ROI region instance segmentation model and the road object instance segmentation model, performing edge detection and instance segmentation reasoning on the inspection robot video stream data frame, combining multi-faceted data processing and feature judgment to identify foreign objects, and updating the model based on unclassified foreign objects, the technical means effectively solves the problems of low accuracy in tunnel road foreign object detection, difficulty in discovering new foreign object types, and poor model adaptability in the existing technology, thereby achieving the technical effect of timely and accurate detection of tunnel road foreign objects, reducing safety hazards, ensuring smooth tunnel traffic, and improving the level of intelligent and automated tunnel inspection.

[0024] 2. By using instance segmentation reasoning on video stream data frames to obtain ROI area mask data and road object mask data respectively, and filtering and recording the bounding box data of classified foreign objects through confidence thresholds, the problem of difficulty in accurately identifying classified foreign objects and inability to quickly locate their positions in existing technologies is effectively solved. This allows for accurate identification of classified foreign objects and rapid location of foreign objects, providing a basis for subsequent warning and processing, and improving the efficiency of foreign object detection and response in tunnels.

[0025] 3. The method adopts the technical means of removing filling, interpolating and scaling, dilating, bitwise logical operation and merging the ROI area mask data, road object mask data and edge binary map, and finally obtaining the remaining object contour map data through contour detection. Therefore, it effectively solves the problems of difficulty in integrating multi-source data and low accuracy in identifying unclassified foreign objects in the existing technology, thereby achieving a clearer presentation of possible foreign object contours and improving the accuracy of unclassified foreign object identification by combining edge information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flow chart of a method for detecting foreign objects on a road based on the vision of a tunnel inspection robot in an embodiment of the present application; Figure 2 This is a schematic diagram of the data processing flow of the road foreign object detection method based on the vision of the tunnel inspection robot in an embodiment of the present application; Figure 3 This is a ROI dynamic change diagram of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application; Figure 4 This is an original frame image containing foreign objects in the method for detecting foreign objects on the road based on the vision of the tunnel inspection robot in an embodiment of the present application; Figure 5 This is an edge detection diagram of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application; Figure 6 is an unclassified foreign object image of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application; Figure 7 This is a schematic diagram of the physical device structure of the server in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0029] For ease of understanding, the following describes the process of the method provided by this implementation. Figure 1 , which is a flow chart of a method for detecting foreign objects on the road based on the vision of a tunnel inspection robot in an embodiment of the present application.

[0030] S101. Based on the existing tunnel robot video data set, train a ROI region instance segmentation model and a road object instance segmentation model; The server collects video data captured by the tunnel robot at different time periods, different tunnel locations, and different camera perspectives, extracts frames to create an image training set, and copies the image training set into two parts: one is the ROI image training set, and the other is the road object image set. Both image sets are segmented and labeled for instance. The ROI is the road monitoring foreign object area, that is, the curbs / maintenance road parts on both sides are excluded, and the road part about 100 meters away is excluded. Only the road area within 100 meters is retained for labeling. The road objects mainly include the following categories: lane lines, vehicles, pedestrians, non-motor vehicles, and foreign objects. The foreign objects are foreign object data that have been recorded and confirmed in the historical traffic video, and are defined as classified foreign objects.

[0031] Then, we use mature instance segmentation models in the industry, such as Mask-RCNN / YOLO, to train models on two major data sets, and obtain the trained ROI region segmentation model MODEL_ROI and road object segmentation model MODEL_STUFF.

[0032] The beneficial effect of this step is: unlike fixed-point cameras, for mobile robot scenes, its ROI area often changes with the movement of the robot, as shown in the following figure. Figure 3 As shown, Figure 3 This will be explained in detail later and will not be repeated here. This system uses deep learning for recognition, rather than the traditional fixed ROI approach. This allows for dynamic and accurate ROI recognition, unaffected by the robot's position and camera angle. Furthermore, the ROI and road objects are separated into two models rather than a single model for training and inference because the ROI is a large area covering both the vehicle and lane markings. If combined for training, the ROI in the inference result will be obscured by the vehicle, affecting the final detection result.

[0033] S102, obtaining a video stream data frame sent by the inspection robot and performing edge detection on the video stream data frame to generate a corresponding edge binary image; The tunnel inspection robot travels in the tunnel according to the preset inspection path and speed. It uses its high-definition camera to obtain the video stream data frame FRM_n in the tunnel in real time and converts it into a tensor in the GPU. Its number of channels is 3 (RGB), the width is WIDTH_org, and the height is HEIGHT_org. Figure 4 Example. Figure 4 This will be explained in detail later and will not be repeated here.

[0034] For each acquired video data frame, perform Gaussian filtering to remove noise, and then perform grayscale processing. Then use edge detection algorithms such as Canny / Sobel to obtain the edge binary image EDGE_ORG. Figure 5 Example, Figure 5 This will be explained in detail later and will not be repeated here.

[0035] The beneficial effect of this step is that foreign objects often have a color difference from the road background. Using edge detection can record the outline of foreign objects with a high probability, laying the foundation for the subsequent step of eliminating irrelevant objects.

[0036] S103: Inferring the original data frame acquired by the camera based on the ROI region instance segmentation model and the road object instance segmentation model to obtain an inference result, which includes ROI region mask data and road object mask data. If classified foreign object segmentation data appears in the inference result and the confidence level of the foreign object segmentation data exceeds a preset confidence threshold, the corresponding bounding box data is recorded as BBOX_ABNORMAL_1st; Perform instance segmentation reasoning on the previously acquired data frame FRM_n, and obtain the ROI area mask data tensor MSK_ROI through model MODEL_ROI reasoning. The mask data tensor MSK_ROI is a binary tensor with a value of 0 or 1, a width of WIDTH_msk, a height of HEIGHT_msk, and a dimension of [1, WIDTH_msk, HEIGHT_msk].

[0037] An example of its masked data tensor is as follows:

[0038] 1 indicates that the pixel belongs to the ROI region, and 0 indicates background. The model MODEL_STUFF is used to infer the road object mask data tensor MSK_STUFF and bounding box data BBOX_STUFF. Its dimension is [k, WIDTH_msk, HEIGHT_msk], where k is the number of identified segmentation instances. For example, if two lane lines, five vehicles, and one foreign object are identified, then k = 2 + 5 + 1 = 8. If MSK_STUFF contains a foreign object mask and its confidence is greater than a preset threshold, it indicates that it is a classified road foreign object. Its bounding box data is then extracted from BBOX_STUFF and recorded as BBOX_ABNORMAL_1st.

[0039] S104, processing the edge binary image, the ROI region mask data, and the road object mask data, and merging them to obtain a merged data image, performing a morphological dilation operation on the merged data image, and obtaining remaining object contour image data using a contour detection algorithm; Specifically, the server removes padding from MSK_ROI and MSK_STUFF (the inference model adds padding to the image during inference to meet inference requirements), and then interpolates and scales them to the size of the original frame image, that is, its width becomes WIDTH_org and its height becomes HEIGHT_org.

[0040] The morphological dilation operation is performed on the mask data in MSK_STUFF to expand the contour area of the identified lane lines, cars, people, foreign objects, etc., and cover the redundant noise points to facilitate subsequent processing.

[0041] Perform bitwise logical inversion operation on the mask data in MSK_STUFF, that is, change the original 1 to 0 and the original 0 to 1, and obtain MSK_STUFF_INV.

[0042] Concatenate MSK_STUFF_INV and MSK_ROI along the first dimension to obtain the tensor MSK_CAT, whose dimension is [1+k, WIDTH_org, HEIGHT_org]. Perform torch.prod (dim=0) on MSK_CAT, which multiplies and merges the tensors along the first dimension by positional element values. Then, byte the resulting tensor and multiply all its elements by 255 to obtain the final masked data tensor MSK_DONE, whose dimension is [1, WIDTH_org, HEIGHT_org].

[0043] Then perform bitwise AND operation on the edge map tensor EDGE_ORG and the mask map tensor MSK_DONE, perform morphological expansion operation, and perform contour detection algorithm on it to obtain the contour map data list LEFT_CONTOUR_LIST of the remaining objects, such as Figure 6 The figure shows the unclassified foreign bodies that have been filtered out. Figure 6 This will be explained in detail later and will not be repeated here.

[0044] This method operates directly on the masked tensor, rather than on segmented contour data. This greatly improves computational speed and ensures real-time detection in practical applications. Furthermore, by using a step-by-step peeling method to gradually reduce the data close to the target foreign object, recognition accuracy can be greatly guaranteed.

[0045] S105. Determine again whether the remaining object is noise data based on the area feature, speed feature, and continuity feature of the remaining object in the remaining object outline data. Determine the remaining object that is not noise data as an unclassified road foreign object, and determine the corresponding bounding box data as BBOX_ABNORMAL_2nd. Specifically, the server calculates the area of each contour in the LEFT_CONTOUR_LIST and filters out contours with an area greater than a preset threshold, removing contours with an area too small. The remaining contours are tracked frame by frame using a tracking algorithm to obtain the contour's tracking ID and the coordinates of the center point of the lower edge [center_x, center_y]. Preferably, the BYTE_TRACK algorithm is used for the tracking algorithm. The duration of the tracking ID is then recorded, and contours with a duration less than the threshold are filtered out to select contour data with good continuity.

[0046] The coordinates of the center point of the lower edge of the contour obtained previously are converted into tunnel space coordinates through the coordinate conversion algorithm. The conversion matrix needs to be calibrated in advance. Considering the simplicity of calibration and the acceptable range of distance accuracy, only the conversion matrix at the position of several robots is calibrated. Assume that the conversion matrix at a certain robot position is M_trans, and its value is:

[0047] The conversion method is projection transformation:

[0048] Where Tx, Ty is the position of the point in the tunnel horizontal coordinate system, in meters. Note that the tunnel coordinate system here is a horizontal coordinate system and does not consider the altitude dimension.

[0049] Calculate the relative moving speed V_tunnel of the center of the lower edge of the contour in the tunnel coordinate system during the specified time interval (default 1s), and calculate the actual moving speed of the contour:

[0050] Among them, V_robot is the absolute moving speed of the robot.

[0051] The contours whose moving speed is less than the specified threshold are retained and determined to be real unclassified road foreign objects. Their bounding box data is BBOX_ABNORMAL_2nd and their contour data is ABNORMAL_CONTOUR.

[0052] In actual application scenarios, this can lead to anomalies, such as a frame where the segmentation inference yields no results or a frame where the background is misclassified as the foreground. These situations can cause noise in the target recognition. In actual applications, it has been found that using area, speed, and continuity as thresholds for filtering can minimize recognition noise and improve the final accuracy of foreign object detection.

[0053] S106. Report the BBOX_ABNORMAL_1st and the BBOX_ABNORMAL_2nd to the preset management terminal according to the preset warning rules; Specifically, according to the preset alarm rules, including warning interval setting, warning duration setting, warning level setting, warning priority setting and other rules, the foreign object data BBOX_ABNORMAL_1st and BBOX_ABNORMAL_2nd are reported through the preset communication rules.

[0054] S107 , automatically generating a labeling file based on the unclassified road foreign object, and loading the corresponding image frame into a training database for model training, thereby obtaining a reloaded and trained road object instance segmentation model.

[0055] Specifically, at the same time as the alarm is triggered, the image frames containing previously identified unclassified foreign body data are automatically saved. The contour data ABNORMAL_CONTOUR is then used to automatically create annotation files, and the original instance segmentation model MODEL_STUFF is retrained / fine-tuned using the image frames and annotation files. The server then loads the trained MODEL_STUFF to perform inference on the real-time frames. Implementing this method allows for real-time iterative updates of the model's recognition capabilities by adding newly emerging foreign body data to training. This eliminates the need to rely solely on recognition, and allows for neural network recognition to be used first. This is equivalent to adding a layer of recognition processing mechanism, which can improve the final foreign body detection rate.

[0056] In the embodiments of the present application, due to the use of technical means such as collecting multi-source data to train models, multi-step processing and multi-model reasoning of video stream data frames, comprehensive multi-data fusion analysis, and using unclassified foreign objects to update models, the foreign object detection capability can be improved from multiple dimensions, effectively solving the problems of low accuracy in tunnel road foreign object detection, difficulty in discovering new foreign object types, difficulty in multi-source data integration, and poor model adaptability in the existing technology, thereby achieving timely and accurate detection of tunnel road foreign objects, reducing safety hazards, ensuring smooth tunnel traffic, and improving the level of intelligence and automation of tunnel inspections.

[0057] After combining the above content, the following is a more detailed description of the data flow of the method provided by this implementation. Figure 2 , which is a schematic diagram of the data processing flow of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application.

[0058] exist Figure 2 The entire process from image acquisition to foreign object detection and model update is demonstrated. The specific steps are as follows: The mobile inspection robot's camera acquires image frames and performs preprocessing, including grayscale conversion and edge detection, to obtain a binary edge map. The preprocessed image is processed using an instance segmentation ROI model to obtain ROI mask data. Simultaneously, an instance segmentation road object model is used to obtain an object mask, which is then element-wise negated. The ROI mask data and the negated object mask are concatenated into tensors and their element-wise product is calculated. Morphological dilation, contour detection, and feature denoising are then performed to filter out unclassified foreign objects. Classified foreign objects are recorded, and detected foreign objects are alerted according to the alarm rules. Unclassified foreign objects are generated into a new foreign object training set, and the model is then trained or fine-tuned online before being updated and deployed. This method effectively extracts foreign object-related information from complex images. Through operations such as tensor concatenation and element-wise product, as well as subsequent morphological processing, unclassified foreign objects can be accurately filtered out, achieving accurate foreign object detection in tunnels and improving detection accuracy.

[0059] See below Figure 3 , is a ROI dynamic change diagram of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application: Figure 3 The demonstration shows the dynamic changes of ROI in the road foreign object detection method based on the tunnel inspection robot vision. Figure 3Two visible light images of a tunnel are shown. These are examples of video stream data frames captured by a tunnel inspection robot at different times. The acquisition times are indicated above the two images: 18:50:12, Monday, November 18, 2024, and 18:53:33, Monday, November 18, 2024, respectively. The words "visible light" are labeled in the lower right corner of each image. The images depict a tunnel scene, illuminated by lights at the top and sides. Vehicles are traveling on the road, and the region of interest (ROI) is indicated in the image. This region covers the lane markings and part of the area where the vehicles are located. As can be seen, the position and extent of the ROI change over time as the robot moves, demonstrating the dynamic nature of the ROI in mobile robot scenarios.

[0060] In mobile robot scenarios, unlike fixed-point cameras, their ROIs (Region of Interest) (ROI) change with the robot's movement. Traditional fixed ROI methods have limitations, while this system utilizes deep learning technology to identify ROIs, offering significant advantages. In actual tunnel inspections, robots are constantly moving, and their positions and camera angles are constantly changing. Traditional fixed ROI methods struggle to adapt to these dynamic changes, resulting in blind spots or inaccuracies. This system's deep learning recognition approach is unaffected by these factors and accurately determines the ROI based on the robot's real-time state, ensuring comprehensive and accurate detection. The ROI covers a large area, including vehicles and lane markings. If the ROI and road objects are combined into a single model for training and inference, vehicles and other objects may obstruct the ROI, leading to discrepancies in the ROI recognition results and ultimately affecting foreign object detection. This system separates these two models: the ROI instance segmentation model focuses on identifying the ROI, while the road object instance segmentation model detects various road objects. This avoids mutual interference and improves detection accuracy.

[0061] See below Figure 4 , is an original frame image containing foreign objects in the method for detecting foreign objects on the road based on the vision of the tunnel inspection robot in an embodiment of the present application: Figure 4This is a visible light image of a tunnel, showing an example of a video stream data frame acquired in real time by a high-definition camera when the tunnel inspection robot is traveling according to a preset inspection path and speed. A suspected foreign object (white box) placed on the tunnel road is shown in the lower left corner of the image, and there are other vehicles traveling on the road. The top and both sides of the tunnel are illuminated, creating a lighting environment inside the tunnel. The time when the data frame was acquired is marked above the image as 18:49:01, Monday, November 18, 2024, and the word "visible light" is marked in the lower right corner, indicating that the image was acquired through a visible light camera. The acquired video stream data frame FRM_n can be converted into a tensor in the GPU, with 3 channels (RGB), a width of WIDTH_org, and a height of HEIGHT_org. The video stream data frame is acquired in real time through a high-definition camera, providing raw image data for subsequent foreign object detection and other analyses. See below Figure 5 , which is an edge detection diagram of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application: Figure 5 This example shows the results of a series of image processing operations on video data frames captured by a tunnel inspection robot. The raw video data frames are first subjected to Gaussian filtering to remove noise and reduce interference in the image. Grayscale processing then converts the color image into a grayscale image, simplifying the image's color information for easier processing. An edge detection algorithm (such as the Canny or Sobel algorithm) is then applied to the grayscale image to highlight edge information, resulting in a binary edge map (EDGE_ORG). The image shows the tunnel's interior contours, road lines, suspected foreign objects (white boxes), and the edges of other objects within the tunnel, displayed as lines against a black background. The time "2021-11-18 Monday 18:49:01" is annotated above the image, and the word "visible light" is annotated in the lower right corner. This reduces random interference in the image, making it smoother and clearer.

[0062] See below Figure 6 , Figure 6 This is an unclassified foreign object diagram of the road foreign object detection method based on the tunnel inspection robot vision in an embodiment of the present application: Figure 6 The image shows an unclassified foreign object image filtered out after a series of image processing operations. It is the result of a bitwise AND operation on the edge map tensor EDGE_ORG and the mask map tensor MSK_DONE, followed by a morphological dilation operation and the application of a contour detection algorithm. The image shows only the outline of a suspected foreign object (a white box shape) against a black background. This outline belongs to the content of the contour map data list LEFT_CONTOUR_LIST of the remaining objects and represents the detected unclassified foreign object.

[0063] The following describes the server in the embodiment of the present invention from the perspective of hardware processing. Figure 7 , is a schematic diagram of a physical device structure of a server in an embodiment of the present application.

[0064] It should be noted that Figure 7 The structure of the server shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0065] like Figure 7 As shown, the server includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes, such as the methods described in the above embodiments, based on programs stored in a read-only memory (ROM) 702 or programs loaded from a storage unit 708 into a random access memory (RAM) 703. RAM 703 also stores various programs and data required for system operation. CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0066] The following components are connected to the I / O interface 705: an input section 706 including an audio input device, push button switches, and the like; an output section 707 including a liquid crystal display (LCD), an audio output device, indicator lights, and the like; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. Removable media 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read from the removable media can be installed in the storage section 708 as needed.

[0067] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 709 and / or installed from removable media 711. When executed by the central processing unit (CPU) 701, the computer program performs the various functions defined in the present invention.

[0068] It should be noted that specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0070] Specifically, the server of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the road foreign object detection method based on the tunnel inspection robot vision provided in the above embodiment is implemented.

[0071] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the server described in the above embodiments, or may exist independently and not incorporated into the server. The storage medium carries one or more computer programs. When executed by a processor of the server, the server implements the road foreign object detection method based on tunnel inspection robot vision as described in the above embodiments.

[0072] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0073] As used in the above embodiments, the term “when” may be interpreted to mean “if” or “after” or “in response to determining that” or “in response to detecting that”, depending on the context. Similarly, the phrases “upon determining that” or “if (stated condition or event) is detected” may be interpreted to mean “if determining that” or “in response to determining that” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0074] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A road foreign object detection method based on tunnel inspection robot vision, applied to a server, characterized in that: The method comprises: Based on the existing tunnel robot video data set, the ROI region instance segmentation model and the road object instance segmentation model are trained; Obtaining the video stream data frame sent by the inspection robot and performing edge detection on the video stream data frame to generate a corresponding edge binary image; Inference is performed on the raw data frames acquired by the camera based on the ROI region instance segmentation model and the road object instance segmentation model to obtain an inference result, which includes ROI region mask data and road object mask data. If classified foreign object segmentation data appears in the inference result and the confidence level of the foreign object segmentation data exceeds a preset confidence threshold, the corresponding bounding box data is recorded as BBOX_ABNORMAL_1st; The edge binary image, the ROI region mask data, and the road object mask data are processed and merged to obtain a merged data image, a morphological dilation operation is performed on the merged data image, and remaining object contour image data is obtained by a contour detection algorithm; Determining whether the remaining object is noise data again based on the area feature, speed feature, and continuity feature of the remaining object in the remaining object outline data, determining the remaining object that is not noise data as an unclassified road foreign object, and determining the corresponding bounding box data as BBOX_ABNORMAL_2nd; Report the BBOX_ABNORMAL_1st and the BBOX_ABNORMAL_2nd to the preset management terminal according to the preset warning rules; A labeling file is automatically generated according to the unclassified road foreign objects, and the corresponding image frames are loaded into a training database for model training to obtain a reloaded trained road object instance segmentation model.

2. The method according to claim 1, characterized in that The steps of training the ROI region instance segmentation model and the road object instance segmentation model based on the existing tunnel robot video data set include: Collect video data collected by the tunnel robot at different time periods, different tunnel locations, and different camera angles, and extract frames to create an image training set; After copying the image training set, a ROI image training set and a road object image set are obtained, and instance segmentation and annotation are performed on each of them. The ROI image training set is the road monitoring foreign object area, that is, the curbs or maintenance roads on both sides are excluded, and the road surface beyond a set distance is excluded, and only the road area within the set distance is retained for annotation. The road object image set includes at least lane lines, vehicles, pedestrians, non-motor vehicles and foreign objects. If the foreign object has confirmed foreign object data recorded in the traffic history video, it is considered a classified foreign object; A preset instance segmentation model is used to perform model training on the ROI image training set and the road object image set to obtain a ROI region instance segmentation model and a road object instance segmentation model.

3. The method according to claim 1, characterized in that The steps of obtaining the video stream data frame sent by the inspection robot and performing edge detection on the video stream data frame to generate a corresponding edge binary map specifically include: After obtaining the video stream data frame in the tunnel in real time through the high-definition camera carried by the inspection robot in the tunnel, the video stream data frame is converted into a tensor in the GPU. The number of channels of the tensor is 3, the width is WIDTH_org, and the height is HEIGHT_org; Performing Gaussian filtering and denoising on each of the acquired video stream data frames, and then performing grayscale processing to obtain a grayscale image; An edge detection algorithm is used on the grayscale image to obtain an edge binary image.

4. The method according to claim 1, wherein The raw data frames acquired by the camera are inferred based on the ROI region instance segmentation model and the road object instance segmentation model to obtain an inference result, wherein the inference result includes ROI region mask data and road object mask data. If classified foreign object segmentation data appears in the inference result and the confidence level of the foreign object segmentation data exceeds a preset confidence threshold, the corresponding bounding box data is recorded as BBOX_ABNORMAL_1st, specifically including the following steps: Perform instance segmentation reasoning on the video stream data frame, and obtain the tensor MSK_ROI of ROI region mask data through the ROI region instance segmentation model reasoning, wherein the tensor MSK_ROI is a binary tensor with a value of 0 or 1, and the corresponding width is WIDTH_msk, the height is HEIGHT_msk, and the dimension is [1, WIDTH_msk, HEIGHT_msk]; The road object instance segmentation model is used to infer the mask data tensor MSK_STUFF and the bounding box data BBOX_STUFF of the road object. The dimension of the tensor MSK_STUFF is [k, WIDTH_msk, HEIGHT_msk], where k is the number of identified segmentation instances. If the tensor MSK_STUFF contains a foreign object mask and its confidence is greater than the preset confidence threshold, it means that the object represented by the foreign object mask is a classified road foreign object. In this case, the bounding box data of the road foreign object is extracted from the bounding box data BBOX_STUFF and recorded as BBOX_ABNORMAL_1st.

5. The method according to claim 1, wherein The step of processing and merging the edge binary image, the ROI region mask data, and the road object mask data to obtain a merged data image, performing a morphological dilation operation on the merged data image, and obtaining the remaining object contour image data by a contour detection algorithm specifically includes: Performing a fill removal operation on the ROI region mask data and the road object mask data, and then interpolating and scaling them to the size of the original frame image, where the width of the original frame image is WIDTH_org and the height is HEIGHT_org; Performing a morphological dilation operation on the mask data in the tensor MSK_STUFF, wherein the morphological dilation operation is used to expand the contour area of the identified data including at least lane lines, vehicles, people, and foreign objects; Performing a bitwise logical inversion operation on the mask data in the tensor MSK_STUFF to obtain MSK_STUFF_INV; Concatenate the MSK_STUFF_INV and the tensor MSK_ROI in the first dimension to obtain a tensor MSK_CAT, where the dimension of the tensor MSK_CAT is [1+k, WIDTH_org, HEIGHT_org]. Multiply and merge the tensor MSK_CAT by positional element values in the direction of the first dimension to obtain a result tensor, then byte-ize the result tensor and multiply all elements of the result tensor by 255 to obtain a final mask data tensor MSK_DONE. The dimension of the mask data tensor MSK_DONE is [1, WIDTH_org, HEIGHT_org]. The edge binary map tensor EDGE_ORG and the mask data tensor MSK_DONE are bitwise ANDed, and then a morphological dilation operation is performed. The remaining object contour map data is obtained according to the contour detection algorithm. The remaining object contour map data includes a contour map data list LEFT_CONTOUR_LIST.

6. The method according to claim 1, characterized in that The step of determining whether the remaining objects are noise data again based on the area features, speed features, and continuity features of the remaining objects in the remaining object outline data, determining the remaining objects that are not noise data as unclassified road foreign objects, and determining the corresponding bounding box data as BBOX_ABNORMAL_2nd specifically includes: Calculate the area of each contour in LEFT_CONTOUR_LIST in the remaining object contour map data, filter out contours with an area greater than a preset upper threshold, and filter out contours with an area less than a preset lower threshold; Track the remaining contours frame by frame using a tracking algorithm to obtain the tracking ID of the remaining contours and the coordinates of the center point of the lower edge of the contour [center_x, center_y]. The remaining contours refer to the contours that are smaller than the upper limit of the preset threshold and larger than the lower limit of the preset threshold. Record the existence time of the tracking ID and filter out the contours whose existence time is less than the set time; Converting the coordinates of the center point of the lower edge of the contour into tunnel space coordinates through a coordinate conversion algorithm; Calculate the relative moving speed V_tunnel of the center point coordinates of the lower edge of the contour in the tunnel coordinate system during the specified time interval, and calculate the actual moving speed of the corresponding contour: , where V_robot is the absolute moving speed of the robot; The outline of the teammate whose actual moving speed is less than the set speed threshold is retained, and the outline of the teammate is determined to be a real unclassified road foreign object. The bounding box data of the road foreign object is BBOX_ABNORMAL_2nd, and the contour data of the road foreign object is ABNORMAL_CONTOUR.

7. The method according to claim 1, characterized in that The steps of automatically generating a labeling file based on the unclassified road foreign objects, loading the corresponding image frames into a training database for model training, and obtaining a reloaded trained road object instance segmentation model specifically include: While reporting the information to the preset management terminal, automatically saving the image frame where the unclassified road foreign object appears; Use the contour data ABNORMAL_CONTOUR to automatically create annotation files; Re-adjusting the original road object instance segmentation model using the image frame and the annotation file; The real-time frame is inferred by loading the trained road object instance segmentation model.

8. A server, characterized in that: The server includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the server to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a server, the server is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a server, the server is caused to perform the method according to any one of claims 1 to 7.

Images