Method and system for identifying construction procedures in tunnel based on visual information

Abstract

The invention discloses a method and a system for identifying construction procedures in a tunnel based on visual information, and relates to the technical field of tunnel construction, and the method comprises the following steps: obtaining a real-time monitoring video in a tunnel construction environment; extracting feature points and descriptors of the feature points in each frame of the video by using an SIFT feature detection algorithm, matching the feature points in adjacent frames by using a feature matching algorithm, determining matching points in each frame, and judging and identifying key frames in the monitoring video according to the number of the matching points of every two adjacent frames and the change of displacement; and inputting the key frame into a target detection model, extracting multi-scale features through a plurality of multi-scale attention modules sequentially arranged in a backbone network, performing up-down sampling and splicing fusion on the multi-scale features through a reinforced feature fusion network, and finally extracting and outputting target information through a network detection head. And identifying the current tunnel construction process based on the target information. According to the method, different tunnel construction procedures can be efficiently and accurately identified, and smooth tunnel construction is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of tunnel construction, and in particular to a method and system for identifying construction procedures in a tunnel based on visual information. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] During tunnel construction, the identification and management of different construction processes are crucial to ensuring construction progress, improving safety, and ensuring construction quality. Traditional tunnel construction management mainly relies on manual monitoring, paper records, or simple electronic monitoring methods. It is often difficult to accurately identify specific processes in the face of the complex environment in the tunnel, and there are problems such as low efficiency and prone to errors. In addition, factors such as insufficient ambient light, small space, and diverse equipment in the tunnel further increase the difficulty of monitoring the construction process, and traditional methods are difficult to effectively track the progress of the process.

[0004] In recent years, visual recognition technology based on image processing and deep learning has been gradually applied to the construction field, which provides new possibilities for the automation and intelligent management of the tunnel construction process. However, due to the unique complex environment characteristics in the tunnel, this poses new challenges to the vision-based recognition method or system. Existing visual recognition technologies such as Faster R-CNN and YOLO series still have many problems when facing complex environments such as large changes in lighting in tunnels, complex backgrounds, and difficulties in multi-scale feature extraction: Faster R-CNN has poor adaptability to lighting, and fixed convolution kernels are difficult to capture target features under different lighting; the YOLO algorithm's multi-scale feature extraction is insufficient, and the lack of background suppression mechanism is prone to misjudgment. In other words, when facing the complex environment in the tunnel, the existing image recognition model is difficult to fully extract and learn effective features, and it is difficult to balance the complexity and real-time performance of the calculation, resulting in poor recognition accuracy and efficiency, and it is difficult to accurately identify the tunnel construction process. Summary of the invention

[0005] In order to address the deficiencies of the above-mentioned prior art, the present invention provides a method and system for identifying construction procedures in tunnels based on visual information. By processing the video stream in the complex environment of the tunnel, key frames in the video stream are extracted, and a deep learning algorithm is introduced to build a target detection model based on the C2RG network and the SFF network. The target detection model is used to perform accurate target recognition on the key frames, so as to automatically identify and classify different tunnel construction procedures, thereby effectively improving the construction management efficiency and ensuring the smooth progress of tunnel construction.

[0006] In a first aspect, the present invention provides a method for identifying construction procedures in a tunnel based on visual information.

[0007] A method for identifying construction procedures in a tunnel based on visual information, comprising:

[0008] Obtain real-time monitoring video in tunnel construction environment;

[0009] The SIFT feature detection algorithm is used to extract feature points and their descriptors in each frame of the surveillance video. Based on the feature point descriptors, the feature matching algorithm is used to match the feature points in adjacent frames to determine and locate the matching points in each frame. The key frames in the surveillance video are identified based on the number of matching points and the change in displacement between two adjacent frames.

[0010] The key frames are input into the target detection model, and multi-scale features are extracted through multiple multi-scale attention modules set in sequence in the backbone network. The multi-scale features are up-sampled and down-sampled and spliced ​​through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

[0011] In a second aspect, the present invention provides a system for identifying construction procedures in a tunnel based on visual information.

[0012] A tunnel construction process recognition system based on visual information, comprising:

[0013] A monitoring video acquisition module is used to obtain real-time monitoring videos in a tunnel construction environment;

[0014] The key frame extraction module is used to extract the feature points and their descriptors in each frame of the surveillance video using the SIFT feature detection algorithm, match the feature points in adjacent frames using the feature matching algorithm based on the feature point descriptors, determine and locate the matching points in each frame; and identify the key frames in the surveillance video based on the change in the number and displacement of matching points between two adjacent frames;

[0015] The target detection and process identification module is used to input the key frames into the target detection model, and extract multi-scale features through multiple multi-scale attention modules set in sequence in the backbone network. The multi-scale features are up-sampled and down-sampled and spliced ​​and fused through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

[0016] In a third aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method as described in the first aspect is implemented.

[0017] In a fourth aspect, the present invention further provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the steps of the method described in the first aspect are completed.

[0018] In a fifth aspect, the present invention further provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the method described in the first aspect.

[0019] One or more of the above technical solutions have the following beneficial effects:

[0020] 1. The present invention provides a method and system for identifying construction procedures in a tunnel based on visual information, builds a target detection model based on a C2RG network and an SFF network, and uses a multi-layer convolutional network structure in the model to process the collected on-site video frame image data, effectively extract and analyze the feature information in the construction scene, automatically identify and accurately classify the targets in the tunnel construction environment (i.e., tunnel construction procedures), thereby achieving accurate discrimination of various construction procedures or behaviors, effectively improving the construction management efficiency, and ensuring the smooth progress of tunnel construction; the present invention can adapt to the environment with complex lighting conditions, strong background interference, and high spatial limitations in the tunnel, supports long-term monitoring and multi-scale feature extraction, and by feeding back the process recognition results to the construction management system in real time, it can optimize the construction process, reduce manual intervention, and significantly improve the efficiency and safety of tunnel construction.

[0021] 2. Considering that the gap between frames of long-term events in video surveillance is small, and videos usually contain a large number of frames, and the content between many frames does not change much, the present invention proposes a video key frame extraction method with feature point changes, which can realize key frame recognition in dynamic backgrounds and complex environments, and is more suitable for complex environments of tunnel construction. Moreover, by extracting key frames and then analyzing and processing them, the amount of data can be significantly reduced, which is convenient for storage and transmission, and at the same time improves data processing efficiency.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a flow chart of a method for identifying construction procedures in a tunnel based on visual information according to an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of the structure of the target detection model in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that the following detailed descriptions are exemplary only, are intended to describe specific embodiments, are intended to provide further explanation of the present invention, and are not intended to limit exemplary embodiments according to the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] Embodiment 1

[0028] Considering that tunnel construction requires simultaneous analysis of spatial and temporal information events, such as rapid movement or wandering behavior, etc., analysis only at the image level cannot provide a comprehensive response, and the temporal and contextual information of the video has not been fully investigated and utilized. To this end, this embodiment detects events by analyzing the spatiotemporal information in the video, including identifying the movement pattern of objects and the changes in the relationship between objects and scenes. Therefore, this embodiment provides a method for identifying construction processes in tunnels based on visual information. For real-time monitoring videos in complex environments of tunnel construction, a deep learning algorithm is used for analysis and processing, key frames in the video stream are extracted, and accurate target recognition is performed on the key frames, so as to automatically identify and classify different tunnel construction processes. This method is as follows Figure 1 As shown, the specific steps include:

[0029] Step S1, obtaining real-time monitoring video in a tunnel construction environment.

[0030] Step S2: Use the SIFT feature detection algorithm to extract feature points and their descriptors in each frame of the surveillance video. According to the feature point descriptors, use the feature matching algorithm to match the feature points in adjacent frames, determine and locate the matching points in each frame; judge according to the change in the number and displacement of matching points between two adjacent frames, and identify the key frames in the surveillance video.

[0031] Specifically, considering that the gap between frames of long-term events in surveillance videos is small, and videos usually contain a large number of frames, the content between many frames does not change much, therefore, key frame extraction can significantly reduce the amount of data, facilitate storage and transmission, and also improve data processing efficiency. Therefore, this embodiment proposes a video key frame extraction method based on feature point changes, which is suitable for key frame recognition in dynamic backgrounds and complex environments. Key frames can be extracted from surveillance videos to reduce the amount of data for analysis and processing while ensuring detection accuracy.

[0032] Step S2.1: Use the SIFT feature detection algorithm to extract feature points and their descriptors in each frame of the surveillance video. The specific steps are as follows:

[0033] Step S2.1.1, scale space construction. In order to achieve scale invariance, SIFT first performs multi-scale processing on the image through a Gaussian blur filter to gradually reduce the resolution of the image. Specifically, a scale space pyramid is constructed, and a Gaussian blur filter is used to calculate the Gaussian difference (DoG) image at each scale layer of the scale space pyramid to locate potential feature points.

[0034] Step S2.1.2: Key point detection: By comparing each pixel with its surrounding pixels at the current scale and adjacent scales, the local maximum or minimum value is determined, thereby detecting and determining potential key points (i.e., feature points) in the DoG image.

[0035] Step S2.1.3: Accurately locate key points. Use the Taylor expansion formula to optimize the key point positions at the sub-pixel level, remove points with low contrast or strong edge response, and thus improve the stability and accuracy of key points (i.e., feature points).

[0036] Specifically, the Taylor expansion formula is a formula that uses the information of a function at a certain point to describe the value of its vicinity. The Taylor expansion formula is used to optimize the key point position at the sub-pixel level, including:

[0037] First, let the initial estimate of the key point position be (x0, y0), and let the objective function about the key point position be F(x, y), which is a function related to image features (such as gradient), and its value reaches an extreme value at the key point position, such as a maximum value or a minimum value;

[0038] Secondly, Taylor expansion is performed on the objective function F(x,y) at the initial estimated value (x0,y0). In order to obtain the extreme value of the objective function, the first-order partial derivative of the Taylor expansion formula is set to 0 (the partial derivative can be calculated by the pixel value of the image). By solving the system of equations, the sub-pixel level estimate of the optimized key point position can be obtained, thereby completing the sub-pixel level optimization of the key point position.

[0039] Step S2.1.4, direction assignment: According to the gradient direction histogram of the key point neighborhood, one or more main directions are assigned to each key point, so that the key point can remain unchanged under rotation transformation.

[0040] Step S2.1.5, generate descriptors. Select a neighborhood area around the key point, calculate its gradient and direction information, and generate a 128-dimensional feature descriptor, which is highly robust to illumination changes, rotations, and small-scale affine transformations.

[0041] Specifically, after selecting a neighborhood area around the key point, the gradient of each pixel is calculated using discrete difference or Sobel filter, and then the gradient direction can be calculated. The neighborhood area is then divided into several sub-areas, and the gradient direction histogram of each sub-area is statistically calculated (for example, divided into 8 intervals) and the gradient amplitude is normalized. Finally, the normalized histogram vectors of all sub-areas are concatenated to generate a 128-dimensional feature descriptor that is highly robust to illumination changes, rotations, and small-scale affine transformations.

[0042] The feature points and descriptors generated in the above manner can be directly used to detect significant changes between frames, and further assist in determining the position of key frames. The scale invariance and rotation invariance of SIFT enable this embodiment to achieve accurate key frame extraction under dynamic backgrounds and complex lighting conditions.

[0043] Step S2.2: According to the feature point descriptors, feature points in adjacent frames are matched using a feature matching algorithm to determine and locate matching points in each frame.

[0044] After the feature point detection of the above video frames is completed, a feature point matching algorithm (such as nearest neighbor matching KNN or BFMatcher) can be used to perform matching analysis on the feature points of adjacent frames to identify significant changes between frames.

[0045] In this embodiment, first, feature point matching is performed. Specifically, the nearest neighbor matching method is used to calculate the Euclidean distance between feature points of adjacent frames based on the feature point descriptors, and the nearest neighbor search is performed based on the distance, that is, the feature descriptor of the previous frame and the feature descriptor of the next frame are searched for the closest matching point, so as to determine the feature point matching relationship between the frames, and match the feature points in the adjacent frames one by one.

[0046] Secondly, perform no-match filtering. In order to improve the accuracy of matching, cross-validation or ratio testing is usually performed, that is, for each feature point, the two closest feature points in adjacent frames are determined by matching, and the distances between the current feature point and the two closest feature points are compared. According to the comparison results, it is determined whether to discard the matching point pair. Among them, if the distance of the best match is significantly smaller than the distance of the second closest match (for example, 0.75 times), the matching point pair is retained, otherwise the matching point pair is discarded.

[0047] Preferably, the random sample consensus (RANSAC) algorithm can be used to further filter outliers and retain only true matching points. Specifically, the RANSAC algorithm estimates the parameters of a mathematical model by repeatedly randomly extracting samples from a data set, and then uses the model to evaluate whether other data points conform to the model, thereby distinguishing between inliers (i.e., data points that conform to the model) and outliers (i.e., outliers). The RANSAC algorithm is used to filter outliers, specifically:

[0048] (1) Random sampling: randomly select a minimum sample set from the set of matched feature point pairs to estimate a geometric transformation model (such as a homography matrix or a basic matrix, etc.);

[0049] (2) Model estimation: Estimate the geometric transformation model parameters based on the selected minimum sample set;

[0050] (3) Inlier calculation: For the remaining matching feature point pairs (i.e., except for the minimum sample set used to estimate the model), the coordinates of the previous frame are transformed through the estimated model, and then the Euclidean distance between the transformed coordinates and the actual coordinates of the next frame is calculated. If the distance of a point pair is less than a preset threshold, the point pair is considered to be an inlier;

[0051] (4) Repeat the above steps of random sampling, model estimation and inlier calculation. After a set number of iterations, the model parameters corresponding to the largest number of inliers are selected as the final model based on the number of inliers recorded each time and the corresponding model parameters. The inliers corresponding to this model are the valid matching points after filtering by the RANSAC algorithm, and the outliers (i.e., outliers) are filtered out.

[0052] Finally, after feature matching and matching filtering, the matching points in each frame are determined and located.

[0053] Step S2.3: judging based on the change in the number of matching points and displacement between two adjacent frames, and identifying the key frames in the surveillance video.

[0054] Specifically, after matching the feature points of adjacent frames through the feature matching algorithm, the number of matching points and the displacement distribution and other change characteristics are analyzed. If the number of matching points between frames is significantly reduced or the displacement amplitude exceeds the set threshold, the frame is marked as a key frame, including:

[0055] (1) After filtering out false matches, count the number of matching points in each frame of two adjacent frames. The number of matching points represents the degree of feature similarity between frames. If the difference in the number of matching points between two frames exceeds the set threshold, that is, when the number of matching points decreases significantly, it indicates that the content between the two frames has changed significantly, and a new scene or significant action may have appeared. At this time, the next frame is marked as a key frame.

[0056] (2) Based on the descriptors of the matching points, the displacement vectors between all pairs of matching points in two adjacent frames are calculated, that is, the position difference of the feature points in the two frames (including the displacement size and direction). Based on the displacement vectors of all matching points, the average value and standard deviation of the displacement vectors are calculated. If the average value and standard deviation exceed the set threshold range, it is judged that the current content has changed significantly, and the next frame is regarded as the key frame.

[0057] Furthermore, corresponding thresholds such as the number of matching points, the average value and standard deviation of the displacement vector are set through experiments. When the number of matching points is lower than the set threshold, or the average value and standard deviation of the displacement exceed the set range, the latter frame of the two frames is determined to be the key frame. This ensures that the key frame is extracted in time when the content changes significantly, while the frames that do not change significantly are ignored.

[0058] Through the above-mentioned judgment method based on the number of matching points and displacement changes, changes such as scene switching and rapid motion in the video can be effectively identified, while avoiding redundant storage of non-key frames, providing support for efficient analysis and processing of video content.

[0059] Step S3, input the key frame into the target detection model (TOLO-TP model), extract multi-scale features through multiple multi-scale attention modules set in sequence in the backbone network, and perform up-down sampling and splicing fusion of the multi-scale features through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

[0060] In this embodiment, in order to more accurately detect the target in the video frame, a multi-scale attention module, namely the C2RG module, is designed. The module can flexibly capture the deformation and posture changes of the target, and quickly obtain multi-scale information to adapt to targets of different scales. On this basis, this embodiment proposes an improved backbone network C2RGNet. To address the problem of large changes in multi-scale targets, multi-level feature extraction and fusion are performed through multiple C2RG modules in sequence, and global context information is adaptively combined to enhance the model's perception of global information. Finally, by adjusting the fusion strategy in the feature fusion network, the diversity of feature fusion is further enriched to ensure the accuracy of the final target detection.

[0061] like Figure 2As shown in the figure, the constructed target detection model includes a backbone network, an enhanced feature fusion network and a network detection head; the backbone network adopts a multi-scale attention network, namely C2RGNet, which includes multiple C2RG modules connected in sequence, for continuously using multi-scale information to capture attention and fuse, and extract multi-scale features; the enhanced feature fusion network adopts SFFNet, which includes multiple UCC modules and 3C downsampling feature fusion modules, for up- and down-sampling and splicing features of multiple scales, and extracting feature maps of multiple scales to adapt to targets of different sizes; the network detection head is used to extract target information from feature maps of multiple scales and output it.

[0062] (1) Backbone Network - C2RGNet

[0063] Due to the complex environment and diverse processes in the tunnel, the process identification technology has the problem of difficulty in feature extraction. For this reason, this embodiment designs C2RGNet as the backbone network. The core of this network is to deal with the problem of large changes in multi-scale targets by continuously using multi-scale information to capture attention (C2RG), while maintaining computational efficiency and being able to adapt to inputs of different scales, ultimately achieving efficient feature extraction of the model.

[0064] C2RGNet is used as the backbone network, and its input image size is 3×W×H. The key frame is input to the backbone network, and preliminary feature extraction is performed through the initial convolution layer, and then it passes through multiple C2RG modules in sequence for multi-level feature extraction and fusion. Each C2RG module includes 1x1 convolution and 3x3 depth convolution, which are used to refine the feature representation, and integrate features through upsampling and splicing operations at different scales, so that multi-scale features can be comprehensively utilized to improve the richness of feature representation; the fused feature map is passed to the SPPF module, and finally the SPPF module performs spatial pyramid pooling at different spatial scales to generate rich and robust feature representation, thereby significantly improving the feature focusing ability and classification performance of the model in complex scenes.

[0065] Regarding the C2RG module, considering that it is difficult to effectively focus on the key feature area in the process detection task, and the target information changes on a large scale, resulting in insufficient performance of the model in complex scenes, this embodiment designs the multi-scale attention model (i.e., the C2RG module) to enhance the feature focusing capability. The C2RG module can effectively capture multi-scale semantic information and reduce the redundancy of the self-attention mechanism.

[0066] Specifically, the key frame is input to the backbone network, and preliminary feature extraction is performed through the 1x1 convolution layer. The extracted preliminary features are used as input features and input to the C2RG module. In the C2RG module, the input features are input to multiple parallel paths, and in each path, they are respectively passed through the 1x1 convolution, BatchNorm layer and SiLU activation function to extract refined features. The BatchNorm layer and SiLU activation function can effectively prevent the gradient from disappearing during the model training process. The refined features in the first path are then bottlenecked by the Bottleneck module to extract fine-grained features. The fine-grained features are concatenated and fused with the refined features extracted by the second path. The generated features are further refined through 1x1 convolution, BatchNorm layer and SiLU activation function to further optimize the feature representation. The features extracted again are concatenated and fused with the refined features extracted by the third path. The concatenated and fused features are input into two parallel paths and extracted through multiple deep convolutions (DWConv) and ghost convolutions (GhostConv). The features extracted by the two paths are concatenated and output, which can enhance the richness and diversity of features while retaining computational efficiency.

[0067] Through the design of the above-mentioned C2RG module, efficient feature extraction and fusion can be achieved, which can flexibly capture the deformation and posture changes of the target, quickly obtain multi-scale information, adapt to the changes of targets at different scales, and significantly enhance the feature focusing ability and classification performance of the model.

[0068] Furthermore, the feature map output by the last layer of the C2RG module is input into the SPPF module, which performs spatial pyramid pooling at different spatial scales to generate rich and robust feature representations, and then the pooled features are input into SFFNet.

[0069] (2) Strengthening the feature fusion network-SFFNet

[0070] The SFF network includes multiple UCC modules and a 3C downsampling feature fusion module. The UCC module includes an upsampling layer, a splicing layer and a C2RG module, which are used to fuse feature maps of different scales through upsampling and splicing operations, and the fused feature map is refined by the C2RG module. The UCC module fuses features through upsampling and splicing operations, which can enhance the model's detection ability for targets with large deformation and posture changes; the 3C downsampling feature fusion module includes a downsampling convolution layer, a splicing layer and a C2RG module, which are used to fuse feature maps of different scales output by the UCC module through downsampling and splicing operations, and the fused feature map is refined by the C2RG module, and outputs feature maps of multiple different sizes to adapt to targets of different sizes. In this embodiment, the feature map is divided into three scales according to its size: S, M, and L, so as to adapt to targets of different sizes.

[0071] Through the above structural design, the constructed YOLO-TP network has high robustness and adaptability in feature fusion and target detection. It can effectively handle complex tunnel construction scenes and diverse targets, and effectively improve the accuracy and practicality of target detection.

[0072] (3) Network detection head

[0073] The network detection head is used to extract target information from the feature map and output the detection results. The detection head mainly includes the following modules:

[0074] (3.1) Anchor box generation module, which is used to generate a set of predefined target anchor boxes based on the input feature map for target detection at different scales. The size and proportion of each anchor box are set according to the target characteristics of the training data to achieve compatibility with multiple target sizes.

[0075] (3.2) Position regression and classification prediction module, which is used to predict the category probability, bounding box position offset and confidence score of each target anchor box through the convolution layer, and generate the final detection information. Among them, this module specifically includes a position regression module and a classification prediction module. The regression module outputs the center coordinates and width and height offset values ​​of the bounding box, and the classification prediction module gives the prediction score of the target category. These generated values ​​are processed by the activation function to generate the final detection output.

[0076] (3.3) Multi-scale detection module, which is used to output detection information of multiple scales based on the input feature maps of different scales, and can effectively improve the detection performance of small targets.

[0077] Through the design of the above network detection head, through modules such as target anchor box, regression and classification, the network can quickly and accurately detect targets of different scales.

[0078] In fact, the target of the final detection output is the construction process to be identified. Specifically, in the training process of the target detection model constructed above, the monitoring video of the tunnel construction environment with the construction process marked is first obtained, and then the marked video key frames are extracted to construct a training data set; then the training data set is used to train the model, and after continuous iteration, a trained target detection model is obtained; finally, the trained target detection model is used to perform target detection, and the detected target information is output, and the current tunnel construction process can be identified and determined based on the target information.

[0079] Embodiment 2

[0080] This embodiment provides a system for identifying construction procedures in a tunnel based on visual information, which specifically includes the following modules:

[0081] A monitoring video acquisition module is used to obtain real-time monitoring videos in a tunnel construction environment;

[0082] The key frame extraction module is used to extract the feature points and their descriptors in each frame of the surveillance video using the SIFT feature detection algorithm, match the feature points in adjacent frames using the feature matching algorithm based on the feature point descriptors, determine and locate the matching points in each frame; and identify the key frames in the surveillance video based on the change in the number and displacement of matching points between two adjacent frames;

[0083] The target detection and process identification module is used to input the key frames into the target detection model, and extract multi-scale features through multiple multi-scale attention modules set in sequence in the backbone network. The multi-scale features are up-sampled and down-sampled and spliced ​​and fused through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

[0084] Embodiment 3

[0085] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps in the method for identifying construction procedures in a tunnel based on visual information as described above are completed.

[0086] Embodiment 4

[0087] This embodiment also provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the method for identifying construction procedures in a tunnel based on visual information as described above are completed.

[0088] Embodiment 5

[0089] This embodiment provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0090] The steps involved in the above embodiments 2 to 5 correspond to the method embodiment 1. For the specific implementation, please refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0091] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0092] The above description is only a preferred embodiment of the present invention. Although the specific implementation mode of the present invention is described in conjunction with the accompanying drawings, it is not a limitation of the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for identifying construction procedures in a tunnel based on visual information, characterized in that: include: Obtain real-time monitoring video in tunnel construction environment; The SIFT feature detection algorithm is used to extract feature points and their descriptors in each frame of the surveillance video. Based on the feature point descriptors, the feature matching algorithm is used to match the feature points in adjacent frames to determine and locate the matching points in each frame. The key frames in the surveillance video are identified based on the number of matching points and the change in displacement between two adjacent frames. The key frames are input into the target detection model, and multi-scale features are extracted through multiple multi-scale attention modules set in sequence in the backbone network. The multi-scale features are up-sampled and down-sampled and spliced ​​through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

2. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 1, characterized in that: According to the feature point descriptors, feature matching algorithms are used to match feature points in adjacent frames, and matching points in each frame are determined and located, including: The nearest neighbor matching method is used to calculate the Euclidean distance between feature points of adjacent frames according to the feature point descriptors, and the nearest neighbor search is performed based on the distance to match the feature points in adjacent frames one by one; For each feature point, determine the two closest feature points in adjacent frames through matching, compare the distances between the current feature point and the two closest feature points, and determine whether to discard the matching point pair based on the comparison results; After feature matching and matching filtering, the matching points in each frame are determined and located.

3. The method for identifying construction procedures in a tunnel based on visual information according to claim 1, characterized in that: According to the change of the number of matching points and displacement of two adjacent frames, the key frames in the surveillance video are identified, including: Count the number of matching points in each of the two adjacent frames. If the difference in the number of matching points between the two frames exceeds the set threshold, it is determined that the current content has changed significantly, and the next frame is considered a key frame. Alternatively, based on the descriptors of the matching points, the displacement vectors between all the matching point pairs in two adjacent frames are calculated, and the average value and standard deviation of the displacement vectors are calculated. If the average value and standard deviation exceed the set threshold range, it is determined that the current content has changed significantly, and the next frame is the key frame.

4. The method for identifying construction procedures in a tunnel based on visual information according to claim 1, characterized in that: The target detection model includes a backbone network, an enhanced feature fusion network and a network detection head; The backbone network adopts a multi-scale attention network, namely a C2RG network, which includes a plurality of C2RG modules connected in sequence, for continuously using multi-scale information to capture attention and fuse, and extract multi-scale features; The enhanced feature fusion network adopts an SFF network, which includes multiple UCC modules and a 3C downsampling feature fusion module, which is used to upsample and downsample features of multiple scales and splice and fuse them, and extract feature maps of multiple scales to adapt to targets of different sizes; The network detection head is used to extract target information from feature maps of multiple scales and output it.

5. The method for identifying construction procedures in a tunnel based on visual information according to claim 4, characterized in that: The key frame is input into the backbone network, and the preliminary features are extracted through convolution. The preliminary features are used as input features and input into the C2RG module. In the C2RG module, the input features are input to multiple parallel paths, and in each path they pass through convolution, BatchNorm layer and SiLU activation function in turn to extract refined features; the refined features in the first path are then bottlenecked by the Bottleneck module to extract fine-grained features, which are then concatenated and fused with the refined features extracted by the second path, and the generated features are further refined through convolution, BatchNorm layer and SiLU activation function, and the features extracted after further refinement are concatenated and fused with the refined features extracted by the third path; the concatenated and fused features are then input to two parallel paths, and features are extracted through multiple deep convolutions and phantom convolutions respectively, and the features extracted by the two paths are concatenated and output.

6. The method for identifying construction procedures in a tunnel based on visual information according to claim 4, characterized in that: The multi-scale features are upsampled and spliced ​​by the UCC module in the enhanced feature fusion network to generate multi-scale fusion features, which are then downsampled and spliced ​​by the 3C module. The UCC module includes an upsampling layer, a splicing layer and a C2RG module, which is used to fuse feature maps of different scales through upsampling and splicing operations, and the fused feature maps are refined by the C2RG module; The 3C downsampling feature fusion module includes a downsampling convolution layer, a splicing layer and a C2RG module, which is used to fuse the feature maps of different scales output by the UCC module through downsampling and splicing operations, and the fused feature maps are refined by the C2RG module to output feature maps of multiple different sizes to adapt to targets of different sizes.

7. A tunnel construction process recognition system based on visual information, characterized in that: include: A monitoring video acquisition module is used to obtain real-time monitoring videos in a tunnel construction environment; The key frame extraction module is used to extract the feature points and their descriptors in each frame of the surveillance video using the SIFT feature detection algorithm, match the feature points in adjacent frames using the feature matching algorithm based on the feature point descriptors, determine and locate the matching points in each frame; and identify the key frames in the surveillance video based on the change in the number and displacement of matching points between two adjacent frames; The target detection and process identification module is used to input the key frames into the target detection model, and extract multi-scale features through multiple multi-scale attention modules set in sequence in the backbone network. The multi-scale features are up-sampled and down-sampled and spliced ​​and fused through the enhanced feature fusion network. The fused multi-scale features are then extracted and output by the network detection head, and the current tunnel construction process is identified based on the target information.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a method for identifying construction procedures in a tunnel based on visual information is implemented as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a method for identifying construction procedures in a tunnel based on visual information as described in any one of claims 1 to 6 are completed.

10. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of a method for identifying construction procedures in a tunnel based on visual information as described in any one of claims 1-6.

Images