Method and system for identifying construction procedures in tunnel based on visual information
By constructing a multi-scale spatiotemporal feature and optical flow feature extraction network, and combining a sparse sampling strategy and a spatial attention mechanism, efficient identification of tunnel construction procedures is achieved. This solves the problems of low efficiency and increased costs caused by improper organization of procedures in tunnel construction, and realizes intelligent management of the construction process.
Patent Information
- Application Number
- CN202411092754.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-09
AI Technical Summary
In tunnel construction, especially when facing parallel construction with railway lines that are about to undergo joint commissioning and testing, the pressure of the construction period is high, and existing technologies are difficult to organize the construction procedures in a reasonable way, resulting in low construction efficiency and increased costs.
A visual information-based method for identifying construction procedures within tunnels is adopted. By constructing a multi-scale spatiotemporal feature extraction network and an optical flow feature extraction network, and combining a uniform sparse sampling strategy and a spatial attention mechanism, efficient identification of construction procedures within tunnels is achieved.
It improves the efficiency and accuracy of identifying construction procedures within the tunnel, ensures intelligent and automated management of the construction process, and reduces construction costs and schedule impact.
Smart Images

Figure CN119007077A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of tunnel construction, and in particular relates to a method and system for identifying tunnel construction procedures based on visual information. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] With more and more transportation infrastructure construction, parallel construction of lines is common. During construction, there are few tunnel construction working surfaces and the space in the single-line tunnel is very limited. Especially when facing parallel construction with the railway line that is about to undergo joint commissioning and testing, the pressure on the construction period is even greater. If the implementation and connection of the construction process cannot be reasonably organized, after the proposed line enters the joint commissioning and testing, the tunnel under construction will directly face the construction of the existing line, which will inevitably significantly reduce the effectiveness of on-site construction, bring huge impact on the project schedule, and also greatly increase the later construction costs. Summary of the invention
[0004] In order to solve the above problems, the present invention proposes a method and system for identifying construction procedures in a tunnel based on visual information. The present invention can improve the efficiency and accuracy of identifying construction procedures in a tunnel, and is conducive to the control of construction procedures in a tunnel.
[0005] According to some embodiments, the present invention adopts the following technical solutions:
[0006] A method for identifying construction procedures in a tunnel based on visual information comprises the following steps:
[0007] Obtain construction monitoring video in the tunnel;
[0008] The acquired video is dynamically sampled using a uniform sparse sampling strategy;
[0009] Constructing a multi-scale spatiotemporal feature extraction network, and using the constructed extraction network to extract and fuse multi-scale spatiotemporal features of the sampled video image. After extracting high- and low-layer spatiotemporal features, the multi-scale spatiotemporal feature extraction network uses a semantic feature embedding fusion method to introduce semantic information of high-layer spatiotemporal features into low-layer spatiotemporal features;
[0010] Introducing optical flow data information, using an optical flow feature extraction network to process the optical flow data information and extract optical flow data features, the optical flow feature extraction network is added with a spatial attention mechanism, the key information is screened out through the spatial attention mechanism, and then the key information is sent to a residual network for feature extraction;
[0011] The multi-scale spatiotemporal features and optical flow data features are mapped to a feature fusion space, and classification is performed based on the mapped and fused data to obtain a process recognition result.
[0012] As an optional implementation, the specific process of dynamically sampling the acquired video using a uniform sparse sampling strategy includes: assuming that the current video clip has N feature maps, the current sampling value S is expressed as:
[0013] S=[N / L]
[0014] Where S represents the sampling value of the current video, N represents the number of image frames of the current video after data preprocessing, and L represents the amount of data input to the network. The feature map is indexed according to the obtained sampling value S to obtain the model input L = [L(0), L(1)S, ...L(l-1)S, L(l)S].
[0015] As an optional implementation, the multi-scale spatiotemporal feature extraction network includes a multi-scale convolution module, a basic skeleton network and a multi-feature information aggregation module. The multi-scale convolution module is used to obtain the global features of the image, the basic skeleton network is used to generate high- and low-layer spatiotemporal features, and the multi-feature information aggregation module is used to introduce the semantic information of high-level spatiotemporal features into low-level spatiotemporal features by using semantic feature embedding and fusion, thereby enhancing the semantic expression of low-level spatiotemporal features and making the contextual spatiotemporal information and scale information complement each other.
[0016] Furthermore, the multi-scale convolution module includes three 3D convolution kernels of different sizes connected in sequence.
[0017] Furthermore, the basic skeleton network includes 8 3D convolutional layers, pooling layers and fully connected layers, wherein the 8 3D convolutional layers are connected in sequence, each 3D convolutional layer or every two convolutional layers is connected to a pooling layer, and finally two fully connected layers are connected, the first 3D convolutional layer is a multi-scale convolution module, and each pooling layer adopts a maximum pooling operation.
[0018] As a further step, in the multi-scale spatiotemporal feature extraction process, the 3D convolution kernel moves simultaneously in the x, y, and z directions, and the calculation process of the j-th feature map of the i-th layer at (x, y, z) is as follows:
[0019]
[0020] In the formula, m represents the feature map connected to the current feature map in the i-1th layer; is the value of the jth feature map of the i-th layer at (x, y, z), f is the Re lu activation function, usually expressed as f(x) = max(0, x), To sum all M feature maps connected to the current feature map in the i-1th layer, To sum the convolution kernel size H(i) in the time dimension (z direction), To sum the convolution kernel size W(i) in width (y direction), To sum over the convolution kernel size L(i) in length (x direction), W (i,j,m) Represents the convolution kernel weight connected to the m-th feature map of the i-1 layer; It is represented by the value of the mth feature map of the i-1th layer at (x+l, y+w, z+h), b (i,j) Represents the bias of the jth feature map in the i-th layer.
[0021] As a further step, the multi-feature information aggregation module includes 4 parallel 1×1×1 convolution kernels, which are used to set the channel values of high- and low-level spatiotemporal features to set set values, resample the high-level features and fuse them with the next-high-level features from top to bottom through semantic embedding, use the high-level semantic information to improve the low-level detail information, and then resample the fused features and fuse them with the next-level features to enhance the semantic expression of the low-level spatiotemporal features;
[0022] Convolution kernels with different step sizes are used to map the spatiotemporal feature maps into feature maps with the same dimensions, and then the high- and low-level spatiotemporal features embedded with semantic information are fused.
[0023] As an optional implementation, the optical flow feature extraction network uses the deep residual network ResNet101 model as the basic structure, and adds the Transformer attention mechanism to the basic structure. Specifically, a maximum pooling layer and an average pooling layer are used to obtain two feature maps of size 1×H×W, and then a 7×7 convolutional layer is used to obtain point-to-point spatial information. The spatial information is activated using the SIGMOID function, and the final spatial attention activation map is obtained.
[0024] As an optional implementation, the feature vector is mapped into a probability sequence using a SoftMax function, the corresponding result generated by the classification sequence through the encoder is extracted, and the result is classified to obtain the final classification result.
[0025] A tunnel construction process recognition system based on visual information includes the following steps:
[0026] A video acquisition module is configured to acquire construction monitoring videos in the tunnel;
[0027] A dynamic sampling module, configured to dynamically sample the acquired video using a uniform sparse sampling strategy;
[0028] The multi-scale spatiotemporal feature extraction module is configured to construct a multi-scale spatiotemporal feature extraction network, and use the constructed extraction network to extract and fuse multi-scale spatiotemporal features of the sampled video image. After extracting high- and low-layer spatiotemporal features, the multi-scale spatiotemporal feature extraction network uses a semantic feature embedding fusion method to introduce semantic information of high-layer spatiotemporal features into low-layer spatiotemporal features;
[0029] An optical flow data feature extraction module is configured to introduce optical flow data information, process the optical flow data information using an optical flow feature extraction network, and extract optical flow data features. The optical flow feature extraction network is added with a spatial attention mechanism, and key information is screened out through the spatial attention mechanism, and then the key information is sent to a residual network for feature extraction;
[0030] The fusion recognition module is configured to map the multi-scale spatiotemporal features and optical flow data features to a feature fusion space, and perform classification based on the mapped fused data to obtain a process recognition result.
[0031] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.
[0032] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] The present invention takes the full video time domain modeling as the starting point, extracts the global spatiotemporal features that can adapt to the changes in viewing angle based on the feature extraction network based on multi-column convolution, and obtains the deep features of the optical flow data based on the feature extraction network guided by the Transformer attention mechanism; then, feature fusion is performed in the fully connected layer and the final behavior recognition is completed using the SoftMax classifier. The final recognition result is accurate and efficient.
[0035] When acquiring the long-term temporal information of the entire video segment and establishing a video-level feature extraction network, the present invention introduces a uniform sparse sampling strategy to complete the temporal modeling of the entire video segment, thereby fully retaining the long-term temporal information while reducing the redundancy of the video frames.
[0036] The present invention constructs a multi-feature information aggregation module for aggregating high- and low-level spatiotemporal features, which solves the problem that missing high-level semantic information or low-level spatial detail information will affect the final event recognition result and lead to reduced accuracy.
[0037] When extracting optical flow information, the present invention can effectively locate key information in the image through the spatial attention mechanism, thereby effectively improving network performance.
[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0040] Figure 1 It is a schematic diagram of the process identification framework;
[0041] Figure 2 It is a schematic diagram of the multi-scale spatiotemporal feature extraction network;
[0042] Figure 3 It is a schematic diagram of the multi-scale module convolution structure;
[0043] Figure 4 It is a schematic diagram of the multi-scale convolutional block structure;
[0044] Figure 5 It is a schematic diagram of the C3D network structure;
[0045] Figure 6 It is a schematic diagram of the optical flow feature extraction network structure;
[0046] Figure 7 It is a schematic diagram of the spatial attention module. DETAILED DESCRIPTION
[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0048] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0050] Embodiment 1
[0051] Construction process recognition methods based on visual information, such as Figure 1 As shown, the following steps are included:
[0052] S1, using uniform sparse sampling strategy to perform data sampling, and then complete the time domain modeling of the entire video segment;
[0053] S2, obtain multi-scale spatiotemporal features through multi-column convolution to weaken the interference of perspective changes on video images;
[0054] S3, introduce optical flow data information, and obtain the deep features of optical flow data through the feature extraction network guided by the spatial attention mechanism;
[0055] S4, the acquired multi-scale spatiotemporal features and optical flow information are integrated in the fully connected layer of the network to achieve end-to-end long video behavior recognition;
[0056] S5. Input the continuous video frame data processed by the data conversion layer into the SM classifier for classification.
[0057] The following is a detailed introduction to each step. First, in step S1, this embodiment mainly uses video surveillance for identification. However, considering that the gap between frames of long-term events is small, random sampling easily introduces a large amount of redundant information and eliminates the correlation of video images in the time dimension. Therefore, when obtaining the long-term temporal information of the entire video segment and establishing a video-level feature extraction network, this embodiment introduces a uniform sparse sampling strategy to complete the temporal modeling of the entire video segment, and fully retains the long-term information while reducing the redundancy of the video frame. Assuming that the current video clip has N feature maps, the current sampling value S can be expressed as:
[0058] S=[N / L]
[0059] Where S represents the sampling value of the current video, N represents the number of image frames of the current video after data preprocessing, and L represents the amount of data input to the network. The feature map is indexed according to the obtained sampling value S to obtain the model input L = [L(0), L(1)S, ...L(l-1)S, L(l)S].
[0060] This embodiment ensures the consistency of the network input data volume through dynamic sampling values, and does not need to adjust other parameters, and can be applied to video data of different lengths.
[0061] In step S2, a multi-scale spatiotemporal feature extraction network is used, such as Figure 2As shown in the figure, it mainly includes three parts: multi-scale convolution module, basic skeleton network C3D and multi-feature information aggregation. First, the global features of the original image are obtained through the multi-scale convolution module, and then the high- and low-level spatiotemporal features are generated using the basic skeleton network. Finally, the semantic feature embedding and fusion method of the multi-feature aggregation module is used to introduce more semantic information contained in the high-level spatiotemporal features into the low-level spatiotemporal features, enhance the semantic expression of the low-level spatiotemporal features, make the contextual spatiotemporal information and scale information complement each other, and improve the network's ability to represent spatiotemporal features.
[0062] Multi-scale convolution is a multi-scale convolution module based on a multi-column structure for spatiotemporal feature extraction. The specific structure is as follows Figure 3 As shown in the figure. Since there is often a dynamic switching of perspectives when shooting videos, the video images have large scale changes, and single-column convolution is difficult to deal with the scale change problem in video images. In the multi-scale convolution module, three 3D convolution kernels of different sizes are used to learn scale-related features from the original input image blocks to achieve effective acquisition of multi-scale information. The multi-scale convolution block structure used in this paper is as follows: Figure 4 As shown in Figure 5, the convolution kernels of 3×3×3, 5×5×5, and 7×7×7 can effectively aggregate the global spatiotemporal information.
[0063] The basic skeleton network C3D performs feature extraction. The network uses stacked video RGB frames as input data and then uses 3D convolution kernels for feature extraction. The size of the convolution kernel determines the effectiveness of extracting video features. Since video images have problems such as dynamic occlusion and perspective changes, this requires that the features extracted by the network must be universal and effective. At the same time, in the time dimension, the connection between video features must be compact. Based on this, the size of all convolution kernels in the 8 3D convolution layers contained in the C3D network are set to 3×3×3; the pooling layers all use maximum pooling operations, where the pool 1 kernel is 1×2×2, and the remaining pooling kernels are 2×2×2; the network has 2 fully connected layers, which are mainly used to reduce the dimension of feature vectors. The network structure is as follows Figure 5 In order to deal with the scale change problem in video images, the first convolution layer of the network is replaced by a multi-scale convolution module, and the global features of the original image are obtained through multi-scale convolution.
[0064] In the multi-feature fusion, in the multi-scale spatiotemporal feature extraction process, the 3D convolution kernel moves simultaneously in the x, y, and z directions, and the calculation process of the j-th feature map of the i-th layer of the neural network at (x, y, z) is as follows:
[0065]
[0066] In the formula, m represents the feature map connected to the current feature map in the i-1th layer; is the value of the jth feature map of the i-th layer at (x, y, z), f is the Re lu activation function, usually expressed as f(x) = max(0, x), To sum all M feature maps connected to the current feature map in the i-1th layer, To sum the convolution kernel size H(i) in the time dimension (z direction), To sum the convolution kernel size W(i) in width (y direction), To sum over the convolution kernel size L(i) in length (x direction), W (i,j,m) Represents the convolution kernel weight connected to the m-th feature map of the i-1 layer; It is represented by the value of the mth feature map of the i-1th layer at (x+l, y+w, z+h), b (i,j) Represents the bias of the jth feature map in the i-th layer.
[0067] As the convolutional layer network deepens, the convolution process will lose some feature information. Since the receptive field of the high-level spatiotemporal feature network is relatively large, the extracted high-level spatiotemporal features contain more semantic information and less spatial detail features; the receptive field of the low-level spatiotemporal feature network is relatively small, and the extracted low-level spatiotemporal features contain more spatial detail information and less high-level semantic information. If the high-level semantic information or the low-level spatial detail information is missing, it will affect the final event recognition results and lead to reduced accuracy.
[0068] To address this problem, this embodiment constructs a multi-feature information aggregation module for aggregating high- and low-level spatiotemporal features. First, four parallel 1×1×1 convolution kernels are used to set the channel values of high- and low-level spatiotemporal features to 512; then, through semantic embedding, the high-level features are resampled and fused with the next-high-level features from top to bottom, and the high-level semantic information is used to improve the low-level detail information. The fused features are then resampled and fused with the next-level features to enhance the semantic expression of the low-level spatiotemporal features. After that, 3×3×3 convolution kernels with different step sizes are used to map the spatiotemporal feature map to a feature map with the same dimension; finally, the high- and low-level spatiotemporal features after embedding the semantic information are fused, and the calculation formula for the fused high- and low-level spatiotemporal features is as follows:
[0069]
[0070] Among them, F hl : Represents the sum of all eigenvalues FlF_lFl within the set range (from lminl_{min}lmin to lmaxl_{max}lmax), and finally calculates a new eigenvalue FhlF_{hl}Fhl.
[0071] lminl_{min}lmin: is the starting point of the accumulation operation, indicating which eigenvalue to start accumulation from.
[0072] lmaxl_{max}lmax: is the termination point of the accumulation operation, indicating at which eigenvalue the accumulation ends.
[0073] FlF_lFl: It is the accumulated eigenvalues. As lll changes from lminl_{min}lmin to lmaxl_{max}lmax, all these eigenvalues will be accumulated in turn to form the final eigenvalue FhlF_{hl}Fhl.
[0074] In step S3, optical flow data is introduced as another input modality of the model, and an optical flow feature extraction network is used.
[0075] For the feature map F, first pass through a maximum pooling layer and an average pooling layer to obtain two feature maps of size 1×H×W, then pass through a 7×7 convolution layer to obtain point-to-point spatial information, and then use the SI GMO ID function to activate the spatial information to obtain the final spatial attention activation map. The specific structure is as follows Figure 6 shown.
[0076] F is a comprehensive feature obtained by layer-by-layer fusion of multi-layer features, combining high-level semantic information and low-level detail information. The introduction of optical flow data and the use of deep residual networks further improve the effect of feature extraction. The introduction of optical flow features reduces the interference of lighting changes and complex backgrounds, and the spatial attention mechanism effectively locates the significant areas where actions occur, improving network performance.
[0077] The main reason for using optical flow information is that optical flow is the instantaneous speed of pixel movement of a moving object in space on the observation plane, which can reflect the speed, direction and other information of the moving subject in the video image; optical flow has apparent invariance, and the complex background in the video and the differences of the moving subject itself will not affect the expression of optical flow. The optical flow map is used as another input mode of the network to reduce the interference of factors such as lighting changes and complex backgrounds.
[0078] In the past, the network structure used to extract optical flow features was relatively shallow, and the extraction of optical flow information focused more on shallow detail information, while ignoring the deeper high-level semantic information in the optical flow. In order to fully explore the potential features of optical flow data, this embodiment uses the deep residual network ResNet101 model as the basic structure. Considering that the key information in the optical flow map is often concentrated in the area where the action occurs, the present invention adds a spatial attention mechanism to the basic network, selects the key information through spatial attention, and then sends it to the residual network for feature extraction. The essence of the attention mechanism is to locate the information related to the current task and suppress irrelevant information. Since the content presented in the optical flow map is the area where the action changes significantly, the key information in the image can be effectively located through the spatial attention mechanism, effectively improving the network performance.
[0079] In S4, the 4096-dimensional multi-scale spatiotemporal features and the 4096-dimensional optical flow data features are mapped to a 4096-dimensional feature fusion space. The advantage of this feature fusion method is that the model can learn the characteristic parameters of the two parallel networks during the training phase and independently complete the coordinated feedback, thus realizing end-to-end training of the model.
[0080] In S5, the SoftMax function is selected to map the feature vector into a probability sequence to retain more original information of the features. The corresponding results generated by the encoder are extracted from the classification sequence. The obtained results are input into the MLP module to compare the probabilities of each behavior. The event with the highest probability is the final event classification result.
[0081] The MLP module and SoftMax classifier work together, the former is used for feature learning and processing, and the latter is used for the final probability mapping and classification decision. The MLP module further processes the feature vector output by the encoder, learns more complex feature relationships, and generates the final feature vector for classification.
[0082] Finally, through the application of feature fusion, encoder processing, MLP module and SoftMax function, real-time detection and classification of tunnel construction processes can be achieved. The pre-stored event categories correspond to each specific construction process. The feature representation of each process is learned during the model training phase. When applied to actual scenarios, it can accurately identify and classify the current construction process, ensuring the intelligent and automated management of the construction process.
[0083] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.
Claims
1. A method for identifying construction processes in tunnels based on visual information. Its characteristics are: The following steps are involved: Obtain construction monitoring video in the tunnel; The acquired video is dynamically sampled using a uniform sparse sampling strategy; Constructing a multi-scale spatiotemporal feature extraction network, and using the constructed extraction network to extract and fuse multi-scale spatiotemporal features of the sampled video image. After extracting high- and low-layer spatiotemporal features, the multi-scale spatiotemporal feature extraction network uses a semantic feature embedding fusion method to introduce semantic information of high-layer spatiotemporal features into low-layer spatiotemporal features; Introducing optical flow data information, using an optical flow feature extraction network to process the optical flow data information and extract optical flow data features, the optical flow feature extraction network is added with a spatial attention mechanism, the key information is screened out through the spatial attention mechanism, and then the key information is sent to a residual network for feature extraction; The multi-scale spatiotemporal features and optical flow data features are mapped to a feature fusion space, and classification is performed based on the mapped and fused data to obtain a process recognition result.
2. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 1, Its characteristics are: The specific process of dynamically sampling the acquired video using the uniform sparse sampling strategy includes: assuming that the current video clip has N feature maps, the current sampling value S is expressed as: S=[N / L] Where S represents the sampling value of the current video, N represents the number of image frames of the current video after data preprocessing, and L represents the amount of data input to the network. The feature map is indexed according to the obtained sampling value S to obtain the model input L = [L(0), L(1)S, ...L(l-1)S, L(l)S].
3. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 1, Its characteristics are: The multi-scale spatiotemporal feature extraction network includes a multi-scale convolution module, a basic skeleton network and a multi-feature information aggregation module. The multi-scale convolution module is used to obtain the global features of the image, the basic skeleton network is used to generate high- and low-layer spatiotemporal features, and the multi-feature information aggregation module is used to introduce the semantic information of high-level spatiotemporal features into low-level spatiotemporal features by using a semantic feature embedding fusion method, enhance the semantic expression of low-level spatiotemporal features, and make the contextual spatiotemporal information and scale information complement each other.
4. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 3, Its characteristics are: The multi-scale convolution module includes three 3D convolution kernels of different sizes connected in sequence.
5. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 3, Its characteristics are: The basic skeleton network includes 8 3D convolutional layers, pooling layers and fully connected layers, wherein the 8 3D convolutional layers are connected in sequence, each 3D convolutional layer or every two convolutional layers is connected with a pooling layer, and finally two fully connected layers are connected, the first 3D convolutional layer is a multi-scale convolution module, and each pooling layer adopts the maximum pooling operation.
6. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 3, Its characteristic is that In the process of multi-scale spatiotemporal feature extraction, the 3D convolution kernel moves simultaneously in the x, y, and z directions. The calculation process of the j-th feature map of the i-th layer at (x, y, z) is as follows: In the formula, m represents the feature map connected to the current feature map in the i-1th layer; is the value of the jth feature map of the i-th layer at (x, y, z), f is the ReLU activation function, usually expressed as f(x) = max(0, x), To sum all M feature maps connected to the current feature map in the i-1th layer, To sum the convolution kernel size H(i) in the time dimension (z direction), To sum the convolution kernel size W(i) in width (y direction), To sum over the convolution kernel size L(i) in length (x direction), W (i,j,m) Represents the convolution kernel weight connected to the m-th feature map of the i-1 layer; It is represented by the value of the mth feature map of the i-1th layer at (x+l, y+w, z+h), b (i,j) Represents the bias of the jth feature map of the i-th layer.
7. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 3, Its characteristics are: The multi-feature information aggregation module includes 4 parallel 1×1×1 convolution kernels, which are used to set the channel values of high- and low-level spatiotemporal features to set set values, resample the high-level features and fuse them with the next-high-level features from top to bottom through semantic embedding, use the high-level semantic information to improve the low-level detail information, and then resample the fused features and fuse them with the next-level features to enhance the semantic expression of the low-level spatiotemporal features; Convolution kernels with different step sizes are used to map the spatiotemporal feature maps into feature maps with the same dimensions, and then the high- and low-level spatiotemporal features embedded with semantic information are fused.
8. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 1, Its characteristics are: The optical flow feature extraction network uses the deep residual network ResNet101 model as the basic structure, and adds the Transformer attention mechanism to the basic structure. Specifically, a maximum pooling layer and an average pooling layer are used to obtain two feature maps of size 1×H×W, and then a 7×7 convolutional layer is used to obtain point-to-point spatial information. The spatial information is activated using the SIGMOID function, and the final spatial attention activation map is obtained.
9. A method for identifying construction procedures in a tunnel based on visual information as claimed in claim 1, Its characteristics are: The SoftMax function is used to map the feature vector into a probability sequence, the corresponding results generated by the classification sequence through the encoder are extracted, and the results are classified to obtain the final classification results.
10. A tunnel construction process recognition system based on visual information, Its characteristics are: The following steps are involved: A video acquisition module is configured to acquire construction monitoring videos in the tunnel; A dynamic sampling module, configured to dynamically sample the acquired video using a uniform sparse sampling strategy; The multi-scale spatiotemporal feature extraction module is configured to construct a multi-scale spatiotemporal feature extraction network, and use the constructed extraction network to extract and fuse multi-scale spatiotemporal features of the sampled video image. After extracting high- and low-layer spatiotemporal features, the multi-scale spatiotemporal feature extraction network uses a semantic feature embedding fusion method to introduce semantic information of high-layer spatiotemporal features into low-layer spatiotemporal features; An optical flow data feature extraction module is configured to introduce optical flow data information, process the optical flow data information using an optical flow feature extraction network, and extract optical flow data features. The optical flow feature extraction network is added with a spatial attention mechanism, and key information is screened out through the spatial attention mechanism, and then the key information is sent to a residual network for feature extraction; The fusion recognition module is configured to map the multi-scale spatiotemporal features and optical flow data features to a feature fusion space, and perform classification based on the mapped fused data to obtain a process recognition result.
Citation Information
Patent Citations
Time sequence adaptive video classification method based on multi-source motion feature fusion
CN111209883A
Process identification method and device, electronic equipment and storage medium
CN114241363A
Computer-implemented method, data processing apparatus and computer program for object detection
US20230298335A1
