A vehicle bridge impact behavior detection method and computer device based on image recognition

Abstract

The present invention provides a method and computer equipment for detecting a vehicle bridge collision based on image recognition, and relates to the technical field of vehicle bridge collision detection. The method comprises the following steps: obtaining a road monitoring image; and inputting the road monitoring image into a trained vehicle detection model to obtain a vehicle detection frame, and then judging whether a vehicle bridge collision occurs according to the positional relationship between the vehicle detection frame and the collision detection line. The vehicle detection model in the present invention is based on a Swin Transformer network, and is improved and trained in combination with a deconvolution module, a feature fusion layer, and a Bbox detection head. Compared with traditional vehicle speed measurement methods and methods of judging by camera imaging by the human eye, the present invention utilizes the real-time image analysis and recognition capabilities of computer vision to achieve real-time and high-precision vehicle bridge collision detection, and is suitable for scenes with large scale changes in vehicle detection and complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle bridge impact detection, and in particular to a vehicle bridge impact behavior detection method based on image recognition and a computer device. Background Art

[0002] With the continuous growth of road transportation and urban population, the development of three-dimensional transportation modes such as overpasses and viaducts has become a key measure to reduce the pressure on urban roads. However, due to various reasons, accidents of vehicles hitting bridge piers also occur from time to time, which not only damages the bridge structure, but also causes serious harm to people's lives and property.

[0003] Traditional vehicle speed measurement methods are only applicable to specific speed measurement sections and cannot monitor vehicle speed in real time. In addition, sudden vehicle and bridge accidents require manual monitoring to be discovered and cannot be responded to immediately. Road abnormality detection relies on cameras on the road, but these cameras are usually unable to perform complex real-time image analysis and recognition, and are prone to false positives and missed positives. With the rapid development of computer vision technology, it has become possible to detect the direction of vehicle-bridge collision behavior in real time through image recognition technology. The key is to consider the large scale changes, complex background and high precision of vehicle target detection when performing image recognition. Summary of the invention

[0004] The purpose of the present invention is to provide a vehicle bridge impact behavior detection method and computer equipment based on image recognition, which realizes the real-time and high-precision detection of vehicle bridge impact behavior.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] In one aspect, the present invention provides a method for detecting a vehicle bridge collision behavior based on image recognition, comprising the following steps:

[0007] A road monitoring image is obtained; the road monitoring image includes a collision detection line and a vehicle to be identified.

[0008] The road monitoring image is input into the vehicle detection model to obtain the vehicle detection frame; the vehicle detection model is a target detection model that has been trained; the target detection model is a model obtained by combining the deconvolution module, feature fusion layer and Bbox detection head on the basis of the SwinTransformer network; the vehicle detection frame is the labeling frame of the vehicle to be identified by the vehicle detection model.

[0009] According to the positional relationship between the vehicle detection frame and the collision detection line, it is determined whether a vehicle-bridge collision occurs.

[0010] Optionally, obtaining a road monitoring image specifically includes:

[0011] Obtain road surveillance video.

[0012] According to the road monitoring video, a road monitoring video frame is extracted.

[0013] The collision detection lines are marked on the road monitoring video frames to obtain the road monitoring images.

[0014] Optionally, the Swin Transformer network includes a feature extraction layer; the deconvolution module is connected after the feature extraction layer, and the deconvolution module is used to sequentially perform upsampling and deconvolution processing on the image features output by the feature extraction layer to obtain deconvolution features.

[0015] Optionally, a feature fusion layer is connected after the deconvolution module. The feature fusion layer is used to perform residual processing, global maximum pooling processing, dimensionality reduction processing, channel information interaction processing, convolution processing and dimensionality restoration processing on the deconvolution features in sequence to obtain reprocessed features, and fuse the reprocessed features with the deconvolution features to obtain fused features.

[0016] Optionally, the Bbox detection head is connected to the output of the feature fusion layer; the Bbox detection head is used to output a vehicle detection frame according to the fused features.

[0017] Optionally, before inputting the road monitoring image into the vehicle detection model, the method includes:

[0018] Obtain a training sample data set; the training sample data set includes a plurality of training samples; the training samples include road monitoring images and vehicle detection frame annotations;

[0019] According to the training sample data set, the target detection model is trained to obtain a vehicle detection model.

[0020] Optionally, when training the target detection model, a combination of the Smooth-L1 loss function and the GIoU loss function is used as the total loss function, and the model parameters of the target detection model are updated with the goal of minimizing the total loss function to obtain a vehicle detection model.

[0021] Optionally, the Smooth-L1 loss function is as follows:

[0022]

[0023] in, represents the Smooth-L1 loss function value, b σ(i) represents the vehicle detection box annotation of the i-th index, represents the anchor box of the i-th index, and N is the total number of anchor boxes.

[0024] Optionally, the GIoU loss function is as follows:

[0025]

[0026] Among them, GIoU is the GIoU loss function value, A is the area of ​​the vehicle detection box, B is the area of ​​the vehicle detection box annotation, and C is the area of ​​the minimum circumscribed box of the vehicle detection box and the vehicle detection box annotation.

[0027] On the other hand, corresponding to the aforementioned vehicle bridge impact behavior detection method based on image recognition, the present invention also provides a computer device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the aforementioned vehicle bridge impact behavior detection method based on image recognition.

[0028] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0029] The present invention provides a vehicle bridge collision behavior detection method and computer equipment based on image recognition, the method comprising the following steps: obtaining a road monitoring image; and inputting the road monitoring image into a vehicle detection model to obtain a vehicle detection frame, and then judging whether a vehicle bridge collision behavior occurs based on the positional relationship between the vehicle detection frame and the collision detection line. Compared with traditional vehicle speed measurement methods and methods of judging by camera imaging by the human eye, the vehicle detection model obtained by the present invention is based on the SwinTransformer network, combined with a deconvolution module, a feature fusion layer and a Bbox detection head, and model training. It can utilize the real-time image analysis and recognition capabilities of computer vision technology to achieve real-time and high-precision vehicle bridge collision behavior detection, and is suitable for scenes with large scale changes and complex backgrounds for vehicle target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 A flow chart of a method for detecting vehicle-bridge impact behavior based on image recognition provided in Example 1 of the present invention;

[0032] Figure 2 A flowchart of step A1 in the vehicle bridge impact behavior detection method based on image recognition provided in Example 1 of the present invention;

[0033] Figure 3 A flowchart of steps B1-B2 in the vehicle bridge impact behavior detection method based on image recognition provided in Example 1 of the present invention;

[0034] Figure 4 A schematic diagram of the structure of an improved SwinTransformer network in the vehicle-bridge collision behavior detection method based on image recognition provided in Example 1 of the present invention;

[0035] Figure 5 A schematic diagram of a feature fusion layer in a vehicle bridge impact behavior detection method based on image recognition provided in Example 1 of the present invention;

[0036] Figure 6 A schematic diagram of weighted multi-model fusion using an adaptive fuzzy Gaussian kernel module in the vehicle-bridge impact behavior detection method based on image recognition provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] The purpose of the present invention is to provide a vehicle bridge impact behavior detection method and computer equipment based on image recognition, which realizes the real-time and high-precision detection of vehicle bridge impact behavior.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] Embodiment 1:

[0041] This embodiment provides a method for detecting vehicle-bridge collision behavior based on image recognition, such as Figure 1 The flowchart shown includes the following steps:

[0042] A1. Obtain a road monitoring image; the road monitoring image includes a collision detection line and a vehicle to be identified.

[0043] A2. Input the road monitoring image into the vehicle detection model to obtain the vehicle detection frame. The vehicle detection model is a target detection model that has been trained. The target detection model is a model obtained by combining the deconvolution module, feature fusion layer and Bbox detection head on the basis of the Swin Transformer network. The vehicle detection frame is the label frame of the vehicle to be identified by the vehicle detection model.

[0044] Specifically, the Swin Transformer network includes a feature extraction layer; a deconvolution module is connected after the feature extraction layer, and the deconvolution module is used to sequentially upsample and deconvolve the image features output by the feature extraction layer to obtain deconvolution features.

[0045] On this basis, the feature fusion layer is connected after the deconvolution module; the feature fusion layer is used to perform residual processing, global maximum pooling processing, dimensionality reduction processing, channel information interaction processing, convolution processing and dimensionality restoration processing on the deconvolution features in sequence to obtain reprocessed features, and fuse the reprocessed features with the deconvolution features to obtain fused features.

[0046] The newly added Bbox detection head is connected to the output of the feature fusion layer; the Bbox detection head is used to output the vehicle detection frame based on the fused features.

[0047] A3. Determine whether a vehicle-bridge collision occurs based on the positional relationship between the vehicle detection frame and the collision detection line.

[0048] In this embodiment, Figure 2 In the flowchart shown, step A1 acquires a road monitoring image, which specifically includes the following steps:

[0049] A11. Obtain road surveillance video.

[0050] A12. Extract road monitoring video frames based on the road monitoring video.

[0051] A13. Annotate the road monitoring video frame with collision detection lines to obtain a road monitoring image.

[0052] Before inputting the road monitoring image into the vehicle detection model, Figure 3 As shown in the flowchart, the method provided in this embodiment also includes the following steps:

[0053] B1. Obtain a training sample data set; the training sample data set includes a number of training samples; the training samples include road monitoring images and vehicle detection frame annotations.

[0054] B2. According to the training sample data set, the target detection model is trained to obtain a vehicle detection model. Specifically, when training the target detection model, the combination of the Smooth-L1 loss function and the GIoU loss function is used as the total loss function, and the model parameters of the target detection model are updated with the minimum total loss function as the goal to obtain a vehicle detection model.

[0055] The Smooth-L1 loss function is as follows:

[0056]

[0057] in, represents the Smooth-L1 loss function value, b σ(i) represents the vehicle detection box annotation of the i-th index, represents the anchor box of the i-th index, and N is the total number of anchor boxes.

[0058] The GIoU loss function is as follows:

[0059]

[0060] Among them, GIoU is the GIoU loss function value, A is the area of ​​the vehicle detection box, B is the area of ​​the vehicle detection box annotation, and C is the area of ​​the minimum circumscribed box of the vehicle detection box and the vehicle detection box annotation.

[0061] The following is a specific example to illustrate the vehicle bridge collision behavior detection method based on image recognition provided by this embodiment:

[0062] First, image frames are extracted from road surveillance videos, and these images are cropped and annotated for subsequent analysis. Based on the public vehicle dataset, photos of road vehicles at road monitoring angles are collected and sorted out through retrieval. At the same time, in order to ensure that road vehicles can still be detected well in the event of vehicle rollover or other abnormal conditions during a collision, some images of vehicles in abnormal conditions are retrieved and added to the dataset.

[0063] Subsequently, the improved Swin Transformer network is used to extract image features and detect vehicles in surveillance videos to determine the location and features of vehicles in the image. In order to solve the problem of local information loss and excessive parameters caused by the fully connected layer during window splicing in the traditional Swin Transformer network, this example introduces a deconvolution Deconv module, a feature fusion layer, and a bbox regression head based on the Swin Transformer network. Figure 4 The model structure diagram shown. Deconvolution Deconv module, Bbox Head module and feature fusion layer are added to SwinTransformer; in this embodiment, an adaptive fuzzy Gaussian kernel module is also added. The Deconv network structure consists of one convolution and one deconvolution. A high-resolution feature map is obtained by upsampling, and a convolution operation is performed on the high-resolution feature map to capture more local information. The network structure of the regression head part consists of two convolutions. The adaptive fuzzy Gaussian kernel and the regression head are calculated according to the high-resolution feature map, and the center point regression prediction is realized for the high-resolution feature map to enhance the model feature expression ability.

[0064] Specifically, by introducing the deconvolution Deconv module in the Swin Transformer network, the feature extraction capability of local information is enhanced, so as to solve the problem of local zero feature information loss; the deconvolution Deconv module is introduced to perform deconvolution operation on the input feature map and insert zero-value pixels. The convolution kernel is used to perform convolution operation on these pixels, and the size of the feature map is expanded several times to extract more feature information. Figure 5 As shown, the feature input to the feature fusion layer is a C×H×W-dimensional feature X. After being processed by the residual block, it is pooled through the global maximum pooling layer to extract each feature dimension and reduce the dimension to obtain a C×1-dimensional feature. After that, through channel information interaction and convolution, the feature dimension is restored to a C×1×1-dimensional feature. Finally, the processed feature is fused with the original feature X to form the final fused feature. In an embodiment of the present invention, designing a residual structure in the feature fusion layer can ensure gradient backpropagation and enhance network training convergence.

[0065] At the same time, in this embodiment, an adaptive fuzzy Gaussian kernel is added after the feature fusion layer, which can adjust the target size and center point position in the output fusion feature map, solving the problem that sample points in low-dimensional space cannot be accurately divided. The degree of blur is adaptively adjusted according to the characteristics of the data to capture the edges and details of the target in the image. The fractional nature of the sample in the high-dimensional space is used to determine the position of the target object, and the distribution of the target object in the feature space is analyzed to determine its center position. The Bayesian posterior theorem is applied to determine the final result in combination with the output result of the model.

[0066] Assume that the multi-model consists of M sub-models, and the output is obtained by the sub-models according to the weighted method. Its basic form is:

[0067]

[0068] Where F(x) is the multi-model output; F i (x) is the sub-model output; w i For sub-model F i The weight of (x).

[0069] In order to solve the problem that the number of clusters is difficult to determine when the data set is too large, an adaptive fuzzy Gaussian kernel clustering is used to consider the distance between the data point and the cluster center and assign a membership value to each data point, such as dividing the data set into three categories: small car, medium pickup truck, and large bus. Based on clustering, this method calculates the posterior probability of the corresponding class according to each subset, and uses n j Indicates that it belongs to class w j (j=1,2,…,M) samples, then class w j The probability is:

[0070]

[0071] In the formula In order to calculate the class conditional probability density at the sample point, we can use x i The conditional probability of falling into the neighborhood replaces the point x i The class conditional probability density at , let R be the radius of the ball containing all sample sets, Let 0≤λ≤1, and take S(x i ; λR)={x|||xx i ||<λR} is the neighborhood of , and the sample set falls into S(x i ;λR) belongs to class w j The number of sample points is The class conditional probability of a sample point is According to the Bayesian principle, the posterior probability at the sample point is The obtained class posterior probability is used as the weight of the sub-model, and the multi-model is fused by probability weighting to achieve the desired purpose.

[0072] In addition, the Bbox regression head added to the fully connected layer after the feature extraction is completed fine-tunes the bounding box of the target according to the features extracted by the previous network, and regresses the center point position, size and offset of the target to more accurately frame the position of the target. Classification Block For each anchor box or prediction box, the classification block outputs a vector containing the score or probability of each category. This vector is normalized by a normalization function to obtain the probability distribution of each category. For example, if there are N categories, the length of the output vector will be N. Bbox Block For each anchor box or prediction box, the bounding box regression block outputs a vector containing the coordinate offset of the bounding box.

[0073] After improving the SwinTransformer network, the improved SwinTransformer network was trained and tested using a public vehicle dataset. The network training and testing was completed based on the public vehicle dataset. The experimental platform used was Ubuntu 20.04, the development language was python 3.7, the deep learning framework was pytorch 1.7, the CUDA version was 11.1, the CPU was Xeon Platinum 8350, the main frequency was 2.60 GHz, the memory was 43 GB, the hard disk was 1 TB, and the graphics card was RTX 3090, with 24 GB video memory. The constructed network model was transferred and frozen, the feature layer was frozen, and only the fully connected layer was trained. The SwinTransformer network weights were loaded and the parameters were initialized. The learning rate was set to 3e in the first 80 epochs.-5 , batchsize is set to 16, dropt is 0.3, then the model is unfrozen and the learning rate is adjusted to 5e -5 , batchsize is set to 8, dropt is 0.1, and training is performed for 30 epochs. The loss change rate is set to end the training early. During training, Smooth-L1 is combined with GIoU as the loss function so that the model can further converge to a higher accuracy.

[0074] The GIOU loss function takes into account the non-overlapping areas of the two boxes, which can effectively reflect the overlap of the two boxes. When the two boxes do not overlap, the gradient can also be returned for optimization.

[0075]

[0076] In the formula, A represents the predicted box area, B represents the real box area, C represents the minimum box area that can enclose A and B at the same time, and C\(A∪B) represents the relative difference between set C and (A∪B).

[0077] In the first 50 rounds of training, the numerical difference between the anchor box and the calibration box is large. Smooth-L1 can suppress the back-propagation gradient to avoid gradient explosion. In the last 50 rounds of training, the numerical difference between the anchor box and the calibration box is small. At this time, the gradient can continue to be returned to make the model converge to a higher accuracy. The Smooth-L1 loss function uses the horizontal and vertical coordinate values ​​and length and width values ​​of the anchor box and the calibration box when calculating the loss. Smooth-L1 is defined as:

[0078]

[0079] Where b σ(i) represents the calibration box of the i-th index, Represents the anchor box of the i-th index. When calculating the regression loss, the GIoU loss function is introduced to calculate the overlap loss between the anchor box and the calibration box. In the first 50 rounds of training, the numerical difference between the anchor box and the calibration box is large. Smooth-L1 suppresses the back-propagation gradient to avoid gradient explosion. In the last 50 rounds of training, the numerical difference between the anchor box and the calibration box is small, and the gradient is continuously returned to further converge the model to a higher accuracy. By combining the above two loss functions as the regression loss of this article, the training efficiency of the model is improved.

[0080] After completing the training and optimization of the model, the specific steps of the detection process for axle impact detection using the trained detection model are as follows:

[0081] (1) Operate the surveillance video of the bridge section that needs to be inspected and draw inspection lines at locations that are prone to impact.

[0082] (2) Input the road surveillance video into the detection model and extract the video image frames.

[0083] (3) The target vehicle in the image frame is detected through the Swin Transformer target detection model, and the information of the detected target vehicle detection box and the coordinates of the vehicle representative point are regressed.

[0084] (4) The detection model determines the positional relationship between the vehicle and the detection line. If the vehicle has not crossed the line, "Normal" is displayed. If it has crossed the line, a warning sign "Collison warning" is displayed on the screen.

[0085] (5) Display the axle impact detection results.

[0086] The present embodiment provides a method for detecting a vehicle bridge impact behavior based on image recognition. Compared with traditional vehicle speed measurement methods and methods of judging by camera imaging by the human eye, the present embodiment uses a Swin Transformer network, combined with a deconvolution module, a feature fusion layer, an adaptive fuzzy Gaussian kernel module and a Bbox detection head, to obtain a vehicle detection model after training. The real-time image analysis and recognition capabilities of computer vision technology can be used to achieve real-time and high-precision vehicle bridge impact behavior detection, which is suitable for scenarios with large scale changes and complex backgrounds for vehicle target detection.

[0087] Embodiment 2:

[0088] In addition, the method of embodiment 1 of the present invention can also be implemented with the help of a computer device architecture. The computer device provided in this embodiment includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the vehicle bridge collision behavior detection method based on image recognition described in embodiment 1.

[0089] Specific examples are used in this article, but the above description is only to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas; technicians in this field should understand that the modules or steps of the present invention can be implemented by a general computer device, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps in them can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0090] Meanwhile, for those skilled in the art, according to the concept of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting vehicle-bridge collision behavior based on image recognition, characterized in that: include: Acquire a road monitoring image: the road monitoring image includes a collision detection line and a vehicle to be identified; The road monitoring image is input into a vehicle detection model to obtain a vehicle detection frame; the vehicle detection model is a target detection model that has been trained; the target detection model is a model obtained by combining a deconvolution module, a feature fusion layer and a Bbox detection head on the basis of a Swin Transformer network; the vehicle detection frame is a labeling frame of the vehicle to be identified by the vehicle detection model; The Swin Transformer network includes a feature extraction layer; the deconvolution module is connected after the feature extraction layer, and the deconvolution module is used to sequentially perform upsampling processing and deconvolution processing on the image features output by the feature extraction layer to obtain deconvolution features; The feature fusion layer is connected after the deconvolution module, and is used to perform residual processing, global maximum pooling processing, dimensionality reduction processing, channel information interaction processing, convolution processing and dimension restoration processing on the deconvolution features in sequence to obtain reprocessed features, and fuse the reprocessed features with the deconvolution features to obtain fused features; An adaptive fuzzy Gaussian kernel is added after the feature fusion layer to adjust the target size and center point position in the output fusion feature map, and its blur degree is adaptively adjusted according to the characteristics of the data to capture the edge and details of the target in the image; Whether a vehicle-bridge collision occurs is determined based on the positional relationship between the vehicle detection frame and the collision detection line.

2. The vehicle bridge collision behavior detection method based on image recognition according to claim 1 is characterized in that: Obtain road monitoring images, including: Obtain road surveillance video; Extracting a road monitoring video frame according to the road monitoring video; The road monitoring video frame is annotated with collision detection lines to obtain a road monitoring image.

3. The vehicle bridge collision behavior detection method based on image recognition according to claim 1, characterized in that: The Bbox detection head is connected to the output of the feature fusion layer; the Bbox detection head is used to output the vehicle detection frame according to the fusion feature.

4. The vehicle bridge collision behavior detection method based on image recognition according to claim 1, characterized in that: Before inputting the road monitoring image into the vehicle detection model, the method includes: Obtaining a training sample data set: the training sample data set includes a plurality of training samples: the training samples include road monitoring images and vehicle detection frame annotations; The target detection model is trained according to the training sample data set to obtain a vehicle detection model.

5. The vehicle bridge collision behavior detection method based on image recognition according to claim 4 is characterized in that: When training the target detection model, the Smooth-L1 loss function and the GIoU loss function are combined as the total loss function, and the model parameters of the target detection model are updated with the goal of minimizing the total loss function to obtain a vehicle detection model.

6. The vehicle bridge collision behavior detection method based on image recognition according to claim 5 is characterized in that: The Smooth-L1 loss function is as follows: in, represents the Smooth-L1 loss function value, b σ(i) represents the calibration box of the i-th index, represents the anchor box of the i-th index, and N is the total number of anchor boxes.

7. The vehicle-bridge collision behavior detection method based on image recognition according to claim 5, It is characterized in that The GIoU loss function is as follows: Among them, GIoU is the GIoU loss function value, A is the area of ​​the vehicle detection box, B is the area of ​​the vehicle detection box annotation, and C is the area of ​​the minimum circumscribed box of the vehicle detection box and the vehicle detection box annotation.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the vehicle bridge impact behavior detection method based on image recognition as described in any one of claims 1-7.

Images