A road monitoring video abnormal behavior detection system, method, computer equipment and storage medium

Abstract

The present invention discloses a road monitoring video abnormal behavior detection system, method, computer equipment and storage medium, which belongs to the field of computer vision technology. The camera posture detection module constructed by the present invention estimates the camera posture change through the feature map, calculates the posture change matrix using the camera posture change, reconstructs the posture change matrix through the ground depth construction module to obtain the ground depth map, fuses the feature map and the ground depth map through the ground depth fusion module to obtain a fused depth feature map, and finally, uses a 3D detector to detect the fused depth feature map to obtain the detection result. Therefore, by correcting the camera posture change appearing in the road monitoring video, the posture change is prevented from having a greater impact on the 3D detection accuracy, making the prediction of the object more accurate, so that the abnormal behavior in the road monitoring video can be more accurately detected and judged, which has strong practicality and wide applicability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to a road monitoring video abnormal behavior detection system, method, computer equipment and storage medium. Background Art

[0002] Road traffic surveillance cameras are everywhere in cities. As an important part of the city's security system, road traffic surveillance is meant to record everything that happens in road traffic. 3D vehicle detection based on road traffic surveillance videos can detect and judge abnormal vehicle behavior in road traffic from these surveillance videos.

[0003] Compared with 2D detection, 3D detection has the following advantages: (1) More accurate spatial perception: 3D detection can provide more accurate object position and direction of movement, which helps to accurately determine the target's movement trajectory and predict possible traffic accidents. (2) More sensitive scale change recognition capability: In road traffic monitoring videos, the size of objects such as cars and pedestrians will change with their distance from the camera. 3D detection technology can better adapt to this scale change and improve the accuracy and reliability of detection results.

[0004] In the prior art, 3D detection methods based on the vehicle-side perspective (i.e., cameras or radar sensors mounted on vehicles) are mostly applied to vehicle autonomous driving. In urban road monitoring, most cameras are installed at fixed positions on the lanes, with a large monitoring angle and range, and the camera installation has a large downward angle. Therefore, there is a big difference between 3D detection based on the road monitoring perspective and 3D detection based on the vehicle-side perspective. At present, there are still some challenges in the 3D detection methods based on the road monitoring perspective: (1) Most existing 3D detection methods focus on the vehicle-side perspective, and the existing algorithm modules are also designed for the characteristics of vehicle-side perspective images. Since the position and orientation of the camera in the road monitoring perspective and the vehicle-side perspective are quite different, these existing modules are difficult to directly apply to 3D detection in the road monitoring perspective. (2) In the vehicle-side perspective, the camera and the car have the same orientation and will not change their posture. In the road monitoring perspective, the camera will rotate and its orientation is not parallel to the ground. There is a large downward angle, which makes it difficult to calculate the camera parameters and may affect the effect of 3D detection. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the deficiencies in the prior art, the present invention provides a road surveillance video abnormal behavior detection system, method, computer equipment and storage medium, which can provide algorithm support for 3D detection in public transportation safety systems to address the deficiencies in 3D detection of traffic monitoring perspectives.

[0007] 2. Technical solution

[0008] The purpose of the present invention is achieved through the following technical solutions.

[0009] A road monitoring video abnormal behavior detection system, comprising: an input module, a feature extraction module, a detection scene correction module, a ground depth fusion module, and a 3D detector;

[0010] The input module collects road monitoring videos and inputs road monitoring images;

[0011] The feature extraction module extracts a feature map of the road monitoring image;

[0012] The detection scene correction module includes a camera posture detection module and a ground depth construction module; the camera posture detection module estimates the camera posture change through the feature map, and obtains the posture change matrix by calculating the camera posture change; the ground depth construction module reconstructs the posture change matrix to obtain the ground depth map;

[0013] The ground depth fusion module fuses the feature map and the ground depth map to obtain a fused depth feature map;

[0014] The 3D detector detects the fused depth feature map to obtain a detection result.

[0015] Furthermore, the camera posture detection module estimates the camera posture change through the feature map, and obtains the posture change matrix by calculating the camera posture change, including:

[0016] Input the feature map into the posture detection network of the camera posture detection module to detect the horizon, vanishing point and original feature map;

[0017] The camera roll angle is calculated using the horizon, and the roll angle transformation matrix is ​​calculated using the camera roll angle. The camera pitch angle is calculated using the vanishing point, and the pitch angle transformation matrix is ​​calculated using the camera pitch angle. The local transformation matrix is ​​obtained using the roll angle transformation matrix and the pitch angle transformation matrix.

[0018] Pre-train visual cue features, concatenate the visual cue features and the original feature map to perform lane line detection, and use the local transformation matrix to correct the lane line detection results to obtain the corrected lane lines;

[0019] Set the initial position image, match the corrected lane line with the initial position image, calculate the horizontal displacement of the camera through the matched lane line to get the camera yaw angle, and calculate the yaw angle transformation matrix through the camera yaw angle;

[0020] The attitude change matrix is ​​obtained through the local transformation matrix and the yaw angle transformation matrix.

[0021] Furthermore, the setting of the initial position image and performing lane line matching between the corrected lane line and the initial position image includes:

[0022] Set the camera position, determine the two endpoints of the lane line, set the two endpoints of the lane line to be located on both sides of the camera, and perform lane line matching by calculating the similarity score of the features of the two endpoints of the lane line;

[0023] The similarity score is calculated as:

[0024]

[0025] Among them, i and j are natural numbers, F (i,j) represents the similarity between the i-th lane line in the initial position image and the j-th lane line in the road monitoring image, Represents the feature of the endpoint on the i-th lane line that is closer to the camera in the initial position image, represents the feature of the endpoint farthest from the camera on the i-th lane line in the initial position image, f j h represents the feature of the end point on the jth lane line in the road monitoring image that is closer to the camera, f j t Represents the feature of the endpoint on the jth lane line in the road monitoring image that is farthest from the camera.

[0026] Furthermore, the calculation formula for calculating the horizontal displacement of the camera through the matched lane line is:

[0027]

[0028] Where d represents the horizontal displacement of the camera, n represents the number of lane lines, Indicates the pixel coordinate of the end point of the i-th lane line in the initial position image that is closer to the camera on the pixel coordinate system v-axis of the initial position image, Represents the pixel coordinate of the end point of the i-th lane line in the initial position image that is farther from the camera on the pixel coordinate system v-axis of the initial position image, represents the pixel coordinate of the end point of the i-th lane line in the road monitoring image that is closer to the camera on the pixel coordinate system v-axis of the road monitoring image, Represents the pixel coordinates of the endpoint farther from the camera of the i-th lane line corresponding to the road monitoring image on the v-axis of the pixel coordinate system of the road monitoring image.

[0029] Furthermore, the ground depth construction module reconstructs the posture change matrix to obtain a ground depth map, including:

[0030] The camera coordinate system at the initial position is converted into the camera coordinate system after the transformed posture through the posture change matrix;

[0031] Obtaining a ground depth map of the initial position image, establishing a three-dimensional coordinate system, and reversely projecting the pixel points of the ground depth map of the initial position image into the three-dimensional coordinate system to obtain initial three-dimensional coordinate points;

[0032] The initial three-dimensional coordinate point is converted into the camera coordinate system after the posture transformation through the posture change matrix to obtain the posture change point;

[0033] Project the posture change point onto the road monitoring image to obtain the pixel coordinates in the road monitoring image under the current camera posture;

[0034] The depth value is calculated by the pixel coordinates, and the ground depth map is reconstructed by the depth value.

[0035] Furthermore, the ground depth fusion module fuses the feature map with the current scene ground depth map to obtain a depth feature map, including:

[0036] Input the feature map into the ground contact point detection network of the ground depth fusion module to obtain the ground contact point;

[0037] Project the ground contact points into the ground depth map, discretize the ground depth map into depth intervals, calculate the fusion weights of the depth intervals, and superimpose the depth intervals and fusion weights to obtain a depth sequence;

[0038] Input the feature map into the encoder of the self-attention mechanism to calculate the serialized features;

[0039] The serialized features and depth sequences are input into the decoder of the self-attention mechanism to output the fused depth feature map.

[0040] Furthermore, the calculation formula of the fusion weight of the depth interval is:

[0041]

[0042] Among them, λ represents the fusion weight sequence of the depth interval, λ i represents the fusion weight of the depth interval, p represents the total number of ground contact points, and p i It represents the number of ground contact points within the depth interval, i represents the sequence number of the depth interval, and c represents the total number of depth intervals after discretization.

[0043] A method for detecting abnormal behavior in road monitoring videos comprises the following steps:

[0044] Constructing a road surveillance video abnormal behavior detection system as described above;

[0045] Collect road surveillance videos and input road surveillance images;

[0046] Extracting a feature map of the road monitoring image through a feature extraction module;

[0047] The camera posture detection module estimates the camera posture change through the feature map, and uses the camera posture change to calculate the posture change matrix. The ground depth construction module reconstructs the posture change matrix to obtain the ground depth map.

[0048] The ground depth fusion module fuses the feature map and the ground depth map to obtain a fused depth feature map;

[0049] The fused depth feature map is detected by a 3D detector to obtain the detection result.

[0050] A computer device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the method when executing the computer program.

[0051] A computer-readable storage medium stores a computer program, and the computer program executes the method when executed by a processor.

[0052] 3. Beneficial effects

[0053] Compared with the prior art, the advantages of the present invention are:

[0054] (1) The present invention provides a road surveillance video abnormal behavior detection system, method, computer device and storage medium, which construct a detection scene correction module, including a camera posture detection module and a ground depth construction module. The camera posture detection module is used to estimate the camera posture change and calculate the posture change matrix. The ground depth construction module is used to reconstruct the ground depth map in the image. Therefore, through the detection scene correction module, the problem that camera movement affects depth perception can be solved, and the problem of large deviation in depth estimation can be effectively avoided. It is a plug-and-play divide-and-conquer module.

[0055] (2) The present invention provides a road surveillance video abnormal behavior detection system, method, computer equipment and storage medium. In the ground depth fusion module, the depth area where the target is located is highlighted through the ground contact point, thereby increasing the depth information of the depth area and better fusing the depth information. Other depth areas that do not contain the target contribute less to the feature fusion depth information and do not need to be focused on, which effectively saves time cost and improves detection efficiency.

[0056] (3) The present invention provides a road surveillance video abnormal behavior detection system, method, computer device and storage medium, which corrects the camera posture changes in the road surveillance video, thereby preventing the posture changes from having a significant impact on the 3D detection accuracy, making the prediction of objects more accurate, and thus enabling more accurate detection and judgment of abnormal behaviors in the road surveillance video, and has strong practicality and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the overall structure of the detection system according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of the structure of a camera posture detection module according to an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of roll angle transformation according to an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of pitch angle transformation according to an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of yaw angle transformation according to an embodiment of the present invention;

[0062] Figure 6 Schematic diagram of the projection transformation process according to an embodiment of the present invention;

[0063] Figure 7 It is a schematic diagram of the structure of the ground depth fusion module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0065] Example

[0066] like Figure 1 As shown, a road monitoring video abnormal behavior detection system provided by this embodiment includes an input module, a feature extraction module, a detection scene correction module, a ground depth fusion module and a 3D detector.

[0067] Specifically in this embodiment, in the input module, the road monitoring video is collected and the road monitoring image is input.

[0068] In the feature extraction module, the feature map F of the road monitoring image is extracted by the feature extraction module. i It should be noted that, in this embodiment, the feature extraction module used is the existing technology. For example, the existing DLA-34 network is used as the image extraction network.

[0069] The detection scene correction module includes a camera posture detection module and a ground depth construction module. In this embodiment, the detection scene correction module is composed of a camera posture detection module and a ground depth construction module in series. The camera posture detection module uses the feature map F i The camera pose change is estimated and the pose change matrix is ​​calculated using the camera pose change. The ground depth construction module reconstructs the pose change matrix to obtain the ground depth map.

[0070] Specifically, Figure 2 As shown, in the camera posture detection module, the camera posture detection module uses the feature map F i Estimate the camera posture change and use the camera posture change to calculate the posture change matrix, which specifically includes the following steps:

[0071] Input feature map F i The horizon y is detected in the posture detection network of the camera posture detection module gp , vanishing point y vp and the original feature map F t In this embodiment, the posture detection network can be an existing deep learning model such as AlexNet and VGG-16. In this embodiment, AlexNet is selected as the posture detection network. The AlexNet network includes 5 convolutional layers and 3 fully connected layers. It should be noted that in this embodiment, horizon y gp Refers to the straight line formed by the intersection of the far ground and the sky in the road monitoring image. The position in the road monitoring image is represented by the straight line equation y=ax+b, where a represents the coefficient of the straight line equation, that is, the slope, and b represents the intersection of the straight line and the y-axis. vp It means that the parallel lane lines on the actual road intersect at one point in the road monitoring image after perspective transformation. The vanishing point coordinates are set as (x vp ,y vp ), where x vp represents the horizontal coordinate of the vanishing point on the road monitoring image plane, y vp Represents the vertical coordinate of the vanishing point on the road monitoring image plane. Original feature map F t It refers to the feature map output in the middle process of the posture detection network.

[0072] In this embodiment, the horizon y is used gp The camera roll angle is calculated, and then the roll angle transformation matrix is ​​calculated by the camera roll angle. The calculation formula is:

[0073] θ R =arctan(a)

[0074]

[0075] Among them, θR represents the camera roll angle, a represents the coefficient of the straight line equation, arctan represents the inverse tangent trigonometric function, A R Represents the roll angle transformation matrix.

[0076] Using the vanishing point y vp The camera pitch angle is calculated, and then the pitch angle transformation matrix is ​​calculated by the camera pitch angle. The calculation formula is:

[0077]

[0078] Among them, θ P Indicates the camera pitch angle, x vp represents the horizontal coordinate of the vanishing point on the road monitoring image plane, x c represents the horizontal coordinate of the center point of the road monitoring image, f represents the focal length of the camera, A P Represents the pitch angle transformation matrix.

[0079] Therefore, the roll angle transformation matrix A R And the pitch angle transformation matrix A P Get the local transformation matrix, which is expressed as:

[0080] M p ′=A R A P

[0081] Among them, M p ′ represents the local transformation matrix.

[0082] Furthermore, the visual cue feature is pre-trained, and the lane line detection is performed after the visual cue feature and the original feature map are spliced. The lane line detection result is corrected using the local transformation matrix to obtain the corrected lane line. In this embodiment, the pre-trained visual cue feature is a learnable feature, and the size of the visual cue feature is set to be the same as the size of the original feature map. Then, the visual cue feature and the original feature map are spliced ​​to perform lane line detection to obtain the lane line detection result. After obtaining the lane line detection result, the local transformation matrix M is used to correct the lane line detection result. p 'Correct the lane line detection result to obtain the corrected lane line. In this embodiment, the local transformation matrix M p ′Correct the lane line detection result to eliminate the roll angle θ R and the pitch angle θ P The changes in these two dimensions relative to the initial position.

[0083] Furthermore, an initial position image is set, and the corrected lane line is matched with the initial position image for lane line matching. The horizontal displacement of the camera is calculated through the matched lane line to obtain the camera yaw angle, and the yaw angle transformation matrix is ​​calculated through the camera yaw angle. In this embodiment, in the collected road monitoring video, a road monitoring image is selected and set as the initial position image. Then, the corrected lane line is matched with the initial position image for lane line matching. The specific steps are: setting the camera position; determining the two endpoints of the lane line. It should be noted that in this embodiment, the lane line is determined by the two endpoints; setting the two endpoints of the lane line to be located on both sides of the camera respectively. In this embodiment, one endpoint of the lane line is set to be closer to the camera, and the other endpoint of the lane line is set to be farther from the camera. Lane line matching is performed by calculating the similarity score of the features of the two endpoints of the lane line; in this embodiment, the calculation formula of the similarity score is:

[0084]

[0085] Among them, i and j are natural numbers, F (i,j) represents the similarity between the i-th lane line in the initial position image and the j-th lane line in the road monitoring image, Represents the feature of the endpoint on the i-th lane line in the initial position image that is closer to the camera, represents the feature of the endpoint farthest from the camera on the i-th lane line in the initial position image, f j h represents the feature of the end point on the jth lane line in the road monitoring image that is closer to the camera, f j t Represents the feature of the endpoint on the jth lane line in the road monitoring image that is farthest from the camera.

[0086] Therefore, by calculating the similarity score between the features of the lane lines in the road monitoring image and the initial position image, a threshold α is set. If the similarity score is greater than the threshold α, the two lane lines with the largest similarity score are selected and matched as a group until the lane lines in the detection scene are matched, or the remaining lane lines cannot be matched.

[0087] In this embodiment, since there is no roll angle θ R and the pitch angle θ P The influence of changes in two dimensions, therefore, the horizontal displacement of the camera can be calculated by the matched lane line to obtain the camera yaw angle. Since the camera yaw angle is only related to the transformation in the horizontal direction, the horizontal transformation of the camera is calculated by the horizontal distance between the corresponding lane lines, which is the camera yaw angle. In this embodiment, the calculation formula for calculating the horizontal displacement of the camera by the matched lane line is:

[0088]

[0089] Where d represents the horizontal displacement of the camera, n represents the number of lane lines, Indicates the pixel coordinate of the end point of the i-th lane line in the initial position image that is closer to the camera on the pixel coordinate system v-axis of the initial position image, Represents the pixel coordinate of the end point of the i-th lane line in the initial position image that is farther from the camera on the pixel coordinate system v-axis of the initial position image, represents the pixel coordinate of the end point of the i-th lane line in the road monitoring image that is closer to the camera on the pixel coordinate system v-axis of the road monitoring image, Represents the pixel coordinates of the endpoint farther from the camera of the i-th lane line corresponding to the road monitoring image on the v-axis of the pixel coordinate system of the road monitoring image.

[0090] The camera's horizontal displacement d is used to calculate the camera's yaw angle, and the yaw angle transformation matrix is ​​obtained through the camera's yaw angle calculation. The calculation formula is:

[0091]

[0092] Among them, θ Y represents the camera yaw angle, A Y Represents the yaw angle transformation matrix.

[0093] Furthermore, the attitude change matrix is ​​obtained through the local transformation matrix and the yaw angle transformation matrix. The attitude change matrix is ​​expressed as:

[0094] M P =M P 'A Y =A R A P A Y

[0095] Among them, M P Represents the posture change matrix.

[0096] In this embodiment, the posture change matrix M P The loss function is expressed as:

[0097]

[0098] Among them, L pose represents the loss function of the posture change matrix, A represents the true transformation matrix, i.e., Ground Truth, and || || represents the second normal form calculation.

[0099] Therefore, in this embodiment, Figure 3-Figure 5 As shown, through the roll angle θ R , pitch angle θ P and yaw angle θY Represents the transformation of the camera in three dimensions, thereby determining the change in camera posture.

[0100] In this embodiment, after obtaining the posture change matrix M P After that, the posture change matrix M is constructed by the ground depth module P Reconstructing the ground depth map includes the following steps:

[0101] The camera coordinate system at the initial position is converted to the camera coordinate system after the transformed posture through the posture change matrix. Specifically, the camera perspective transformation is represented by the posture change matrix between the two coordinate systems. The camera coordinate system at the initial position is set as C 0 , through the posture change matrix M P The camera coordinate system after the transformed posture is C', and the camera coordinate system C' after the transformed posture is expressed as:

[0102] C″=C 0 M P

[0103] Further, the ground depth map of the initial position image is obtained, a three-dimensional coordinate system is established, and the pixel points of the ground depth map of the initial position image are reversely projected into the three-dimensional coordinate system to obtain the initial three-dimensional coordinate points. In this embodiment, in the ground depth map of the initial position image, when the camera posture is unchanged, the ground depth is manually measured and collected in advance, so it is known. A pixel coordinate system is established, and the pixel point on the initial position image is (u 0 , v 0 ), where u 0 represents the coordinates on the u-axis of the pixel coordinate system of the image at the initial position, v 0 Represents the coordinates on the v-axis of the pixel coordinate system of the image at the initial position. Figure 6 As shown, for the pixel point (u 0 , v 0 ), and its corresponding depth value is The pixel points of the ground depth map of the initial position image are reversely projected into the three-dimensional coordinate system to obtain the initial three-dimensional coordinate points. The calculation formula for reverse projection is:

[0104]

[0105] in, Represents the coordinate of the x-axis of the three-dimensional coordinate system in the image scene at the initial position, represents the coordinate of the y-axis of the three-dimensional coordinate system in the image scene at the initial position, f x Indicates the focal length in pixels on the horizontal axis of the image at the initial position, f y represents the focal length in pixels on the vertical axis of the image at the initial position, c xIndicates the coordinate of the center point of the initial position image plane on the x-axis, c y Indicates the coordinate of the center point of the initial position image plane on the y-axis. x and c y It is a fixed parameter of the camera and will not change. It should be noted that the parameter f x 、f y 、c x 、c y is the built-in parameter of the camera, which is fixed and available. Then, the initial three-dimensional coordinate point is expressed as

[0106] Furthermore, the initial three-dimensional coordinate point is converted into the camera coordinate system after the transformed posture through the posture change matrix to obtain the posture change point, which is expressed as:

[0107]

[0108] Among them, x′ 3d Represents the coordinate of the x-axis in the three-dimensional coordinate system of the road monitoring image scene, y′ 3d Represents the coordinate of the y-axis in the three-dimensional coordinate system of the road monitoring image scene, z′ 3d Represents the coordinate of the z-axis in the three-dimensional coordinate system of the road monitoring image scene.

[0109] Furthermore, the posture change point (x′ 3d , y′ 3d , z′ 3d ) is projected onto the road monitoring image to obtain the pixel coordinates (u′, v′) in the road monitoring image under the current camera posture.

[0110] Since the height of the surveillance camera from the ground is fixed at the traffic intersection, recorded as EL, the depth value is calculated by the pixel coordinates, and the calculation formula is:

[0111]

[0112] Wherein, z' represents the depth value.

[0113] Then, the ground depth map is reconstructed by the depth value, and the ground depth map is expressed as:

[0114] D={z′|0<i<W,0<j<H}

[0115] Wherein, D represents the ground depth map, W represents the width of the ground depth map, and H represents the height of the ground depth map.

[0116] Therefore, in this embodiment, the constructed detection scene correction module is a plug-and-play divide-and-conquer module, which is used to solve the problem that camera movement affects depth perception and can effectively avoid the problem of large deviation in depth estimation.

[0117] In the ground depth fusion module, the feature map and the ground depth map are fused to obtain a fused depth feature map. In this embodiment, in order to fuse the feature map and the ground depth map, the feature map and the ground depth map need to be aligned. Therefore, it is necessary to use some special positioning points to align the feature map and the ground depth map. In this embodiment, the contact point between the object and the ground is the key point for locating the depth of the target. The depth area where the target is located can be highlighted, thereby increasing the depth information of the depth area and better fusing the depth information. Other depth areas that do not contain targets contribute less to the feature fusion depth information. Therefore, there is no need to focus on them, which effectively saves time costs.

[0118] Specifically, Figure 7 As shown, first, the feature map F i Input into the ground contact point detection network of the ground depth fusion module to obtain the ground contact point P, P = {P1, P2, ..., P n}, where n represents a natural number. It should be noted that, in this embodiment, the ground contact point network adopts the existing VPGNet as the backbone network, which is composed of 8 layers of convolution and pooling layers, so that the ground contact point P can better locate the depth position of the target, so that the depth area is focused.

[0119] Furthermore, the ground contact point P is projected into the ground depth map D, and the ground depth map is discretized into depth intervals L D , L D ={d1,d2,d3,....,d c}, where d i (1≤i≤c) represents a depth interval. For example, if the road monitoring image contains a scene that is 10 meters to 90 meters away from the camera, then 10 meters is a depth interval, and it will be divided into 8 depth intervals {d1, d2, d3, ...., d8}. The more depth intervals d containing ground contact points P, the more depth intervals d i It is important to note that this depth interval will contain more targets, so the depth interval d i The corresponding fusion weight will also be higher. In this embodiment, the calculation formula for the fusion weight of the depth interval is:

[0120]

[0121] Among them, λ represents the fusion weight sequence of the depth interval, λ i represents the fusion weight of the depth interval, p represents the total number of ground contact points, and p i It represents the number of ground contact points within the depth interval, i represents the sequence number of the depth interval, and c represents the total number of depth intervals after discretization.

[0122] Furthermore, the feature map F i Input it into the encoder of the self-attention mechanism Transformer to calculate the serialized features. Specifically, the feature map F i ∈R W×H×C Flattened to sequence feature H img ∈R N×C , where R represents the vector space and W represents the feature map F i In the size of the first dimension, H represents the feature map F i In the second dimension, C represents the feature map F i In the third dimension, N represents the size of H img The size in the first dimension, H img In this embodiment, the serialization feature H img The query feature Q, key feature K, and value feature V of the self-attention module in the encoder are calculated by the position code position. The calculation formula is:

[0123]

[0124] Among them, l represents a natural number, Q l represents the query features of the lth training stage, K l represents the key features of the lth training stage, V l represents the value feature of the lth training stage, Represents the serialized feature H in the lth stage of the self-attention mechanism Transformer img Transformed into a learnable weight matrix of query features, Represents the serialized feature H in the lth stage of the self-attention mechanism Transformer img Transformed into a learnable weight matrix of key features, Represents the serialized feature H in the lth stage of the self-attention mechanism Transformer img Converted into a learnable weight matrix of value features. It should be noted that, in this embodiment, the position encoding position used is a trainable parameter.

[0125] Therefore, in the self-attention module in the encoder, for the serialized feature H img , then:

[0126]

[0127] in, Represents the serialized features obtained by iterating the l+1th stage in the encoder Encoder.

[0128] Furthermore, the depth interval L D And the fusion weight λ i The depth sequence H is obtained by superposition dep , where H dep ∈R N×1 , and its calculation formula is:

[0129] H dep =L D +λ={d i +λ i |1≤i≤c}

[0130] Furthermore, the serialized feature H img , depth sequence H dep Input to the decoder of the self-attention mechanism Transformer to output the fused deep feature map. In this embodiment, in the decoder of the self-attention mechanism Transformer, the calculation formula of the query feature Q, key feature K and value feature V of the self-attention module is:

[0131]

[0132] in, represents the depth sequence obtained by iterating the lth stage in the decoder, and concat(·) represents the concatenation operation.

[0133] Therefore, the output fusion depth feature map in the decoder is expressed as:

[0134] M f =reshape2D(normalize(Attention(Q l ,K l ,V l ))

[0135] Among them, M f represents the fusion of deep feature maps, Attention(·) represents the self-attention module calculation, normalize(·) represents the normalization operation, and reshape2D(·) represents the process of inverting serialized features into 2D feature maps.

[0136] Therefore, in this embodiment, the constructed deep ground fusion module uses the self-attention mechanism Transformer to extract the feature map F obtained by the feature extraction network i It is fused with the reconstructed ground depth map D to obtain a fused feature map M containing depth information. f .

[0137] Finally, the fused depth feature map is detected by a 3D detector to obtain the detection result. The detection result includes the object category cls, the center position (x, y, z) of the 3D detection box, the size (l, w, h), the depth position d and the yaw angle θ. It should be noted that in this embodiment, the 3D detector used adopts the existing standard Mono3D detector as the underlying detection core.

[0138] The present embodiment also provides a method for detecting abnormal behavior of road surveillance videos, comprising the following steps: constructing a road surveillance video abnormal behavior detection system as described above; collecting road surveillance videos and inputting road surveillance images; extracting feature maps of road surveillance images through feature extraction modules; a camera posture detection module estimating camera posture changes through feature maps, calculating posture change matrices using camera posture changes, and a ground depth construction module reconstructing posture change matrices to obtain ground depth maps; a ground depth fusion module fusing feature maps and ground depth maps to obtain fused depth feature maps; and detecting fused depth feature maps through 3D detectors to obtain detection results.

[0139] Therefore, the present embodiment provides a method for detecting abnormal behavior in road surveillance videos, which corrects the camera posture changes in the road surveillance videos, thereby preventing the posture changes from having a significant impact on the 3D detection accuracy, making the prediction of objects more accurate, thereby enabling more accurate detection and judgment of abnormal behaviors in road surveillance videos, and having strong practicality and wide applicability.

[0140] This embodiment also provides a computer device. A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video anomaly detection method when executing the computer program.

[0141] This embodiment also provides a computer-readable storage medium. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, a video anomaly detection method described in this embodiment is executed. The computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] The above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic features of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited thereto, and any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and an embodiment similar to the technical solution are designed without creativity, which should all belong to the protection scope of the present invention. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate the name, and do not indicate any specific order.

Claims

1. A road monitoring video abnormal behavior detection system, characterized in that: include: Input module, feature extraction module, detection scene correction module, ground depth fusion module, 3D detector; The input module collects road monitoring videos and inputs road monitoring images; The feature extraction module extracts a feature map of the road monitoring image; The detection scene correction module includes a camera posture detection module and a ground depth construction module; the camera posture detection module estimates the camera posture change through the feature map, and obtains the posture change matrix by calculating the camera posture change; the ground depth construction module reconstructs the posture change matrix to obtain the ground depth map; The ground depth fusion module fuses the feature map and the ground depth map to obtain a fused depth feature map, including: inputting the feature map into the ground contact point detection network of the ground depth fusion module to obtain the ground contact points; projecting the ground contact points into the ground depth map, discretizing the ground depth map into depth intervals, calculating the fusion weights of the depth intervals, and superimposing the depth intervals and the fusion weights to obtain a depth sequence; inputting the feature map into the encoder of the self-attention mechanism to calculate the serialized features; inputting the serialized features and the depth sequence into the decoder of the self-attention mechanism to output the fused depth feature map; The 3D detector detects the fused depth feature map to obtain a detection result.

2. A road monitoring video abnormal behavior detection system according to claim 1, characterized in that: The camera posture detection module estimates the camera posture change through the feature map, and calculates the posture change matrix using the camera posture change, including: Input the feature map into the posture detection network of the camera posture detection module to detect the horizon, vanishing point and original feature map; The camera roll angle is calculated using the horizon, and the roll angle transformation matrix is ​​obtained through the camera roll angle calculation; the camera pitch angle is calculated using the vanishing point, and the pitch angle transformation matrix is ​​obtained through the camera pitch angle calculation; the local transformation matrix is ​​obtained through the roll angle transformation matrix and the pitch angle transformation matrix; Pre-train visual cue features, concatenate the visual cue features and the original feature map to perform lane line detection, and use the local transformation matrix to correct the lane line detection results to obtain the corrected lane lines; Set the initial position image, match the corrected lane line with the initial position image, calculate the horizontal displacement of the camera through the matched lane line to get the camera yaw angle, and calculate the yaw angle transformation matrix through the camera yaw angle; The attitude change matrix is ​​obtained through the local transformation matrix and the yaw angle transformation matrix.

3. A road monitoring video abnormal behavior detection system according to claim 2, characterized in that: The setting of the initial position image and performing lane line matching between the corrected lane line and the initial position image includes: Set the camera position, determine the two endpoints of the lane line, set the two endpoints of the lane line to be located on both sides of the camera, and perform lane line matching by calculating the similarity score of the features of the two endpoints of the lane line; The similarity score is calculated as: Among them, i and j both represent natural numbers. represents the similarity between the i-th lane line in the initial position image and the j-th lane line in the road monitoring image, Represents the feature of the endpoint on the i-th lane line that is closer to the camera in the initial position image, Represents the feature of the endpoint on the i-th lane line in the initial position image that is farthest from the camera, Indicates the feature of the end point on the jth lane line in the road monitoring image that is closer to the camera. Represents the feature of the endpoint on the jth lane line in the road monitoring image that is farthest from the camera.

4. A road monitoring video abnormal behavior detection system according to claim 3, characterized in that: The calculation formula for calculating the horizontal displacement of the camera through the matched lane line is: Where d represents the horizontal displacement of the camera, n represents the number of lane lines, Indicates the pixel coordinate of the end point of the i-th lane line in the initial position image that is closer to the camera on the pixel coordinate system v-axis of the initial position image, Represents the pixel coordinate of the end point of the i-th lane line in the initial position image that is farther from the camera on the pixel coordinate system v-axis of the initial position image, represents the pixel coordinate of the end point of the i-th lane line in the road monitoring image that is closer to the camera on the pixel coordinate system v-axis of the road monitoring image, Represents the pixel coordinates of the endpoint farther from the camera of the i-th lane line corresponding to the road monitoring image on the v-axis of the pixel coordinate system of the road monitoring image.

5. A road monitoring video abnormal behavior detection system according to claim 4, characterized in that: The ground depth construction module reconstructs the posture change matrix to obtain a ground depth map, including: The camera coordinate system at the initial position is converted into the camera coordinate system after the transformed posture through the posture change matrix; Obtaining a ground depth map of the initial position image, establishing a three-dimensional coordinate system, and reversely projecting the pixel points of the ground depth map of the initial position image into the three-dimensional coordinate system to obtain initial three-dimensional coordinate points; The initial three-dimensional coordinate point is converted into the camera coordinate system after the posture transformation through the posture change matrix to obtain the posture change point; Project the posture change point onto the road monitoring image to obtain the pixel coordinates in the road monitoring image under the current camera posture; The depth value is calculated by the pixel coordinates, and the ground depth map is reconstructed by the depth value.

6. A road monitoring video abnormal behavior detection system according to claim 1, characterized in that: The calculation formula of the fusion weight of the depth interval is: Among them, λ represents the fusion weight sequence of the depth interval, λ i represents the fusion weight of the depth interval, p represents the total number of ground contact points, and p i It represents the number of ground contact points within the depth interval, i represents the sequence number of the depth interval, and c represents the total number of depth intervals after discretization.

7. A method for detecting abnormal behavior in road surveillance video, comprising the following steps: Constructing a road surveillance video abnormal behavior detection system based on any one of claims 1-6; Collect road surveillance videos and input road surveillance images; Extracting a feature map of the road monitoring image through a feature extraction module; The camera posture detection module estimates the camera posture change through the feature map, and uses the camera posture change to calculate the posture change matrix. The ground depth construction module reconstructs the posture change matrix to obtain the ground depth map. The ground depth fusion module fuses the feature map and the ground depth map to obtain a fused depth feature map, including: inputting the feature map into the ground contact point detection network of the ground depth fusion module to obtain the ground contact points; projecting the ground contact points into the ground depth map, discretizing the ground depth map into depth intervals, calculating the fusion weights of the depth intervals, and superimposing the depth intervals and the fusion weights to obtain a depth sequence; inputting the feature map into the encoder of the self-attention mechanism to calculate the serialized features; inputting the serialized features and the depth sequence into the decoder of the self-attention mechanism to output the fused depth feature map; The fused depth feature map is detected by a 3D detector to obtain the detection result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the method according to claim 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program executes the method as claimed in claim 7 when executed by a processor.

Images