Vehicle climbing detection method based on segmentation model

Abstract

The invention relates to the technical field of target detection, in particular to a vehicle climbing detection method based on a segmentation model. The method comprises the following steps: S1, collecting a large amount of image data of different types of vehicles in a factory environment, and marking contours of the vehicles and pedestrians in combination with images in an open source COCO data set; s2, training a model by using an open source deep learning framework and labeled vehicle data; and S3, decoding the input video stream by using an H264 decoder, and performing acceleration processing in combination with hardware. S4, processing continuous frames in the video by using a deep learning optical flow model RAFT; s5, using a YOLOv8s vehicle detection model to detect vehicle and pedestrian targets in the video frame, and calculating the intersection-to-union ratio of the bounding rectangle of the moving target to the vehicle rectangular frame; s6, solving circumscribed rectangles of the vehicles and the pedestrians in the static state, and sending pictures intercepted from the rectangles into a YOLOv81 segmentation model for segmenting masks of the vehicles and the pedestrians; s7, judging a climbing behavior; and S8, result display and integration.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more specifically, to a vehicle climbing detection method based on a segmentation model. Background Technology

[0002] Vehicle climbing detection based on segmentation models is a technique that utilizes deep learning, particularly convolutional neural networks (CNNs), to accurately identify and locate vehicles and their climbing behavior. The core of this method is to train a deep neural network to understand image content and assign each pixel in the image to a predefined category, such as vehicle, climber, or other background. Furthermore, vehicle climbing detection is a complex security monitoring task, requiring the system to accurately identify the vehicle's outline from a video stream and determine whether climbing behavior has occurred.

[0003] Existing vehicle climbing detection methods based on segmentation models are usually trained on image data acquired in daytime scenes. However, the lighting conditions in nighttime scenes are different from those in daytime scenes, which may lead to poor image segmentation results in nighttime scenes. In real-world environments, vehicle climbing scenarios are highly variable, and a single dataset is unlikely to cover all possible situations. Therefore, this paper proposes a vehicle climbing detection method based on a segmentation model. Summary of the Invention

[0004] The purpose of this invention is to provide a vehicle climbing detection method based on a segmentation model, in order to address the variability of vehicle climbing scenarios in real-world environments, as mentioned in the background section, where a single dataset is insufficient to cover all possible situations.

[0005] To achieve the above objectives, the present invention aims to provide a method for invoking a data system interface via natural language, comprising the following steps:

[0006] S1. Collect a large amount of image data of different types of vehicles in the factory environment, and combine them with vehicle images in the open source COCO dataset to perform object detection and semantic segmentation annotation on these images, and annotate the outlines of vehicles and pedestrians.

[0007] S2. Train the model using an open-source deep learning framework and labeled vehicle data;

[0008] S3. Decode the input video stream using an H264 decoder and combine it with GPU hardware acceleration processing;

[0009] S4. Use the deep learning optical flow model RAFT to process consecutive frames in the video;

[0010] S5. Use the YOLOv8s vehicle detection model to detect vehicles in the video frame and the YOLOv9e detection model to detect pedestrian targets. Calculate the intersection-union ratio of the bounding rectangle of the moving target and the vehicle rectangle.

[0011] S6. For stationary vehicles and pedestrians, find their bounding rectangles. The images cropped from these rectangles are fed into the YOLOv8I segmentation model to segment the masks of vehicles and pedestrians.

[0012] S7. Use contour analysis to check the mask intersection to determine if there is any unauthorized climbing behavior;

[0013] S8. The results are integrated through the GstPlugin interface, and the segmentation model optimized by TensorRT is loaded through the nvinfer interface of DeepStream. The detection boxes and detection regions are displayed in conjunction with OSD, and finally displayed on the desktop through EGLISink.

[0014] As a further improvement to this technical solution, the specific steps involved in marking the outlines of vehicles and pedestrians in step S1 are as follows:

[0015] Target detection annotation:

[0016] To draw a bounding box around vehicles and pedestrians, denoted as BBox = x min y min x max y max ;

[0017] Among them, (x min y min (x) represents the coordinates of the top-left corner; max y max () represents the coordinates of the bottom right corner;

[0018] Semantic segmentation annotation:

[0019] Assign a category label to each pixel in the image, denoted as Mask = 0, 1;

[0020] In this context, 0 represents the background and 1 represents the target.

[0021] As a further improvement to this technical solution, the specific steps for training the model in S2 are as follows:

[0022] S3.1. Train the YOLOv8s network to obtain the vehicle detection model;

[0023] Training the YOLOv8s network includes:

[0024] Bounding box localization loss:

[0025] L1oc =ΣSmoothL1Loss(pred bbox -true bbox );

[0026] Among them, L loc This represents the bounding box localization loss; pred bbox Indicates the bounding box coordinates predicted by the model; true bbox Represents the actual bounding box coordinates;

[0027] Category probability loss:

[0028] L cls =ΣCrossEntropyLoss(pred cls true cls );

[0029] Among them, L cls Represents class probability loss; pred cls Indicates the target category predicted by the model; true cls This represents the true target category of the model;

[0030] Confidence loss:

[0031] L conf =ΣCrossEntropyLoss(pred conf true conf );

[0032] Among them, L conf Indicates confidence loss; pred conf This indicates the confidence that the bounding box predicted by the model contains the target; true. conf The label represents the true confidence level of the model;

[0033] Total loss:

[0034] L total =λ loc *L loc +λ cls *L cls +λ conf *L conf ;

[0035] Among them, L total λ represents the total loss. loc λ represents the bounding box localization loss coefficient. cls λ represents the class probability loss coefficient. conf Indicates the confidence loss coefficient;

[0036] Backpropagation:

[0037]

[0038] in, This represents the gradient operator, specifically the loss function L. total The gradient with respect to the model parameters θ; θ represents the set of parameters of the vehicle detection model; α represents the learning rate;

[0039] S3.2 Training the YOLOv8I network yields a binary classification semantic segmentation model used to distinguish between people and vehicles;

[0040] Training the YOLOv8I network includes:

[0041] Loss function:

[0042] L seg =∑CrossEntropyLoss(pred mask true mask );

[0043] Among them, L seg Represents the loss function; pred mask The mask representing the model's prediction; true mask The mask that represents the true value;

[0044] Backpropagation:

[0045]

[0046] in, This represents the gradient operator, specifically the loss function L. seg The gradient with respect to the model parameters θ1; θ1 represents the set of parameters of the semantic segmentation model; α represents the learning rate;

[0047] S3.3. Use the open-source YOLOv9e detection model as the pedestrian detection model;

[0048] The YOLOv9e detection model is as follows:

[0049] Bounding box localization loss:

[0050] L loc1 =ΣSmoothL1Loss(pred bbox1 -true bbox1 );

[0051] Among them, L loc1 This represents the bounding box localization loss; pred bbox1 Indicates the bounding box coordinates predicted by the model; true bbox1 Represents the actual bounding box coordinates;

[0052] Category probability loss:

[0053] L cls1 =ΣCrossEntropyLoss(pred cls1 true cls1 );

[0054] Among them, L cls1 Represents class probability loss; pred cls1 Indicates the target category predicted by the model; true cls1 This represents the true target category of the model;

[0055] Confidence loss:

[0056] L conf1 =ΣCrossEntropyLoss(pred conf1 true conf1 );

[0057] Among them, L conf1 Indicates confidence loss; pred conf1 This indicates the confidence that the bounding box predicted by the model contains the target; true. conf1 The label represents the true confidence level of the model;

[0058] Total loss:

[0059] L total1 =λ loc1 *L loc1 +λ cls1 *L cls1 +λ conf1 *L conf1 ;

[0060] Among them, L total1 λ represents the total loss; loc1 λ represents the bounding box localization loss coefficient. cls1 λ represents the class probability loss coefficient. conf1 Indicates the confidence loss coefficient;

[0061] Backpropagation:

[0062]

[0063] in, This represents the gradient operator, specifically the loss function L. total1 The gradient of the model parameter θ2; θ2 represents the set of parameters of the pedestrian detection model; α represents the learning rate.

[0064] As a further improvement to this technical solution, the detailed steps in S3 are as follows:

[0065] S4.1 First, you need to create an H264 decoder instance and specify the format of the input data for it;

[0066] The S4.2 H264 decoder will read compressed H264 data from the network and local files;

[0067] The S4.3 and H264 decoders will parse the NAL units in the input data and extract the encoded data of each image.

[0068] S4.4. The encoded data is obtained and decoded into the original image using GPU hardware acceleration technology;

[0069] GPU hardware acceleration technologies include:

[0070] Transform coding:

[0071]

[0072] Where X[u][v] represents the frequency domain coefficients obtained after DCT exchange; α(u)α(v) represents the normalization factor; x[x][y] represents the spatial domain element value of the original image; N represents the size of the image block; and coS represents the cosine function.

[0073] Quantification:

[0074]

[0075] Where Q[u][v] represents the quantized frequency domain coefficients; round represents the rounding function; Q factor Indicates the quantification factor;

[0076] Inverse transform coding:

[0077]

[0078] Where, x[x][y] 逆 Y[u][v] represents the frequency domain coefficients obtained after inverse DCT transformation; Y[u][v] represents the frequency domain coefficients after quantization.

[0079] The S4.5 and H264 decoders will output the decoded image.

[0080] As a further improvement to this technical solution, the specific steps involved in processing consecutive frames in the video in step S4 are as follows:

[0081] S5.1 Calculate the motion amplitude of each pixel:

[0082]

[0083] Where M(x, y) represents the motion amplitude of pixel (x, y), and Vx V represents the component of the optical flow vector in the x-direction; y This represents the component of the optical flow vector in the y-direction;

[0084] S5.2 Extract target contours with a motion amplitude greater than 1 unit;

[0085] S5.3, Find its circumscribed rectangle:

[0086] Circumscribed rectangle side length:

[0087] width = x max -x min ;

[0088] height = y max -y min ;

[0089] Where width represents the width of the bounding rectangle; height represents the height of the bounding rectangle; x min x represents the minimum value of x among all contour points; max The x-value represents the maximum value among all contour points; y max This represents the minimum value of y among all contour points; y min This represents the maximum value of y among all contour points;

[0090] The coordinates of the four vertices of the circumscribed rectangle are: A = (x... min y min B = (x) max y min C = (x) max y max D=(x min y max ).

[0091] As a further improvement to this technical solution, the specific steps for extracting the target contour in S5.2 are as follows:

[0092] S5.21. Set a threshold T, T = 1;

[0093] S5.22. Traverse all pixels and extract those pixels whose motion amplitude is greater than the threshold.

[0094] S5.23, Form the outline of the target C = (x, y) | M(x, y) > T;

[0095] Where C represents the coordinates of all pixels that meet the conditions.

[0096] As a further improvement to this technical solution, the detailed steps in S5 are as follows:

[0097] S6.1 Load the YOL0v8s model and use it to perform object detection on the input image to obtain the bounding box set (BBox) for vehicles and pedestrians. car and BBox person ;

[0098] S6.2 Calculate the moving target rectangle:

[0099]

[0100] w = x2 - x1;

[0101] h = y2 - y1;

[0102] Where (x1, y1) represents the coordinates of the top left corner of the rectangle; (x2, y2) represents the coordinates of the bottom right corner of the rectangle; (cx, cy) represents the coordinates of the center point; w represents the width of the rectangle; and h represents the height of the rectangle.

[0103] S6.3 Calculate the intersection-union ratio of the vehicle rectangle and the target rectangle;

[0104] S6.4 Determine the vehicle status. Vehicles with a crossover ratio greater than 0.7 are considered to be in a moving state; otherwise, they are considered to be in a stationary state.

[0105] As a further improvement to this technical solution, the specific steps for segmenting the masks for vehicles and pedestrians in step S6 are as follows:

[0106] S7.1 Detect the circumscribed rectangle and obtain the coordinates of the circumscribed rectangle for vehicles and pedestrians;

[0107] The specific coordinates of the bounding rectangles for vehicles and pedestrians are as follows:

[0108] The coordinates of the vehicle's circumscribed rectangle are: (a1, b1, a2, b2);

[0109] The coordinates of the circumscribed rectangle for the pedestrian are: (c1, d1, c2, d2).

[0110] Where (a1, b1) represents the coordinates of the upper left corner of the vehicle's bounding rectangle; (a2, b2) represents the coordinates of the lower right corner of the vehicle's bounding rectangle; (c1, d1) represents the coordinates of the upper left corner of the pedestrian's bounding rectangle; and (c2, d2) represents the coordinates of the lower right corner of the pedestrian's bounding rectangle.

[0111] S7.2 For each detected bounding box, extract the corresponding region from the original image;

[0112] cropped image [i] = original image [y min [i]:y max [i], x min[i]:x max [i]];

[0113] Among them, cropped image [i] represents the cropped image patch of the i-th object; original image Represents the original image; (x min [i], y min [i]) represents the coordinates of the top-left corner of the detected object: (x max [i], y max [i]) represents the coordinates of the bottom right corner of the object being detected;

[0114] S7.3, Preprocessing the image;

[0115] preprocessed image [i]=normalize(resize(cropped image [i]));

[0116] Among them, preprocessed image [i] represents the i-th image patch after preprocessing; normalize() is a function that normalizes the pixel values ​​of an image; resize() is a function that resizes an image to a specified size;

[0117] S7.4 Input the preprocessed image blocks into the YOLOv81 segmentation model, and the YOLOv81 segmentation model outputs the segmentation mask for each image block;

[0118] The YOLOv8I segmentation model is specifically as follows:

[0119] The YOLOv8I model employs a single-stage detection method, directly inputting images into a neural network and outputting information such as the location, category, and confidence score of predicted bounding boxes. This model extracts features from the image through multiple operations including convolutional and pooling layers, then converts the feature maps into candidate bounding boxes, and trains the model based on the IoU value between the candidate boxes and the ground truth boxes.

[0120] S7.5 Post-process the segmentation mask output by the YOLOv81 segmentation model;

[0121] The post-processing specifically includes:

[0122] postprocessed mask[i] =postprocess(mask[i]);

[0123] Among them, postprocessed mask[i]`mask[i]` represents the i-th segmentation mask after post-processing; `mask[i]` represents the segmentation mask of the i-th image patch; `postprocess()` represents the post-processing operation function for a segmentation mask.

[0124] S7.6. Overlay the segmentation mask back onto the original image or the cropped image patch and analyze the segmentation results.

[0125] As a further improvement to this technical solution, the specific steps for determining whether unauthorized climbing behavior exists in S7 are as follows:

[0126] S8.1 Check if there is a union between the masks for vehicles and pedestrians:

[0127]

[0128] Where Intersection represents the number of non-zero elements in the intersection of the two masks; Mask vehicle [i] represents Mask vehicle The i-th element in the array; Mask pedestrian [i] represents Mask pedestrian The i-th element in the array; ∧ represents the logical AND operation; This represents the summation of all elements i from 0 to n-1; n represents the total number of elements in the array.

[0129] S8.2 Check whether the alarm key points set by the pedestrian rectangle are located within the outline of the vehicle mask;

[0130] The outline of the vehicle mask:

[0131] C vehicle =[(a1,b1),(a2,b2),...,(a n b n )];

[0132] Among them, C vehicle The set of outline points representing the vehicle mask;

[0133] Check if key points are located within the vehicle's outline:

[0134]

[0135] Where IsInside represents a variable; its value is 1 if the keypoint is inside the outline, and 0 otherwise; (x key y key () represents the coordinates of the key points of the pedestrian rectangle; A function indicating whether a point is inside the contour;

[0136] S8.3 Check if the y-value of the top left corner of the pedestrian rectangle is less than the y-value of the top left corner of the vehicle rectangle:

[0137]

[0138] in, This represents the y-value of the top-left corner of the pedestrian rectangle; This represents the y-value of the top-left corner of the vehicle's bounding box.

[0139] S8.4 If all three conditions above are met, it is determined that the pedestrian has engaged in illegal climbing behavior, and an alarm is triggered.

[0140] As a further improvement to this technical solution, the result display and integration in S8 are specifically as follows:

[0141] S9.1. Load the segmentation model optimized by TensorRT using DeepStream's nvinfer interface, and display the detection box and detection region using OSD.

[0142] TensorRT optimizations include:

[0143] Quantification:

[0144]

[0145] Where q represents the quantized integer value; round() represents the function that performs rounding; x represents the original floating-point number; S represents the scaling factor; and Z represents the zero point.

[0146] Inverse quantization:

[0147] x′=(qZ)×S;

[0148] Where x′ represents the floating-point number converted back from the quantized integer value;

[0149] Calculation of scaling factor and zero point:

[0150]

[0151] Among them, S act The scaling factor representing the activation value; max act This represents the maximum value of the activation value; min act Z represents the minimum activation value. act Indicates the zero point of the activation value; N represents the quantization precision;

[0152] Batch normalized parameters are fused into convolutional weights:

[0153]

[0154] Among them, newweight Represents the new weight matrix; weight represents the weight matrix of the original convolutional layer; new bias γ represents the new bias term; bias represents the bias term of the original convolutional layer; γ represents the scaling factor of the batch normalization layer; β represents the translation factor of the batch normalization layer; ε represents a constant; mean represents the mean of the batch data calculated during training.

[0155] Computational cost after layer fusion:

[0156]

[0157] Among them, compute fused The computational cost after layer fusion is represented by M; the width of the input feature map is represented by M; and the height of the input feature map is represented by N. Indicates the size of the convolution kernel;

[0158] S9.2, Use EGL Sink for desktop display;

[0159] Among them, EGLSink:

[0160]

[0161] Among them, Rendered Frame Output represents the rendered video frame. Frame This represents the video frame output from the inference module; This represents the detection results plotted on a video frame; This represents the final output after rendering using EGL Sink; Sink This indicates that video frames will be rendered onto the display device.

[0162] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0163] 1. In this vehicle climbing detection method based on a segmentation model, the outlines of vehicles and pedestrians are marked by object detection and semantic segmentation of image data to facilitate subsequent data processing. The model is trained using an open-source deep learning framework and labeled vehicle data. A vehicle detection model is obtained by training a YOLOv8s network, and a binary classification semantic segmentation model is obtained by training a YOLOv8I network to distinguish between people and vehicles. The open-source YOLOv9e detection model is used as the pedestrian detection model. By separating different models, it is possible to achieve good detection even in changing environments.

[0164] 2. In this vehicle climbing detection method based on a segmentation model, the results are integrated through the GstPlugin interface, and the segmentation model optimized by TensorRT is loaded through the nvinfer interface of DeepStream. The detection box and detection area are displayed in conjunction with OSD, and finally displayed on the desktop through EGLISink, which ensures good detection results, improves alarm efficiency, and avoids accidents. Attached Figure Description

[0165] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0166] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0167] Example:

[0168] Please see Figure 1 As shown, this embodiment provides a vehicle climbing detection method based on a segmentation model, including the following steps:

[0169] S1. Collect a large amount of image data of different types of vehicles in the factory environment, and combine them with vehicle images in the open source COCO dataset to perform object detection and semantic segmentation annotation on these images, and annotate the outlines of vehicles and pedestrians.

[0170] In this example, the specific steps involved in marking the outlines of vehicles and pedestrians are as follows:

[0171] Target detection annotation:

[0172] To draw a bounding box around vehicles and pedestrians, denoted as BBox = x min y min x max y max ;

[0173] Among them, (x min y min (x) represents the coordinates of the top-left corner; max y max () represents the coordinates of the bottom right corner;

[0174] Semantic segmentation annotation:

[0175] Assign a category label to each pixel in the image, denoted as Mask = 0, 1;

[0176] In this context, 0 represents the background and 1 represents the target.

[0177] S2. Train the model using an open-source deep learning framework and labeled vehicle data;

[0178] In this example, the specific steps for training the model are as follows:

[0179] S3.1. Train the YOLOv8s network to obtain the vehicle detection model;

[0180] Training the YOLOv8s network includes:

[0181] Bounding box localization loss:

[0182] L loc =ΣSmoothL1Loss(pred bbox -true bbox );

[0183] Among them, L loc This represents the bounding box localization loss; pred bbox Indicates the bounding box coordinates predicted by the model; true bbox Represents the actual bounding box coordinates;

[0184] Category probability loss:

[0185] L cls =ΣCrossEntropyLoss(pred cls true cls );

[0186] Among them, L cls Represents class probability loss; pred cls Indicates the target category predicted by the model; true cls This represents the true target category of the model;

[0187] Confidence loss:

[0188] L conf =ΣCrossEntropyLoss(pred conf trUe conf );

[0189] Among them, L conf Indicates confidence loss; pred conf This indicates the confidence that the bounding box predicted by the model contains the target; true. conf The label represents the true confidence level of the model;

[0190] Total loss:

[0191] Ltotal =λ loc *L loc +λ cls *L cls +λ conf *L conf ;

[0192] Among them, L total λ represents the total loss; loc λ represents the bounding box localization loss coefficient. cls λ represents the class probability loss coefficient. conf Indicates the confidence loss coefficient;

[0193] Backpropagation:

[0194]

[0195] in, This represents the gradient operator, specifically the loss function L. total The gradient with respect to the model parameters θ; θ represents the set of parameters of the vehicle detection model; α represents the learning rate;

[0196] S3.2 Training the YOLOv8I network yields a binary classification semantic segmentation model used to distinguish between people and vehicles;

[0197] Training the YOLOv8I network includes:

[0198] Loss function:

[0199] L seg =∑CrossEntropyLoss(pred mask true mask );

[0200] Among them, L seg Represents the loss function; pred mask The mask representing the model's prediction; true mask The mask that represents the true value;

[0201] Backpropagation:

[0202]

[0203] in, This represents the gradient operator, specifically the loss function L. seg The gradient with respect to the model parameters θ1; θ1 represents the set of parameters of the semantic segmentation model; α represents the learning rate;

[0204] S3.3. Use the open-source YOLOv9e detection model as the pedestrian detection model.

[0205] The YOLOv9e detection model is as follows:

[0206] Bounding box localization loss:

[0207] L loc1 =ΣSmoothL1Loss(pred bbox1 -true bbox1 );

[0208] Among them, L loc1 This indicates the bounding box localization loss; pred bbox1 Indicates the bounding box coordinates predicted by the model; true bbox1 Represents the actual bounding box coordinates;

[0209] Category probability loss:

[0210] L cls1 =ΣCrossEntropyLoss(pred cls1 true cls1 );

[0211] Among them, L cls1 Represents class probability loss; pred cls1 Indicates the target category predicted by the model; true cls1 This represents the true target category of the model;

[0212] Confidence loss:

[0213] L conf1 =ΣCrossEntropyLoss(pred conf1 true conf1 );

[0214] Among them, L conf1 Indicates confidence loss; pred conf1 This indicates the confidence that the bounding box predicted by the model contains the target; true. conf1 The label represents the true confidence level of the model;

[0215] Total loss:

[0216] L total1 =λ loc1 *L loc1 +λ cls1 *L cls1 +λ conf1 *L conf1 ;

[0217] Among them, L total1 λ represents the total loss; loc1 λ represents the bounding box localization loss coefficient. cls1 λ represents the class probability loss coefficient.conf1 Indicates the confidence loss coefficient;

[0218] Backpropagation:

[0219]

[0220] in, This represents the gradient operator, specifically the loss function L. total1 The gradient of the model parameter θ2; θ2 represents the set of parameters of the pedestrian detection model; α represents the learning rate.

[0221] S3. Decode the input video stream using an H264 decoder and combine it with GPU hardware acceleration processing;

[0222] In this example, the detailed steps in S3 are as follows:

[0223] S4.1 First, you need to create an H264 decoder instance and specify the format of the input data for it;

[0224] The S4.2 H264 decoder will read compressed H264 data from the network and local files;

[0225] The S4.3 and H264 decoders will parse the NAL units in the input data and extract the encoded data of each image.

[0226] S4.4. The encoded data is obtained and decoded into the original image using GPU hardware acceleration technology;

[0227] GPU hardware acceleration technologies include:

[0228] Transform coding:

[0229]

[0230] Where X[u][v] represents the frequency domain coefficients obtained after DCT exchange; α(u)α(v) represents the normalization factor; x[x][y] represents the spatial domain element value of the original image; N represents the size of the image patch; and cos represents the cosine function.

[0231] Quantification:

[0232]

[0233] Where Q[u][v] represents the quantized frequency domain coefficients; round represents the rounding function; Q factor Indicates the quantification factor;

[0234] Inverse transform coding:

[0235]

[0236] Where, x[x][y] 逆 Y[u][v] represents the frequency domain coefficients obtained after inverse DCT transformation; Y[u][v] represents the frequency domain coefficients after quantization.

[0237] The S4.5 and H264 decoders will output the decoded image.

[0238] Specifically, an H.264 decoder is a software or hardware tool used to decode the H.264 video encoding format. H.264, also known as MPEG-4 Part 10 or AVC (Advanced Video Coding), is a widely used video compression standard that offers a high data compression ratio while maintaining good video quality. This makes H.264 the preferred video encoding format for applications such as digital television broadcasting, video conferencing, online video streaming, and multimedia messaging.

[0239] An H.264 decoder works by receiving H.264 encoded video data and converting it back into a visual video signal. The decoding process involves multiple steps, such as decompression, motion compensation, transform, and entropy coding, to reconstruct the original video content. While the H.264 standard defines the decoding method, it allows manufacturers the freedom to develop competitive decoders, provided they achieve the same results as those used in the standard.

[0240] S4. Use the deep learning optical flow model RAFT to process consecutive frames in the video;

[0241] The RAFT optical flow model is applicable to a variety of computer vision tasks, including action recognition, video inpainting, frame interpolation, 3D reconstruction, and synthesis. Due to its efficiency and accuracy, the RAFT model is extremely useful in practical high-resolution optical flow applications.

[0242] In this example, the specific steps involved in processing consecutive frames in a video are as follows:

[0243] S5.1 Calculate the motion amplitude of each pixel:

[0244]

[0245] Where M(x, y) represents the motion amplitude of pixel (x, y), and V x V represents the component of the optical flow vector in the x-direction; y This represents the component of the optical flow vector in the y-direction;

[0246] S5.2 Extract target contours with a motion amplitude greater than 1 unit;

[0247] In this example, the specific steps for extracting the target contour are as follows:

[0248] S5.21. Set a threshold T, T = 1;

[0249] S5.22. Traverse all pixels and extract those pixels whose motion amplitude is greater than the threshold.

[0250] S5.23, Form the outline of the target C = (x, y) | M(x, y) > T;

[0251] Where C represents the coordinates of all pixels that meet the conditions.

[0252] S5.3, Find its circumscribed rectangle:

[0253] Circumscribed rectangle side length:

[0254] width = x max -x min ;

[0255] height = y max -y min ;

[0256] Where width represents the width of the bounding rectangle; height represents the height of the bounding rectangle; x min x represents the minimum value of x among all contour points; max The x-value represents the maximum value among all contour points; y max This represents the minimum value of y among all contour points; y min This represents the maximum value of y among all contour points;

[0257] The coordinates of the four vertices of the circumscribed rectangle are: A = (x... min y min B = (x) max y min C = (x) max y max D=(x min y max ).

[0258] S5. Use the YOL0v8s vehicle detection model to detect vehicles in the video frame and the YOL0v9e detection model to detect pedestrian targets. Calculate the intersection-union ratio of the bounding rectangle of the moving target and the vehicle rectangle.

[0259] In this example, the detailed steps in S5 are as follows:

[0260] S6.1 Load the YOL0v8s model and use it to perform object detection on the input image to obtain the bounding box set (BBox) for vehicles and pedestrians. car and BBoxperson ;

[0261] S6.2 Calculate the moving target rectangle:

[0262]

[0263] w = x2 - x1;

[0264] h = y2 - y1;

[0265] Where (x1, y1) represents the coordinates of the top left corner of the rectangle; (x2, y2) represents the coordinates of the bottom right corner of the rectangle; (cx, cy) represents the coordinates of the center point; w represents the width of the rectangle; and h represents the height of the rectangle.

[0266] S6.3 Calculate the intersection-union ratio of the vehicle rectangle and the target rectangle;

[0267] S6.4 Determine the vehicle status. Vehicles with a crossover ratio greater than 0.7 are considered to be in a moving state; otherwise, they are considered to be in a stationary state.

[0268] S6. For stationary vehicles and pedestrians, find their bounding rectangles. The images cropped from these rectangles are fed into the YOLOv81 segmentation model to segment the masks of vehicles and pedestrians.

[0269] In this example, the specific steps for segmenting vehicles and pedestrians using masks are as follows:

[0270] S7.1 Detect the circumscribed rectangle and obtain the coordinates of the circumscribed rectangle for vehicles and pedestrians;

[0271] The specific coordinates of the bounding rectangles for vehicles and pedestrians are as follows:

[0272] The coordinates of the vehicle's circumscribed rectangle are: (a1, b1, a2, b2);

[0273] The coordinates of the circumscribed rectangle for the pedestrian are: (c1, d1, c2, d2).

[0274] Where (a1, b1) represents the coordinates of the upper left corner of the vehicle's bounding rectangle; (a2, b2) represents the coordinates of the lower right corner of the vehicle's bounding rectangle; (c1, d1) represents the coordinates of the upper left corner of the pedestrian's bounding rectangle; and (c2, d2) represents the coordinates of the lower right corner of the pedestrian's bounding rectangle.

[0275] S7.2 For each detected bounding box, extract the corresponding region from the original image;

[0276] cropped image [i] = original image [y min [i]:y max [i], xmin [i]:x max [i]];

[0277] Among them, cropped image [i] represents the cropped image patch of the i-th object; original image Represents the original image; (x min [i], y min [i]) represents the coordinates of the top-left corner of the detected object; (x max [i], y max [i]) represents the coordinates of the bottom right corner of the object being detected;

[0278] S7.3, Preprocessing the image;

[0279] preprocessed image [i]=normalize(resize(cropped image [i]));

[0280] Among them, preprocessed image [i] represents the i-th image patch after preprocessing; normalize() is a function that normalizes the pixel values ​​of an image; resize() is a function that resizes an image to a specified size;

[0281] S7.4 Input the preprocessed image blocks into the YOLOv8I segmentation model, and the YOLOv8I segmentation model outputs the segmentation mask for each image block;

[0282] The YOLOv8I segmentation model is specifically as follows:

[0283] The YOLOv8I model employs a single-stage detection method, directly inputting images into a neural network and outputting information such as the location, category, and confidence score of predicted bounding boxes. This model extracts features from the image through multiple operations including convolutional and pooling layers, then converts the feature maps into candidate bounding boxes, and trains the model based on the IoU value between the candidate boxes and the ground truth boxes.

[0284] S7.5 Post-process the segmentation mask output by the YOLOv8I segmentation model;

[0285] The post-processing specifically includes:

[0286] postprocessed mask[i] =postprocess( mask [i]);

[0287] Among them, postprocessed mask[i]`mask[i]` represents the i-th segmentation mask after post-processing; `mask[i]` represents the segmentation mask of the i-th image patch; `postprocess()` represents the post-processing operation function for a segmentation mask.

[0288] S7.6. Overlay the segmentation mask back onto the original image or the cropped image patch and analyze the segmentation results.

[0289] S7. Use contour analysis to check the mask intersection to determine if there is any unauthorized climbing behavior;

[0290] S8. The results are integrated through the GstPlugin interface, and the segmentation model optimized by TensorRT is loaded through the nvinfer interface of DeepStream. The detection boxes and detection regions are displayed in conjunction with OSD, and finally displayed on the desktop through EGLISink.

[0291] In this example, the results are displayed and integrated as follows:

[0292] S9.1. Load the segmentation model optimized by TensorRT using DeepStream's nvinfer interface, and display the detection box and detection region using OSD.

[0293] TensorRT optimizations include:

[0294] Quantification:

[0295]

[0296] Where q represents the quantized integer value; round() represents the function that performs rounding; x represents the original floating-point number; S represents the scaling factor; and Z represents the zero point.

[0297] Inverse quantization:

[0298] x′=(qZ)×S;

[0299] Where x′ represents the floating-point number converted back from the quantized integer value;

[0300] Calculation of scaling factor and zero point:

[0301]

[0302] Among them, S act The scaling factor representing the activation value; max act This represents the maximum value of the activation value; min act Z represents the minimum activation value. act Indicates the zero point of the activation value; N represents the quantization precision;

[0303] Batch normalized parameters are fused into convolutional weights:

[0304]

[0305] Among them, new weight Represents the new weight matrix; weight represents the weight matrix of the original convolutional layer; new bias represents the new bias term; bias represents the bias term of the original convolutional layer; Y represents the scaling factor of the batch normalization layer; β represents the translation factor of the batch normalization layer; ε represents a constant; mean represents the mean of the batch data calculated during training.

[0306] Computational cost after layer fusion:

[0307]

[0308] Among them, compute fused The computational cost after layer fusion is represented by M; the width of the input feature map is represented by M; and the height of the input feature map is represented by N. Indicates the size of the convolution kernel;

[0309] S9.2, Use EGL Sink for desktop display;

[0310] Among them, EGLSink:

[0311]

[0312] Among them, Rendered Frame Output represents the rendered video frame. Frame This represents the video frame output from the inference module; This represents the detection results plotted on a video frame; This represents the final output after rendering using EGL Sink; Sink This indicates that video frames will be rendered onto the display device.

[0313] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle climbing detection method based on a segmentation model, characterized in that: Includes the following steps: S1. Collect a large amount of image data of different types of vehicles in the factory environment, and combine them with vehicle images in the open source COCO dataset to perform object detection and semantic segmentation annotation on these images, and annotate the outlines of vehicles and pedestrians. S2. Train the model using an open-source deep learning framework and labeled vehicle data; S3. Decode the input video stream using an H264 decoder and combine it with GPU hardware acceleration processing; S4. Use the deep learning optical flow model RAFT to process consecutive frames in the video; S5. Use the YOLOv8s vehicle detection model to detect vehicles in the video frame and the YOLOv9e detection model to detect pedestrian targets. Calculate the intersection-union ratio of the bounding rectangle of the moving target and the vehicle rectangle. S6. For stationary vehicles and pedestrians, find their bounding rectangles. The images cropped from these rectangles are fed into the YOLOv8I segmentation model to segment the masks of vehicles and pedestrians. S7. Check the mask intersection based on the contour analysis method to determine whether there is any illegal climbing behavior; S8. The results are integrated through the GstPlugin interface, and the segmentation model optimized by TensorRT is loaded through the nvinfer interface of DeepStream. The detection boxes and detection regions are displayed in conjunction with OSD, and finally displayed on the desktop through EGLISink.

2. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In step S1, the specific steps involved in marking the outlines of vehicles and pedestrians are as follows: Target detection annotation: To draw a bounding box around vehicles and pedestrians, denoted as BBox = x min ,y min x max y max ; Among them, (x min y min (x) represents the coordinates of the top-left corner; (x) max y max () represents the coordinates of the bottom right corner; Semantic segmentation annotation: Assign a category label to each pixel in the image, denoted as Mask = 0, 1; In this context, 0 represents the background and 1 represents the target.

3. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In S2, the specific steps for training the model are as follows: S3.

1. Train the YOLOv8s network to obtain the vehicle detection model; S3.2 Training the YOLOv8 I network yields a binary classification semantic segmentation model used to distinguish between people and vehicles; S3.

3. Use the open-source YOLOv9e detection model as the pedestrian detection model.

4. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: The detailed steps in S3 are as follows: S4.1 First, you need to create an H264 decoder instance and specify the format of the input data for it; The S4.2 H264 decoder will read compressed H264 data from the network and local files; The S4.3 and H264 decoders will parse the NAL units in the input data and extract the encoded data of each image. S4.

4. The encoded data is obtained and decoded into the original image using GPU hardware acceleration technology; The S4.5 and H264 decoders will output the decoded image.

5. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In step S4, the specific steps involved in processing consecutive frames in the video are as follows: S5.1 Calculate the motion amplitude of each pixel: Where M(x, y) represents the motion amplitude of pixel (x, y), and V x V represents the component of the optical flow vector in the x-direction; y This represents the component of the optical flow vector in the y-direction; S5.2 Extract target contours with a motion amplitude greater than 1 unit; S5.3, Find its circumscribed rectangle: Circumscribed rectangle side length: width=x max -x min ; height=y max -and min ; Where width represents the width of the bounding rectangle; height represents the height of the bounding rectangle; x min x represents the minimum value of x among all contour points; max The x-value represents the maximum value among all contour points; y max ymi represents the minimum value of y among all contour points. n This represents the maximum value of y among all contour points; The coordinates of the four vertices of the circumscribed rectangle are: A = (x... min ,y min B = (x) max ,y min C = (x) max y max D=(x min ,y max ).

6. The vehicle climbing detection method based on a segmentation model according to claim 5, characterized in that: In step S5.2, the specific steps for extracting the target contour are as follows: S5.

21. Set a threshold T, T = 1; S5.

22. Traverse all pixels and extract those pixels whose motion amplitude is greater than the threshold. S5.23, Form the outline of the target C = (x, y) | M(x, y) > T; Where C represents the coordinates of all pixels that meet the conditions.

7. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: The detailed steps in S5 are as follows: S6.1 Load the YOL0v8s model and use it to perform object detection on the input image to obtain the bounding box set (BBox) for vehicles and pedestrians. car and BBox person ; S6.2 Calculate the target rectangle to be moved; S6.3 Calculate the intersection-union ratio of the vehicle rectangle and the target rectangle; S6.4 Determine the vehicle status. Vehicles with a crossover ratio greater than 0.7 are considered to be in a moving state; otherwise, they are considered to be in a stationary state.

8. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In step S6, the specific steps for segmenting the masks for vehicles and pedestrians are as follows: S7.1 Detect the circumscribed rectangle and obtain the coordinates of the circumscribed rectangle for vehicles and pedestrians; S7.2 For each detected bounding box, extract the corresponding region from the original image: S7.3, Preprocessed Image: S7.4 Input the preprocessed image blocks into the YOLOv81 segmentation model, and the YOLOv81 segmentation model outputs the segmentation mask for each image block; S7.5 Post-process the segmentation mask output by the YOLOv81 segmentation model; S7.

6. Overlay the segmentation mask back onto the original image or the cropped image patch and analyze the segmentation results.

9. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In step S7, the specific steps for determining whether there is any unauthorized climbing behavior are as follows: S8.1 Check if there is a union between the masks for vehicles and pedestrians: Where Intersection represents the number of non-zero elements in the intersection of the two masks; Mask vehicle [i] represents Mask vehicle The i-th element in the array; Mask pedestrian [i] represents Mask pedestrian The i-th element in the array; ∧ represents the logical AND operation; This represents the summation of all elements i from 0 to n-1; n represents the total number of elements in the array. S8.2 Check whether the alarm key points set by the pedestrian rectangle are located within the outline of the vehicle mask; The outline of the vehicle mask: C vehicle =[(a1, b1), (a2, b2), ..., (a n b n )]; Among them, C vehicle The set of outline points representing the vehicle mask; Check if key points are located within the vehicle's outline: Where IsInside represents a variable; its value is 1 if the keypoint is inside the outline, and 0 otherwise; (z key ,y key () represents the coordinates of the key points of the pedestrian rectangle; A function indicating whether a point is inside the contour; S8.3 Check if the y-value of the top left corner of the pedestrian rectangle is less than the y-value of the top left corner of the vehicle rectangle: in, This represents the y-value of the top-left corner of the pedestrian rectangle; This represents the y-value of the top-left corner of the vehicle's bounding box. S8.4 If all three conditions above are met, it is determined that the pedestrian has engaged in illegal climbing behavior, and an alarm is triggered.

10. The vehicle climbing detection method based on a segmentation model according to claim 1, characterized in that: In S8, the result display and integration are specifically as follows: S9.

1. Load the segmentation model optimized by TensorRT using DeepStream's nvinfer interface, and display the detection bounding boxes and detection regions using OSD: TensorRT optimizations include: Quantification: Where q represents the quantized integer value; round() represents the function that performs rounding; x represents the original floating-point number; S represents the scaling factor; and Z represents the zero point. Inverse quantization: x′=(qZ)×S; Where x′ represents the floating-point number converted back from the quantized integer value; Calculation of scaling factor and zero point: Among them, S act The scaling factor representing the activation value; max act This represents the maximum value of the activation value; min act Z represents the minimum activation value. act Indicates the zero point of the activation value; N represents the quantization precision; Batch normalized parameters are fused into convolutional weights: Among them, new weight Represents the new weight matrix; weight represents the weight matrix of the original convolutional layer; new bias γ represents the new bias term; bias represents the bias term of the original convolutional layer; γ represents the scaling factor of the batch normalization layer; β represents the translation factor of the batch normalization layer; ε represents a constant; mean represents the mean of the batch data calculated during training. Computational cost after layer fusion: Among them, compute fused The computational cost after layer fusion is represented by M; the width of the input feature map is represented by M; and the height of the input feature map is represented by N. Indicates the size of the convolution kernel; S9.2, Use EGL Sink for desktop display; Among them, EGLSink: Among them, Rendered Frame Output represents the rendered video frame. Frame This represents the video frame output from the inference module; This represents the detection results plotted on a video frame; This represents the final output after rendering using EGL Sink; Sink This indicates that video frames will be rendered onto the display device.

Images