A key frame acquisition method for bus inspection videos

Abstract

The present invention provides a method that specifically includes: performing target detection on each frame of a bus inspection video using an improved YOLOX algorithm; removing invalid data based on the detection results obtained in step 1; obtaining a usable frame sequence based on the processed detection results obtained in step 2 using an available frame search algorithm; and inputting the usable frame sequence into a DGoogLeNet network to obtain key frames from the bus inspection video. The key frame acquisition algorithm proposed in the present invention can significantly improve the efficiency of anomaly detection for bus stop signs.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a key frame acquisition method for public transportation inspection videos. Background Art

[0002] With the rapid development of the field of artificial intelligence, computer vision technology has gradually been integrated into people's lives and is changing people's way of life, business, and production, providing stronger support for the development of all walks of life.

[0003] However, for enterprises, in the context of the big data era, how to process and utilize the massive data collected in production activities plays a vital role in improving enterprise efficiency.

[0004] In the video field, there is a lot of redundant information, which wastes a lot of computing power when searching for frames containing specific targets in the video. If key frames are extracted from the video, a lot of time can be saved.

[0005] With the rapid development of public transportation, a large number of smart bus stop signs have been damaged. Currently, most bus companies still rely on manual inspections. Since most buses are now equipped with cameras, they capture a large amount of bus stop sign video. Therefore, analyzing and processing bus inspection videos and identifying anomalies can improve company efficiency. This invention can help us obtain key frames containing bus stop signs in bus inspection videos, improving analysis and processing speed.

[0006] The emergence of computer vision technology has greatly improved the efficiency of production operations, avoided the accuracy of detection results affected by subjective judgment, and saved a lot of manpower. Summary of the Invention

[0007] In light of the shortcomings of existing technologies and the characteristics of bus inspection videos, the present invention aims to provide a method for acquiring key frames for specific scene videos. This method employs the following technical solutions: first, based on target detection technology (such as the YOLO algorithm), a frame search algorithm is used to acquire a sequence of available frames in the video; further, a convolutional neural network is used to search for key frames in the video. This method significantly improves the efficiency of analyzing and processing bus inspection videos during production.

[0008] The technical solution for achieving the purpose of the present invention is: a key frame acquisition method for bus inspection video, comprising the following steps:

[0009] Step 1: Use the improved YOLOX algorithm to detect targets in each frame of the bus inspection video;

[0010] Step 2: Remove invalid data from the test results obtained in step 1;

[0011] Step 3: Based on the processed detection result obtained in step 2, an available frame search algorithm is used to obtain an available frame sequence;

[0012] Step 4: Input the available frame sequence into the DGoogLeNet network to obtain the key frames in the bus inspection video.

[0013] A further improvement of the present invention is that in step 1, the loss function of YOLOX is replaced with the Focal Loss loss function. During the training process, the accuracy of the samples that are easy to classify in the data set can reach 99%, while the accuracy of the samples that are difficult to classify is very poor. Focal loss reduces the weight of the correctly classified samples instead of giving the same weight to all samples. This has the same effect as giving the training samples more difficult-to-classify samples. Therefore, we replace the loss function in YOLOX with the Focal Loss loss function;

[0014] A further improvement of the present invention is that in step 2, an intersection-over-union strategy is used to remove invalid detection results so as not to affect subsequent processing results.

[0015] A further improvement of the present invention is that an available frame search algorithm is proposed in step three, which classifies frames according to parameters of necessary parts of the detection results in step two, and uses the available frame sequence as input to the model in step four.

[0016] A further improvement of the present invention is that in step 4, GoogLeNet is used to construct a DGoogLeNet twin network to implement an image quality assessment model. The twin network shares weights, and after pre-training and fine-tuning with the IQA dataset, key frames are obtained with the help of the model.

[0017] The beneficial effects of the present invention are:

[0018] This paper creates a target detection dataset and an image quality assessment dataset based on bus inspection videos, trains target detection models and image quality assessment models, and proposes a keyframe acquisition method for bus inspection videos. This method retrieves keyframes from inspection videos through available frame search and image evaluation. Compared to methods that rely on manual inspection, this method significantly improves the efficiency of subsequent anomaly detection for bus stop signs, avoids factors such as subjective judgment by inspection workers, and achieves more accurate and rapid keyframe acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Attachment Figure 1 A schematic diagram of an available frame search algorithm of the present invention;

[0020] Attachment Figure 2 This is a flowchart of key frame acquisition in the present invention. DETAILED DESCRIPTION

[0021] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments and in conjunction with the accompanying drawings:

[0022] The technical solution proposed in the present invention is a key frame acquisition method for bus inspection videos, which specifically includes the following steps:

[0023] Step 1: Use the improved YOLOX algorithm to detect targets in each frame of the bus inspection video;

[0024] Step 2: Remove invalid data from the test results obtained in step 1;

[0025] Step 3: Based on the processed detection result obtained in step 2, an available frame sequence is obtained according to an available frame search algorithm;

[0026] Step 4: Input the available frame sequence into the DGoogLeNet network and obtain the key frames in the bus inspection video based on the network output.

[0027] Furthermore, the specific process of step one is as follows:

[0028] (1.1) Create a target detection dataset

[0029] Some frames were extracted from bus inspection videos and manually labeled. The labeled categories are bus_station, board_ok, board_wrong, box, leave, and ad, which represent the bus stop sign as a whole, the announcement area (normal), the announcement area (abnormal), the box area, the leaves, and the advertisements, respectively.

[0030] (1.2) Establishing a target detection model

[0031] Use the target detection algorithm (YOLOX) to build a target detection model, and select PhotoMetricDistortion, RandomExpand, MinloURandomCrop, and RandomFlip data augmentation methods to enhance the dataset;

[0032] (1.3) Experimental environment

[0033] The experimental environment is set up with Ubuntu 18.0.4, GTX1080Ti GPU, Core i7 CPU, 64GB memory, and PyTorch as the deep learning framework.

[0034] (1.4) Focal Loss Function

[0035] During training, the accuracy of samples that are easy to classify in the dataset can reach 99%, while the accuracy of samples that are difficult to classify is very poor. Focal loss reduces the weight of correctly classified samples instead of giving equal weight to all samples. This has the same effect as giving more difficult-to-classify samples to the training samples. Therefore, we replaced the loss function in YOLOX with the Focal Loss loss function;

[0036] (1.5) Training parameters

[0037] The training parameters are set as follows: epoch 128, SGD as the optimizer, initial learning rate 0.02, momentum 0.9, weight_delay 0.0001, and batch_size 4;

[0038] (1.6) Verify model performance

[0039] Use the validation dataset (including 100 images extracted from different bus stop inspection videos) to verify the trained model and output the AP value.

[0040] Furthermore, the specific process of step 2 is as follows:

[0041] (2.1) Remove low confidence boxes

[0042] In the target detection output, the confidence score represents the likelihood that the model believes the box actually contains an object. We need to filter out boxes with scores below a certain threshold.

[0043] (2.2) Normalize the bounding box scale that exceeds the range

[0044] In the target detection output results, some detection boxes may exceed the image scale, affecting subsequent steps. The specific normalization strategy is as follows: parts with a scale less than 0 are normalized to 0, and parts larger than the image length and width are uniformly changed to the corresponding length and width scale;

[0045] (2.3) Remove the board_ok / board_wrong detection boxes with low confidence

[0046] Since the processed results can only retain the normal station announcement area or the abnormal station announcement area, and usually the detection results will contain both parts, it is necessary to remove one of them. In this invention, we retain the corresponding results by comparing the confidence levels;

[0047] (2.4) Remove the detection box outside the bus_station domain

[0048] In bus inspection videos, some detection frames (ad and leave) appear outside the bus_station frame. To remove these detection frames, we calculate the intersection-over-union ratio of the bus_station detection frame with the ad and leave detection frames, and remove the ad or leave detection frames whose value is less than the threshold.

[0049] Furthermore, the specific process of step three is as follows:

[0050] (3.1) Available frame search

[0051] like Figure 1 As shown, we start from the head and tail of the frame sequence, use two pointers (pointing to the middle of the sequence) to detect the frame sequence according to the process of steps one and two, and then classify the frames according to the detection results;

[0052] (3.2) Classification criteria

[0053] Through the test results of step 1 on the test set, it is found that the box detection frame has the highest accuracy. Therefore, by processing the box frame in the frame detection result, the frame can be classified;

[0054] (3.3) Type A frame

[0055] A type frame is a frame without a station sign. If the detection result is empty, it means that there is no necessary part in the frame, then it is a type A frame;

[0056] (3.4) Type B frame

[0057] Type B frames are frames with incomplete bus stop signs. If the head pointer (tail pointer) points to the detection result of the frame and the box detection box area continues to increase, it indicates that the bus stop sign in the frame is incomplete, then it is a Type B frame;

[0058] (3.5) C-type frame

[0059] A C-type frame is a usable frame. If the head pointer (tail pointer) in the detection result points to the detection result of the frame and the box area no longer increases, it is a C-type frame. At this time, the head pointer (or tail pointer) stops and no longer moves to the other pointer until the frames pointed to by both pointers are C-type frames.

[0060] Furthermore, the specific process of step 4 is as follows:

[0061] (4.1) Main Process

[0062] Step 4: The process of obtaining key frames using available frame sequences is as follows: Figure 2As shown in the figure, the main process is: pre-training DGoogleNet with the TID2013 dataset, fine-tuning the DGoogLeNet network with the self-made IQA dataset, inputting the available frame sequence into the network, and obtaining the key frames corresponding to the inspection video;

[0063] (4.2) Prepare the TID2013 dataset as a pre-training set

[0064] Download the TID2013 dataset, a full-reference image quality visual assessment dataset designed to evaluate the match between image quality assessment models and average human perception. This dataset includes 25 reference images and 3,000 distorted images, representing 25 reference images x 24 distortion types x 5 distortion levels. The dataset includes 24 distortion types, including: altered color saturation, multi-Gaussian noise, comfort noise, lossy compression, color image quantization, color aberration, and sparse sampling. Higher MOS scores in this dataset correspond to higher image visual quality, with scores ranging from 0 to 9.

[0065] (4.3) Building a twin network

[0066] The twin neural network is used to measure the similarity between two inputs. This paper expands the GoogLeNet network into a DGoogLeNet twin network. The two networks share weights and the network inputs are paired images, each pair of images is a different distortion type of the same image.

[0067] (4.4) Set the number of network outputs

[0068] The number of outputs of the GoogLeNet twin network is 1, indicating the quality of the two pairs of images predicted by the model;

[0069] (4.5) Loss function selection

[0070] The loss function used is the Hinge Loss function. The idea of ​​this loss function is to make the distance between those that are not correctly classified and those that are correctly classified far enough. When the model predicts the quality of the image 1 correctly, no loss is added. When the prediction is wrong, the gradient updates the network parameter value. The Hinge Loss loss function formula is:

[0071]

[0072] in Represents the predicted output, and y represents the correct category;

[0073] (4.6) Training parameters

[0074] The network is implemented using Pytorch, with epoch set to 256 times and batch-size to 4;

[0075] (4.7) Experimental environment

[0076] The experimental environment is set up with Ubuntu 18.0.4, GTX1080Ti GPU, Core i7 CPU, 64GB memory, and PyTorch as the deep learning framework.

[0077] (4.8) Building an image evaluation network

[0078] Use the trained twin network, save its parameters, and take one of the models to fine-tune it on the self-made IQA dataset;

[0079] (4.9) Create IQA dataset

[0080] The self-made IQA dataset consists of frames from bus inspection videos. In addition to the HD frames and distorted frames present in the videos, different types of noise are added to the HD frames using a program. Finally, personnel are organized to make subjective quality assessments of the images in the dataset.

[0081] (4.10) Fine-tuning the dataset

[0082] Fine-tune the model using a self-made IQA dataset and save its parameters;

[0083] (4.11) Get inspection video key frames

[0084] The available frame sequence obtained in step 3 is input into the network. Based on the image quality scores, the four images with the highest quality are retained as the key frames of the bus inspection video.

Claims

1. A key frame acquisition method for bus inspection videos, characterized by: The specific process of the method is carried out as follows: Step 1: Use the improved YOLOX algorithm to detect targets in each frame of the bus inspection video; Step 2: Remove invalid data from the test results obtained in step 1; Step 3: Based on the processed detection result obtained in step 2, an available frame search algorithm is used to obtain an available frame sequence; Step 4: Input the available frame sequence into the DGoogLeNet network to obtain the key frames in the bus inspection video; In the above steps, an available frame search algorithm is used to obtain an available frame sequence. The specific process is as follows: 1) Available frame search Starting from the head and tail of the frame sequence, use two pointers to detect the frame sequence according to the process of steps 1 and 2, and then classify the frames according to the detection results; 2) Classification benchmark By processing the box in the frame detection result, the frame is classified; 3) Type A frame A type frame is a frame without a station sign. If the detection result is empty, it means that there is no necessary part in the frame, so it is a type A frame; 4) Type B frame Type B frames are frames with incomplete bus stop signs. If the head pointer or tail pointer points to the detection result of the frame, and the area of ​​the box detection box continues to increase, it indicates that the bus stop sign in the frame is incomplete, then it is a Type B frame; 5) C-type frame A C-type frame is a usable frame. If the box area no longer increases in the detection result of the frame pointed to by the head pointer or the tail pointer in the detection result, it is a C-type frame. At this time, the head pointer or the tail pointer stops and no longer moves toward the other pointer until the frames pointed to by both pointers are C-type frames.

2. The key frame acquisition method for bus inspection videos according to claim 1, characterized in that: In step 1, an improved target detection model is constructed based on the characteristics of bus inspection videos. The specific process is as follows: 1) Create a target detection dataset Some frames were extracted from bus inspection videos and manually labeled. The labeled categories are bus_station, board_ok, board_wrong, box, leave, and ad, which represent the bus stop sign as a whole, normal station announcement area, abnormal station announcement area, box area, leaves, and advertisements, respectively. 2) Build a target detection model Use the target detection algorithm (YOLOX) to build a target detection model, and select PhotoMetricDistortion, RandomExpand, MinloURandomCrop, and RandomFlip data augmentation methods to enhance the dataset; 3) Experimental environment The experimental environment is set up with Ubuntu 18.0.4, GTX1080Ti GPU, Core i7 CPU, 64GB memory, and PyTorch as the deep learning framework. 4) Focal Loss function Replace the loss function in YOLOX with the Focal Loss loss function; 5) Training parameters The training parameters are set as follows: epoch 128, SGD as the optimizer, initial learning rate 0.02, momentum 0.9, weight_delay 0.0001, and batch_size 4; 6) Verify model performance Use the validation dataset to verify the trained model and output the value of the evaluation index AP.

3. The key frame acquisition method for bus inspection video according to claim 1, characterized in that: The second step uses the intersection-over-union strategy to remove invalid data based on the detection results of the first step. The specific process is as follows: 1) Remove low confidence boxes Filter out boxes with scores below a certain threshold; 2) Normalize the bounding box scale that exceeds the range 3) Remove the board_ok detection frame and board_wrong detection frame with low confidence; 4) Remove the detection box outside the bus_station domain; Calculate the intersection-and-union ratio of the bus_station detection frame, the ad detection frame, and the leave detection frame respectively, and remove the ad detection frame and the leave detection frame whose intersection-and-union ratio is less than the threshold.

4. The key frame acquisition method for bus inspection videos according to claim 1, characterized in that: In step 4, the available frame sequence is input into the DGoogLeNet network, and the key frames in the bus inspection video are obtained according to the network output. The specific process is as follows: 1) Main Process The main process of step 4 is as follows: pre-training DGoogleNet with the TID2013 dataset, fine-tuning the DGoogLeNet network with the self-made IQA dataset, inputting the available frame sequence into the network, and obtaining the key frames corresponding to the inspection video; 2) Prepare the TID2013 dataset as a pre-training set 3) Building a twin network Expand the GoogLeNet network into a DGoogLeNet twin network. The two networks share weights and the network input is paired images, each pair of images is a different distortion type of the same image; 4) Set the number of network outputs The number of outputs of the DGoogLeNet twin network is 1, indicating the quality of the two pairs of images predicted by the model; 5) Loss function selection The loss function uses the hinge loss function (Hinge Loss); 6) Training parameters The network is implemented using Pytorch, with epoch set to 256 times and batch-size to 4; 7) Experimental environment The experimental environment is set up with Ubuntu 18.0.4, GTX1080Ti GPU, Core i7 CPU, 64GB memory, and PyTorch as the deep learning framework. 8) Build an image evaluation network Use the trained twin network, save its parameters, and take one of the models to fine-tune it on the self-made IQA dataset; 9) Create IQA dataset The self-made IQA dataset consists of frames from bus inspection videos. In addition to the HD frames and distorted frames present in the videos, different types of noise are added to the HD frames using a program. Finally, personnel are organized to make subjective quality assessments of the images in the dataset. 10) Fine-tune the dataset Fine-tune the model using a self-made IQA dataset and save its parameters; 11) Get inspection video key frames The available frame sequence obtained in step 3 is input into the network. Based on the image quality scores, the four images with the highest quality are retained as the key frames of the bus inspection video.

Images