An adaptive moving target visual detection method

Abstract

The application discloses a kind of self-adapting moving target visual detection methods, comprising the following steps: data preparation and pre-processing: prepare the data set of good mark, data set contains infrared and visible light image, and mark the common and unique features of both;Image pre-processing stage: the infrared and visible light image are fused using ATFuse network, and the new image containing the information advantage of both is generated by injecting common and unique features during the fusion process;Model selection and loading: load the pre-trained YOLOv3 model and its weight parameters;Feature extraction and target detection stage: the input data is extracted using the backbone network of YOLOv3;Target detection and classification: the features are classified and located using the prediction network of YOLOv3;Self-adapting adjustment and post-processing: adjust the detection parameters using non-maximum suppression, and post-process the detected target;Result output and visualization: the detected target is drawn on the original image or the new image after fusion.

Description

Technical Field

[0001] This invention relates to the field of visual detection technology for moving targets, and more particularly to an adaptive visual detection method for moving targets. Background Technology

[0002] Adaptive moving target visual detection is a crucial component of autonomous driving technology, and its development and application have driven continuous innovation and progress in this field. By continuously optimizing detection algorithms and models, the perception capabilities and decision-making levels of autonomous driving systems have been improved. Safety enhancement: This technology can monitor moving targets on the road in real time, providing accurate target information for autonomous driving systems, which helps reduce traffic accidents and improve road safety.

[0003] In existing technologies, autonomous vehicles primarily operate during the day or in well-lit environments, where visible light images may be sufficient for target detection. However, if the vehicle needs to operate at night or in low-light conditions, or if it needs to detect hidden or camouflaged targets, infrared images may be more necessary. Fusion of infrared and visible light images combines the advantages of both, incorporating the target detection capabilities of infrared images and the color and texture information of visible light images, providing more comprehensive and accurate scene information. However, infrared thermal imaging images typically contain significant noise, and extracting effective information and removing noise is another challenge in the fusion process. Especially under low contrast and low resolution conditions, image quality is poor, and noise easily interferes with information extraction. Therefore, an adaptive visual detection method for moving targets is proposed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies, such as the difficulty in extracting effective information and removing noise from infrared thermal imaging images which contain a lot of noise. Therefore, this invention proposes an adaptive visual detection method for moving targets.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An adaptive visual detection method for moving targets includes the following steps:

[0007] Data preparation and preprocessing: Prepare a labeled dataset, which includes bounding boxes and category labels for moving targets (such as pedestrians, vehicles, traffic signs, etc.). The dataset contains infrared and visible light images, and the common and unique features of both are labeled.

[0008] Image preprocessing stage: The infrared and visible light images are fused using the ATFuse network, and a new image containing the advantages of both is generated by injecting common and unique features;

[0009] Model selection and loading: Select a YOLOv3 model trained on a specific dataset to improve detection accuracy and real-time performance, load the pre-trained YOLOv3 model and its weight parameters, the specific dataset includes KITTI dataset, Cityscapes dataset and BDD100K dataset;

[0010] Feature extraction and target detection stage: Preprocess the input video sequence, including image scaling, cropping, etc., for the fused new image, control its size and format to match the input requirements of YOLOv3 model, use the backbone network of YOLOv3 (Darknet-53) to extract features from the input data;

[0011] Target detection and classification: After feature extraction, use the prediction network of YOLOv3 to classify and locate the features, classify according to class probability and confidence to determine the class of the target, locate by calculating the coordinates and height of the bounding box to determine the position and size of the target;

[0012] Adaptive adjustment and post-processing: According to the size and number of detected targets, adjust the detection parameters using non-maximum suppression, and perform post-processing on the detected targets, such as filtering, de-duplication, etc., to exclude redundant detection results and improve the accuracy and stability of target detection;

[0013] Result output and visualization: Draw the detected target on the original image or the fused new image, including the bounding box, class label and confidence, store the detection result as an image or video file.

[0014] The above further includes:

[0015] Further, the common features of the infrared and visible light images include target contour, target position and motion trajectory, the unique features of the infrared image include temperature information, and the unique features of the visible light image include texture information and reflection and shadow.

[0016] Further, the ATFuse network is used to fuse the infrared and visible light images, including the following steps:

[0017] Preprocessing: Perform denoising, contrast enhancement and other preprocessing operations on the infrared and visible light images to improve image quality;

[0018] Feature extraction: Use pyramid transformation to extract features of infrared and visible light images at different scales, learn over-complete dictionary and sparse coding through sparse representation method, perform sparse representation on the image, and extract features;

[0019] Difference information and common information injection: analyze the unique features of infrared and visible light images, inject the unique features as additional features into the fusion process, identify the common features in infrared and visible light images, and the common features help to maintain the consistency of the image in the fusion process, and use non-decreasing sampling contour wave transform for fusion;

[0020] Fusion image optimization: post-processing and optimization of the fused image to make it more consistent with human visual perception.

[0021] Further, in the fusion image optimization, a segmented pixel loss function is designed, which focuses more on preserving the features of the infrared image in the temperature information rich area, and focuses more on preserving the features of the visible light image in the texture and brightness information rich area, including the following steps:

[0022] Region division: divide the fused image into different regions by image segmentation algorithm, each region has similar features, divide the image into temperature information rich area, texture information rich area and brightness information rich area;

[0023] Loss function design: for the divided regions, design different loss functions, in the temperature information rich area, use the loss function based on temperature difference; in the texture information rich area, use the loss function based on texture similarity; in the brightness information rich area, use the loss function based on brightness difference;

[0024] Weighted fusion: in the fusion process, according to the loss function of each region, weighted fusion of infrared and visible light images is carried out, and the weighted coefficient can be dynamically adjusted according to the result of the loss function to ensure that the fused image can retain the advantage information in different regions;

[0025] Optimization iteration: optimize the fused image by gradient descent method, and adjust the weighted coefficient according to the result of the segmented pixel loss function in each iteration until the convergence condition or the preset iteration number is reached.

[0026] Further, the backbone network (Darknet-53) of YOLOv3 is used to extract features from the input video frame, and the specific steps of feature extraction are as follows:

[0027] The preprocessed input video frame is sent into the Darknet-53 network;

[0028] The convolution layer in the network performs convolution operation on the input image to extract local features;

[0029] Batch normalization layer normalizes the convolution result to speed up the training process and improve the stability of the model;

[0030] Leaky ReLU activation function performs nonlinear transformation on the normalized result, increasing the network's nonlinear expression ability.

[0031] Residual connection fuses features of different layers, which helps to alleviate the gradient vanishing problem in deep networks.

[0032] After the combination of multiple convolutional blocks and residual blocks, the high-level feature representation of the input image is obtained.

[0033] Further, YOLOv3 uses a logistic regression function to predict the probability of multiple classes for each grid cell, and uses a softmax function (in some versions of YOLOv3, sigmoid may be used instead of softmax for multi-label classification to simplify calculations and improve efficiency) to predict a confidence for each bounding box, indicating the probability of the existence of the target in the bounding box, and the class probability indicates the probability of the target belonging to class i in the case of the existence of the target, and the confidence indicates the product of the probability of the existence of the target in the bounding box and the intersection over union of the bounding box and the true target box.

[0034] Further, YOLOv3 predicts multiple bounding boxes for each grid cell, which includes four coordinate values: the horizontal coordinate x and the vertical coordinate y of the center point, and the width w and the height h of the bounding box, which are normalized with respect to the grid cell to maintain consistency on feature maps of different scales. The bounding box coordinates are (x, y, w, h), where x and y represent the horizontal and vertical coordinates of the center point of the bounding box (with respect to the grid cell), and w and h represent the width and height of the bounding box (normalized with respect to the width and height of the entire image).

[0035] Further, according to the size and number of detected targets, non-maximum suppression is used to adjust the detection parameters, and post-processing is performed on the detected targets, including the following steps:

[0036] Confidence sorting: all detected bounding boxes are sorted in descending order according to their confidence (i.e. the score of the existence of the object);

[0037] Select the highest scoring bounding box: select the bounding box with the highest confidence from the sorted list, and consider it as the current optimal bounding box, and add it to the final detection result list;

[0038] Calculate the intersection over union: for each remaining bounding box, calculate the intersection over union with the current optimal bounding box, which is the ratio of the intersection area of the two bounding boxes to the union area, used to measure the degree of overlap between the two bounding boxes, and the formula for calculating the intersection over union is

[0039] Remove overlapping bounding boxes: if the intersection over union of a bounding box with the current best bounding box exceeds a preset threshold (such as 0.5), it is considered that the two bounding boxes represent the same target, at this time, remove the bounding box with lower confidence, only keep the one with higher confidence;

[0040] Repeat the above process: continue to select the one with the highest confidence from the remaining bounding boxes, and repeat the process of calculating the intersection over union and removing overlapping bounding boxes until all bounding boxes are checked or no more bounding boxes are added to the final detection result list.

[0041] The present application has the following beneficial effects:

[0042] In the present application, common and unique features of infrared and visible light images are labeled, and a new image containing the advantages of both is generated by injecting common and unique features during fusion, which helps to weaken or eliminate noise in the infrared image in the fusion result. Select YOLOv3 model as the target detection algorithm, which has strong feature expression ability and real-time performance. Through optimizing the model structure and parameters, and adopting efficient feature extraction and detection strategy, the real-time performance of the method is ensured, which meets the real-time detection requirements of the unmanned driving system. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A step diagram of an adaptive moving target visual detection method is proposed in the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0045] Please refer to Figure 1 The present application is an adaptive moving target visual detection method, which includes the following steps:

[0046] Data preparation and preprocessing: prepare a labeled data set containing the bounding boxes and class labels of moving targets (such as pedestrians, vehicles, traffic signs, etc.), the data set contains infrared and visible light images, and the common and unique features of the two are labeled;

[0047] Image preprocessing stage: fuse the infrared and visible light images using the ATFuse network to generate a new image containing the advantages of both;

[0048] Model selection and loading: Select a YOLOv3 model trained on a specific dataset to improve detection accuracy and real-time performance, load the pre-trained YOLOv3 model and its weight parameters, the specific dataset includes KITTI dataset, Cityscapes dataset and BDD100K dataset;

[0049] Feature extraction and target detection stage: Preprocess the input video sequence, including image scaling, cropping, etc., for the fused new image, control its size and format to match the input requirements of YOLOv3 model, use the backbone network of YOLOv3 (Darknet-53) to extract features from the input data;

[0050] Target detection and classification: After feature extraction, use the prediction network of YOLOv3 to classify and locate the features, classify according to class probability and confidence to determine the class of the target, locate by calculating the coordinates and height of the bounding box to determine the position and size of the target;

[0051] Adaptive adjustment and post-processing: According to the size and number of detected targets, adjust the detection parameters using non-maximum suppression, post-process the detected targets, such as filtering, de-duplication, etc., to exclude redundant detection results, improve the accuracy and stability of target detection;

[0052] Result output and visualization: Draw the detected targets on the original image or the fused new image, including bounding boxes, class labels and confidence, store the detection results as image or video files.

[0053] In one embodiment, the common features of the infrared and visible light images include target contour, target position and motion trajectory, the unique features of the infrared image include temperature information, and the unique features of the visible light image include texture information and reflection and shadow.

[0054] In one embodiment, the ATFuse network is used to fuse the infrared and visible light images, including the following steps;

[0055] Preprocessing: Denoising, contrast enhancement and other preprocessing operations are performed on the infrared and visible light images to improve image quality;

[0056] Feature extraction: Use pyramid transformation to extract features of infrared and visible light images at different scales, learn over-complete dictionary and sparse coding through sparse representation method, and extract features through sparse representation of images;

[0057] Difference information and common information injection: analyze the unique features of infrared and visible light images, inject the unique features as additional features into the fusion process, identify the common features in infrared and visible light images, and the common features help to maintain the consistency of the image in the fusion process, and use non-decreasing sampling contour wave transform for fusion;

[0058] Fusion image optimization: post-processing and optimization of the fused image to make it more consistent with human visual perception.

[0059] In one embodiment, in the fusion image optimization, a segmented pixel loss function is designed to focus more on preserving the features of the infrared image in areas rich in temperature information, and focus more on preserving the features of the visible light image in areas rich in texture and brightness information, including the following steps:

[0060] Region division: divide the fusion image into different regions by image segmentation algorithm, each region has similar features, and the image is divided into temperature information rich area, texture information rich area and brightness information rich area;

[0061] Loss function design: for the divided regions, design different loss functions, in the temperature information rich area, use the loss function based on temperature difference; in the texture information rich area, use the loss function based on texture similarity; in the brightness information rich area, use the loss function based on brightness difference;

[0062] Weighted fusion: in the fusion process, according to the loss function of each region, weighted fusion is carried out on the infrared and visible light images, and the weighting coefficient can be dynamically adjusted according to the result of the loss function to ensure that the fused image can retain the respective advantage information in different regions;

[0063] Optimization iteration: optimize the fusion image by gradient descent method, and adjust the weighting coefficient according to the result of the segmented pixel loss function in each iteration until the convergence condition or the preset iteration number is reached.

[0064] In one embodiment, the backbone network (Darknet-53) of YOLOv3 is used to extract features from the input video frame, and the specific steps of feature extraction are as follows:

[0065] The preprocessed input video frame is sent into the Darknet-53 network;

[0066] The convolution layer in the network performs convolution operation on the input image to extract local features;

[0067] The batch normalization layer normalizes the convolution result to speed up the training process and improve the stability of the model;

[0068] The Leaky ReLU activation function performs a nonlinear transformation on the normalized result, increasing the nonlinear expressive power of the network.

[0069] Residual connections fuse features from different layers, which helps to alleviate the vanishing gradient problem in deep networks.

[0070] By combining multiple convolutional blocks and residual blocks, a high-level feature representation of the input image is obtained.

[0071] In one embodiment, YOLOv3 uses a logistic regression function to predict the probability of multiple classes for each grid cell and a softmax function (in some versions of YOLOv3, sigmoid may be used instead of softmax for multi-label classification to simplify computation and improve efficiency) to predict a confidence score for each bounding box, representing the probability that an object exists within the bounding box. The class probability represents the probability that the object belongs to class i if the object is present, and the confidence score represents the product of the probability that an object exists within the bounding box and the intersection-union ratio (IUU) of the bounding box and the ground truth bounding box.

[0072] In one embodiment, YOLOv3 predicts multiple bounding boxes for each grid cell. Each bounding box includes four coordinate values: the x-coordinate of the center point, the y-coordinate, the width w, and the height h. These coordinate values ​​are normalized relative to the grid cell to maintain consistency across feature maps at different scales. The bounding box coordinates are: (x, y, w, h), where x and y represent the x-coordinate and y-coordinate of the center point of the bounding box (relative to the grid cell), and w and h represent the width and height of the bounding box (normalized relative to the width and height of the entire image).

[0073] In one embodiment, based on the size and number of detected targets, non-maximum suppression is used to adjust the detection parameters, and post-processing is performed on the detected targets, including the following steps:

[0074] Confidence ranking: Sort all detected bounding boxes in descending order according to their confidence score (i.e., the score for the presence of objects);

[0075] Select the highest-scoring bounding box: Select the bounding box with the highest confidence from the sorted list, regard it as the current best bounding box, and add it to the final detection results list;

[0076] Calculate the intersection-union ratio (CURRR): For each remaining bounding box, calculate its CURRR with the current best bounding box. The CURRR is the ratio of the intersection area to the union area of ​​the two bounding boxes, used to measure the degree of overlap between the two bounding boxes. The formula for calculating the CURRR is:

[0077] Remove overlapping bounding boxes: If the intersection-union ratio of a bounding box with the current best bounding box exceeds a preset threshold (e.g., 0.5), it is considered that the two bounding boxes represent the same target. In this case, the bounding box with lower confidence is removed, and only the one with higher confidence is kept.

[0078] Repeat the above process: continue to select the remaining bounding box with the highest confidence, repeat the process of calculating the intersection-union ratio and removing overlapping bounding boxes, until all bounding boxes have been checked, or no more bounding boxes are added to the final detection result list.

[0079] Suppose that in an autonomous driving scenario, the object detection algorithm processes the input image and outputs the following three bounding boxes and their confidence scores:

[0080] Bounding box A: x1 = 100, y1 = 100, x2 = 200, y2 = 200, confidence level = 0.9

[0081] Bounding box B: x1 = 110, y1 = 110, x2 = 210, y2 = 210, confidence level = 0.8

[0082] Bounding box C: x1 = 250, y1 = 250, x2 = 350, y2 = 350, confidence level = 0.7

[0083] Where x1, y1 are the coordinates of the top left corner of the bounding box, and x2, y2 are the coordinates of the bottom right corner of the bounding box.

[0084] Confidence ranking: Sort by confidence in descending order to get bounding box A (0.9), bounding box B (0.8), and bounding box C (0.7).

[0085] Select the highest-scoring bounding box: Select bounding box A as the current best bounding box.

[0086] Calculate IoU:

[0087] A high IoU (over 0.5) between bounding box A and bounding box B indicates that they overlap.

[0088] A low IoU (assuming it's below 0.5) between bounding boxes A and C indicates that they do not overlap.

[0089] Remove overlapping bounding boxes: Remove bounding box B with lower confidence because it overlaps with bounding box A and has lower confidence.

[0090] Repeat the above process: select bounding box C as the next optimal bounding box (because only bounding box C remains unprocessed at this point), and add it to the final detection result list.

[0091] Finally, after nonmaximum suppression, the resulting list of detection results includes bounding boxes A and C, which represent two different targets in the image.

[0092] In one embodiment, the input image is preprocessed by resizing the original image to the input size required by the YOLOv3 model (e.g., 416x416) and normalizing it.

[0093] Feature extraction: The preprocessed image is fed into the Darknet-53 network for feature extraction to obtain high-level feature maps.

[0094] Prediction network processing: High-level feature maps are fed into the prediction network for classification and localization. For each grid cell, the probabilities of multiple classes and the confidence scores and coordinates of multiple bounding boxes are predicted.

[0095] Non-maximum suppression (NMS): Since each grid cell may predict multiple bounding boxes, and these bounding boxes may overlap, the NMS algorithm is needed to remove redundant bounding boxes and retain the best bounding box as the final detection result.

[0096] Post-processing: Based on the outputs of the classification and localization parts, the category, location, and size of the target are determined, and the detection results are plotted on the original image.

[0097] In a driverless scenario, the input image contains a car and a pedestrian. After extracting features from the input image, the YOLOv3 model predicts the following output:

[0098] For a given grid cell, the probability of predicting the car category is 0.8, the confidence level is 0.9, and the bounding box coordinates are (x = 0.3, y = 0.5, w = 0.2, h = 0.4).

[0099] For another grid cell, the predicted probability of identifying a pedestrian as a human is 0.7, with a confidence level of 0.85, and the bounding box coordinates are (x = 0.6, y = 0.7, w = 0.1, h = 0.2).

[0100] After processing by the NMS algorithm, the best bounding box is retained as the final detection result, and the category, location, and size of cars and pedestrians are plotted on the original image.

[0101] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A self-adapting visual detection method for moving targets, characterized in that, The method comprises the following steps: Prepare a labeled data set containing the bounding box and class label of the moving target, the data set contains infrared and visible light images, and the common and unique features of the two are labeled, the common features of the infrared and visible light images include target contour, target position and motion trajectory, the unique features of the infrared image include temperature information, and the unique features of the visible light image include texture information and reflection and shadow; Fuse the infrared and visible light images by using the ATFuse network, and generate new images containing the information advantages of the two by injecting common and unique features during the fusion process, including: Preprocessing the infrared and visible light images; Extracting features of the infrared and visible light images at different scales by using pyramid transformation, learning over-complete dictionary and sparse coding by using sparse expression method, and performing sparse expression on the images and extracting features; Analyzing the unique features of the infrared and visible light images, injecting the unique features as additional features into the fusion process, identifying the common features in the infrared and visible light images, and fusing by using non-down-sampling contour wave transformation; Optimizing the fused image, designing a segmented pixel loss function, so that the characteristics of the infrared image are more focused on in the area rich in temperature information, and the characteristics of the visible light image are more focused on in the area rich in texture and brightness information, including: Dividing the fused image into different regions by using image segmentation algorithm, each region has similar features, and the image is divided into temperature information rich area, texture information rich area and brightness information rich area; Designing different loss functions for the divided regions to optimize the fused image, using temperature difference based loss function in the temperature information rich area, using texture similarity based loss function in the texture information rich area, and using brightness difference based loss function in the brightness information rich area; According to the loss function of each region, weighted fusion is performed; Optimizing the fused image by using gradient descent method, in each iteration, the weighted coefficient is adjusted according to the result of the segmented pixel loss function until the convergence condition is reached or the preset iteration number is reached; Select a YOLOv3 model trained by a specific data set, load the pre-trained YOLOv3 model and its weight parameters; Preprocessing the input video sequence, controlling the size and format of the fused new image to match the input requirements of the YOLOv3 model, and extracting features of the input data by using the backbone network of YOLOv3; After feature extraction, the prediction network of YOLOv3 is used to classify and locate the features, the classification is determined according to the class probability and confidence, and the location is determined by calculating the coordinates and height of the bounding box to determine the position and size of the target; According to the size and number of the detected targets, adjust the detection parameters by using non-maximum suppression, and post-process the detected targets; Draw the detected targets on the original image or the fused new image, including the bounding box, class label and confidence, and store the detection results as image or video files.

2. The adaptive moving object vision detection method of claim 1, wherein, The backbone network of YOLOv3 is used to extract features from the input video frame. The specific steps of feature extraction are as follows: After preprocessing, the input video frame is sent to the Darknet-53 network; The convolutional layer in the network performs convolution operation on the input image to extract local features; The batch normalization layer normalizes the convolution result; The Leaky ReLU activation function performs nonlinear transformation on the normalized result; Residual connection fuses features from different layers; After the combination of multiple convolution blocks and residual blocks, the high-level feature representation of the input image is obtained.

3. The adaptive moving object vision detection method of claim 1, wherein, YOLOv3 uses a logistic regression function to predict the probability of multiple classes for each grid cell, and uses a softmax function to predict a confidence for each bounding box, which represents the probability of the existence of the target in the bounding box. The class probability represents the probability of the target belonging to class i in the case of the existence of the target, and the confidence represents the product of the probability of the existence of the target in the bounding box and the intersection over union of the bounding box and the real target box.

4. The adaptive moving object vision detection method of claim 1, wherein, YOLOv3 predicts multiple bounding boxes for each grid cell, which includes four coordinate values: the horizontal coordinate x and the vertical coordinate y of the center point, the width w and the height h of the bounding box. The coordinate values are normalized with respect to the grid cell. The bounding box coordinates are (x, y, w, h), where x and y represent the horizontal and vertical coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box.

5. The adaptive moving object vision detection method of claim 1, wherein, According to the size and number of detected targets, the non-maximum suppression is used to adjust the detection parameters, and the detected targets are post-processed, including the following steps: Confidence sorting: all detected bounding boxes are sorted in descending order according to their confidence; Select the highest scoring bounding box: select the bounding box with the highest confidence from the sorted list, and consider it as the current optimal bounding box, and add it to the final detection result list; Calculate the intersection over union: for each bounding box remaining, calculate its intersection over union with the current best bounding box, which is the ratio of the area of the intersection of the two bounding boxes to the area of the union, which measures the degree of overlap of the two bounding boxes, the formula for calculating the intersection over union is ; Remove overlapping bounding boxes: if the intersection over union of a bounding box and the current optimal bounding box exceeds a preset threshold, it is considered that the two bounding boxes represent the same target, and the bounding box with lower confidence is removed, and only the one with higher confidence is kept; Repeat the above process: continue to select the one with the highest confidence from the remaining bounding boxes, and repeat the process of calculating the intersection over union and removing overlapping bounding boxes, until all bounding boxes are checked, or no more bounding boxes are added to the final detection result list.

Images