Multi-source visual perception vehicle target collaborative detection and ID cross-view distribution method

Abstract

The invention discloses a multi-source visual perception vehicle target collaborative detection and ID cross-view angle distribution method, which is suitable for scenes such as traffic state monitoring, unmanned driving test and control and the like, and comprises the following steps: acquiring an image frame sequence from vehicle-mounted, roadside and aerial view angles, and carrying out synchronization, denoising and scale standardization processing on an image; marking the image, generating a label file in a YOLO format, and dividing the label file into a training set and a verification set; improving a YOLOv8 model to process different view angle image input, and adjusting a convolution kernel and a detection layer to adapt to different targets; extracting a high-dimensional feature vector of a detection target by using a ResNet50 model; performing target matching and ID distribution by using a Hungary algorithm based on the feature similarity matrix; and outputting a detection result as a YOLO format tag file containing the category, the position and the ID. Through feature matching and an ID updating mechanism, the problem of target detection consistency under multiple view angles is solved, and the stability of target tracking is ensured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation systems, and in particular to a method for collaborative detection of vehicle targets and ID cross-viewing angle allocation based on multi-source visual perception. Background Art

[0002] With the development of intelligent transportation systems, more and more traffic monitoring equipment is being used in smart cities, unmanned driving, traffic management and other fields. These devices are usually composed of multi-source equipment such as vehicle-mounted cameras, roadside monitoring and drones, which can obtain rich traffic scene data from different perspectives. These data not only include dynamic targets such as vehicles and pedestrians, but also static targets such as traffic signals and signs.

[0003] In complex traffic scenes, the data collected by devices with different perspectives often show obvious feature differences between the same target due to differences in perspective, target occlusion, illumination changes, and distance. This makes it difficult for traditional single-perspective or time-series-based target detection and tracking methods to effectively solve the problem of target detection consistency under multiple perspectives. Especially in high-dynamic scenes, the rapid movement of targets and scene switching further increase the difficulty of detection and tracking.

[0004] Existing traffic monitoring systems usually rely on a single device (such as a vehicle-mounted camera or a roadside camera) for target detection and tracking. This method shows obvious limitations when facing multi-view data. The detection results of the same target under different viewpoints may be different, and even the consistency of the target ID cannot be guaranteed. Many existing target tracking systems rely on the timing information between consecutive frames for ID allocation and tracking. However, when the scene switches or the viewpoint switches, the interruption of the timing information will cause the target ID tracking to be interrupted. For example, when the vehicle switches from the viewpoint of the vehicle-mounted camera to the viewpoint of the drone, the traditional timing tracking method cannot maintain the consistency of the target ID. Since the appearance, size, and position of the target vary greatly under different viewpoints, it is difficult for the traditional detection method based on manually designed features to accurately handle these differences, thus affecting the accuracy of detection and tracking. Existing systems usually lack a flexible ID allocation and update mechanism. Especially when facing multi-source data, how to achieve stable ID allocation and update in the case of target detection, scene switching, and inconsistent detection is a bottleneck of the existing technology. Summary of the invention

[0005] The purpose of the present invention is to provide a method for collaborative detection of vehicle targets and ID cross-viewpoint allocation based on multi-source visual perception. The method combines data from multiple perspectives such as vehicle-mounted, roadside, and aerial, and performs target matching and ID allocation through an improved YOLOv8 target detection algorithm, ResNet50 model feature extraction, and Hungarian algorithm to ensure the ID consistency of the same target under multiple perspectives. In addition, based on the theory of vehicle following and lane changing behavior, the present invention introduces an ID allocation and update mechanism that considers the similarity of vehicle features, which can achieve continuous tracking of targets during scene switching, and effectively improve the stability of detection and tracking. The method designed by the present invention not only makes up for the deficiencies of the prior art in multi-perspective consistency, temporal information dependence, and target feature difference processing, but also provides a new solution for multi-source data fusion and target detection in intelligent transportation systems.

[0006] To achieve the above functions, the present invention designs a vehicle target collaborative detection and ID cross-viewing assignment method based on multi-source visual perception, and executes the following steps S1 to S6 to complete the detection of the target object:

[0007] Step S1: for the target road section, image data are acquired from the vehicle-mounted perspective, the roadside perspective and the aerial perspective respectively to form a multi-source image frame sequence, and the image data of each perspective is pre-processed respectively;

[0008] Step S2: labeling the image data in the multi-source image frame sequence, the labels include the target category, position, width and height of the target, the labels conform to the YOLO format, and generating a label text document corresponding to the image data; the labeled image data and labels are divided into a training set and a validation set according to a preset ratio;

[0009] Step S3: Based on the YOLOv8 model, an improved YOLOv8 target detection model is constructed, the image data of each perspective in the training set is used as input, and the detected target and the corresponding label are used as output, the improved YOLOv8 target detection model is trained to obtain a trained improved YOLOv8 target detection model; the test data set is executed in step S1, and the image data in the verification set is loaded into the trained improved YOLOv8 target detection model, and the inference results of the image data and labels are output and saved;

[0010] Step S4: extract features of the target object detected in step S3, and use the pre-trained ResNet50 model to generate a high-dimensional feature vector of the target object;

[0011] Step S5: Target matching and ID assignment based on feature similarity: Based on the feature similarity matrix of the target objects under different viewing angles, the Hungarian algorithm is used to perform target matching, and a unique ID is assigned to each target object; a new ID is assigned to the target object that is not matched;

[0012] Step S6: Output the detected target information as a label file in YOLO format to complete the detection of the target.

[0013] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0014] 1. Efficient target detection from multiple perspectives: This invention uses data from multiple perspectives such as vehicle-mounted, roadside, and aerial, combined with an improved YOLOv8 target detection model, to achieve efficient target detection from different perspectives. By optimizing the data loader and convolutional layer parameters of the YOLOv8 model, this invention can adaptively handle target detection tasks from different perspectives, especially for the size and perspective changes of targets in highly dynamic traffic scenes, which can improve detection accuracy and reduce missed detections and false detections;

[0015] 2. Consistent ID allocation under multiple perspectives: Through the calculation of feature similarity and target matching of the Hungarian algorithm, the present invention can effectively solve the problem of inconsistent target ID allocation under multiple perspectives. The same target under different perspectives can maintain ID consistency through feature matching, especially when the scene is switched or the perspective is changed, and the target can still be accurately tracked, ensuring the stability of target detection and tracking in highly dynamic traffic scenes;

[0016] 3. Flexibility of ID allocation and update mechanism: The present invention introduces a dynamic ID update mechanism, which can allocate a new ID to a new target when the scene switches or the new target fails to match, and track the target through a continuous feature extraction and matching mechanism. This mechanism ensures the consistency of the target ID in multiple scenes and perspectives, reduces the ID inconsistency problem caused by changes in target features, and improves the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of a method for collaborative detection of vehicle targets and ID cross-viewing angle assignment based on multi-source visual perception provided by an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of an image file naming method provided according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a tag file naming method provided according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of a YOLO format file provided according to an embodiment of the present invention;

[0021] Figure 5 is a workflow diagram of a data loader provided according to an embodiment of the present invention;

[0022] Figure 6 is a schematic diagram of a target detection result image provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0024] The vehicle target collaborative detection and ID cross-viewing assignment method based on multi-source visual perception provided by the embodiment of the present invention is referred to Figure 1 , perform the following steps S1 to S6 to complete the detection of the target object:

[0025] Step S1: for the target road section, image data are acquired from the vehicle-mounted (Ego) perspective, the roadside (Infra) perspective, and the aerial (UAV) perspective to form a multi-source image frame sequence, and the image data of each perspective is preprocessed respectively;

[0026] The preprocessing of the image data from each perspective includes image synchronization, denoising and scale normalization to ensure that data from different perspectives can be processed at the same time point.

[0027] Among them, image synchronization ensures that the image frames of each perspective have the same timestamp, and the multi-perspective image matching is ensured by the file naming rules, which are named in the format of "image number_perspective_images", such as "2409162329_01_ego_images", where the image file and the label file are saved separately, such as Figure 2 and Figure 3 shown.

[0028] As well as image enhancement processing, the purpose is to improve the robustness of the model under different lighting conditions and adapt to target detection in various weather conditions. This includes performing random rotation, scaling, color adjustment, and contrast adjustment on the image.

[0029] Step S2: labeling the image data in the multi-source image frame sequence, the labels include the target category, position, width and height of the target, the labels conform to the YOLO format, and generating a label text document corresponding to the image data; the labeled image data and labels are divided into a training set and a validation set according to a preset ratio;

[0030] The labeling of image data and the division of training set and validation set include:

[0031] Reference Figure 4, use the LabelImg tool to label the objects in the image data using the YOLO format. The label information includes the object category class, such as the vehicle Car, represented by the number 0; and the center point position x center and center , the width width and height height of the target;

[0032] The image data is divided into a training set and a validation set according to a preset ratio, and the ratio of the training set to the validation set is set to 80% for training and 20% for validation.

[0033] Step S3: Based on the YOLOv8 model, an improved YOLOv8 target detection model is constructed, the image data of each perspective in the training set is used as input, and the detected target and the corresponding label are used as output, the improved YOLOv8 target detection model is trained to obtain a trained improved YOLOv8 target detection model; the test data set is executed in step S1, and the image data in the verification set is loaded into the trained improved YOLOv8 target detection model, and the inference results of the image data and labels are output and saved;

[0034] The data loader of the improved YOLOv8 target detection model is modified, and three perspective images and label data of the same scene are loaded respectively through three data channels, so that the model can adaptively process image inputs of different perspectives. The model adjusts the number of convolution kernels and detection layers to adapt to target detection under different perspectives and scenes;

[0035] In one embodiment, the data loader of YOLOv8 is modified to support multi-view input. The original YOLOv8 model can only process single-view input. The present invention modifies its data loading part so that it can simultaneously process multiple images from vehicle-mounted, roadside and aerial perspectives. In this embodiment, YOLO, a mainstream method for target detection, is used, and the specific model is yolov8x. In order to improve the effect of model training by coping with data of three different perspectives, the data loader in the original yolov8 base code is modified, that is, a function for data loading is added under the class YOLO Dataset (BaseDataset) in the dataset.py file, such as Figure 5 shown.

[0036] Improved YOLOv8 target detection model For the YOLOv8 model, its convolution kernel size and detection layer parameters are modified. The convolution kernels of the vehicle-mounted view, roadside view and aerial view image data are from small to large. For the vehicle-mounted view, the target is usually close, so the convolution kernel size is set to 3×3 with a short step size. For the aerial view, due to the long distance of the target, the convolution kernel size is appropriately increased (for example, 5×5 or 7×7), and the step size is adjusted accordingly to ensure the effective detection of small targets at a long distance. According to the changes in target size and light conditions in different scenes, the detection accuracy of the model is improved by adjusting the parameters of the detection layer. The detection layer parameters are adjusted to adjust the anchor box size (anchor boxes) of the detection layer for targets in different scenes to increase the detection accuracy of small and large targets. The anchor box sizes of the vehicle-mounted view, roadside view and aerial view image data are from large to small. The anchor box size of the vehicle-mounted view is larger to adapt to the detection of large targets (such as vehicles and pedestrians); the anchor box size of the aerial view is smaller to improve the detection accuracy of small targets.

[0037] Step S4: extract features of the target object detected in step S3, and use the pre-trained ResNet50 model to generate a high-dimensional feature vector of the target object;

[0038] The specific method of step S4 is as follows:

[0039] The pre-trained ResNet50 model is used to extract features of the target object detected in step S3 to generate a high-dimensional feature vector of the target object; the ResNet50 model extracts deep features of the image through multiple residual blocks, and the feature vector dimension finally generated is 2048 dimensions; the calculation formula of the residual block is:

[0040] y=F(x,{W i})+x

[0041] Among them, x is the input feature, F(x,{W i}) is the feature extracted by multiple convolutional layers, W i is the convolution kernel parameter, y is the output feature; through the jump connection method, the gradient disappearance is avoided and the higher-level semantic features in the image are extracted;

[0042] The feature extraction process of the ResNet50 model is divided into five stages, each of which consists of multiple convolutional layers and residual blocks. By gradually narrowing the receptive field and increasing the number of convolution kernels in different convolutional layers, target features of different scales are extracted. The output of feature extraction is a vector containing high-dimensional features of the target. The dimension of the feature vector is determined by the number of output channels of the convolutional layer. The commonly used final feature vector dimension of the ResNet50 model is 2048. The calculation formula for feature extraction is:

[0043] F = ResNet50 (I)

[0044] Where I represents the input image, F is the high-dimensional feature vector extracted by the ResNet50 model, and the output high-dimensional feature vector is used for target feature matching and ID assignment in subsequent steps.

[0045] The target feature extraction in step S4 includes:

[0046] A feature similarity matrix is ​​constructed, and the weighted average method is used to calculate the comprehensive feature similarity. The calculation formula of the feature similarity matrix is:

[0047]

[0048] Among them, F i 、F j and F k They are target feature vectors from different perspectives, S(i,j,k) is the comprehensive feature similarity, and w 1 、w 2 and w 3 is the weight coefficient between different perspectives.

[0049] Step S5: Target matching and ID assignment based on feature similarity: Based on the feature similarity matrix of the target objects under different viewing angles, the Hungarian algorithm is used to perform target matching, and a unique ID is assigned to each target object; a new ID is assigned to the target object that is not matched;

[0050] Object matching based on feature similarity includes:

[0051] Based on the feature similarity matrix, the Hungarian algorithm is used to minimize the matching cost of the feature similarity matrix and optimize the target matching process under multiple perspectives. The Hungarian algorithm is used to solve the matching problem of target features under multiple perspectives and minimize the cost function to achieve the best matching between targets. The cost function formula is as follows:

[0052]

[0053] Among them, C(i,j) is an element in the feature similarity matrix, and X(i,j) is a binary variable indicating whether to select a match.

[0054] The target ID allocation and update mechanism includes:

[0055] In the case of scene switching, the consistency of the target ID is maintained through feature extraction and feature matching mechanisms; for targets that fail to pass feature similarity matching, new IDs are assigned, and continuous feature extraction and matching mechanisms are used in subsequent scenes to keep the targets tracked.

[0056] Step S6: Output the detected target information as a label file in YOLO format to complete the detection of the target.

[0057] Reference Figure 6 The output of the test results includes:

[0058] The output is a label file in YOLO format, which includes the target category, location coordinates, target width and height, and assigned ID; the label file format is:

[0059] <class><x center ><y center > <width> <height> <id>

[0060] Among them, class represents the target category, x center and center Indicates the coordinates of the center point of the target, width and height indicate the width and height of the target, and ID indicates the unique identifier of the target.

[0061] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.< / id> < / height> < / width> < / class>

Claims

1. A vehicle target collaborative detection and ID cross-view distribution method based on multi-source visual perception, characterized in that: Execute the following steps S1 to S6 to complete the detection of the target object: Step S1: for the target road section, image data are acquired from the vehicle-mounted perspective, the roadside perspective and the aerial perspective respectively to form a multi-source image frame sequence, and the image data of each perspective is pre-processed respectively; Step S2: labeling the image data in the multi-source image frame sequence, the labels include the target category, position, width and height of the target, the labels conform to the YOLO format, and generating a label text document corresponding to the image data; the labeled image data and labels are divided into a training set and a validation set according to a preset ratio; Step S3: Based on the YOLOv8 model, an improved YOLOv8 target detection model is constructed, the image data of each perspective in the training set is used as input, and the detected target and the corresponding label are used as output, the improved YOLOv8 target detection model is trained to obtain a trained improved YOLOv8 target detection model; the test data set is executed in step S1, and the image data in the verification set is loaded into the trained improved YOLOv8 target detection model, and the inference results of the image data and labels are output and saved; Step S4: extract features of the target object detected in step S3, and use the pre-trained ResNet50 model to generate a high-dimensional feature vector of the target object; Step S5: Target matching and ID assignment based on feature similarity: Based on the feature similarity matrix of the target objects under different viewing angles, the Hungarian algorithm is used to perform target matching, and a unique ID is assigned to each target object; a new ID is assigned to the target object that is not matched; Step S6: Output the detected target information as a label file in YOLO format to complete the detection of the target.

2. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The preprocessing performed on the image data of each viewing angle in step S1 includes: image synchronization, denoising and scale normalization processing, and image enhancement processing, including performing random rotation, scaling, color adjustment and contrast adjustment on the image.

3. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: In step S2, labeling the image data and dividing the training set and the validation set include: Use the YOLO format to label the objects in the image data. The label information includes the object category class, the center point position x center and center , the width width and height height of the target; The image data is divided into a training set and a validation set according to a preset ratio, and the ratio of the training set to the validation set is set to 80% for training and 20% for validation.

4. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The improved YOLOv8 target detection model in step S3 modifies the convolution kernel size and detection layer parameters of the YOLOv8 model, wherein the convolution kernels of the vehicle-mounted perspective, roadside perspective, and aerial perspective image data are from small to large; for the detection layer, the anchor frame sizes of the vehicle-mounted perspective, roadside perspective, and aerial perspective image data are from large to small.

5. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The specific method of step S4 is as follows: The pre-trained ResNet50 model is used to extract features of the target object detected in step S3 to generate a high-dimensional feature vector of the target object; the calculation formula of the residual module for deep feature extraction of image data in the ResNet50 model is: y=F(x,{W i })+x Among them, x is the input feature, F(x,{W i }) is the feature extracted by multiple convolutional layers, W i is the convolution kernel parameter, y is the output feature; The feature extraction process of the ResNet50 model is divided into five stages, each of which consists of multiple convolutional layers and residual blocks. By gradually narrowing the perceptual field of view and increasing the number of convolution kernels in different convolutional layers, target features of different scales are extracted. The calculation formula for feature extraction is: F = ResNet50 (I) Where I represents the input image, F is the high-dimensional feature vector extracted by the ResNet50 model, and the output high-dimensional feature vector is used for target feature matching and ID assignment in subsequent steps.

6. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The target feature extraction in step S4 includes: A feature similarity matrix is ​​constructed, and the weighted average method is used to calculate the comprehensive feature similarity. The calculation formula of the feature similarity matrix is: Among them, F i 、F j and F k The target feature vectors are from different perspectives, S(i,j,k) is the comprehensive feature similarity, and w1, w2 and w3 are the weight coefficients between different perspectives.

7. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The target matching based on feature similarity in step S5 includes: Based on the feature similarity matrix, the Hungarian algorithm is used to solve the matching problem of target features under multiple perspectives, and the cost function is minimized to achieve the best match between targets. The cost function formula is as follows: Among them, C(i,j) is an element in the feature similarity matrix, and X(i,j) is a binary variable indicating whether to select a match.

8. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The target ID allocation and update mechanism in step S5 includes: In the case of scene switching, the consistency of the target ID is maintained through feature extraction and feature matching mechanisms; for targets that fail to pass feature similarity matching, new IDs are assigned, and continuous feature extraction and matching mechanisms are used in subsequent scenes to keep the targets tracked.

9. The method for collaborative detection of vehicle targets and ID cross-view distribution based on multi-source visual perception according to claim 1 is characterized in that: The output of the detection result in step S6 includes: The output is a label file in YOLO format, which includes the target category, location coordinates, target width and height, and assigned ID; the label file format is: <class><x center ><y center > <width> <height> <id>< / id> < / height> < / width> < / class> Among them, class represents the target category, x center and center Indicates the coordinates of the center point of the target, width and height indicate the width and height of the target, and ID indicates the unique identifier of the target.

Images