Multi-source visual perception-based collaborative vehicle target detection and cross-view ID assignment method

Abstract

This invention discloses a multi-source visual perception-based collaborative vehicle target detection and cross-view ID assignment method, applicable to scenarios such as traffic condition monitoring, autonomous driving testing, and control. The method includes: acquiring image frame sequences from vehicle-mounted, roadside, and aerial perspectives; performing synchronization, denoising, and scale normalization on the images; labeling the images to generate YOLO format label files, and dividing them into training and validation sets; improving the YOLOv8 model to handle image inputs from different perspectives, and adjusting the convolutional kernels and detection layers to adapt to different targets; using the ResNet50 model to extract high-dimensional feature vectors of the detected targets; performing target matching and ID assignment based on the feature similarity matrix using the Hungarian algorithm; and outputting the detection results as YOLO format label files containing category, location, and ID. This invention solves the problem of target detection consistency under multiple perspectives through feature matching and ID update mechanisms, ensuring the stability of target tracking.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation systems, specifically to a method for collaborative vehicle target detection and cross-view ID allocation based on multi-source visual perception. Background Technology

[0002] With the development of intelligent transportation systems, an increasing number of traffic monitoring devices are being applied in fields such as smart cities, autonomous driving, and traffic management. These devices typically consist of multi-source equipment, including vehicle-mounted cameras, roadside monitoring systems, and drones, enabling them to acquire rich traffic scene data from different perspectives. This data includes not only dynamic targets such as vehicles and pedestrians, but also static targets such as traffic signals and signs.

[0003] In complex traffic scenarios, data collected by devices from different perspectives often exhibits significant feature differences for the same target across different devices due to variations in viewpoint, target occlusion, lighting changes, and distance. This makes traditional single-viewpoint or time-series-based target detection and tracking methods ineffective in addressing the consistency problem of target detection across multiple perspectives. Especially in highly dynamic scenes, rapid target movement and scene transitions further increase the difficulty of detection and tracking.

[0004] Existing traffic monitoring systems typically rely on single devices (such as vehicle-mounted cameras or roadside cameras) for target detection and tracking. This approach exhibits significant limitations when dealing with multi-view data. Detection results for the same target may differ across different viewpoints, and even target ID consistency cannot be guaranteed. Many existing target tracking systems rely on temporal information between consecutive frames for ID allocation and tracking; however, breaks in temporal information during scene or viewpoint transitions can lead to interruptions in target ID tracking. For example, when a vehicle switches from the viewpoint of a vehicle-mounted camera to that of a drone, traditional temporal tracking methods cannot maintain target ID consistency. Because the appearance, size, and position of targets vary considerably across different viewpoints, traditional detection methods based on manually designed features struggle to accurately handle these differences, thus affecting detection and tracking accuracy. Existing systems often lack flexible ID allocation and update mechanisms, especially when dealing with multi-source data. Achieving stable ID allocation and updates under conditions of target detection, scene transitions, and detection inconsistencies remains a bottleneck for current technology. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-source visual perception-based collaborative vehicle target detection and cross-view ID allocation method. This method combines data from multiple perspectives, including vehicle-mounted, roadside, and aerial perspectives. It utilizes an improved YOLOv8 target detection algorithm, ResNet50 model feature extraction, and the Hungarian algorithm for target matching and ID allocation, ensuring ID consistency for the same target across multiple perspectives. Furthermore, based on vehicle car-following and lane-changing behavior theory, this invention introduces an ID allocation and update mechanism that considers vehicle feature similarity, enabling continuous target tracking during scene transitions and effectively improving the stability of detection and tracking. The method designed in this invention not only overcomes the shortcomings of existing technologies in multi-view consistency, temporal information dependence, and target feature difference processing, but also provides a new solution for multi-source data fusion and target detection in intelligent transportation systems.

[0006] To achieve the above functions, this invention designs a multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method, which executes the following steps S1-S6 to complete the detection of the target object:

[0007] Step S1: For the target road segment, image data are acquired from the vehicle view, roadside view, and aerial view to form a multi-source image frame sequence, and the image data from each view is preprocessed.

[0008] Step S2: Label the image data in the multi-source image frame sequence. The labels include the target object category, location, width and height of the target object. The labels conform to the YOLO format, and generate the label text document corresponding to the image data. The labeled image data and labels are divided into training set and validation set according to a preset ratio.

[0009] Step S3: Based on the YOLOv8 model, construct an improved YOLOv8 object detection model. Take the image data from each viewpoint in the training set as input and the detected target and corresponding label as output to train the improved YOLOv8 object detection model and obtain the trained improved YOLOv8 object detection model. Perform step S1 on the dataset to be tested and load the image data from the validation set into the trained improved YOLOv8 object detection model. Output and save the inference results of image data and labels.

[0010] Step S4: Extract features from the target objects detected in step S3, and generate high-dimensional feature vectors of the target objects using a pre-trained ResNet50 model;

[0011] Step S5: Target matching and ID assignment based on feature similarity. Based on the feature similarity matrix of the target objects from different perspectives, the Hungarian algorithm is used to perform target matching and assign a unique ID to each target object; new IDs are assigned to unmatched target objects.

[0012] Step S6: Output the detected target information as a YOLO format tag file to complete the target detection.

[0013] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0014] 1. Highly Efficient Target Detection from Multiple Perspectives: This invention utilizes data from multiple perspectives, including vehicle-mounted, roadside, and aerial sources, combined with an improved YOLOv8 target detection model, to achieve highly efficient target detection from various perspectives. By optimizing the data loader and convolutional layer parameters of the YOLOv8 model, this invention can adaptively handle target detection tasks from different perspectives, especially in highly dynamic traffic scenarios where target size and perspective changes, thereby improving detection accuracy and reducing missed detections and false detections.

[0015] 2. Consistent ID Allocation Across Multiple Views: By calculating feature similarity and using the Hungarian algorithm for target matching, this invention effectively solves the problem of inconsistent target ID allocation across multiple views. The same target under different views can maintain ID consistency through feature matching, especially during scene switching or viewpoint transformation, ensuring accurate target tracking and maintaining stability of target detection and tracking in highly dynamic traffic scenarios.

[0016] 3. Flexibility of ID Allocation and Update Mechanism: This invention introduces a dynamic ID update mechanism, which can assign a new ID to a target when the scene changes or a new target fails to match, and track the target through continuous feature extraction and matching. This mechanism ensures the consistency of the target ID across multiple scenes and perspectives, reduces ID inconsistency caused by changes in target features, and improves the robustness of the system. Attached Figure Description

[0017] Figure 1 This is a flowchart of a multi-source visual perception vehicle target cooperative detection and ID cross-view allocation method provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of an image file naming method provided according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of the tag file naming method provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of a YOLO format file provided according to an embodiment of the present invention;

[0021] Figure 5 This is a flowchart of the data loader workflow provided according to an embodiment of the present invention;

[0022] Figure 6 This is a schematic diagram of a target detection result image provided according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0024] The multi-source visual perception vehicle target cooperative detection and ID cross-view allocation method provided in this embodiment of the invention refers to... Figure 1 Perform the following steps S1-S6 to complete the detection of the target object:

[0025] Step S1: For the target road segment, image data is acquired from the vehicle (Ego) view, roadside (Infra) view, and aerial (UAV) view to form a multi-source image frame sequence, and the image data from each view is preprocessed.

[0026] The preprocessing performed on the image data for each viewpoint includes image synchronization, denoising, and scale normalization to ensure that data from different viewpoints can be processed at the same point in time.

[0027] Image synchronization ensures that image frames from different viewpoints have the same timestamp. Multi-view image matching is guaranteed through file naming rules, using the format "image number_viewpoint_images", such as "2409162329_01_ego_images". Image files and tag files are saved separately. Figure 2 and Figure 3 As shown.

[0028] Image enhancement processing is also included to improve the model's robustness under different lighting conditions and adapt it to target detection in various weather conditions. This includes performing random rotation, scaling, color adjustment, and contrast adjustment on the images.

[0029] Step S2: Label the image data in the multi-source image frame sequence. The labels include the target object category, location, width and height of the target object. The labels conform to the YOLO format, and generate the label text document corresponding to the image data. The labeled image data and labels are divided into training set and validation set according to a preset ratio.

[0030] The labeling of image data and the division of training and validation sets include:

[0031] Reference Figure 4Using the LabelImg tool, objects in the image data are labeled in YOLO format. The label information includes the object category (e.g., car, represented by the number 0) and the center point position (x). center and y center The width and height of the target object;

[0032] The image data is divided into a training set and a validation set according to a preset ratio, with 80% used for training and 20% for validation.

[0033] Step S3: Based on the YOLOv8 model, construct an improved YOLOv8 object detection model. Take the image data from each viewpoint in the training set as input and the detected target and corresponding label as output to train the improved YOLOv8 object detection model and obtain the trained improved YOLOv8 object detection model. Perform step S1 on the dataset to be tested and load the image data from the validation set into the trained improved YOLOv8 object detection model. Output and save the inference results of image data and labels.

[0034] The data loader of the improved YOLOv8 object detection model has been modified so that the images and label data from three perspectives of the same scene are loaded through three data channels respectively, enabling the model to adaptively process image inputs from different perspectives. The model adjusts the number of convolutional kernels and detection layers to adapt to object detection under different perspectives and scenes.

[0035] In one embodiment, the YOLOv8 data loader is modified to support multi-view input. The original YOLOv8 model can only handle single-view input; this invention modifies its data loading part to simultaneously process multiple images from vehicle-mounted, roadside, and aerial perspectives. This embodiment uses YOLO, a mainstream method for object detection, specifically the yolov8x model. To handle data from three different perspectives and improve model training performance, the data loader in the original yolov8 base code is modified. Specifically, a function for data loading is added to the `class YOLO Dataset(BaseDataset)` class in the `dataset.py` file, such as... Figure 5 As shown.

[0036] This paper improves the YOLOv8 object detection model by modifying its convolutional kernel size and detection layer parameters. The convolutional kernel size for vehicle-view, roadside-view, and aerial-view image data increases sequentially from small to large. For the vehicle-view, where targets are typically closer, the kernel size is set to 3×3 with a short stride. For the aerial-view, due to the greater target distance, the kernel size is appropriately increased (e.g., 5×5 or 7×7), and the stride is adjusted accordingly to ensure effective detection of small, distant targets. The detection layer parameters are adjusted to improve the model's detection accuracy based on variations in target size and lighting conditions in different scenarios. Specifically, the anchor box size is adjusted for different target scenarios to increase detection accuracy for both small and large targets. The anchor box size for vehicle-view, roadside-view, and aerial-view image data decreases sequentially from large to small. The larger anchor box size for the vehicle-view is used to detect large targets (such as vehicles and pedestrians), while the smaller anchor box size for the aerial-view improves detection accuracy for small targets.

[0037] Step S4: Extract features from the target objects detected in step S3, and generate high-dimensional feature vectors of the target objects using a pre-trained ResNet50 model;

[0038] The specific method for step S4 is as follows:

[0039] The pre-trained ResNet50 model is used to extract features from the target objects detected in step S3, generating high-dimensional feature vectors for the target objects. The ResNet50 model extracts deep features from the image through multiple residual blocks, ultimately generating a feature vector with a dimension of 2048. The calculation formula for the residual blocks is as follows:

[0040] y=F(x,{W i})+x

[0041] Where x is the input feature, F(x,{W i}) represents the features extracted through multiple convolutional layers, W i y represents the convolution kernel parameters and y represents the output features; by using skip connections, gradient vanishing is avoided and higher-level semantic features in the image are extracted.

[0042] The feature extraction process of the ResNet50 model consists of five stages, each composed of multiple convolutional layers and residual blocks. By progressively reducing the receptive field and increasing the number of convolutional kernels in different convolutional layers, target features at different scales are extracted. The output of feature extraction is a vector containing high-dimensional target features. The dimension of the feature vector is determined by the number of output channels of the convolutional layer; the commonly used final feature vector dimension of the ResNet50 model is 2048. The calculation formula for feature extraction is as follows:

[0043] F = ResNet50(I)

[0044] Where I represents the input image, F is the high-dimensional feature vector extracted by the ResNet50 model, and the output high-dimensional feature vector is used for target feature matching and ID assignment in subsequent steps.

[0045] Step S4, target feature extraction, includes:

[0046] A feature similarity matrix is ​​constructed, and the comprehensive feature similarity is calculated using a weighted average method. The formula for calculating the feature similarity matrix is ​​as follows:

[0047]

[0048] Among them, F i F j and F k The target feature vectors are from different perspectives, S(i,j,k) is the comprehensive feature similarity, and w1, w2 and w3 are the weight coefficients between different perspectives.

[0049] Step S5: Target matching and ID assignment based on feature similarity. Based on the feature similarity matrix of the target objects from different perspectives, the Hungarian algorithm is used to perform target matching and assign a unique ID to each target object; new IDs are assigned to unmatched target objects.

[0050] Feature-similarity-based target matching includes:

[0051] Based on the feature similarity matrix, the Hungarian algorithm is used to minimize the matching cost of the feature similarity matrix, thus optimizing the target matching process under multiple perspectives. The Hungarian algorithm solves the target feature matching problem under multiple perspectives, minimizing the cost function to achieve optimal matching between targets. The cost function formula is as follows:

[0052]

[0053] Where C(i,j) are elements in the feature similarity matrix, and X(i,j) are binary variables representing whether to select a match.

[0054] The target ID allocation and update mechanism includes:

[0055] In the event of scene switching, the consistency of target ID is maintained through feature extraction and feature matching mechanisms; for targets that fail to pass feature similarity matching, a new ID is assigned, and continuous tracking of the target is maintained in subsequent scenes through continuous feature extraction and matching mechanisms.

[0056] Step S6: Output the detected target information as a YOLO format tag file to complete the target detection.

[0057] Reference Figure 6 The output of the detection results includes:

[0058] The output is a tag file in YOLO format, which includes the target object category, location coordinates, target object width and height, and assigned ID; the tag file format is:

[0059] <class><x center ><y center > <width> <height> <id>

[0060] Where class represents the target object category, x center and y center This indicates the coordinates of the center point of the target object, width and height represent the width and height of the target object, and ID represents the unique identifier of the target object.

[0061] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.< / id> < / height> < / width> < / class>

Claims

1. A method for collaborative vehicle target detection and cross-view ID allocation based on multi-source visual perception, characterized in that, Perform the following steps S1-S6 to complete the detection of the target object: Step S1: For the target road segment, image data are acquired from the vehicle view, roadside view, and aerial view to form a multi-source image frame sequence, and the image data from each view is preprocessed. Step S2: Label the image data in the multi-source image frame sequence. The labels include the target object category, location, width and height of the target object. The labels conform to the YOLO format, and generate the label text document corresponding to the image data. The labeled image data and labels are divided into training set and validation set according to a preset ratio. Step S3: Based on the YOLOv8 model, construct an improved YOLOv8 object detection model. Take the image data from each viewpoint in the training set as input and the detected target and corresponding label as output to train the improved YOLOv8 object detection model and obtain the trained improved YOLOv8 object detection model. Perform step S1 on the dataset to be tested and load the image data from the validation set into the trained improved YOLOv8 object detection model. Output and save the inference results of image data and labels. The data loader of the improved YOLOv8 target detection model has been modified. For the YOLOv8 model, the size of its convolution kernel and the parameters of the detection layer have been modified. Specifically, the convolution kernels of the vehicle view, roadside view, and aerial view image data are increased in size in that order. For the detection layer, the anchor box size of the vehicle view, roadside view, and aerial view image data is decreased in size in that order. Step S4: Extract features from the target objects detected in step S3, and generate high-dimensional feature vectors of the target objects using a pre-trained ResNet50 model; Target feature extraction includes: A feature similarity matrix is ​​constructed, and the comprehensive feature similarity is calculated using a weighted average method. The formula for calculating the feature similarity matrix is ​​as follows: Among them, F i F j and F k The target feature vectors are from different perspectives, S(i,j,k) is the comprehensive feature similarity, and w1, w2 and w3 are the weight coefficients between different perspectives; Step S5: Target matching and ID assignment based on feature similarity. Based on the feature similarity matrix of the target objects from different perspectives, the Hungarian algorithm is used to perform target matching and assign a unique ID to each target object; new IDs are assigned to unmatched target objects. Step S6: Output the detected target information as a YOLO format tag file to complete the target detection.

2. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, The preprocessing performed on the image data for each viewpoint in step S1 includes: image synchronization, denoising and scale normalization, and image enhancement, including random rotation, scaling, color adjustment and contrast adjustment of the image.

3. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, Step S2, which involves labeling the image data and dividing it into training and validation sets, includes: The YOLO format is used to label the objects in the image data. The label information includes the object category (class) and the center point position (x). center and y center The width and height of the target object; The image data is divided into a training set and a validation set according to a preset ratio, with 80% used for training and 20% for validation.

4. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, The specific method for step S4 is as follows: The pre-trained ResNet50 model is used to extract features from the target objects detected in step S3, generating high-dimensional feature vectors for the target objects; the calculation formula for the residual module of the ResNet50 model for deep feature extraction of image data is as follows: y=F(x,{W i })+x Where x is the input feature, F(x,{W i }) represents the features extracted through multiple convolutional layers, W i Here, y represents the convolution kernel parameters, and y represents the output features. The feature extraction process of the ResNet50 model consists of five stages, each composed of multiple convolutional layers and residual blocks. By progressively narrowing the receptive field of view and increasing the number of convolutional kernels in different convolutional layers, target features at different scales are extracted. The calculation formula for feature extraction is as follows: F = ResNet50(I) Where I represents the input image, F is the high-dimensional feature vector extracted by the ResNet50 model, and the output high-dimensional feature vector is used for target feature matching and ID assignment in subsequent steps.

5. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, Step S5, target matching based on feature similarity, includes: Based on the feature similarity matrix, the Hungarian algorithm is used to solve the target feature matching problem from multiple perspectives. The optimal matching between targets is achieved by minimizing the cost function, which is formulated as follows: Where C(i,j) are elements in the feature similarity matrix, and X(i,j) are binary variables representing whether to select a match.

6. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, The target ID allocation and update mechanism in step S5 includes: In the event of scene switching, the consistency of target ID is maintained through feature extraction and feature matching mechanisms; for targets that fail to pass feature similarity matching, a new ID is assigned, and continuous tracking of the target is maintained in subsequent scenes through continuous feature extraction and matching mechanisms.

7. The multi-source visual perception-based vehicle target cooperative detection and ID cross-view allocation method according to claim 1, characterized in that, The output of the detection results in step S6 includes: The output is a tag file in YOLO format, which includes the target object category, location coordinates, target object width and height, and assigned ID; the tag file format is: <class><x center ><y center > <width> <height> <id>< / id> < / height> < / width> < / class> Where class represents the target object category, x center and y center This indicates the coordinates of the center point of the target object, width and height represent the width and height of the target object, and ID represents the unique identifier of the target object.

Images