A road panoramic target detection and tracking method under an edge computing platform

Abstract

The application discloses a road panoramic target detection and tracking method under an edge computing platform, and comprises the following steps: taking a one-stage detector as a basic model, designing a structure for a special scene, introducing an adaptive attention channel, improving useful features and suppressing small features, and optimizing and quantifying; independently detecting and processing a plurality of cameras of a panoramic spliced image on an edge device, using a multi-image synthesis method to accelerate the inference speed, focusing on the region target of the plurality of cameras, and then obtaining the detection information of each image to improve the detection success rate and accuracy of the target; performing matrix mapping on the detection information of the plurality of cameras, for the repeated detection problem of the spliced edge target, adopting a method of removing a high threshold detection frame based on the improved NMS idea to fuse and process the redundant edge detection information, adopting an improved multi-scale filtering algorithm for tracking, and improving the detection success rate and accuracy of the target. The efficiency of panoramic target detection under the edge device is greatly improved.

Description

Technical Field

[0001] This invention relates to a target detection and tracking method, and more particularly to a road panoramic target detection and tracking method under an edge computing platform. Background Technology

[0002] With the promotion and application of distributed computing concepts, there is a growing demand in industry for deep learning-based panoramic object detection and tracking on edge devices. The rapid development of deep learning and GPU technologies has significantly improved the training and inference efficiency of neural network algorithms on GPUs, leading to mature applications and solutions for object detection and tracking based on a terminal + GPU server architecture. Real-time panoramic object detection and tracking can be relatively easily achieved with the support of computing power. However, the computing power of edge devices is currently far less than that of GPU servers, causing real-time panoramic object detection and tracking models to fail to achieve the expected results when deployed on edge devices, thus limiting their application scope. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a road panoramic target detection and tracking method under an edge computing platform, which addresses the shortcomings of the existing technology.

[0004] To address the aforementioned technical problems, this invention discloses a road panoramic target detection and tracking method based on an edge computing platform, comprising the following steps:

[0005] Step 1: Acquire panoramic multi-channel video input images, i.e., multi-channel camera images;

[0006] Step 2: Design an adaptive synthesis method for multi-channel video input images to generate composite images from multiple camera images, which can be used to accelerate the inference speed of neural networks.

[0007] Step 3: Construct an object detection network model. Based on a one-stage CNN object network, design the object detection network model structure for specific scenarios.

[0008] Step 4: Train and quantize the model according to the usage scenario;

[0009] Step 5: Using the trained and quantized object detection network model, perform multi-path detection processing on the synthetic image using parallel inference; split the synthetic image to obtain split multi-path camera images, and solve the cross-edge detection allocation problem during the splitting by using a feature proportion method to obtain the independent detection results of the multi-path camera images of the split panoramic stitched image, generate structured data, i.e., multi-path camera detection information, and output it.

[0010] Step 6: Perform matrix mapping on the detection information from multiple cameras to obtain preliminary panoramic detection results;

[0011] Step 7: The improved NMS algorithm is used to filter out the preliminary panoramic detection results output by the matrix mapping to obtain the panoramic detection results.

[0012] Step 8: Use an improved multi-scale filtering algorithm to smooth the detection results and obtain the tracking results.

[0013] The multi-channel video input image adaptive synthesis method described in step 2 of this invention includes:

[0014] The multi-camera images obtained in step 1 are synthesized in pairs. An adaptive dual-image stitching synthesis method is used to accelerate multi-image inference. The specific method is as follows:

[0015] dst=F_concat(F_resize(src1),F_resize(src2))

[0016] Wherein, F_concat is the image compositing function, F_resize is the image size modification function, the image compositing method is to vertically concatenate the matrices describing the first image src1 and the second image src2, and dst is the composite image after compositing the first image src1 and the second image src2.

[0017] The target detection network model described in step 3 of this invention includes three parts: a backbone network, a fusion network, and a detection prediction head.

[0018] The backbone network uses Darknet53 as its base and employs a dual-channel feature extraction mechanism. One channel is used for feature extraction, i.e., the feature channel, while the other channel introduces an adaptive attention mechanism to obtain the importance of each channel in the dual-channel configuration, i.e., the adaptive attention channel.

[0019] The adaptive attention channel uses three computational functions: a compression function, an activation function, and a weight reorganization function; the statistical data z of the d-th layer channel... d The calculation method is as follows:

[0020]

[0021] Among them, F sq For the compression function, z is the statistical data of the d-th layer channel. d This involves compressing the features of the channel feature map of that layer and using them as attention values. The method used is mean pooling, which calculates the mean value of the features in the feature map. d Let (i,j) represent the feature map of the d-th channel, (i,j) represent the data in the i-th row and j-th column of the feature map of the d-th channel, H is the feature channel height of the adaptive attention channel, and W is the feature channel width of the adaptive attention channel.

[0022] The feature channel weights s are calculated as follows:

[0023] s = F ex (z,Weight)=Sigmoid(Relu(z,Weight1),Weight2)

[0024] Among them, F ex The activation function captures the dependencies of feature channels by aggregating information. Weight1 represents the learning parameters of the first activation gating mechanism, Weight2 represents the learning parameters of the second activation gating mechanism, z represents the channel feature vector processed by the compression function, ReLU(z,Weight1) represents the calculation of the corrected linear unit, and Sigmoid() represents the calculation of logistic regression. The gating mechanism with logistic regression Sigmoid activation is used to learn the parameter Weight. The gating mechanism is implemented through the bottleneck of two fully connected layers with non-linear activation functions. The first layer reduces the dimension by selectively outputting the result using ReLU, and the second layer increases the dimension by normalizing the weights to 0-1 using Sigmoid.

[0025] u′ d =F rw (u d ,s d )=u d ·s d

[0026] Among them, F rw To refactor the weighting function, the output of the activation function is multiplied by the original feature map through its channels to obtain a weighted feature map weight, which is used to reset the importance of each feature channel, u′. d s represents the feature channel weights after the reorganization of the d-th channel. d This represents the feature channel weights of the d-th layer channel;

[0027] The fusion network part described in step 3 of this invention adopts a fusion strategy to integrate the backbone network part to obtain multi-scale features;

[0028] The detection prediction head part described in step 3 adopts a lightweight design, that is, it uses 5x5 convolution to increase the model size and expand the convolution field, and uses 1×1 convolution to generate the final output.

[0029] In the target detection network model described in step 3 of this invention:

[0030] Set a dynamic learning rate, which means the learning rate is adjusted by decreasing in multiples of 10, with the initial learning rate set to 0.1;

[0031] The target detection network adopts an end-to-end network structure design;

[0032] The CIoU method is used to calculate the bounding box loss, considering three geometric parameters: overlap area, center-to-center distance, and aspect ratio.

[0033]

[0034]

[0035]

[0036] Among them, b and b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively, where ρ represents the Euclidean distance between the two center points; c represents the diagonal distance of the smallest closure region containing both the predicted and ground truth boxes; a is the weight function, v is used to measure the similarity in aspect ratio; w, h, and w gt h gt These represent the height and width of the predicted bounding box and the height and width of the ground truth bounding box, respectively.

[0037] The training and quantization described in step 4 of this invention include:

[0038] The training is performed using samples from a dedicated scenario dataset. The weight matrix of the trained object detection network model is quantized using INT8 quantization. The parameters are represented by data values ​​of 0-256 in the object detection network. The parameter range, zero-point quantization value, and mapping interval are determined in the weight matrix. Each 32-bit floating-point value in the weight matrix is ​​converted into an 8-bit integer.

[0039] The feature proportion method described in step 5 of this invention solves the cross-edge detection allocation problem during the splitting process. The specific method is as follows:

[0040]

[0041] Where ymin represents the coordinates of the top-left corner of the target, ymax represents the coordinates of the bottom-right corner of the target, and H in This represents the height of the feature map output by the object detection network model;

[0042] ① indicates that the target is assigned to the first image src1 before compositing; ② indicates that the target is assigned to the second image src2 before compositing; ③ indicates that the target is assigned to the first image src1 before compositing, and the portion exceeding the target is assigned the maximum value of the region; ④ indicates that the target is assigned to the second image src2 before compositing, and the portion exceeding the target is assigned the minimum value of the region.

[0043] The matrix mapping of multi-camera detection information described in step 6 of this invention specifically includes:

[0044] To address the issue of mapping targets detected in a single video stream to their corresponding coordinates in the panoramic image, the panoramic mapping matrix `map` is used to obtain the coordinate information of the targets detected in the single video stream onto the panoramic image.

[0045] dst(x,y)=src(map x (x,y),map y (x,y))

[0046] Here, src is the source image, and dst is the target image, implementing a mapping from the source image src to the target image dst. During the mapping process, the data type of the image pixels remains unchanged; map x With map y This is the panoramic stitching mapping matrix, obtained when calculating feature points during panoramic stitching.

[0047] Step 7 of this invention involves filtering the preliminary panoramic detection results output by the matrix mapping using an improved NMS algorithm, including:

[0048] To address the issue of duplicate detection of stitched edges in two video streams, the Intersection over Union (IoU) is set to zero, and the detection information from the adjacency matrix is ​​fused. The following steps are set up:

[0049] Step 7-1: Mark the information of the bbox detection boxes at all seam edges;

[0050] Step 7-2: Obtain information on all bounding boxes (bboxes) under the current target category after marking.

[0051] Step 7-3: Sort the detection boxes according to their confidence level from high to low, and record the detection box with the highest confidence level.

[0052] Step 7-4: Calculate the intersection-union ratio (GIoU) ​​of the bounding box (bbox) corresponding to the highest confidence score and all remaining bounding boxes, incorporating the minimum bounding box as a penalty term. When the result is within the threshold, fuse the bounding boxes. The GIoU calculation method is as follows:

[0053]

[0054] The fusion result is the area A of the minimum closure region. c First, calculate the area A of the minimum closure region between the two boxes. c That is, the area of ​​the smallest box that contains both the predicted box and the ground truth box, then calculate the IoU, then calculate the proportion of the area in the closure region that does not belong to either box, and finally subtract this proportion from the IoU to get the GIoU.

[0055] Step 7-5: For the remaining detection boxes, repeat steps 7-3 and 7-4 until all detection boxes meet the requirements and can no longer be merged.

[0056] The improved multi-scale filtering algorithm described in step 8 of this invention includes:

[0057] Based on Kalman filtering, we introduce the extraction of the target's appearance features for multi-scale nearest neighbor matching, and then use the Hungarian algorithm for data association. The specific method is as follows:

[0058] Multi-scale nearest neighbor matching;

[0059] Prediction: By setting information on the sliding box movement at multiple scales, the position of the current box can be predicted;

[0060] Observation: Using the target detector obtained above, the position of the current bounding box is detected;

[0061] Weighted: Combines the predicted position of the current bounding box with the detected position of the current bounding box.

[0062] Beneficial effects:

[0063] This invention addresses the problem of low forward prediction efficiency in deep learning model deployment on edge device platforms. The proposed method for road panoramic target detection and tracking on edge computing platforms focuses on targets in multi-camera regions, improving the success rate and accuracy of target detection. Compared to existing technologies, it significantly enhances the efficiency of panoramic target detection on edge devices. Attached Figure Description

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0065] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0066] like Figure 1 As shown, this invention proposes a road panoramic target detection and tracking method under an edge computing platform, comprising the following steps:

[0067] Step 1: Acquire panoramic multi-channel video input images, i.e., multi-channel camera images;

[0068] Step 2: Design a multi-channel video input image adaptive synthesis method to generate a synthesized image from multiple camera images to accelerate neural network inference speed; the multi-channel video input image adaptive synthesis method includes:

[0069] The multi-camera images obtained in step 1 are synthesized in pairs. An adaptive dual-image stitching synthesis method is used to accelerate multi-image inference. The specific method is as follows:

[0070] dst=F_concat(F_resize(src1),F_resize(src2))

[0071] Wherein, F_concat is the image compositing function, F_resize is the image size modification function, the image compositing method is to vertically concatenate the matrices describing the first image src1 and the second image src2, and dst is the composite image after compositing the first image src1 and the second image src2.

[0072] Step 3: Construct an object detection network model. Based on a one-stage CNN object network, design the object detection network model structure for specific scenarios.

[0073] The target detection network model comprises three parts: a backbone network, a fusion network, and a detection prediction head.

[0074] The backbone network uses Darknet53 as its base and employs a dual-channel feature extraction mechanism. One channel is used for feature extraction, i.e., the feature channel, while the other channel introduces an adaptive attention mechanism to obtain the importance of each channel in the dual-channel configuration, i.e., the adaptive attention channel.

[0075] The adaptive attention channel uses three computational functions: a compression function, an activation function, and a weight reorganization function; the statistical data z of the d-th layer channel... d The calculation method is as follows:

[0076]

[0077] Among them, F sq For the compression function, z is the statistical data of the d-th layer channel. d This involves compressing the features of the channel feature map of that layer and using them as attention values. The method used is mean pooling, which calculates the mean value of the features in the feature map. d Let (i,j) represent the feature map of the d-th channel, (i,j) represent the data in the i-th row and j-th column of the feature map of the d-th channel, H is the feature channel height of the adaptive attention channel, and W is the feature channel width of the adaptive attention channel.

[0078] The feature channel weights s are calculated as follows:

[0079] s = F ex (z,Weight)=Sigmoid(Relu(z,Weight1),Weight2)

[0080] Among them, F ex The activation function captures the dependencies of feature channels by aggregating information. Weight1 represents the learning parameters of the first activation gating mechanism, Weight2 represents the learning parameters of the second activation gating mechanism, z represents the channel feature vector processed by the compression function, ReLU(z,Weight1) represents the calculation of the corrected linear unit, and Sigmoid() represents the calculation of logistic regression. The gating mechanism with logistic regression Sigmoid activation is used to learn the parameter Weight. The gating mechanism is implemented through the bottleneck of two fully connected layers with non-linear activation functions. The first layer reduces the dimension by selectively outputting the result using ReLU, and the second layer increases the dimension by normalizing the weights to 0-1 using Sigmoid.

[0081] u′ d =F rw (u d ,s d )=u d ·s d

[0082] Among them, F rw To refactor the weighting function, the output of the activation function is multiplied by the original feature map through its channels to obtain a weighted feature map weight, which is used to reset the importance of each feature channel, u′. d s represents the feature channel weights after the reorganization of the d-th channel. d This represents the feature channel weights of the d-th layer channel.

[0083] The fusion network part adopts a fusion strategy to integrate the backbone network part to obtain multi-scale features;

[0084] The detection prediction head uses a lightweight design, which increases the model size and expands the convolution field by using 5x5 convolutions, and uses 1×1 convolutions to generate the final output.

[0085] In the target detection network model:

[0086] Set a dynamic learning rate, which means the learning rate is adjusted by decreasing in multiples of 10, with the initial learning rate set to 0.1;

[0087] The target detection network adopts an end-to-end network structure design;

[0088] The CIoU method is used to calculate the bounding box loss, considering three geometric parameters: overlap area, center-to-center distance, and aspect ratio.

[0089]

[0090]

[0091]

[0092] Among them, b and b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively, where ρ represents the Euclidean distance between the two center points; c represents the diagonal distance of the smallest closure region containing both the predicted and ground truth boxes; a is the weight function, v is used to measure the similarity in aspect ratio; w, h, and w gt h gt These represent the height and width of the predicted bounding box and the height and width of the ground truth bounding box, respectively.

[0093] Step 4: Train and quantize the model according to the usage scenario; the training and quantization include:

[0094] The training is performed using samples from a dedicated scenario dataset. The weight matrix of the trained object detection network model is quantized using INT8 quantization. The parameters are represented by data values ​​of 0-256 in the object detection network. The parameter range, zero-point quantization value, and mapping interval are determined in the weight matrix. Each 32-bit floating-point value in the weight matrix is ​​converted into an 8-bit integer.

[0095] Step 5: Using the trained and quantized object detection network model, perform multi-path detection processing on the synthetic image using parallel inference; split the synthetic image to obtain split multi-path camera images, and solve the cross-edge detection allocation problem during the splitting by using a feature proportion method to obtain the independent detection results of the multi-path camera images of the split panoramic stitched image, generate structured data, i.e., multi-path camera detection information, and output it.

[0096] The feature proportion-based method addresses the cross-edge detection allocation problem during the splitting process. The specific method is as follows:

[0097]

[0098] Where ymin represents the coordinates of the top-left corner of the target, ymax represents the coordinates of the bottom-right corner of the target, and H in This represents the height of the feature map output by the object detection network model;

[0099] ① indicates that the target is assigned to the first image src1 before compositing; ② indicates that the target is assigned to the second image src2 before compositing; ③ indicates that the target is assigned to the first image src1 before compositing, and the portion exceeding the target is assigned the maximum value of the region; ④ indicates that the target is assigned to the second image src2 before compositing, and the portion exceeding the target is assigned the minimum value of the region.

[0100] Step 6: Perform matrix mapping on the multi-camera detection information to obtain preliminary panoramic detection results. Specific methods include:

[0101] To address the issue of mapping targets detected in a single video stream to their corresponding coordinates in the panoramic image, the panoramic mapping matrix `map` is used to obtain the coordinate information of the targets detected in the single video stream onto the panoramic image.

[0102] dst(x,y)=src(map x (x,y),map y (x,y))

[0103] Here, src is the source image, and dst is the target image, implementing a mapping from the source image src to the target image dst. During the mapping process, the data type of the image pixels remains unchanged; map x With map y This is the panoramic stitching mapping matrix, obtained when calculating feature points during panoramic stitching.

[0104] Step 7: The improved NMS algorithm is used to filter out the preliminary panoramic detection results output by the matrix mapping to obtain the panoramic detection results.

[0105] The aforementioned filtering process using the improved NMS algorithm to remove errors from the initial panoramic detection results output by the matrix mapping includes:

[0106] To address the issue of duplicate detection of stitched edges in two video streams, the Intersection over Union (IoU) is set to zero, and the detection information from the adjacency matrix is ​​fused. The following steps are followed:

[0107] Step 7-1: Mark the information of the bbox detection boxes at all seam edges;

[0108] Step 7-2: Obtain information on all bounding boxes (bboxes) under the current target category after marking.

[0109] Step 7-3: Sort the detection boxes according to their confidence level from high to low, and record the detection box with the highest confidence level.

[0110] Step 7-4: Calculate the intersection-union ratio (GIoU) ​​of the bounding box (bbox) corresponding to the highest confidence score and all remaining bounding boxes, incorporating the minimum bounding box as a penalty term. When the result is within the threshold, fuse the bounding boxes. The GIoU calculation method is as follows:

[0111]

[0112] The fusion result is the area A of the minimum closure region. c First, calculate the area A of the minimum closure region between the two boxes. cThat is, the area of ​​the smallest box that contains both the predicted box and the ground truth box, then calculate the IoU, then calculate the proportion of the area in the closure region that does not belong to either box, and finally subtract this proportion from the IoU to get the GIoU.

[0113] Step 7-5: For the remaining detection boxes, repeat steps 7-3 and 7-4 until all detection boxes meet the requirements and can no longer be merged.

[0114] Step 8: Use an improved multi-scale filtering algorithm to smooth the detection results and obtain the tracking results.

[0115] The improved multi-scale filtering algorithm includes:

[0116] Based on Kalman filtering, we introduce the extraction of the target's appearance features for multi-scale nearest neighbor matching, and then use the Hungarian algorithm for data association. The specific method is as follows:

[0117] Multi-scale nearest neighbor matching;

[0118] Prediction: By setting information on the sliding box movement at multiple scales, the position of the current box can be predicted;

[0119] Observation: Using the target detector obtained above, the position of the current bounding box is detected;

[0120] Weighted: Combines the predicted position of the current bounding box with the detected position of the current bounding box.

[0121] Example 1:

[0122] like Figure 1 As shown, the present invention provides a method for road panoramic target detection and tracking under edge devices, comprising:

[0123] Step 1: Design the target detection network, which consists of three parts: the backbone network, the fusion network Neck, and the detection prediction head.

[0124] Backbone uses Darknet53 (the backbone network used in YOLOv3, see paper: YOLOv3: An Incremental Improvement) as its foundation, employs dual-channel feature extraction, and introduces an adaptive attention channel to determine the importance of each channel, thereby enhancing useful features and suppressing features with little effect.

[0125] The adaptive attention channel uses three computational functions: a compression function, an activation function, and a weight reorganization function.

[0126]

[0127] Compression function F sq Compressed channel data compresses the features of each feature map as attention values ​​for that feature layer. The method used is mean pooling, which calculates the mean value of the features within the feature layer. d Z represents the feature map of the d-th channel. d This represents the statistical data for the d-th layer channel.

[0128] s = F ex (z,W)=Sigmoid(Relu(z,W1),W2)

[0129] Excitation function F ex It uses aggregated information to capture dependencies in feature channels:

[0130] This method uses a gating mechanism with sigmoid activation to learn the parameter weights and generate weights s for the feature channels. The gating mechanism is implemented by leveraging the bottleneck of two fully connected layers with non-linear activation functions. The first layer reduces dimensionality by selectively outputting results using ReLU, while the second layer increases dimensionality by normalizing the weights to 0-1 using sigmoid, making the results more non-linear while significantly reducing the number of parameters and computation.

[0131] u′ d =F rw (u d ,s d )=u d ·s d

[0132] Reorganization weight function F rw The output of the activation function is multiplied with the original feature map through channels to weight the feature map, thereby resetting the importance of each feature channel.

[0133] Adaptive attention channels offer excellent performance in model building and inference computation. They can achieve squeezing and activation structures without requiring newly designed functions and neural network layers, making model deployment easy.

[0134] The Neck portion employs a fusion strategy, integrating the Backbone portion to obtain multi-scale features.

[0135] The head employs a lightweight design to accelerate inference speed, uses a compact 5x5 convolution with a larger kernel to expand the convolutional field while increasing the model size within a limited range, and uses 1×1 convolution to produce the final output to speed up inference.

[0136] Set a dynamic learning rate, which decreases and adjusts in multiples of 10, with an initial learning rate of 0.1.

[0137] The output layer structure design of the deep learning target recognition model adopts an end-to-end network structure design to accelerate speed and improve accuracy.

[0138] The CIoU method is used to calculate the bounding box loss (refer to the paper: YOLOv4: Optimal Speed ​​and Accuracy of Object Detection), considering three geometric parameters: overlap area, center point distance, and aspect ratio.

[0139]

[0140]

[0141]

[0142] Where, b, b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively, with ρ representing the Euclidean distance between the two center points. c represents the diagonal distance of the smallest closure region that simultaneously contains both the predicted and ground truth boxes. Here, a is the weight function, and v measures the aspect ratio similarity. w, h, and w gt h gt These represent the height and width of the predicted bounding box and the height and width of the ground truth bounding box, respectively.

[0143] Step 2 involves training using a dedicated scenario dataset. The weight matrix is ​​quantized using INT8 quantization (refer to the paper: Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference). This utilizes low-bit data from the network to represent parameters, improving network speed while reducing accuracy loss. The parameter range, zero-point quantization values, and mapping intervals are determined within the weight matrix. Each 32-bit floating-point value in the weight matrix is ​​converted to an 8-bit integer, effectively reducing the memory space required for parameters in the network and facilitating the application of deep learning models on hardware platforms with limited computing resources.

[0144] Step 3 employs a highly modular code design, combined with the multi-core processing capabilities of edge computing devices, to design a multi-path inference method. This involves acquiring multi-channel camera images for panoramic stitching via multi-threading for image inference. Based on the original input size of the images and the processing method of the neural network input images, since the designed target detection network is square-sized, when the image width and height dimensions do not match the network input size, the network will pad the image edges with zeros to meet the target detection network input requirements. However, this prevents the full performance of the network from being utilized. Therefore, the multi-channel video images obtained in Step 1 are synthesized in pairs.

[0145] Based on the original input size of the original image and the processing method of the input image of the designed object detection network, a dual-image adaptive stitching synthesis method is adopted to accelerate the multi-image inference process. The specific method is as follows:

[0146] dst=F_concat(F_resize(src1),F_resize(src2))

[0147] Where F_concat is the image compositing function, F_resize t is the image size modification function, which concatenates the two matrices vertically, and dst is the composite image of src1 and src2.

[0148] Solving the problem of cross-edge detection allocation based on the feature proportion method.

[0149]

[0150] ① The target value is assigned to the first path image target.

[0151] ② The target value is assigned to the second image target.

[0152] ③ The target value is assigned to the first image path, and the portion exceeding the target value is assigned to the maximum value of the region.

[0153] ④ The target value is assigned to the target of the second image path, and the portion exceeding the target value is assigned to the minimum value of the region.

[0154] Step 4: For the problem of mapping the target detected in a single video stream to the panoramic image, a panoramic stitching matrix is ​​used for mapping.

[0155] dst(x,y)=src(map x (x,y),map y (x,y))

[0156] Where src is the source image and dst is the target image, the algorithm implements the mapping from the source image src to the target image dst, and the data type of the image pixels remains unchanged during the mapping process. x With map y This is the panoramic stitching mapping matrix, obtained when calculating feature points during panoramic stitching. This allows us to obtain the corresponding coordinate information of the detected target from a single video stream onto the panoramic image.

[0157] Step 5: An improved NMS algorithm is used to filter out the output layer results. For the problem of repeated detection of stitched edges in both video streams, since post-NMS has already been used after single-stream detection, the traditional IoU-based NMS method will fail, thus preserving all detection information. Therefore, based on this scenario, the detection information from the adjacency matrix with zero IoU is fused. The following steps are set:

[0158] 1) Mark the information of the bbox at all seam edges;

[0159] 2) Obtain information on all bounding boxes (bboxes) under the current target category after marking;

[0160] 3) Sort the bounding boxes in descending order of confidence, and record the bounding box with the highest current confidence.

[0161] 4) Calculate the GIoU between the bounding box corresponding to the maximum confidence and all the remaining bounding boxes. If the result is within the threshold, fuse the bounding boxes.

[0162]

[0163] The fusion result is the area A of the minimum closure region. c First, calculate the area A of the minimum closure region between the two boxes. c (The area of ​​the smallest box that includes both the predicted box and the ground truth box), then calculate the IoU, then calculate the proportion of the area in the closure region that does not belong to either box, and finally subtract this proportion from the IoU to get the GIoU.

[0164] 5) For the remaining bboxes, repeat steps (3) and (4) until all bboxes meet the requirements (i.e., no more bboxes can be merged);

[0165] Step 6: Based on the detection information, an improved multi-scale filtering algorithm is used for tracking to obtain better detection and tracking results.

[0166] Example 2:

[0167] To address the challenges of using neural networks for target detection in panoramic scenarios and deploying deep learning models on edge devices, the HiSilicon Hi3559 edge device was deployed. Tested with eight cameras in a two-channel panoramic scene, compared to the YOLOv3 network, the Hi3559 platform showed improved recognition accuracy and a detection rate 1.5 times that of YOLOv3 on the same platform. Under maximum power consumption, the Hi3559 platform achieved a single frame time of less than 80ms. With the addition of tracking algorithms, it can meet the requirements for panoramic target recognition scenarios.

[0168] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a road panoramic target detection and tracking method under an edge computing platform, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0169] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0170] This invention provides an idea and method for road panoramic target detection and tracking under an edge computing platform. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for road panoramic target detection and tracking under an edge computing platform, characterized in that, Includes the following steps: Step 1: Acquire panoramic multi-channel video input images, i.e., multi-channel camera images; Step 2: Design an adaptive synthesis method for multi-channel video input images to generate composite images from multiple camera images, which can be used to accelerate the inference speed of neural networks. Step 3: Construct an object detection network model. Based on a one-stage CNN object network, design the object detection network model structure for specific scenarios. Step 4: Train and quantize the model according to the usage scenario; Step 5: Using the trained and quantized object detection network model, perform multi-path detection processing on the synthetic image using parallel inference; The synthesized image is split to obtain split multi-camera images. The cross-edge detection allocation problem during the split is solved by the feature proportion method. The independent detection results of the multi-camera images of the split panoramic stitched image are obtained, and structured data, namely multi-camera detection information, is generated and output. Step 6: Perform matrix mapping on the detection information from multiple cameras to obtain preliminary panoramic detection results; Step 7: The improved NMS algorithm is used to filter out the preliminary panoramic detection results output by the matrix mapping to obtain the panoramic detection results. Step 8: Use an improved multi-scale filtering algorithm to smooth the detection results and obtain the tracking results; Step 7, which involves filtering the preliminary panoramic detection results output by the matrix mapping using the improved NMS algorithm, includes: To address the issue of duplicate detection of stitched edges in two video streams, the Intersection over Union (IoU) is set to zero, and the detection information from the adjacency matrix is ​​fused. The following steps are set up: Step 7-1: Mark the information of the bbox detection boxes at all seam edges; Step 7-2: Obtain information on all bounding boxes (bboxes) under the current target category after marking. Step 7-3: Sort the detection boxes according to their confidence level from high to low, and record the detection box with the highest confidence level. Step 7-4: Calculate the intersection-union ratio (GIoU) ​​of the bounding box (bbox) corresponding to the highest confidence score and all remaining bounding boxes, incorporating the minimum bounding box as a penalty term. When the result is within the threshold, fuse the bounding boxes. The GIoU calculation method is as follows: The fusion result is the area A of the minimum closure region. c First, calculate the area A of the minimum closure region between the two boxes. c That is, the area of ​​the smallest box that contains both the predicted box and the ground truth box, then calculate the IoU, then calculate the proportion of the area in the closure region that does not belong to either box, and finally subtract this proportion from the IoU to get the GIoU. Step 7-5: For the remaining detection boxes, repeat steps 7-3 and 7-4 until all detection boxes meet the requirements and can no longer be merged.

2. The road panoramic target detection and tracking method under an edge computing platform as described in claim 1, characterized in that, The multi-channel video input image adaptive synthesis method described in step 2 includes: The multi-camera images obtained in step 1 are synthesized in pairs. An adaptive dual-image stitching synthesis method is used to accelerate multi-image inference. The specific method is as follows: dst=F_concat(F_resize(src1),F_resize(src2)) Wherein, F_concat is the image compositing function, F_resize is the image size modification function, the image compositing method is to vertically concatenate the matrices describing the first image src1 and the second image src2, and dst is the composite image after compositing the first image src1 and the second image src2.

3. The road panoramic target detection and tracking method under an edge computing platform as described in claim 2, characterized in that, The target detection network model described in step 3 includes three parts: a backbone network, a fusion network, and a detection prediction head. The backbone network uses Darknet53 as its base and employs a dual-channel feature extraction mechanism. One channel is used for feature extraction, i.e., the feature channel, while the other channel introduces an adaptive attention mechanism to obtain the importance of each channel in the dual-channel configuration, i.e., the adaptive attention channel. The adaptive attention channel uses three computational functions: a compression function, an activation function, and a weight reorganization function; the statistical data z of the d-th layer channel... d The calculation method is as follows: Among them, F sq For the compression function, z is the statistical data of the d-th layer channel. d This involves compressing the features of the channel feature map of that layer and using them as attention values. The method used is mean pooling, which calculates the mean value of the features in the feature map. d Let (i,j) represent the feature map of the d-th channel, (i,j) represent the data in the i-th row and j-th column of the feature map of the d-th channel, H is the feature channel height of the adaptive attention channel, and W is the feature channel width of the adaptive attention channel. The feature channel weights s are calculated as follows: s=F ex (z,Weight)=Sigmoid(Relu(z,Weight1),Weight2) Among them, F ex The activation function captures the dependencies of feature channels by aggregating information. Weight1 represents the learning parameters of the first activation gating mechanism, Weight2 represents the learning parameters of the second activation gating mechanism, z represents the channel feature vector processed by the compression function, ReLU(z,Weight1) represents the calculation of the corrected linear unit, and Sigmoid() represents the calculation of logistic regression. The gating mechanism with logistic regression Sigmoid activation is used to learn the parameter Weight. The gating mechanism is implemented through the bottleneck of two fully connected layers with non-linear activation functions. The first layer reduces the dimension by selectively outputting the result using ReLU, and the second layer increases the dimension by normalizing the weights to 0-1 using Sigmoid. in ′ d =F rw (in d ,with d )=in d ·with d Among them, F rw To restructure the weighting function, the output of the activation function is multiplied by the original feature map through each channel to obtain a weighted feature map weight, which is used to reset the importance of each feature channel. ′ d s represents the feature channel weights after the reorganization of the d-th channel. d This represents the feature channel weights of the d-th layer channel.

4. The road panoramic target detection and tracking method under an edge computing platform as described in claim 3, characterized in that, The fusion network part described in step 3 adopts a fusion strategy to integrate the backbone network part to obtain multi-scale features; The detection prediction head part described in step 3 adopts a lightweight design, that is, it uses 5x5 convolution to increase the model size and expand the convolution field, and uses 1×1 convolution to generate the final output.

5. The road panoramic target detection and tracking method under an edge computing platform as described in claim 4, characterized in that, In the target detection network model described in step 3: Set a dynamic learning rate, which means the learning rate is adjusted by decreasing in multiples of 10, with the initial learning rate set to 0.1; The target detection network adopts an end-to-end network structure design; The CIoU method is used to calculate the bounding box loss, considering three geometric parameters: overlap area, center-to-center distance, and aspect ratio. Among them, b and b gt ρ and c represent the center points of the predicted and ground truth boxes, respectively, where ρ represents the Euclidean distance between the two center points; c represents the diagonal distance of the smallest closure region containing both the predicted and ground truth boxes; a is the weight function, v is used to measure the similarity in aspect ratio; w, h, and w gt h gt These represent the height and width of the predicted bounding box and the height and width of the ground truth bounding box, respectively.

6. The road panoramic target detection and tracking method under an edge computing platform as described in claim 5, characterized in that, The training and quantization described in step 4 include: The training is performed using samples from a dedicated scenario dataset. The weight matrix of the trained object detection network model is quantized using INT8 quantization. The parameters are represented by data values ​​of 0-256 in the object detection network. The parameter range, zero-point quantization value, and mapping interval are determined in the weight matrix. Each 32-bit floating-point value in the weight matrix is ​​converted into an 8-bit integer.

7. The road panoramic target detection and tracking method under an edge computing platform as described in claim 6, characterized in that, The feature proportion method described in step 5 solves the cross-edge detection allocation problem during the splitting process. The specific method is as follows: Where ymin represents the coordinates of the top-left corner of the target, ymax represents the coordinates of the bottom-right corner of the target, and H in This represents the height of the feature map output by the object detection network model; ① indicates that the target is assigned to the first image src1 before compositing; ② indicates that the target is assigned to the second image src2 before compositing; ③ indicates that the target is assigned to the first image src1 before compositing, and the portion exceeding the target is assigned the maximum value of the region; ④ indicates that the target is assigned to the second image src2 before compositing, and the portion exceeding the target is assigned the minimum value of the region.

8. The road panoramic target detection and tracking method under an edge computing platform as described in claim 7, characterized in that, Step 6, which involves matrix mapping of the multi-camera detection information, specifically includes the following methods: To address the issue of mapping targets detected in a single video stream to their corresponding coordinates in the panoramic image, the panoramic mapping matrix `map` is used to obtain the coordinate information of the targets detected in the single video stream onto the panoramic image. dst(x,y)=src(map x (x,y),map y (x,y)) Here, src is the source image, and dst is the target image, implementing a mapping from the source image src to the target image dst. During the mapping process, the data type of the image pixels remains unchanged; map x With map y This is the panoramic stitching mapping matrix, obtained when calculating feature points during panoramic stitching.

9. The road panoramic target detection and tracking method under an edge computing platform as described in claim 8, characterized in that, The improved multi-scale filtering algorithm described in step 8 includes: Based on Kalman filtering, we introduce the extraction of the target's appearance features for multi-scale nearest neighbor matching, and then use the Hungarian algorithm for data association. The specific method is as follows: Multi-scale nearest neighbor matching; Prediction: By setting information on the sliding box movement at multiple scales, the position of the current box can be predicted; Observation: Using the target detector obtained above, the position of the current bounding box is detected; Weighted: Combines the predicted position of the current bounding box with the detected position of the current bounding box.

Images