Robust aerocar air-ground target matching method and system

Abstract

The invention discloses a robust hovercar air-ground target matching method and system, and belongs to the field of hovercar computer vision. The target matching method comprises the following steps: receiving images of a ground driving mode and a flight mode of the hovercar; based on the YOLOv5s network, introducing a CA attention mechanism and improving a loss function to detect the received image; carrying out target tracking processing based on a DeepSORT algorithm according to the detection result, and carrying out re-identification preprocessing on the tracking result to obtain a to-be-detected data set; for the target image and the to-be-detected data set, target re-identification processing based on a FastReID algorithm is carried out; performing cosine similarity calculation matching on a re-identification result, and performing re-tracking; according to the invention, the information of the flight view angle and the ground driving view angle is fused, so that the tracking accuracy and robustness are remarkably improved, and accurate positioning and tracking of the same target image under double view angles are realized.

Description

Technical Field

[0001] The present invention belongs to the field of flying car computer vision, and in particular relates to a robust flying car air-ground target matching method and system. Background Art

[0002] The rapid development of flying car technology and its application in urban transportation, logistics and emergency rescue have put forward higher and higher requirements for the autonomous navigation, safety control and collaborative operation of vehicles. Among them, it is crucial to achieve accurate target recognition and tracking of flying cars in the process of air-to-ground mode switching. However, existing target recognition and tracking technologies face many challenges in flying car scenarios: single-view target tracking is easily affected by view limitation and target occlusion; the target size is reduced in flight mode and is easily affected by lighting, atmospheric interference, etc.; in ground driving mode, the details of medium and long-distance targets are missing; target recognition and matching in different modes (flight mode and ground driving mode) are also difficult. Existing technologies such as YOLOv5s, DeepSORT and FastReID have achieved certain results in their respective fields, but when directly applied to flying car air-to-ground target matching, their robustness and accuracy are still not enough to meet the requirements of high precision and high real-time performance, and it is difficult to effectively solve complex situations such as target occlusion, view change, and lighting change.

[0003] Therefore, there is an urgent need for a robust target matching method that can effectively cope with the challenges of flying car air-to-ground mode switching and multi-vehicle collaborative operations. Summary of the invention

[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a robust flying car air-to-ground target matching method and system, which solves the problems in the prior art.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A robust flying car air-ground target matching method comprises the following steps:

[0007] Receive images of the flying car in ground driving mode and flight mode;

[0008] Based on the YOLOv5s network, the CA attention mechanism is introduced and the loss function is improved to detect the received images;

[0009] According to the detection results, target tracking processing based on the DeepSORT algorithm is performed, and the tracking results are pre-processed for re-identification to obtain the data set to be detected;

[0010] For the target image and the data set to be detected, target re-identification processing based on the FastReID algorithm is performed;

[0011] The cosine similarity calculation and matching are performed on the re-identification results, and re-tracking is performed.

[0012] Furthermore, the CA attention mechanism decomposes the 2D pooling into two 1D average poolings, and performs pooling operations in the horizontal and vertical directions according to two different pooling kernels.

[0013] Furthermore, the improved loss function L MPDIoU for:

[0014] L MPDIoU =1-MPDIoU

[0015]

[0016] in, Represents the upper left and lower right coordinates of the prediction box, represents the upper left and lower right coordinates of the actual frame, w and h represent the width and height of the input image, C represents the minimum enclosed area, A represents the predicted frame range, and B represents the real frame range. represents the square of the Euclidean distance between the upper left corners of the two boxes, Represents the square of the Euclidean distance between the lower right corners of the two boxes.

[0017] Furthermore, the process of re-identification preprocessing of the tracking results is: saving the image of the tracking results and the prediction box txt file frame by frame, and intercepting the saved image according to the prediction box coordinates to obtain the data set to be detected.

[0018] Furthermore, in the target re-identification process, after pre-processing the image, the backbone network is used to generate features, and then the aggregation module is used to generate global features, and then the Head module is used to normalize the global features;

[0019] The backbone network uses ResNet, ResNeXt and ResNest as the network architecture and integrates the IBN mechanism; the Head module combines the normalization head, linear head and simplified head.

[0020] Furthermore, the steps of calculating and matching the cosine similarity of the re-identification results are as follows:

[0021] S51, sequentially read the re-identification result vectors of the data set to be detected, and calculate the cosine similarity with the target image feature vector;

[0022] S52, sorting the cosine similarity results of the data set to be detected by size.

[0023] A robust flying car air-ground target matching system, comprising:

[0024] Image receiving module: receiving images of the flying car in ground driving mode and flight mode;

[0025] The image detection module is based on the YOLOv5s network, introduces the CA attention mechanism and improves the loss function to detect the received images;

[0026] Tracking processing module: Based on the detection results, target tracking processing is performed based on the DeepSORT algorithm, and re-identification preprocessing is performed on the tracking results to obtain the data set to be detected;

[0027] Re-identification module: For the target image and the data set to be detected, the target re-identification processing based on the FastReID algorithm is performed;

[0028] Matching tracking module: performs cosine similarity calculation and matching on the re-identification results, and performs re-tracking.

[0029] A computer storage medium stores a readable program, which can execute the above-mentioned robust flying car air-ground target matching method when the program is running.

[0030] An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0031] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned robust flying car air-to-ground target matching method.

[0032] A computer program product includes computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the above-mentioned robust flying car air-to-ground target matching method.

[0033] Beneficial effects of the present invention:

[0034] 1. The present invention realizes the recognition of small targets from an aerial perspective by improving the loss function and introducing an attention mechanism.

[0035] 2. The present invention realizes ID matching of the same target in two modes of the flying car through an air-ground dual-perspective target matching method, effectively solving the problem of target occlusion in a single perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 is a flow chart of the overall air-to-ground target matching method of the present invention;

[0038] Figure 2 It is a flow chart of the result output of the air-to-ground target matching method of the present invention;

[0039] Figure 3 It is a schematic diagram of the air-to-ground target matching result curve of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Example 1

[0042] like Figure 1 As shown, a robust flying car air-ground target matching method includes the following steps:

[0043] S1, receiving images of the flying car in ground driving mode and flight mode;

[0044] The camera equipped on the flying car is used to receive images of the flying car's ground driving model (UGV mode) and flight mode (UAV mode).

[0045] S2, based on the YOLOv5s network, introduces the CA attention mechanism and improves the loss function to detect the received images to improve the detection accuracy of small targets under the flying perspective and medium and long-distance targets under the ground driving perspective;

[0046] The CA attention mechanism embeds position information into channel attention, so that the grid can reduce parameters to reduce computational complexity while extracting feature information; the CA attention mechanism decomposes 2D pooling into two 1D average pooling, and performs pooling operations in the horizontal and vertical directions according to two different pooling kernels.

[0047] For the cth channel of the input feature map x with a height of h, the output of its information after feature processing is:

[0048]

[0049] Where W is the width of the input feature map, x c (h,i) represents the pixel value of the cth channel at height h and width i.

[0050] For the cth channel of the input feature map x with a width of w, the output of its information after feature processing is:

[0051]

[0052] Where H is the input feature map height, x c (j,w) represents the pixel value of the cth channel at height j and width w.

[0053] After that, the two average pooled feature maps in the vertical and horizontal directions are simplified and concatenated, followed by a one-dimensional convolution and delinearization using the Sigmoid function. The process formula is as follows:

[0054] f=δ(F 1 ([z h ,z w ])) (3)

[0055] Among them, [z h ,z w ] means to set z h ,z w For splicing, F 1 (·) represents a 1×1 convolution operation, δ(·) represents normalization and nonlinear processing, and the final representation contains intermediate features of horizontal and vertical spatial information, and r represents the pooling step size.

[0056] After processing the feature map, it is divided into and Two parts, and f h and f w Adjusted to be connected to the channel of the input feature x, the following formula represents this process:

[0057] g h =σ(F h (f h )) (4)

[0058] g w =σ(F w (f w )) (5)

[0059] Among them, F h (·), F w (·) Use two 1×1 convolutions F h and F w f h and f w Perform channel adjustment function, σ(·) represents Sigmoid function, and multiply the result with the attention weight matrix to get the final output:

[0060]

[0061] Among them, y c (i,j) represents the attention weight matrix, Indicates that the height of the c-th channel is the attention weight parameter of the i-th row, It represents the attention weight parameter of the cth channel width as the jth column. The attention weight matrix is ​​the most critical part of the CA attention mechanism. The weight determines the location of the attention information, thereby realizing the processing of the input data.

[0062] MPDIoU is an improved loss function based on DIoU. By maximizing the DIoU value of two bounding boxes, the network can better learn the position and shape of the bounding box. MPDIoU includes most of the relevant parameters, including but not limited to the center point distance, overlapping and non-overlapping areas, etc. Its calculation process is as follows:

[0063]

[0064]

[0065] in, Represents the upper left and lower right coordinates of the prediction box, represents the upper left and lower right coordinates of the actual frame, w and h represent the width and height of the input image, C represents the minimum enclosed area, A represents the predicted frame range, and B represents the real frame range. represents the square of the Euclidean distance between the upper left corners of the two boxes, Represents the square of the Euclidean distance between the lower right corners of the two boxes.

[0066] The MPDIoU loss function is expressed as follows:

[0067] L MPDIoU =1-MPDIoU (11)

[0068] S3, for the detection results in S2, target tracking processing based on the DeepSORT algorithm is performed, and re-identification preprocessing is performed on the tracking results to obtain a data set to be detected;

[0069] The process of DeepSORT algorithm is:

[0070] Step (1) Initialization: Create trajectory boxes for all targets detected in the first frame of the video, and then use Kalman filtering to predict trajectories;

[0071] Step (2) track update: refresh the detection frame and track frame related information, and check whether each track has been confirmed;

[0072] Step (3) Matching and state determination: For unconfirmed trajectories, use IOU (intersection over union) matching to calculate the cost matrix between the detection box of the current frame and the trajectory box of the previous frame, and output the matching result; the following discussion is made based on the different matching results:

[0073] If the existing trajectory frame is successfully matched, the Kalman filter is used to predict the new trajectory position, and the process returns to step (2) to perform the trajectory update operation, which means that the existing tracking is continuous in this frame;

[0074] If the detection box does not match any trajectory box, create a trajectory box for the target detected by the detection box, assign a new ID, and then execute step (2);

[0075] If the track box does not match the detection box, and if the track box cannot match the corresponding detection box within the threshold range, then delete the track box and the corresponding ID;

[0076] Step (4) Confirm the cascade matching of the trajectory: Perform cascade matching operations on the boxes confirmed by the Kalman filter and output three results:

[0077] When the track frame successfully matches the detection frame, the track update and subsequent matching and judgment related operations are performed.

[0078] When a “track mismatch” occurs, IOU matching is used to associate it with the detection box that the Kalman filter algorithm did not successfully confirm.

[0079] When a “detection mismatch” occurs, IOU matching is used to associate it with the trajectory box that the Kalman filter algorithm did not successfully confirm.

[0080] Step (5) Repeat steps (2), (3) and (4).

[0081] In addition, the DeepSORT algorithm optimizes target tracking by introducing a trajectory state confirmation mechanism (three consecutive frames matched as a confirmed state). While reducing the frequent ID switching, it optimizes trajectory deletion in the case of target occlusion by setting the maximum number of mismatches, effectively improving the robustness of tracking.

[0082] The process of re-identification preprocessing of tracking results is as follows:

[0083] The image of the tracking result and the prediction box txt file are saved frame by frame, and the saved image is intercepted according to the prediction box coordinates to obtain the data set to be detected (i.e. the tracking result after preprocessing).

[0084] S4, performing target re-identification processing based on the FastReID algorithm for the target image and the dataset to be detected;

[0085] The steps of target re-identification processing based on the FastReID algorithm include:

[0086] S41, performing feature processing on the image, wherein the feature processing part is divided into four modules, namely, image pre-processing module, backbone network, aggregation module and Head module.

[0087] First, in the image preprocessing module, since the collected images are not of uniform size, the images need to be scaled to meet the network input requirements. The image is flipped horizontally or vertically to increase data diversity. In addition, random erasing and random repair methods are used to randomly select areas on the image and replace the pixels in the area with random values ​​and random image blocks, thereby enhancing the data and helping the model resist overfitting, and enhancing the model's adaptability to local occlusion of the image.

[0088] For the backbone network of re-identification, ResNet, ResNeXt and ResNest are selected as the main network architectures for training, and Non-local and Instance batch normalization (IBN) mechanisms are integrated to enhance the ability to capture long-distance dependencies and the robustness to background changes.

[0089] After the backbone network generates features, the aggregation module is applied to generate global features, in which multiple pooling methods including maximum pooling, average pooling, generalized pooling and attention pooling are combined.

[0090] The pooling layer is based on X∈R W×H×C As input, and f∈R 1×1×C As the output, W, H, and C are the width, height, and number of channels of the feature map, respectively. The pooling method is defined as follows:

[0091]

[0092] Among them, f cM represents the maximum pooling weight, f cA represents the average pooling weight, f cG represents the generalized pooling weight, f c-At represents the attention pooling weight, x represents a single element, w represents the element attention weight, and a represents the control coefficient of Gem pooling; W c represents the attention weight of the attention pool, |X c | represents the number of pooling elements.

[0093] After the aggregation module generates global features, it is necessary to use the Head module to perform normalization and other operations on them to improve the training stability and performance of the model. The Head module combines the batch normalization head, linear head, and reduction head. The linear head contains only one decision layer, the BN head contains a BN layer and a decision layer, and the reduction head contains conv, BN, relu, and dropout operations, connecting a reduction layer and a decision layer. The design of these Head modules allows the model to apply different normalization and decision strategies at different feature processing stages to optimize the final re-identification performance.

[0094] The BN layer can solve the internal covariant shift. For a given feature vector f∈R m×C , the feature vector calculation formula of the BN layer is:

[0095]

[0096] Among them, f bn represents the eigenvector of the BN layer, γ and β represent the scale and shift parameters, ε represents the numerical stability variance constant, and f i represents the feature vector of the i-th sample, μ represents the batch mean, σ 2 represents the batch variance, and m represents the number of samples.

[0097] S42, model training is performed on the vehicle re-identification dataset VeRi-776 and the pedestrian dataset Market-1501;

[0098] When training the model, multiple loss functions including cross-entropy loss, triplet loss, arcface loss and circle loss are used jointly.

[0099] S43, applying the trained weight file to the target tracking preprocessed result, performing target re-identification detection, and outputting the result (2048-dimensional feature vector).

[0100] S5, performing cosine similarity calculation and matching on the re-identification result of S4, and performing re-tracking;

[0101] The FastReID algorithm is used to obtain the 2048-dimensional feature vector (re-identification result) of the target image and the dataset to be detected. The steps for calculating and matching the cosine similarity of the re-identification result are as follows:

[0102] S51, sequentially read the re-identification result vectors of the data set to be detected, and calculate the cosine similarity with the target image feature vector;

[0103] S52, sort the cosine similarity results of the data set to be tested by size. .

[0104] The steps for further tracking are:

[0105] Keep the screenshot with the highest similarity in each frame, and record the similarity of the screenshot in the corresponding txt document. According to the prediction box output document of target tracking, draw all the prediction boxes corresponding to the ID in the video, and draw a coordinate axis with the number of frames as the horizontal axis. If the detection target is found in the frame, the vertical axis is 1, otherwise it is 0. The results are output and displayed frame by frame. The specific process is as follows: Figure 2 As shown;

[0106] Figure 3 It is clearly shown that the number of frames in which the target is detected by the fusion perspective (green curve) (i.e., the number of horizontal coordinates with y=1) is significantly greater than the number of frames in which the target is detected under a single perspective (including the aerial perspective and the ground perspective). This shows that the target detection and tracking under the air-ground collaboration of the flying car effectively improves the undetectable situation caused by target occlusion and target loss under a single perspective.

[0107] Based on similar inventive concepts, an embodiment of the present invention further provides a computer storage medium storing a readable program, which can execute the above-mentioned robust flying car air-to-ground target matching method when the program is running.

[0108] Based on similar inventive concepts, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0109] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned robust flying car air-to-ground target matching method.

[0110] Based on similar inventive concepts, an embodiment of the present invention further provides a computer program product, including computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the above-mentioned robust flying car air-to-ground target matching method.

[0111] Example 2

[0112] In this embodiment, a robust flying car air-ground target matching system is proposed, including:

[0113] Image receiving module: receiving images of the flying car in ground driving mode and flight mode;

[0114] The image detection module is based on the YOLOv5s network, introduces the CA attention mechanism and improves the loss function to detect the received images;

[0115] Tracking processing module: Based on the detection results, target tracking processing is performed based on the DeepSORT algorithm, and re-identification preprocessing is performed on the tracking results to obtain the data set to be detected;

[0116] Re-identification module: For the target image and the data set to be detected, the target re-identification processing based on the FastReID algorithm is performed;

[0117] Matching tracking module: performs cosine similarity calculation and matching on the re-identification results, and performs re-tracking.

[0118] The method of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0119] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A robust flying car air-ground target matching method, characterized in that: The following steps are involved: Receive images of the flying car in ground driving mode and flight mode; Based on the YOLOv5s network, the CA attention mechanism is introduced and the loss function is improved to detect the received images; According to the detection results, target tracking processing based on the DeepSORT algorithm is performed, and the tracking results are pre-processed for re-identification to obtain the data set to be detected; For the target image and the data set to be detected, target re-identification processing based on the FastReID algorithm is performed; The cosine similarity calculation and matching are performed on the re-identification results, and re-tracking is performed.

2. A robust flying car air-to-ground target matching method according to claim 1, characterized in that: The CA attention mechanism decomposes 2D pooling into two 1D average poolings, and performs pooling operations in the horizontal and vertical directions according to two different pooling kernels.

3. A robust flying car air-to-ground target matching method according to claim 1, characterized in that: Improved loss function L MPDIoU for: L MPDIoU =1-MPDIoU in, Represents the upper left and lower right coordinates of the prediction box, represents the upper left and lower right coordinates of the actual frame, w and h represent the width and height of the input image, C represents the minimum enclosed area, A represents the predicted frame range, and B represents the real frame range. represents the square of the Euclidean distance between the upper left corners of the two boxes, Represents the square of the Euclidean distance between the lower right corners of the two boxes.

4. A robust flying car air-to-ground target matching method according to claim 1, characterized in that: The process of re-identification preprocessing of the tracking results is as follows: the image of the tracking results and the predicted box txt file are saved frame by frame, and the saved image is intercepted according to the predicted box coordinates to obtain the data set to be detected.

5. A robust flying car air-to-ground target matching method according to claim 1, characterized in that: In the target re-identification process, after pre-processing the image, the backbone network is used to generate features, and then the aggregation module is used to generate global features, and then the Head module is used to normalize the global features; The backbone network uses ResNet, ResNeXt and ResNest as the network architecture and integrates the IBN mechanism; the Head module combines the normalization head, linear head and simplified head.

6. A robust flying car air-to-ground target matching method according to claim 4, characterized in that: The steps for calculating and matching the cosine similarity of the re-identification results are: S51, sequentially read the re-identification result vectors of the data set to be detected, and calculate the cosine similarity with the target image feature vector; S52, sorting the cosine similarity results of the data set to be detected by size.

7. A robust flying car air-ground target matching system, characterized in that: include: Image receiving module: receiving images of the flying car in ground driving mode and flight mode; The image detection module is based on the YOLOv5s network, introduces the CA attention mechanism and improves the loss function to detect the received images; Tracking processing module: Based on the detection results, target tracking processing is performed based on the DeepSORT algorithm, and re-identification preprocessing is performed on the tracking results to obtain the data set to be detected; Re-identification module: For the target image and the data set to be detected, the target re-identification processing based on the FastReID algorithm is performed; Matching tracking module: performs cosine similarity calculation and matching on the re-identification results, and performs re-tracking.

8. A computer storage medium storing a readable program, characterized in that: When the program is running, the robust flying car air-ground target matching method described in any one of claims 1 to 6 can be executed.

9. An electronic device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to a robust flying car air-to-ground target matching method as described in any one of claims 1-6.

10. A computer program product comprising computer instructions, characterized in that: The computer instructions instruct the computing device to perform operations corresponding to the robust flying car air-to-ground target matching method as described in any one of claims 1-6.

Images