A method and device for identifying and tracking multiple targets in a road environment

Abstract

The present invention discloses a method and device for multi-target identification and tracking in a road environment, comprising the following steps: establishing a multi-target identification and tracking model, including a YOLOv5 improved network and a Deep Sort improved network, combining the two to increase the transmission of target classification information. The YOLOv5 improved network adds a spatial attention mechanism and a channel attention mechanism, uses an improved BiFPN structure, an improved historical non-maximum suppression algorithm and a slice-assisted reasoning framework in the network. The Deep Sort improved network adds a target classification information verification structure and a still object retention structure. An existing road environment image is obtained, and feature images such as street lights, zebra crossings, pedestrians and vehicles are segmented, and input as data sets into the YOLOv5 improved network and the Deep Sort improved network to obtain a weight file. When a vehicle is driving on the road, road information is obtained in real time through a camera, and after inputting into the model, multi-target tracking of the road environment is realized.

Description

Technical Field

[0001] The invention relates to the field of computer vision and is a method and device for identifying and tracking multiple targets in a road environment. Background Art

[0002] With the development of intelligent vehicles, obtaining and analyzing information about the surrounding environment has become an indispensable link. Vehicles can obtain information about the surrounding road environment through cameras, but only further analysis and understanding can lead to the next decision-making. This requires the help of computer vision algorithms to complete target recognition in images and track multiple targets at the same time. Through multi-target recognition and tracking, the vehicle can enhance its perception of the surrounding environment with the help of artificial intelligence when driving on the road, helping the driver to provide important information on potential dangers. When the vehicle is stationary, due to changes in the surrounding environment such as occlusion or light changes, static objects that were previously successfully identified, such as traffic lights or zebra crossings, may be constantly between detection and non-detection. Summary of the invention

[0003] Purpose of the invention: The present invention aims to solve the defects of the prior art, such as erroneous tracking, easy neglect of small targets, and inability to accurately and continuously track targets due to occlusion or environmental interference, and provides a method for multi-target identification and tracking in a road environment to solve the above problems.

[0004] A method for identifying and tracking multiple targets in a road environment comprises the following steps:

[0005] A multi-target recognition and tracking model is established, including an improved YOLOv5 network and an improved Deep Sort network; the improved YOLOv5 network uses an improved BiFPN structure, in which a spatial attention module and a channel attention module are embedded; an improved historical non-maximum suppression algorithm and a slice-assisted reasoning framework improve the accuracy of target position detection, enhance the detection accuracy when the same type of targets cover each other, and reduce the error rate of small target detection; the improved DeepSort network adds a target classification information verification structure and a static object preservation structure;

[0006] Obtain existing road environment images for target segmentation and annotation, and input them into the multi-target recognition and tracking model as a data set for training; input the road environment images of the video to be processed into the trained model to realize multi-target tracking in the road environment;

[0007] The road environment image of the video to be processed is input into the YOLOv5 improved network to obtain the target feature information; the target feature information is input into the Deep Sort improved network, and the current image and the image processed by the YOLOv5 improved network are subjected to cascade matching of targets and trajectories, target classification matching and IOU matching to obtain the multi-target tracking trajectory.

[0008] Optionally, the improved BiFPN structure includes:

[0009]

[0010]

[0011] The BiFPN structure uses weighted feature fusion, where P6 corresponds to the 6th layer in the YOLOv5 improved network. Corresponding to the 12th layer in the YOLOv5 improved network, Corresponding to the 19th layer in the YOLOv5 improved network, ω1 is the feature information transmitted by P6 to The corresponding weight is ω2, which is the feature information transmitted by P7 after upsampling. The corresponding weight is ω′1, which is the feature information transmitted across P6 to The corresponding weight is ω′2 Pass feature information to The corresponding weight is ω′3 After upsampling, feature information is passed to ; ε is a small value to prevent the denominator from being zero; Conv means performing a convolution operation, Resize means adjusting the image size without losing image information, P7 and Represents a node in the network.

[0012] Optionally, the spatial attention module and the channel attention module specifically include:

[0013] First, perform the channel attention operation, and input the feature map F into two convolutions after the maximum pooling and average pooling respectively. After adding the two, perform sigmoid activation to obtain the feature map M. c1 ; Then perform spatial attention operation on the feature map M c1 Use convolution with a convolution kernel size of 7×7 to perform average pooling and maximum pooling respectively, add the two together, perform convolution processing and sigmoid activation to obtain M c2 ; Finally, M c2 Feature fusion is performed with feature map F to obtain feature map M c3 , complete the attention mechanism;

[0014] M c1 (F)=σ(ω1×MLP(AvgPool(F))+ω2×MLP(MaxPool(F)))

[0015] M c2 (F) = σ(f 7×7 ([AvgPool(F); (MaxPool(F)]))

[0016] M c3 =σ(ω3×MLP(M c2 (M c1 (F))+ω4×MLP(F)))

[0017] Where F is the input feature map, MLP is the convolutional network, and f 7×7 is the convolution processing with a convolution kernel size of 7×7, σ is the sigmoid activation function, AvgPool is the average pooling function, MaxPool is the maximum pooling function; ω1, ω1, ω3 and ω4 are weights respectively.

[0018] Optionally, the current image and the image processed by the YOLOv5 improved network are subjected to cascade matching of targets and trajectories, target classification matching, and IOU matching to obtain multi-target tracking trajectories, specifically including:

[0019] The target frame graph is matched using the improved historical non-maximum suppression algorithm. The specific steps include:

[0020] First, the target detection position and classification information of the previous frame are retained, recorded as boxes3, which contains information (x, y, w, h, c, conf), which are the center coordinates of the target in the previous frame, the width and height of the detection box, the classification information and the confidence; the traditional non-maximum suppression algorithm is performed on this frame to obtain the predicted target information of this frame, recorded as boxes2, which contains information (x, y, w, h, c, conf). The predicted target information eliminated when the non-maximum suppression algorithm is performed on this frame is recorded as boxes1, which contains information (x, y, w, h, c, conf);

[0021] If the IoU between the prediction box boxes2 that is eliminated in this frame and the prediction box boxes3 that has been identified in the previous frame is greater than a certain threshold, HIoU is further used to determine whether it needs to be retained, as shown in the formula:

[0022]

[0023] Among them, ρ 2 (,) is the Euclidean distance, b1 is the prediction box obtained by the traditional non-maximum suppression algorithm in this frame, b2 is the prediction box eliminated by the non-maximum suppression algorithm in this frame, and b0 is the prediction box closest to b2 in the prediction box retained in the previous frame;

[0024] when When , it means that compared with the previous frame, the distance between the two objects of the same type in this frame becomes shorter, closer to each other, and the occlusion is aggravated. b2 can be further analyzed to confirm whether to keep it; when , it may be a misjudgment. In fact, b2 and b1 are the same object, so b2 is not retained.

[0025] Optionally, further analysis of b2 is conducted to confirm whether to retain the specific inclusions;

[0026] b0, b1 and b2 are the center points of the prediction boxes PB0, PB1 and PB2 respectively. PB0 is the prediction box retained in the previous frame, PB1 is the prediction box eliminated in the current frame under the traditional non-maximum suppression algorithm, and PB2 is the prediction box retained in the current frame under the traditional non-maximum suppression algorithm.

[0027]

[0028] s i represents the judgment score, b1 represents the center point of the prediction box with the highest confidence in this frame, b2 represents the center point of the prediction box that needs to be judged whether to be removed in this frame, and ε represents the threshold of historical non-maximum suppression; in one frame, there may be multiple b2s that are removed and retained after HIoU, and multiple b2s need to be screened and s is selected. i The b2 with the highest score is retained.

[0029] Optionally, the frame image to be processed needs to go through the slice-assisted reasoning framework, and the specific steps include:

[0030] First, combine the current frame image Im1 and the target obtained in the previous frame image Im0 to make a comprehensive judgment. If the small target Im obtained in the previous frame image 0a If a small target is not detected in the current frame but is near the location, slice-assisted reasoning is performed;

[0031] According to the small target Im in the previous frame image Im0 0a Information (x 0a ,y 0a , w 0a ,h 0a , c 0a ,conf 0a ), in this frame (x 0a ,y 0a ) around (w 0a ×k0,h 0a ×k0) sliding window in this frame image Im1, slice processing is performed in sequence according to the sliding step size k1 (k1<w0), and the slices can be obtained The number of slices can be controlled, and multiple slices can be spliced ​​together to get the image Im p , control picture Im p The range is in the picture [w min , w max ] and [h min ,h max], the insufficient part is filled with blanks, where k0 is the proportional coefficient; the spliced ​​picture Im p Update Im p Input into the detection network; if multiple small targets Im appear 0b or Im 0c , then the slice obtained after slice-assisted reasoning They are then spliced ​​into a picture in a certain order and re-input into the network for recognition.

[0032] Optionally, the DeepSort improved network performs multi-target matching tracking steps including:

[0033] First, extract the motion feature information and classification information of the obtained target, and classify and match it with the determined trajectory. The specific trajectory data predicted by the Kalman filter in the DeepSort improved network is:

[0034]

[0035] The specific target data obtained by the YOLOv5 improved network are:

[0036] I obj =(x obj ,y obj , γ obj ,h obj , t obj , c obj )

[0037] Where x, y, γ, and h represent the central horizontal coordinate, vertical coordinate, aspect ratio, and height of the candidate box where the target is located, respectively; They represent the change of the horizontal coordinate of the center of the candidate box, the change of the vertical coordinate, the change of the aspect ratio of the candidate box, and the change of the height of the candidate box respectively; t and c represent the classification of the target or trajectory respectively.

[0038] Optionally, the target and trajectory classification information are matched during classification matching. Only when they match can cascade matching and IOU matching be continued;

[0039] When the vehicle is stationary, the features of stationary objects such as zebra crossings, traffic lights and signs are retained. When the stationary objects cannot be detected due to environmental influences, the still features of the previous and next frames are matched.

[0040] A road environment multi-target identification and tracking device comprises a camera, a processor, a memory and a communication bus; the camera obtains the road environment around the vehicle in real time and outputs the image information to the processor; the processor is used to call the program instructions in the memory to execute any of the road environment multi-target identification and tracking methods described in the present invention.

[0041] A computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method for identifying and tracking multiple targets in a road environment as described in any one of the present inventions.

[0042] Compared with the prior art, the present invention has the following significant advantages:

[0043] The present invention adopts an improved YOLOv5 and an improved DeepSort fusion network, adds a BiFPN structure, a spatial attention mechanism and a channel attention mechanism to YOLOv5, an improved historical non-maximum suppression algorithm and a slice-assisted reasoning framework, improves the accuracy of target position detection, strengthens the detection accuracy when the same type of targets cover each other, and reduces the error rate of small target detection. Classification matching and still life matching are added to the DeepSort network to overcome the problems of incorrectly tracking targets of different classifications in traditional DeepSort tracking and being unable to continuously track fixed small targets due to environmental influences when parking. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0045] Figure 1 It is the BiFPN structure in the improved YOLOv5 network in the present invention;

[0046] Figure 2 Schematic diagram of the improved historical non-maximum suppression algorithm in the present invention;

[0047] Figure 3 It is a schematic diagram of the slice-assisted reasoning framework in the present invention;

[0048] Figure 4 It is a schematic diagram of the multi-target identification and tracking process in the present invention;

[0049] Figure 5 It is a schematic diagram of the road environment tracking effect in the present invention;

[0050] Figure 6 It is a schematic diagram of the physical structure of an electronic device according to an example of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The method for multi-target recognition and tracking in a road environment of the present invention comprises the following steps: establishing a multi-target recognition and tracking model, including an improved YOLOv5 network and an improved DeepSort network. The improved YOLOv5 network uses an improved BiFPN structure, a spatial attention module and a channel attention module, an improved historical non-maximum suppression algorithm and a slice-assisted reasoning framework. When image feature information is transmitted in the network, high-level semantic feature information and low-level spatial position information are transmitted in the vertical direction of the network, and information can also be transmitted across the horizontal direction of the network to facilitate the fusion of more feature information. When each frame of target recognition prediction is added, the target information obtained in the previous frame is added. When performing non-maximum suppression processing, the existing target information is compared and analyzed, and the covering situation when the same type of targets are close to each other is considered, so as to reduce the detection error rate caused by the covering of the same type of targets when they are close to each other. A slice-assisted reasoning framework is added, and when a small target appears in a frame, multiple frames around it are sliced ​​and combined, and then input into the network again for detection, so as to enhance the accuracy of continuous detection of small targets. The improved DeepSort network adds a target classification information inspection structure and adds a static object retention structure. Input image information into the YOLOv5 improved network, and extract features through the BiFPN structure, spatial attention module and channel attention module to obtain feature information of the target. Input it into the DeepSort improved network to perform cascade matching, target classification matching and IOU matching with the targets and trajectories of the previous multiple frames to obtain multi-target tracking trajectories;

[0053] Perform target segmentation and annotation in the existing road environment image, input it into the YOLOv5 improved network and the DeepSort improved network for training to obtain the weights of various parameters;

[0054] The real-time image acquired by the camera while the vehicle is driving is input into the YOLOv5 improved network for target detection to obtain the classification information and location information of multiple targets, and perform real-time identification and tracking of multiple targets.

[0055] The addition of the attention mechanism improves the network's understanding of certain targets that often appear in certain locations in the image, making target detection more accurate, such as traffic lights that often appear in the air above the image and zebra crossings that often appear in the lower middle of the image.

[0056] Furthermore, the specific steps of improving the YOLOv5 network to obtain target classification information and location information include:

[0057] The improved BiFPN structure is used to perform weighted feature fusion on some nodes, such as Figure 1 As shown, the image is input into the network, where the nodes are:

[0058]

[0059] in Corresponding to the 12th layer in the YOLOv5 improved network, P6 corresponds to the 6th layer in the YOLOv5 improved network, and ω1 is the feature information transmitted by P6 to The corresponding weight is ω2, which is the feature information transmitted by P7 after upsampling. The corresponding weight is . Among them, ε is a small value to prevent the denominator from being zero, cov means performing a convolution operation, Resize means adjusting the image size without losing image information, and P7 represents a node in the network, such as Figure 1 shown.

[0060]

[0061] in Corresponding to the 19th layer in the YOLOv5 improved network, ω′1 is the feature information transmitted across P6 to The corresponding weight is ω′2 Pass feature information to The corresponding weight is ω′3 After upsampling, feature information is passed to The corresponding weight is Represents a node in the network, such as Figure 1 shown.

[0062] Furthermore, the specific steps of using channel attention and spatial attention mechanisms to process feature maps include:

[0063] First, perform the channel attention operation, and input the feature map F into two convolutions after the maximum pooling and average pooling respectively. After adding the two, perform sigmoid activation to obtain the feature map M. c1 .

[0064] M c1 (F)=σ(ω1×MLP(AvgPool(F))+ω2×MLP(MaxPool(F)))

[0065] Where F is the input feature map, MLP is the convolutional network, σ is the sigmoid activation function, AvgPool is the average pooling function, and MaxPool is the maximum pooling function. The ω weights are different and need to be learned through training in the network.

[0066] Then the spatial attention operation is performed on the feature map M c1 Use convolution with a convolution kernel size of 7×7 to perform average pooling and maximum pooling respectively, add the two together, perform convolution processing and sigmoid activation to obtain M c2 :

[0067] M c2 (F) = σ(f 7×7 ([AvgPool(F); (MaxPool(F)]))

[0068] where f 7×7 It is a convolution process with a convolution kernel size of 7×7;

[0069] Finally, M c2 Feature fusion is performed with feature map F to obtain feature map M c3 :

[0070] M c3 =σ(ω3×MLP(M c2 (M c1 (F))+ω4×MLP(F)))

[0071] Furthermore, an improved historical non-maximum suppression algorithm is used to solve the problem that when the same target objects are close to each other, the detected target information is mistakenly rejected due to cover. The specific steps include:

[0072] First, the target detection position and classification information of the previous frame are retained, recorded as boxes3, which contains the following information: (x, y, w, h, c, conf), which are the center coordinates of the target in the previous frame, the width and height of the detection box, the classification information, and the confidence level.

[0073] The traditional non-maximum suppression algorithm is performed on this frame to obtain the predicted target information of this frame, which is recorded as boxes2 and contains information (x, y, w, h, c, conf). The predicted target information eliminated when the non-maximum suppression algorithm is performed on this frame is recorded as boxes1 and contains information (x, y, w, h, c, conf).

[0074] If the IoU between the prediction box boxes2 that is eliminated in this frame and the prediction box boxes3 that has been identified in the previous frame is greater than a certain threshold, HIoU (History Intersection over Union) is further used to determine whether it needs to be retained, as shown in the formula:

[0075]

[0076] or

[0077] HIoU=IoU-αβ

[0078]

[0079]

[0080] Among them, α is the positive balance parameter, β is the covering possibility, and ρ 2 (,) is the Euclidean distance, Figure 2 b1 in the figure is the prediction box obtained by using the traditional non-maximum suppression algorithm for this frame. Figure 2 b2 is the prediction box eliminated by the non-maximum suppression algorithm in this frame. Figure 2 b0 in the above equation is the prediction box closest to b2 among the prediction boxes retained in the previous frame.

[0081] Considering that the distance between b0 and b1 may be closer, which may lead to misjudgment, we add and Consider. When , it means that compared with the previous frame, the distance between the two objects of the same type in this frame becomes shorter, closer to each other, and the occlusion is aggravated. b2 can be further analyzed to confirm whether to keep it; when , it may be a misjudgment. In fact, b2 and b1 have more objects, so b2 is not retained.

[0082] Figure 2 In the figure, b0, b1 and b2 are the center points of the prediction boxes PB0, PB1 and PB2 respectively, PB0 is the prediction box retained in the previous frame, PB1 is the prediction box eliminated in the current frame under the traditional non-maximum suppression algorithm, and PB2 is the prediction box retained in the current frame under the traditional non-maximum suppression algorithm.

[0083]

[0084] s i Represents the judgment score, b1 represents the center point of the prediction box with the highest confidence in this frame, b2 represents the center point of the prediction box that needs to be judged whether it needs to be moved out in this frame, and ε represents the threshold of historical non-maximum suppression.

[0085] In one frame, there may be multiple b2s that are removed and then retained after HIoU. It is necessary to screen multiple b2s and select s i The b2 with the highest score is retained. The improved historical non-maximum suppression algorithm can alleviate the problem of missed detection caused by mutual occlusion of the same targets in the video.

[0086] Furthermore, the slice-assisted reasoning framework is used to improve the accuracy of small target recognition and solve the problem of being unable to continuously recognize small targets due to environmental reasons. The specific steps include:

[0087] First, combine the current frame image Im1 and the target obtained in the previous frame image Im0 to make a comprehensive judgment. If the small target Im obtained in the previous frame image 0a If the small target is near the position (with the coordinates of the small target in the previous frame as the center and three times the width and height as the side length) but is not detected in the current frame, slice-assisted reasoning is performed.

[0088] According to the small target Im in the previous frame image Im0 0a Information (x 0a ,y 0a , w 0a ,h 0a , c 0a ,conf 0a ), in this frame (x 0a ,y 0a ) around (w 0a ×k0,h 0a ×k0) sliding window in this frame image Im1, slice processing is performed in sequence according to the sliding step size k1 (k1<w0), and the slices can be obtained The number of slices can be controlled, and multiple slices can be spliced ​​together to get the image Im p , control picture Im p The range is in the picture [w min , w max ] and [h min ,h max ], the insufficient part is filled with blanks, where k0 is the proportional coefficient. p Re-Im p Input into the detection network. If there are multiple small targets Im 0b ,Im 0c Etc., then the slice obtained after slice-assisted reasoning They are then spliced ​​into a picture in a certain order and re-input into the network for recognition.

[0089] After identifying a small target, it is reversely mapped to Im1 in a certain order. Slice-assisted reasoning can enhance the detection accuracy of small targets without increasing the computational cost too much.

[0090] The DeepSort improved network performs multi-target matching tracking steps including:

[0091] First, extract the motion feature information and classification information of the obtained target, and classify and match it with the determined trajectory.

[0092] The specific data of the trajectory predicted by the Kalman filter in the DeepSort improved network are:

[0093]

[0094] The specific target data obtained by the YOLOv5 improved network are:

[0095] I obj =(x obj ,y obj , γ obj ,h obj , t obj , c obj )

[0096] Where x, y, γ, and h represent the central horizontal coordinate, vertical coordinate, aspect ratio, and height of the candidate box where the target is located, respectively; Respectively represent the change of the horizontal coordinate of the center of the candidate box, the change of the vertical coordinate, the change of the aspect ratio of the candidate box, and the change of the height of the candidate box; t, c represent the classification of the target or trajectory respectively. When performing classification matching, the target and trajectory classification information are matched. Only when they match can cascade matching and IOU matching be continued.

[0097] Furthermore, when the vehicle is stationary, features of stationary objects such as zebra crossings, traffic lights and ground markers are retained, and when the stationary objects cannot be detected due to environmental influences, still-life features of the previous and next frames are matched.

[0098] Combination Figure 4 The present invention provides a method for identifying and tracking multiple targets in a road environment, comprising the following steps:

[0099] 1. Establish a multi-target recognition and tracking model, including the YOLOv5 improved network and the DeepSort improved network. The YOLOv5 improved network uses the BiFPN structure to improve the image feature information in the network, so that when the image feature information is transmitted in the network, it can transmit high-level semantic feature information and low-level spatial position information in the vertical direction of the network, and can also transmit information across the horizontal direction of the network to facilitate the integration of more feature information.

[0100] The improved YOLOv5 network uses spatial attention modules and channel attention modules to make the detection network pay more attention to the locations where different categories of targets often appear in the image, thereby improving the target recognition accuracy and recall rate.

[0101] The DeepSort improved network adds a target classification information verification structure and a still-life preservation structure.

[0102] 2. Perform target segmentation and annotation in the existing road environment image, and input it into the YOLOv5 improved network and DeepSort improved network for training to obtain the weights of various parameters;

[0103] The actual image captured by the camera when the vehicle is driving is input into the YOLOv5 improved network for target detection to obtain the classification information and location information of multiple targets, and perform real-time identification and tracking of multiple targets.

[0104] 3. The specific steps of the YOLOv5 improved network to obtain target classification information and location information include:

[0105] Combination Figure 1 , the improved BiFPN structure is used to perform weighted feature fusion on some nodes, and the image is input into the network, where the nodes are:

[0106]

[0107] in Corresponding to the 12th layer in the YOLOv5 improved network, P6 corresponds to the 6th layer in the YOLOv5 improved network, and ω1 is the feature information transmitted by P6 to The corresponding weight is ω2, which is the feature information transmitted by P7 after upsampling. The corresponding weight

[0108]

[0109] in Corresponding to the 19th layer in the YOLOv5 improved network, ω′1 is the feature information transmitted across P6 to The corresponding weight is ω′2 Pass feature information to The corresponding weight is ω′3 After upsampling, feature information is passed to , ε is a small value to prevent the denominator from being zero; Conv means performing a convolution operation, Resize means adjusting the image size without losing image information, P7 and Represents a node in the network.

[0110] The specific steps of processing feature maps using channel attention and spatial attention mechanisms include:

[0111] First, perform the channel attention operation, and input the feature map F into two convolutions after the maximum pooling and average pooling respectively. After adding the two, perform sigmoid activation to obtain the feature map M. c1 .

[0112] M c1(F)=σ(ω1×MLP(AvgPool(F))+ω2×MLP(MaxPool(F)))

[0113] Where F is the input feature map, MLP is the convolutional network, σ is the sigmoid activation function, AvgPool is the average pooling function, and MaxPool is the maximum pooling function.

[0114] Then the spatial attention operation is performed on the feature map M c1 Use convolution with a convolution kernel size of 7×7 to perform average pooling and maximum pooling respectively, add the two together, perform convolution processing and sigmoid activation to obtain M c2 :

[0115] M c2 (F) = σ(f 7×7 ([AvgPool(F); (MaxPool(F)]))

[0116] where f 7×7 Convolution processing with a convolution kernel size of 7×7

[0117] Finally, M c2 Feature fusion is performed with feature map F to obtain feature map M c3 :

[0118] M c3 =σ(ω3×MLP(M c2 (M c1 (F))+ω4×MLP(F)));

[0119] ω1, ω1, ω3 and ω4 are the weights of the corresponding positions respectively.

[0120] 4. The specific steps of the improved historical non-maximum suppression algorithm include:

[0121] First, the target detection position and classification information of the previous frame are retained, recorded as boxes3, which contains the following information: (x, y, w, h, c, conf), which are the center coordinates of the target in the previous frame, the width and height of the detection box, the classification information, and the confidence level.

[0122] The traditional non-maximum suppression algorithm is performed on this frame to obtain the predicted target information of this frame, which is recorded as boxes2 and contains information (x, y, w, h, c, conf). The predicted target information eliminated when the non-maximum suppression algorithm is performed on this frame is recorded as boxes1 and contains information (x, y, w, h, c, conf).

[0123] If the IoU between the prediction box boxes2 that is eliminated in this frame and the prediction box boxes3 that has been identified in the previous frame is greater than a certain threshold, HIoU (History Intersection over Union) is further used to determine whether it needs to be retained, as shown in the formula:

[0124]

[0125] or

[0126] HIoU=IoU-αβ

[0127]

[0128]

[0129] Among them, α is the positive balance parameter, β is the covering possibility, and ρ 2 (,) is the Euclidean distance, Figure 2 b1 in the figure is the prediction box obtained by using the traditional non-maximum suppression algorithm for this frame. Figure 2 b2 is the prediction box eliminated by the non-maximum suppression algorithm in this frame. Figure 2 b0 in the above equation is the prediction box closest to b2 among the prediction boxes retained in the previous frame.

[0130] Considering that the distance between b0 and b1 may be closer, which may lead to misjudgment, we add and Consider. When , it means that compared with the previous frame, the distance between the two objects of the same type in this frame becomes shorter, closer to each other, and the occlusion is aggravated. b2 can be further analyzed to confirm whether to keep it; when , it may be a misjudgment. In fact, b2 and b1 have more objects, so b2 is not retained.

[0131] Figure 2 In the figure, b0, b1 and b2 are the center points of the prediction boxes PB0, PB1 and PB2 respectively, PB0 is the prediction box retained in the previous frame, PB1 is the prediction box eliminated in the current frame under the traditional non-maximum suppression algorithm, and PB2 is the prediction box retained in the current frame under the traditional non-maximum suppression algorithm.

[0132]

[0133] s i Represents the judgment score, b1 represents the center point of the prediction box with the highest confidence in this frame, b2 represents the center point of the prediction box that needs to be judged whether it needs to be moved out in this frame, and ε represents the threshold of historical non-maximum suppression.

[0134] In one frame, there may be multiple b2s that are removed and then retained after HIoU. It is necessary to screen multiple b2s and select s i The b2 with the highest score is retained. The improved historical non-maximum suppression algorithm can alleviate the problem of missed detection caused by mutual occlusion of the same targets in the video.

[0135] 5. The specific steps of the slice-assisted reasoning framework include:

[0136] like Figure 3 As shown in , first combine the current frame image Im1 and the target obtained by the previous frame image Im0 to make a comprehensive judgment. If the small target Im obtained by the previous frame image 0a If the small target is near the position (with the coordinates of the small target in the previous frame as the center and three times the width and height as the side length) but is not detected in the current frame, slice-assisted reasoning is performed.

[0137] According to the small target Im in the previous frame image Im0 0a Information (x 0a ,y 0a , w 0a ,h 0a , c 0a ,conf 0a ), in this frame (x 0a ,y 0a ) around (w 0a ×k0,h 0a ×k0) sliding window in this frame image Im1, slice processing is performed in sequence according to the sliding step size k1 (k1<w0), and the slices can be obtained The number of slices can be controlled, and multiple slices can be spliced ​​together to get the image Im p , control picture Im p The range is in the picture [w min , w max ] and [h min ,h max ], the insufficient part is filled with blanks, where k0 is the proportional coefficient. p Re-Im p Input into the detection network. If there are multiple small targets Im 0b ,Im 0c Etc., then the slice obtained after slice-assisted reasoning They are then spliced ​​into a picture in a certain order and re-input into the network for recognition.

[0138] 6. The DeepSort improved network performs multi-target matching tracking steps including:

[0139] First, extract the motion feature information and classification information of the obtained target, and classify and match it with the determined trajectory.

[0140] The specific data of the trajectory predicted by the Kalman filter in the DeepSort improved network are:

[0141]

[0142] The specific target data obtained by the YOLOv5 improved network are:

[0143] I obj =(x obj ,y obj , γ obj ,h obj , t obj , c obj )

[0144] Where x, y, γ, and h represent the central horizontal coordinate, vertical coordinate, aspect ratio, and height of the candidate box where the target is located, respectively; They represent the change of the horizontal coordinate of the center of the candidate box, the change of the vertical coordinate, the change of the aspect ratio of the candidate box, and the change of the height of the candidate box respectively; t and c represent the classification of the target or trajectory respectively.

[0145] 7. When performing classification matching, match the target with the trajectory classification information. Only when they match can cascade matching and IOU matching be continued.

[0146] 8. When the vehicle is stationary, the features of stationary objects such as zebra crossings, traffic lights and signs are retained. When the stationary objects cannot be detected due to environmental influences, the still features of the previous and next frames are matched.

[0147] In this example, the real-time road environment image is obtained through the on-board camera, and the improved convolutional neural network of YOLOv5, which integrates the BiFPN structure, spatial attention mechanism and channel attention mechanism, extracts the target appearance information for target recognition, and transmits the obtained target feature information including position information and classification information to the improved DeepSort network. In the improved DeepSort network, the target and trajectory classification feature matching is first performed, and then the trajectory and target that have been Kalman filtered are cascade matched and IOU matched. For multiple matching trajectories and targets, the Hungarian algorithm is used for matching. When the vehicle is stationary, the position information of some detected targets is retained as the target input for appearance feature matching in the next frame matching, so as to achieve continuous tracking of some static targets.

[0148] 9. Figure 6The figure is a schematic diagram of the physical structure of an electronic device according to an example of the present invention, wherein the electronic device includes a camera, a processor, a memory and a communication bus;

[0149] The camera obtains the road environment around the vehicle in real time and transmits the image information to the processor;

[0150] The processor is used to call the program instructions in the memory to execute the method provided by the above method example, for example, including: receiving the image signal transmitted by the camera, using the YOLOv5 improved network to detect the target, and passing the position information and classification information of the detected target to the improved DeepSort improved network. Filter and remove targets with a confidence level less than 0.5, and perform classification matching and cascade matching with existing trajectories to obtain unmatched trajectories, unmatched targets, and matched trajectories. Perform IOU matching and classification matching on the unmatched trajectories and unmatched targets to obtain unmatched trajectories, unmatched targets, and matched trajectories. Update all matching trajectories and perform Kalman filter prediction, and if the unmatched targets meet the requirements, construct new trajectories. If the vehicle is stationary, retain the stationary detected targets and perform feature matching in the next frame.

[0151] This example provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the above-mentioned methods to implement the provided methods, for example, including receiving image signals transmitted by a camera, using the YOLOv5 improved network to detect targets, and passing the location information and classification information of the detected targets to the improved DeepSort improved network. Targets with a confidence level less than 0.5 are screened and removed, and classification matching and cascade matching are performed with existing trajectories to obtain unmatched trajectories, unmatched targets, and matched trajectories. Unmatched trajectories and unmatched targets are IOU matched and classified matched to obtain unmatched trajectories, unmatched targets, and matched trajectories. All matching trajectories are updated and Kalman filter predictions are performed, and new trajectories are constructed if the unmatched targets meet the requirements. If the vehicle is stationary, the stationary detected targets are retained and feature matching is performed in the next frame.

[0152] Figure 5 This is a frame of the video before and after processing. The target matching in the effect picture is more accurate. The DeepSort improved network integrates the classification information of YOLOv5, and there will be no continuous misclassification and matching. In addition, for stationary objects, such as traffic lights, zebra crossings and ground markers, the target recognition is more accurate.

[0153] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such understanding and thinking, the technical solution of the present invention is essentially or the part that contributes to the existing calculation or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device to perform some of the steps described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0154] The above examples are merely the better application of the present invention. It should be noted that, in the technical field, the present invention may be improved and modified without departing from the principle of the present invention, but these improvements and modifications are still within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying and tracking multiple targets in a road environment, characterized in that: The following steps are involved: A multi-target recognition and tracking model is established, including a YOLOv5 improved network and a Deep Sort improved network; the YOLOv5 improved network uses an improved BiFPN structure, in which a spatial attention module and a channel attention module are embedded, and a channel attention operation is performed first, followed by a spatial attention operation; The Deep Sort improved network adds a target classification information verification structure and a still object preservation structure; Obtain existing road environment images for target segmentation and annotation, and input them into the multi-target recognition and tracking model as a data set for training; input the road environment images of the video to be processed into the trained model to realize multi-target tracking in the road environment; Input the road environment image of the video to be processed into the YOLOv5 improved network to obtain the target feature information; The target feature information is input into the Deep Sort improved network, and the current image and the image processed by the YOLOv5 improved network are matched by cascade matching of targets and trajectories, target classification matching, and IOU matching to obtain multi-target tracking trajectories; The improved BiFPN structure includes: The BiFPN structure uses weighted feature fusion, where Corresponding to the 6th layer in the YOLOv5 improved network, Corresponding to the 12th layer in the YOLOv5 improved network, Corresponding to the 19th layer in the YOLOv5 improved network, for Pass feature information to The corresponding weight is for After upsampling, feature information is passed to The corresponding weight is for Transmit feature information across The corresponding weight is for Pass feature information to The corresponding weight is for After upsampling, feature information is passed to The corresponding weight; ε is a small value to prevent the denominator from being zero; Conv Indicates the convolution operation. Resize Indicates that the image size is adjusted without losing image information. P 7 and Represents a node in the network.

2. The method for identifying and tracking multiple targets in a road environment according to claim 1, characterized in that: The spatial attention module and channel attention module specifically include: First, perform the channel attention operation, and then input the feature map F into two convolutions after the maximum pooling and average pooling respectively. After adding the two, perform sigmoid activation to obtain the feature map ; Then perform spatial attention operation on the feature map The convolution kernel size is The convolution of is respectively average pooling and maximum pooling, and the two are added together and then convolved and sigmoid activated to obtain ; Finally, The feature map is obtained by feature fusion with the feature map F , complete the attention mechanism; ； ； ； Where F is the input feature map, is a convolutional network, The convolution kernel size is The convolution process, is the sigmoid activation function, is the average pooling function, is the maximum pooling function; ω 1. ω 1. ω 3 and ω 4 are weights respectively.

3. The method for identifying and tracking multiple targets in a road environment according to claim 1 or 2, characterized in that: The current image and the image processed by the YOLOv5 improved network are matched by cascade matching of targets and trajectories, target classification matching and IOU matching to obtain multi-target tracking trajectories, including: The target frame graph is matched using the improved historical non-maximum suppression algorithm. The specific steps include: First, the target detection position and classification information of the previous frame are retained, recorded as boxes3, which contains the following information: , are the center coordinates of the target in the previous frame, the width and height of the detection box, the classification information and the confidence; the traditional non-maximum suppression algorithm can be used to obtain the predicted target information of this frame, which is recorded as boxes2, and contains the following information: , the predicted target information eliminated when the non-maximum suppression algorithm is performed in this frame is recorded as boxes1, which contains the information ; If the IoU between the prediction box boxes2 that is eliminated in this frame and the prediction box boxes3 that has been identified in the previous frame is greater than a certain threshold, HIoU is further used to determine whether it needs to be retained, as shown in the formula: ； in, is to find the Euclidean distance, It is the prediction box obtained by using the traditional non-maximum suppression algorithm for this frame. It is the prediction box eliminated by the non-maximum suppression algorithm in this frame. is the distance in the prediction box retained in the previous frame The nearest prediction box; when When , it means that compared with the previous frame, the distance between two objects of the same type in this frame is shorter, closer to each other, and the occlusion is aggravated. Further analysis is conducted to confirm whether to retain the It may be a misjudgment. and are the same object, so they are not retained .

4. The method for identifying and tracking multiple targets in a road environment according to claim 3, characterized in that: right Further analysis is conducted to confirm whether to retain specific inclusions; , and are the center points of the prediction boxes PB0, PB1 and PB2 respectively, PB0 is the prediction box retained in the previous frame, PB1 is the prediction box eliminated in the current frame under the traditional non-maximum suppression algorithm, and PB2 is the prediction box retained in the current frame under the traditional non-maximum suppression algorithm; ； Indicates the judgment score, Indicates the center point of a prediction box with the highest confidence in this frame. Indicates the center point of the prediction box that needs to be determined whether it needs to be moved out in this frame. Indicates the threshold for historical non-maximum suppression; in one frame, there may be multiple values ​​that are removed and retained after HIoU , multiple Filter, select Highest score Make reservations.

5. The method for identifying and tracking multiple targets in a road environment according to claim 1 or 2, characterized in that: The frame image to be processed also needs to go through the slice-assisted reasoning framework. The specific steps include: First combine this frame picture and the previous frame The obtained target is comprehensively judged. If the small target is obtained in the previous frame If a small target is not detected in the current frame but is near the location, slice-assisted reasoning is performed; According to the previous frame Small and medium-sized targets Information , in this frame Around The sliding window is in this frame picture In the sliding step ( ) are sliced ​​sequentially to obtain slices , , , …, ; The number of slices can be controlled, and a picture can be obtained by splicing many slices , control the picture Range in the picture and , the insufficient part is filled with blanks, where is the scale factor; the stitched image renew Input into the detection network; if multiple small targets appear or , then the slice obtained after slice-assisted reasoning , , , , …, , they are spliced ​​into a picture in a certain order and re-input into the network for recognition.

6. The method for identifying and tracking multiple targets in a road environment according to claim 1 or 2, characterized in that: The DeepSort improved network performs multi-target matching tracking steps including: First, extract the motion feature information and classification information of the obtained target, and classify and match it with the determined trajectory. The specific trajectory data predicted by the Kalman filter in the DeepSort improved network is: ； The specific target data obtained by the YOLOv5 improved network are: ； in Respectively represent the central horizontal coordinate, vertical coordinate, aspect ratio and height of the candidate box where the target is located; Respectively represent the change of the central horizontal coordinate of the candidate box, the change of the vertical coordinate, the change of the aspect ratio of the candidate box, and the change of the height of the candidate box; Represents the classification of targets or trajectories respectively.

7. The method for identifying and tracking multiple targets in a road environment according to claim 3, characterized in that: When performing classification matching, the target is matched with the trajectory classification information. Only when they match can the cascade matching and IOU matching be continued; When the vehicle is stationary, the features of stationary objects such as zebra crossings, traffic lights and signs are retained. When the stationary objects cannot be detected due to environmental influences, the still features of the previous and next frames are matched.

8. A road environment multi-target identification and tracking device, characterized in that: It includes a camera, a processor, a memory and a communication bus; the camera obtains the road environment around the vehicle in real time and transmits the image information to the processor; The processor is used to call program instructions in the memory to execute the method for multi-target recognition and tracking in a road environment as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method for identifying and tracking multiple targets in a road environment as described in any one of claims 1-7.

Images