A dynamic multi-target recognition method under intermittent occlusion

Abstract

The application discloses a kind of intermittent shelter under dynamic multi-target identification method.First, the continuous image sequence and video dataset for target identification under the influence of existing intermittent shelter are collected and made, for the training and test of dynamic multi-target identification algorithm.Second, the target identification module based on center network is designed, the boundary box, motion offset and hotspot map feature information of target in adjacent two frames of input video image are obtained.Third, the feature extraction module based on gated recurrent unit network is used to update the feature information of target in current frame, and the feature information of target in previous frame is combined to obtain the motion trajectory of target.Finally, the target trajectory matching module based on hungarian algorithm is used to match the target and its motion trajectory, and assign the corresponding ID value to each target, and then realize dynamic multi-target identification.The application can be applied to dynamic multi-target identification problem under intermittent shelter condition.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a dynamic multi-target recognition method under intermittent occlusion. Background Technology

[0002] Multi-target recognition is a fundamental research problem in computer vision and is being widely applied in many fields such as smart monitoring, industrial inspection and human-computer interaction. It has important research and application value, but it still faces many challenges in complex scenarios, such as multiple targets that occlude each other, significant changes in ambient lighting, and background interference.

[0003] In recent years, in-depth research into deep learning algorithms has further improved environmental perception performance, even reaching or surpassing human recognition accuracy in tasks such as target classification. Therefore, many deep learning-based target recognition methods have been proposed. Compared to traditional methods, deep learning-based target recognition methods have attracted widespread attention from researchers due to their stronger generalization and robustness.

[0004] Target occlusion is very likely to occur in dynamic multi-target recognition. Traditional methods, based on target detection in a single frame image, combine target features from two consecutive frames in the input video to improve the low confidence of individual target recognition and thus improve the target recognition accuracy. However, these methods cannot effectively obtain the feature information of targets between adjacent image frames, so they cannot effectively match targets and their motion trajectories, and are not effective for target recognition problems that are occluded for a long time.

[0005] In recent years, a series of deep learning-based multi-object recognition methods have been proposed. For example, Zhou et al. proposed using CenterNet for dynamic multi-object recognition at the European Conference on Computer Vision. This method extracts the bounding box, heatmap, and motion offset features of the target in the preceding and following frames of the input video and matches the target and its motion trajectory. With a simplified network structure, it achieves higher accuracy in dynamic target recognition. Experiments have demonstrated that this method achieves superior performance in dynamic target recognition, proving the excellent performance of deep learning algorithms in this area. However, these methods cannot effectively extract target feature information from multiple frames of the input video and struggle to accurately identify dynamic targets affected by intermittent occlusion. A search of relevant technologies revealed that there is currently no deep learning-based dynamic multi-object recognition method that satisfies the requirement of intermittent target occlusion. Summary of the Invention

[0006] To address the limitation of central network-based dynamic multi-target recognition methods in handling intermittent occlusion, this invention proposes a novel method for dynamic multi-target recognition under intermittent occlusion. This method, by fusing target feature information from multiple frames of the input video, can effectively identify multiple dynamic targets even when the target is significantly affected by intermittent occlusion, thereby improving the accuracy of dynamic multi-target recognition.

[0007] This invention provides a dynamic multi-target recognition method under intermittent occlusion, which includes two phases: training and testing.

[0008] The training phase consists of a target recognition module based on a central network, a feature extraction module based on a gated recurrent unit network, and a target trajectory matching module based on the Hungarian algorithm. The target recognition module based on the central network includes a baseline network and a feature extraction module. The baseline network adopts the DLA-34 network structure, and the feature extraction module uses the central network.

[0009] After the training phase is completed, the video to be recognized is input into the target recognition module based on the central network. After processing, the target trajectory matching module based on the Hungarian algorithm matches the target and its motion trajectory, thus achieving dynamic multi-target recognition. This method includes the following steps:

[0010] Step 1: Prepare the training dataset.

[0011] Step 2: Input the training video data of total length T frames into the baseline network, and obtain the adjacent frame sequence (x) labeled with target type and bounding box information through the baseline network. t-1 x t ), t=2,…,T, where, x t This represents the t-th frame image labeled with target type and bounding box information. Target type information is used to identify targets, and targets of the same type have the same type information. Bounding box information is used to calculate the location of the target's center point.

[0012] Step 3: Combine the adjacent frame sequences (x) obtained by the baseline network t-1 x t Input the feature extraction module to predict x t+1 The motion offset and heat map feature information of all target center points in the frame sequence are used to calculate the motion trajectory, and the heat map is used to determine whether the marked target type is accurate.

[0013] Step 4: Calculate the loss function, where the overall loss function expression is updated as follows:

[0014]

[0015] Where T represents the sequence length of the input video, λ f , λ s and λ O The hyperparameters λ and λ represent the weights of each branch in the overall loss function. f =1,λ s =0.1, λ O =0.1;

[0016] L f The prediction loss for the target heatmap is represented by a loss function based on local loss, and its expression is as follows:

[0017]

[0018] Among them, Y cab This represents the true heatmap value of the target of type c at position (a, b) in frame t. This value is obtained directly from the input video. Indicates the sequence of adjacent frames (x) t-1 The heatmap of the target at the corresponding position in the t-th frame is predicted by (xt), where N represents the number of targets in the t-th frame image, and α and β are hyperparameters. This represents traversing all types of targets at all positions in the t-th frame of the image.

[0019] L s The prediction loss representing the target bounding box location is expressed using an L1 loss-based loss function, as follows:

[0020]

[0021] in, This represents the bounding box position of the i-th target center point predicted by the baseline network in the t-th frame image. This represents the true value of the bounding box position of the i-th target center point on the t-th frame image. The true value is obtained directly from the input video.

[0022] L o The prediction loss representing the target motion offset is expressed using a regression-based loss function, as shown below:

[0023]

[0024] in Represents each bounding box on the image in frame t. The trajectory of the target i. This represents the true value of the center point position of target i on the detected t-th and t-1-th frames of the image. The true value is obtained directly from the input video.

[0025] This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, the positive positioning head P is used t Supervised learning of the target center point location, This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, a negative positioning head V is used. t Supervised learning of the target center point location. Where M... t The update process is completed by the feature extraction module based on the gated recurrent unit network, which calculates the update gate parameter z in the gated recurrent unit network corresponding to the t-th frame. t Hidden layer recursive representation parameters and reset gate parameter r t Then, the updated target feature information M of the t-th frame image is obtained. t The target feature information includes the target bounding box size, motion offset, and heat map, and the specific calculation formula is as follows:

[0026]

[0027] Among them, z t This indicates updating the gate parameters, M. t-1 M represents the updated target bounding box size, motion offset, and heatmap in frame t-1. In the input image of frame 1, M... 1 =0, This represents the parameters of the hidden layer recursively, and updates the gate parameter z. t The calculation expression is as follows:

[0028] z t =δ(W z F t +U z M t-1 +b z )

[0029] Where δ() represents the Logistic function, and its output interval is (0,1), W Z U z and b z It is the gated recurrent unit network that updates the learnable network parameters of the gate, F. t F represents the target bounding box size, motion offset, and heatmap in frame t before the update. 1 F 2 The values ​​are the target bounding box size, motion offset, and heatmap real values ​​on the first and second frames of the image, respectively. The real values ​​are obtained directly from the input video, and the predicted values ​​are assigned starting from the third frame.

[0030] Hidden layer recursive representation parameters The calculation expression is as follows:

[0031]

[0032] Where ⊙ represents the convolution operation, W M U M and b M These are the learnable network parameters of the hidden layer of a gated recurrent unit network.

[0033] r t To reset the gate parameters, the calculation expression is:

[0034] r t =δ(W r F t +U r M t-1 +b r )

[0035] W r U r and b r It is the gated cyclic unit network that resets the learnable network parameters of the gate.

[0036] Step 5: Based on the intersection-union ratio (IUGR) of the true and predicted values ​​of the target motion trajectory obtained from the target motion offset, the Hungarian algorithm is used to match multiple targets and their motion trajectories that still appear after occlusion based on the IUGR. The calculation process of the IUGR is as follows:

[0037] Define G = {g1, ..., g} T} represents the true value of the target's motion trajectory, which is directly obtained from the input video. We define D = {d1, ..., d2}. T Let} be the predicted value of the target trajectory. The crossover ratio (CRO) function between the two is expressed as follows:

[0038]

[0039] For targets that no longer appear starting from frame t, a pedestrian re-identification method is used to match them with the predicted trajectories. This process is repeated for each target, traversing the training video to complete the training.

[0040] After training, the video to be identified is input into the target recognition module based on the central network. After processing, the target trajectory matching module based on the Hungarian algorithm matches the target and its motion trajectory, thus realizing dynamic multi-target recognition.

[0041] Beneficial effects:

[0042] The advantage of this method is that it can accurately identify dynamic multi-targets affected by intermittent occlusion (≤30FPS), has high robustness, and can identify dynamic multi-targets affected by intermittent occlusion in different scenes (campus, shopping mall, street, etc.) with an accuracy of ≥70%. Attached Figure Description

[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0044] Figure 1 This is a flowchart of the dynamic multi-target recognition method in this invention.

[0045] Figure 2 This is a schematic diagram of the dynamic multi-target recognition method in this invention.

[0046] Figure 3 This is a schematic diagram of the feature extraction module based on the central network in this invention.

[0047] Figure 4 This is a diagram showing the implementation effect of the present invention on a dynamic multi-target recognition dataset that is affected by intermittent occlusion. Detailed Implementation

[0048] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0049] A flowchart of a dynamic multi-target recognition method under the influence of intermittent occlusion is shown below. Figure 1 As shown, a schematic diagram of the method is as follows. Figure 2 As shown, the process includes two phases: training and testing. The training phase is implemented by a target recognition module based on a central network, a feature extraction module based on a gated recurrent unit network, and a target trajectory matching module based on the Hungarian algorithm, as detailed below:

[0050] Step 1: Use a Logitech C920 camera to collect continuous image sequences and video data from multiple different scenes (such as schools, shopping malls, streets, etc.) for dynamic multi-target recognition. Select the continuous image sequences and video data affected by intermittent occlusion and complete the dataset production. This dataset includes a training set, a validation set, and a test set.

[0051] Step 2: In the target recognition module based on the central network, the DLA-34 network structure is used as the baseline network. This network includes an encoder-decoder structure, which can improve the accuracy of target recognition. A total of T frames of training image data are input into the baseline network, and the adjacent frame sequences (x, y, x) labeled with target types and their bounding box information are obtained through the baseline network. t-1 x t ), t=2,…,T, where, x t This represents the t-th frame image labeled with target types and their bounding box information. Target type information is used to identify targets, and targets of the same type have the same type information. Bounding box information is used to calculate the location of the target's center point.

[0052] Step 3: In the target recognition module based on the central network, the feature extraction module uses the central network to extract the adjacent frame sequences (x) obtained by the baseline network. t-1 x t Input the feature extraction module to predict x t+1 The module displays the motion offsets and heatmaps of all target center points in the frame sequence. The motion offsets are used to calculate the motion trajectory, and the heatmaps are used to determine the accuracy of the labeled target type. This module accelerates target feature extraction, reduces network parameters, and improves algorithm speed. A schematic diagram is shown below. Figure 3 As shown.

[0053] Step 4: Calculate the loss function, where the overall loss function expression is as follows:

[0054]

[0055] Where T represents the sequence length of the input video, λ f , λ s and λ O The hyperparameters λ and λ define the weights of each branch in the overall loss function. f =1,λ S =0.1, λ O =0.1.

[0056] L f The prediction loss for the target heatmap is represented by a loss function based on local loss, and its expression is as follows:

[0057]

[0058] Among them, Y cab This represents the ground truth heatmap value of the target of type c at position (a, b) in frame t. The ground truth value is obtained directly from the input video. Indicates the sequence of adjacent frames (x) t-1 x tThe predicted heatmap of the target at the corresponding position in frame t, where N represents the number of targets in frame t of the training data, and α and β are hyperparameters. This indicates that all types of targets are traversed at all positions in the t-th frame of the image.

[0059] L s The prediction loss representing the target bounding box location is expressed using an L1 loss-based loss function, as follows:

[0060]

[0061] in, This represents the bounding box position of the i-th target center point predicted by the baseline network in the t-th frame image. This represents the true value of the bounding box position of the i-th target center point in the t-th frame image. The true value is obtained directly from the input video.

[0062] L o The prediction loss representing the target motion offset is expressed using a regression-based loss function, as shown below:

[0063]

[0064] in Represents each bounding box of the image in frame t. The trajectory of the target i. This represents the true value of the center point position of target i on the t-th and t-1-th frames of the image. The true value is obtained directly from the input video.

[0065] This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, the positive positioning head P is used t Supervised learning of the target center point location, This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, a negative positioning head V is used. t Supervised learning of the target center point location. Where M... t The update process is completed by the feature extraction module based on the gated recurrent unit network, which calculates the update gate parameter z in the gated recurrent unit network corresponding to the t-th frame. t Hidden layer recursive representation parameters and reset gate parameter r t Then, the updated target feature information M of frame t is obtained. t The target feature information includes the target bounding box size, motion offset, and heat map, and the specific calculation formula is as follows:

[0066]

[0067] Among them, z t This indicates updating the gate parameters, M. t-1 M represents the updated target bounding box size, motion offset, and heatmap in frame t-1. In the input image of frame 1, M... 1 =0, This represents the parameters of the hidden layer recursively, and updates the gate parameter z. t The calculation expression is as follows:

[0068] z t =δ(W z F t +U z M t-1 +b z )

[0069] Among them W Z U z and b z It is the gated recurrent unit network that updates the learnable network parameters of the gate, F. t F represents the target bounding box size, motion offset, and heatmap in frame t before the update. 1 F 2 The values ​​are the target bounding box size, motion offset, and heatmap real values ​​on the first and second frames of the image, respectively. The real values ​​are obtained directly from the input video, and the predicted values ​​are assigned starting from the third frame.

[0070] Hidden layer recursive representation parameters The calculation expression is as follows:

[0071]

[0072] Where ⊙ represents the convolution operation, W M U M and b M These are the learnable network parameters of the hidden layer of a gated recurrent unit network.

[0073] r t To reset the gate parameters, the calculation expression is:

[0074] r t =δ(W r F t +U r M t-1 +b r )

[0075] W r U r and b r It is the gated cyclic unit network that resets the learnable network parameters of the gate.

[0076] Step 5: Based on the intersection-over-union (IoU) ratio of the true and predicted values ​​of the target motion trajectory obtained from the target motion offset, the Hungarian algorithm is used to match the predicted multiple targets with their corresponding trajectories. For targets that still appear after occlusion in frame t, matching is performed based on the IoU ratio. The calculation process of the IoU ratio is as follows:

[0077] Define G = {g1, ..., g} T} represents the true value of the target's motion trajectory, which is directly obtained from the input video. We define D = {d1, ..., d2}. T Let} be the predicted value of the target trajectory. The crossover ratio (CRO) function between the two is expressed as follows:

[0078]

[0079] when The process involves matching the target with its corresponding trajectory. For targets that no longer appear starting from frame t, a pedestrian re-identification method is used to match them with the predicted trajectories. This process is repeated for each target in the training video to complete the training.

[0080] After training, the video to be identified is input into the target recognition module based on the central network. After processing, the target trajectory matching module based on the Hungarian algorithm matches the target and its motion trajectory, thus realizing dynamic multi-target recognition.

[0081] The proposed dynamic multi-object recognition algorithm was trained using the publicly available datasets MOT17, KITTI, and COCO. The server used was equipped with an Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHz×16, a quad-core NVIDIA GeForce GTX2080 graphics card, 128GB of RAM, and Ubuntu 16.04 operating system. The algorithm was implemented using Python 3.6 and the PyTorch deep learning framework. The training hyperparameters were: Batchsize=2, Epochs=100, Iterations=1600, Optimizer=Adam. The gated recurrent unit network had a feature dimension of 256 and contained 3×3 filters. The algorithm results are shown below. Figure 4 As shown.

[0082] The advantage of this method is that it can accurately identify dynamic multi-targets affected by intermittent occlusion (≤30FPS), has high robustness, and can identify dynamic multi-targets affected by intermittent occlusion in different scenes (campus, shopping mall, street, etc.) with an accuracy of ≥70%.

[0083] To simplify the explanation of this method, the above figures and text are described as a series of steps. However, it should be understood and understood that these methods are not restricted by the order of operations, because some actions may occur in different orders when implemented according to one or more steps, but those skilled in the art can understand the principle behind the actions.

[0084] Although illustrative specific embodiments of the invention have been described step by step to enable those skilled in the art to understand them, the invention is not limited to the scope of the specific embodiments. Those skilled in the art can make various modifications or variations within the scope of the claims, as long as such variations are within the spirit and scope of the invention as defined and determined by the appended claims.

Claims

1. A dynamic multi-target recognition method under intermittent occlusion, characterized in that: This method includes two phases: training and testing. The training phase is implemented by a target recognition module based on a central network, a feature extraction module based on a gated recurrent unit network, and a target trajectory matching module based on the Hungarian algorithm. Testing phase: After training is completed, the video to be identified is input into the target recognition module based on the central network. After processing, the target trajectory matching module based on the Hungarian algorithm matches the target and its motion trajectory, thus realizing dynamic multi-target recognition. The training process is as follows: Step 1, Prepare the training dataset; Step 2: Input the training video data of total length T frames into the benchmark network, and obtain the adjacent frame sequence (x) labeled with the target type and its bounding box information through the benchmark network. t-1 x t ), t=2,…,T, where, x t This represents the t-th frame image labeled with target types and their bounding box information. Target type information is used to identify targets, and bounding box information is used to calculate the location of the target's center point. Step 3: Combine the adjacent frame sequences (x) obtained by the baseline network t-1 x t Input the feature extraction module to predict x t+1 The motion offsets and heatmaps of all target center points in the frame sequence are used to calculate the motion trajectory, and the heatmaps are used to determine whether the labeled target types are accurate. Step 4: Calculate the loss function, where the overall loss function expression is updated as follows: Where T represents the sequence length of the input video, λ f , λ s and λ O These are hyperparameters, defining the weights of each branch in the overall loss function. L f The prediction loss for the target heatmap is represented by a loss function based on local loss, and its expression is as follows: Among them, Y cab This represents the corresponding value of the target of type c at position (a, b) in the real heat map in frame t. Indicates the sequence of adjacent frames (x) t-1 x t The predicted heatmap of the target at the corresponding position in frame t, where N represents the number of targets in frame t of the training data, and α and β are hyperparameters. This indicates traversing all types of targets at all locations in the t-th frame of the image; L s The prediction loss representing the target bounding box location is expressed using an L1 loss-based loss function, as follows: in, This represents the bounding box position of the i-th target center point predicted by the baseline network in the t-th frame image. The true value of the bounding box position of the i-th target center point in the t-th frame image; L o The prediction loss representing the target motion offset is expressed using a regression-based loss function, as shown below: in Represents each bounding box of the image in frame t. The trajectory of the target i. This represents the true value of the center point position of target i on the images in frame t and frame (t-1). This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, the positive positioning head P is used t Supervised learning of the target center point location, This indicates that the updated target feature information M is used as a basis. t In the input t-th frame image, a negative positioning head V is used. t Supervised learning of the target center point location; where M t The update process is completed by the feature extraction module based on the gated recurrent unit network, which calculates the update gate parameter z in the gated recurrent unit network corresponding to the t-th frame. t Hidden layer recursive representation parameters and reset gate parameter r t Then, the target feature information M on the updated t-th frame image is obtained. t The target feature information includes the target bounding box size, motion offset, and heat map, and the specific calculation formula is as follows: Among them, z t This indicates updating the gate parameters, M. t-1 This represents the updated target bounding box size, motion offset, and heatmap on frame t-1. This represents the parameters of the hidden layer recursively, and updates the gate parameter z. t The calculation expression is as follows: z t =δ(W z F t +U z M t-1 +b z ) Where δ() represents the Logistic function, and its output interval is (0, 1), W Z U z and b z It is the gated recurrent unit network that updates the learnable network parameters of the gate, F. t This represents the target bounding box size, motion offset, and heatmap in frame t before the update; Hidden layer recursive representation parameters The calculation expression is as follows: Where ⊙ represents the convolution operation, W M U M and b M These are the learnable network parameters of the hidden layer of the gated recurrent unit network; r t To reset the gate parameters, the calculation expression is: r t =δ(W r F t +U r M t-1 +b r ) W r U r and b r It is the gated recurrent unit network that resets the learnable network parameters of the gate; Step 5: Based on the intersection-over-union (IoU) ratio of the true and predicted values ​​of the target motion trajectory obtained from the target motion offset, the Hungarian algorithm is used to match the predicted multiple targets with their corresponding trajectories. For targets that still appear after occlusion in frame t, matching is performed based on the IoU ratio. The calculation process of the IoU ratio is as follows: Define G = {g1, ..., g} T } represents the true value of the target's trajectory, D = {d1, ...,d2} T Let} be the predicted value of the target trajectory. The crossover ratio (CRO) function between the two is expressed as follows: For targets that no longer appear starting from frame t, a pedestrian re-identification method is used to match them with the predicted trajectories. This process is repeated for each target, traversing the training video to complete the training.

2. The dynamic multi-target recognition method under intermittent occlusion as described in claim 1, characterized in that: The baseline network adopts the DLA-34 network structure, and the feature extraction module adopts the central network.

Images