A multi-target matching method based on deep learning

Abstract

The present invention discloses a multi-target matching method based on deep learning, comprising: obtaining an image to be identified, wherein the image to be identified includes a frame image and an ID mask image corresponding to the frame image; inputting the image to be identified into a pre-trained deep neural network model, wherein the deep neural network model includes a feature extraction module, a correlation module and a prediction module; processing the image to be identified by using the feature extraction module to obtain a feature map of the image to be identified; processing the feature map by using the correlation module to obtain an input feature map with self-perceived spatiotemporal correlation; processing the input feature map by using the prediction module to obtain a target correlation degree between two frames of images to be identified before and after the prediction; the multi-target matching method uses the deep neural network model to directly complete the multi-target correlation matching process end-to-end, has strong versatility, high real-time performance, fewer error sources, can track for a long time, and has a strong tracking effect robustness.

Description

Technical Field

[0001] The present invention relates to the field of target tracking technology, and in particular to a multi-target matching method based on deep learning. Background Art

[0002] Target matching refers to determining the boundary position of the target of interest in the current frame image based on the temporal and spatial correlation of the boundary position of the target in the previous frame image. It is a core technology in the field of computer vision and has a very wide range of applications. It is an essential technology for many downstream applications, such as motion analysis, behavior recognition, monitoring, and human-computer interaction.

[0003] At present, target matching technology is mainly divided into two categories, as follows:

[0004] 1. Target matching technology based on traditional technology. Representative technologies include Kalman filter tracking, optical flow tracking, template matching tracking, TLD tracking, CT tracking, KCF tracking, etc. The advantages of this type of technology are simple principles, fast operation speed, good results in simpler scenarios, and suitable for short-term tracking. Its disadvantage is that it requires the use of multi-target matching algorithms, has poor robustness, is prone to losing targets or following the wrong targets in slightly more complex scenarios, and cannot adapt to long-term tracking.

[0005] 2. Target matching technology based on deep learning technology. This type of technology mainly uses a target detection plus target matching strategy to complete the target tracking process. The process is to first locate the target position in each frame of the image with the help of a powerful deep learning-based target detection framework (such as: FASTER-RCNN, SSD, YOLO), and then use the nearest neighbor matching algorithm or feature vector matching algorithm to associate the same target in the previous and next frames to complete the target tracking process. The advantage of this type of technology is strong robustness and the ability to track for a long time. Its disadvantage is that it requires a robust multi-target matching algorithm to accurately and efficiently associate the targets in the previous and next frames.

[0006] How to find a multi-target matching algorithm with superior performance is still the main constraint on the development of current target tracking technology. Summary of the Invention

[0007] Based on the technical problems existing in the background technology, the present invention proposes a multi-target matching method based on deep learning, which can achieve end-to-end multi-target matching at a relatively low computational cost. It has strong versatility, high real-time performance, fewer error sources, can track for a long time, and has a strong tracking effect.

[0008] The present invention proposes a multi-target matching method based on deep learning, comprising:

[0009] Acquire an image to be identified, where the image to be identified includes a frame image and an ID mask image corresponding to the frame image;

[0010] Input the image to be identified into a pre-trained deep neural network model, which includes a feature extraction module, a correlation module and a prediction module;

[0011] The image to be identified is processed using a feature extraction module to obtain a feature map of the image to be identified;

[0012] Processing the feature map using a correlation module to obtain an input feature map with self-perceived spatiotemporal correlation;

[0013] The input feature map is processed using a prediction module to obtain a target correlation degree between two frames of images to be identified before and after the prediction.

[0014] Furthermore, the feature extraction module includes a concatenation layer concat0 and a feature extraction layer backbone connected in sequence, the input of the concatenation layer concat0 is used to input the image to be identified, and the output of the feature extraction layer backbone is connected to the input of the correlation module.

[0015] Furthermore, the correlation module includes a splicing layer concat1, a weighted matrix multiplication layer mul_qk, a self-attention map eigenvalue adjustment layer softmax and a matrix multiplication layer mul_v; the input of the splicing layer concat1 is connected to the output of the feature extraction layer backbone in the feature extraction module, the splicing layer concat1, the weighted matrix multiplication layer mul_qk, and the self-attention map eigenvalue adjustment layer softmax are connected in sequence, the output of the splicing layer concat1 and the output of the self-attention map eigenvalue adjustment layer softmax are both connected to the input of the matrix multiplication layer mul_v, and the output of the matrix multiplication layer mul_v is connected to the input of the prediction module.

[0016] Furthermore, the prediction module includes a convolutional layer conv0, a convolutional layer conv1, a global average pooling layer avepooling and a fully connected layer fc connected in sequence; the input of the convolutional layer conv0 is connected to the output of the matrix multiplication layer mul_v in the correlation module, and the fully connected layer fc outputs the target correlation between the two frames of images to be identified before and after the prediction.

[0017] Furthermore, the training steps of the deep neural network model include:

[0018] Collect pedestrian videos in various scenes, various lighting conditions, and various angles, and separate the pedestrian videos into frames to obtain frame images;

[0019] Obtain a sample image training set, where each sample image in the training set is obtained by annotating pedestrian target correlation information in two preceding and following frame images;

[0020] Constructing a target loss function for the deep neural network model;

[0021] The sample image training set is input into a defined deep neural network model for training to obtain the pre-trained deep neural network model.

[0022] Furthermore, the objective loss function L is:

[0023]

[0024] Among them, L id Represents the loss function of each pedestrian target matching. The target matching loss function uses the classic cross entropy loss function. N valid Indicates the number of pedestrian targets appearing in the current sample image, α indicates the loss function calculation switch, and * indicates the product.

[0025] Furthermore, the pedestrian target association information in the two frames is annotated. The specific steps are as follows:

[0026] The deep learning-based pedestrian detection framework obtains the pedestrian position in each frame of the pedestrian video as the pedestrian position information, annotates the pedestrian position information, and obtains the pedestrian target position information;

[0027] Generate the corresponding ID mask image based on the pedestrian target position information in each frame image;

[0028] Arbitrarily select a frame image containing a pedestrian target as the previous frame image, and arbitrarily select an image in the subsequent set frame as the current frame image. The current frame image, the ID mask image of the current frame image, the previous frame image, and the ID mask image of the previous frame image together form a motion image pair;

[0029] Manually review each motion image pair, associate the pedestrian target IDs in different frames, obtain the pedestrian target ID association information, and generate a one-hot type association feature vector of length N+1;

[0030] According to the size of the sequence number value in the pedestrian target's ID association information, the associated feature vector of each pedestrian target is concatenated in sequence.

[0031] Furthermore, the ID mask image generation process is as follows:

[0032] Generate a mask image with the same resolution as the frame image and all pixel values ​​are 0;

[0033] Get the number of all pedestrian targets in the frame image;

[0034] Within this number range, each pedestrian target is randomly assigned a unique ID number;

[0035] Finally, in the mask image, all pixels within the pedestrian target location information range are assigned the ID serial number value of the target to obtain the ID mask image.

[0036] Furthermore, the method for associating the ID association information of the pedestrian target is as follows:

[0037] If the previous and next frame images are of the same pedestrian target, the ID number value of the pedestrian target in the previous frame ID mask image is used as the predicted category number of the target in the current frame image;

[0038] If the previous and next frame images are different pedestrian targets, the predicted category number of the pedestrian target in the current frame image is set to N+1;

[0039] Based on the predicted category number of each target, a one-hot associated feature vector of length N+1 is generated.

[0040] A computer-readable storage medium stores a plurality of acquisition and classification programs, wherein the plurality of acquisition and classification programs are used to be called by a processor and execute the multi-target matching method described above.

[0041] The advantages of the multi-target matching method based on deep learning provided by the present invention are: the multi-target matching method based on deep learning provided in the structure of the present invention, with the help of a deep neural network model, directly completes the multi-target association matching process end-to-end, adaptively obtains the spatiotemporal correlation of the previous and next frame images, and then completes the optimal matching of multiple targets. At a relatively low computational cost, it can achieve end-to-end multi-target matching, has strong versatility, high real-time performance, fewer error sources, can track for a long time, and has a strong tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a structural schematic diagram of the present invention;

[0043] Figure 2 This is the overall structure diagram of the deep neural network model;

[0044] Figure 3 This is the network structure diagram of the feature extraction module;

[0045] Figure 4 It is the network structure diagram of the correlation module;

[0046] Figure 5 It is the network structure diagram of the prediction module;

[0047] The symbol on the left side of each neural network structure layer graph represents the output feature map size of the network structure: feature map width × feature map height × number of feature map channels; DETAILED DESCRIPTION

[0048] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] like Figures 1 to 5 As shown, the present invention proposes a multi-target matching method based on deep learning, comprising the following steps S100 to S500:

[0050] S100: Acquire an image to be identified, where the image to be identified includes a frame image and an ID mask image corresponding to the frame image;

[0051] S200: Inputting the image to be recognized into a pre-trained deep neural network model, where the deep neural network model includes a feature extraction module, a correlation module, and a prediction module;

[0052] S300: Processing the image to be identified using a feature extraction module to obtain a feature map of the image to be identified;

[0053] S400: Processing the feature map using a correlation module to obtain an input feature map with self-perceived spatiotemporal correlation;

[0054] S500: Processing the input feature map using a prediction module to obtain a target correlation between two frames of images to be recognized before and after the prediction.

[0055] This embodiment uses a deep neural network model to directly complete the multi-target association matching process end-to-end, adaptively obtain the spatiotemporal correlation of the previous and next frame images, and then complete the optimal matching of multiple targets. At a relatively low computational cost, it can achieve end-to-end multi-target matching, with strong versatility, high real-time performance, fewer error sources, long-term tracking, and strong tracking effect robustness.

[0056] Before performing multi-target matching on the above-mentioned image to be identified, it is necessary to build and train a deep neural network model, and then use the trained deep neural network model to perform multi-target matching. The overall solution is as follows: Figures 1 to 5 The embodiment shown is applicable to all multi-target tracking scenarios. For the convenience of description, the present invention takes pedestrian multi-target tracking as an example for description:

[0057] S1. Design a deep neural network model. The main function of the deep neural network model designed by the present invention is to directly complete the matching and tracking of pedestrian targets in each frame image with the help of a deep neural network model with self-perception of the spatiotemporal correlation between the previous and next frame images. Since there is no need to manually design a pedestrian target matching method, the entire pedestrian matching and tracking system has a faster operation speed, fewer sources of error, and a more robust tracking effect. The present invention adopts a convolutional neural network (CNN). In order to facilitate the description of the present invention, some terms are defined: feature map resolution refers to feature map height × feature map width, feature map size refers to feature map width × feature map height × number of feature map channels, kernel size refers to kernel width × kernel height, span refers to width direction span × height direction span, and each convolution layer is followed by a batch normalization layer and a nonlinear activation layer. Figure 2 As shown in the figure, the deep neural network model designed in the present invention includes three modules: a feature extraction module backbone module, a correlation module relation module, and a prediction module predict module. The specific design steps are as follows:

[0058] S11, feature extraction module backbone module, the feature extraction module is mainly used to obtain high-level features with high abstraction and rich expression capabilities of the input image. The quality of high-level feature extraction directly affects the performance of subsequent pedestrian target matching. The feature extraction module can adopt any commonly used classic network structure. For the convenience of description, the present invention adopts the classic ResNet network structure, such as Figure 3 As shown in the figure, the input of the feature extraction network consists of a three-channel RGB image with a resolution of 320×320 and a single-channel grayscale image with a resolution of 320×320. The RGB image represents a frame, and the grayscale image represents the ID mask image of all pedestrian targets in the frame. concat0 is the concatenation layer, which mainly stitches the two input images into a four-channel image of the same resolution according to the channel dimension. backbone is obtained by intercepting the ResNet backbone network before the mean pooling layer. out_feature is the output feature map of the feature extraction module, which is used for subsequent pedestrian target matching and tracking. The feature map size of out_feature is 20x20x256.

[0059] S12, correlation module relation module, the correlation module is mainly based on the feature map of the previous and next frame images output by the feature extraction module, through the self-attention theory, in a global range, the features between multiple targets are correlated to improve the accuracy of multi-target matching. The network structure of the correlation module designed by the present invention is as follows Figure 4As shown, it includes a concatenation layer concat1, a weighted matrix multiplication layer mul_qk, a self-attention map eigenvalue adjustment layer softmax, and a matrix multiplication layer mul_v; the input of the concatenation layer concat1 is connected to the output of the feature extraction layer backbone in the feature extraction module, the concatenation layer concat1, the weighted matrix multiplication layer mul_qk, and the self-attention map eigenvalue adjustment layer softmax are connected in sequence, the output of the concatenation layer concat1 and the output of the self-attention map eigenvalue adjustment layer softmax are both connected to the input of the matrix multiplication layer mul_v, and the output of the matrix multiplication layer mul_v is connected to the input of the prediction module.

[0060] Among them, concat1 is a concatenation layer, whose main function is to concatenate multiple input feature maps into a single output feature map according to the channel dimension; mul_qk is a weighted matrix multiplication layer, which is used to obtain the autocorrelation of the input feature map, namely the self-attention map. The specific operation method is to first multiply the input feature map matrix by the transposed matrix of the input feature map matrix, and then divide each element of the resulting calculation matrix by the square root of the number of channels of the input feature map; softmax is a self-attention map eigenvalue adjustment layer, which normalizes the eigenvalues ​​of the self-attention map based on the softmax theory to a range between 0 and 1; mul_v is a matrix multiplication layer, which mainly weights the input feature map based on the self-attention map to obtain an input feature map with self-perceived spatiotemporal correlation. The specific operation method is to multiply the output feature map matrix of the softmax layer by the input feature map matrix. It should be noted that to improve the performance of the correlation module, the correlation module can be used multiple times in series.

[0061] S13, prediction module predict module, the prediction module mainly predicts the target correlation between the two frames based on the input feature map with self-perception spatiotemporal correlation output by the correlation module, and then completes the multi-target matching and tracking process between the two frames. The specific network structure is as follows Figure 5 As shown in the figure, it includes the convolutional layer conv0, the convolutional layer conv1, the global average pooling layer avepooling and the fully connected layer fc connected in sequence; the input of the convolutional layer conv0 is connected to the output of the matrix multiplication layer mul_v in the correlation module, and the fully connected layer fc outputs the target correlation between the two frames of images to be recognized before and after.

[0062] Here, conv0 and conv1 are convolutional layers with a kernel size of 3x3 and a stride of 2x2; avepooling is a global average pooling layer; and fc is a fully connected layer whose output feature map is a feature vector of length Nx(N+1), where N represents N pedestrian targets and the matching of each pedestrian target is represented by an N+1-dimensional feature vector. N represents the maximum number of pedestrian targets allowed in any frame, which must be greater than the number of pedestrian targets that appear in any frame in the application scenario.

[0063] S2. Training the deep neural network model. After designing the deep neural network model, the next step is to collect pedestrian video images in various scenarios, feed them into the deep neural network model, and learn the relevant model parameters. The specific steps are as follows:

[0064] S21. Collect pedestrian videos, mainly collecting pedestrian videos in various scenes, various light conditions, and various angles.

[0065] S22: annotate the pedestrian target correlation information in the previous and next frame images to obtain a sample image training set. This mainly involves annotating the pedestrian position information in each frame image in the video and the correlation information of pedestrian targets between pairs of different frame motion images. The specific steps are as follows S221 to S225:

[0066] S221. Labeling pedestrian target location information. The main method is to use the existing deep learning-based pedestrian detection framework to obtain the pedestrian location in each frame image in the video as the pedestrian target location information.

[0067] S222. Generate an ID mask image, mainly generating a corresponding target ID mask image based on the pedestrian target position in each frame image, hereinafter referred to as the mask image. The specific mask image generation method is: first generate a mask image with the same resolution as the frame image and all pixel values ​​are 0, then obtain the number of all pedestrian targets in the frame image, and then randomly assign a unique ID number to each pedestrian target within the number value range, and finally, in the mask image, assign all pixels within the pedestrian target position range as the ID number value of the target.

[0068] S223. Constructing a motion image pair mainly involves converting the pedestrian video into an image sequence, arbitrarily selecting a frame image containing a pedestrian target as the previous frame image, and then arbitrarily selecting an image within the next 120 frames as the current frame image. Then, the current frame image, the ID mask image corresponding to the current frame image, the previous frame image, and the ID mask image corresponding to the previous frame image are combined to form a motion image pair.

[0069] S224. Obtain the ID association information of pedestrian targets. The main method is to manually review each motion image pair to determine whether the pedestrian targets in the previous and next frames of each motion image pair are the same pedestrian targets or newly appeared pedestrian targets, and then perform ID association of pedestrian targets in different frame images. The specific ID association method is as follows: for the same pedestrian targets in the previous and next frame images, the ID serial number value of the pedestrian target in the previous frame ID mask image is used as the predicted category serial number of the target in the current frame image. For the newly appeared target in the current frame, the predicted category serial number of the target is set to N+1. Finally, based on the predicted category serial number of each target, a one-hot type association feature vector of length N+1 is generated. N represents the maximum number of pedestrian targets allowed to appear in any frame image. This value needs to be greater than the number of pedestrian targets appearing in any frame image in the application scenario.

[0070] S225. Obtain the associated information of the pedestrian target, and sequentially concatenate the associated feature vectors of each pedestrian target according to the size of the pedestrian target ID number value.

[0071] S24, train the deep neural network model, send the organized motion image pair set into the defined deep neural network model, and learn the relevant model parameters. The loss function L during network model training is shown in formula (1), L id Represents the loss function for each pedestrian target matching, which uses the classic cross entropy loss function, N valid Indicates the number of pedestrian targets appearing in the current frame image, * indicates the product, and α indicates the loss function calculation switch. For pedestrian targets appearing in the current frame image, the value of α is 1, and for pedestrian targets not appearing in the current frame image, the value of α is 0.

[0072]

[0073] S3. Using the deep neural network model. After training the deep neural network model, the next step is to use the model in a real environment to perform pedestrian matching and tracking. For any given pedestrian image, the location of the pedestrian target is obtained according to the existing pedestrian detection model, and then the corresponding pedestrian target ID mask image is generated. This is then fed into the matching model to directly output the associated matching relationship between the pedestrian target in the previous and next frame images. The specific steps are as follows:

[0074] S31. Obtain the feature map of the previous frame image. The main method is to obtain the feature map of the first previous frame image by using a feature extraction module. For feature maps other than the first previous frame image, the present invention directly uses the cached feature map of the current frame image as the feature map of the next previous frame image, which can effectively improve the calculation speed.

[0075] S32, obtaining a feature map of the current frame image, mainly by performing calculations based on a feature extraction module.

[0076] S33. Predict the matching relationship of pedestrian targets between the previous and next frame images. The main method is to send the acquired feature maps of the previous and next frame images into the correlation module and the prediction module together to directly predict the matching relationship of pedestrian targets in the previous and next two frame images.

[0077] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A multi-target matching method based on deep learning, comprising: Acquire an image to be identified, where the image to be identified includes a frame image and an ID mask image corresponding to the frame image; Input the image to be identified into a pre-trained deep neural network model, which includes a feature extraction module, a correlation module and a prediction module; The image to be identified is processed using a feature extraction module to obtain a feature map of the image to be identified; Processing the feature map using a correlation module to obtain an input feature map with self-perceived spatiotemporal correlation; Processing the input feature map using a prediction module to obtain a target correlation between the two frames of images to be recognized before and after the prediction; The training steps of the deep neural network model include: Collect pedestrian videos in various scenes, various lighting conditions, and various angles, and separate the pedestrian videos into frames to obtain frame images; Obtain a sample image training set, where each sample image in the training set is obtained by annotating pedestrian target correlation information in two preceding and following frame images; Constructing a target loss function for the deep neural network model; Inputting the sample image training set into a defined deep neural network model for training to obtain the pre-trained deep neural network model; The specific steps for marking the pedestrian target association information in the two frames are as follows: The deep learning-based pedestrian detection framework obtains the pedestrian position in each frame of the pedestrian video as the pedestrian position information, annotates the pedestrian position information, and obtains the pedestrian target position information; Generate the corresponding ID mask image based on the pedestrian target position information in each frame image; Arbitrarily select a frame image containing a pedestrian target as the previous frame image, and arbitrarily select an image in the subsequent set frame as the current frame image. The current frame image, the ID mask image of the current frame image, the previous frame image, and the ID mask image of the previous frame image together form a motion image pair; Manually review each motion image pair, associate the pedestrian target IDs in different frames, obtain the pedestrian target ID association information, and generate a one-hot type association feature vector of length N+1; According to the size of the sequence number value in the pedestrian target's ID association information, the associated feature vector of each pedestrian target is concatenated in sequence.

2. The multi-target matching method based on deep learning according to claim 1, characterized in that: The feature extraction module includes a concatenation layer concat0 and a feature extraction layer backbone connected in sequence. The input of the concatenation layer concat0 is used to input an image to be identified. The output of the feature extraction layer backbone is connected to the input of the correlation module.

3. The multi-target matching method based on deep learning according to claim 2, characterized in that: The correlation module includes a concatenation layer concat1, a weighted matrix multiplication layer mul_qk, a self-attention map eigenvalue adjustment layer softmax, and a matrix multiplication layer mul_v; The input of the concatenation layer concat1 is connected to the output of the feature extraction layer backbone in the feature extraction module. The concatenation layer concat1, the weighted matrix multiplication layer mul_qk, and the self-attention map eigenvalue adjustment layer softmax are connected in sequence. The output of the concatenation layer concat1 and the output of the self-attention map eigenvalue adjustment layer softmax are both connected to the input of the matrix multiplication layer mul_v. The output of the matrix multiplication layer mul_v is connected to the input of the prediction module.

4. The multi-target matching method based on deep learning according to claim 3, characterized in that: The prediction module includes a convolutional layer conv0, a convolutional layer conv1, a global average pooling layer avepooling and a fully connected layer fc connected in sequence; The input of the convolutional layer conv0 is connected to the output of the matrix multiplication layer mul_v in the correlation module, and the fully connected layer fc outputs the target correlation between the two frames of images to be recognized before and after.

5. The multi-target matching method based on deep learning according to claim 1, characterized in that: The objective loss function L is: Among them, L id Represents the loss function of each pedestrian target matching. The target matching loss function uses the classic cross entropy loss function. N valid Indicates the number of pedestrian targets appearing in the current sample image, Indicates the loss function calculation switch, and * indicates the product.

6. The multi-target matching method based on deep learning according to claim 1, characterized in that: The generation process of the ID mask image is as follows: Generate a mask image with the same resolution as the frame image and all pixel values ​​are 0; Get the number of all pedestrian targets in the frame image; Within this number range, each pedestrian target is randomly assigned a unique ID number; Finally, in the mask image, all pixels within the pedestrian target location information range are assigned the ID serial number value of the target to obtain the ID mask image.

7. The multi-target matching method based on deep learning according to claim 1, characterized in that: The association method of the pedestrian target ID association information is as follows: If the previous and next frame images are of the same pedestrian target, the ID number value of the pedestrian target in the previous frame ID mask image is used as the predicted category number of the target in the current frame image; If the previous and next frame images are different pedestrian targets, the predicted category number of the pedestrian target in the current frame image is set to N+1; Based on the predicted category number of each target, a one-hot associated feature vector of length N+1 is generated.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of acquisition and classification programs, and the plurality of acquisition and classification programs are used to be called by a processor and execute the multi-target matching method according to any one of claims 1 to 7.

Images