A segmentation and tracking method based on attention and feature fusion

Abstract

The present application belongs to the technical field of image information processing, and is especially a segmentation and tracking method based on attention and feature fusion, comprising the following steps: step 1, constructing a basic segmentation and tracking framework based on SiamMask; step 2, adding a mixed attention module; step 3, adding feature fusion; step 4, model training, inputting sample pictures into a twin network for training, and the training process is offline; step 5, model testing. The present application enhances the feature learning ability of the network through the mutually dependent channel features and spatial features of the mixed attention template, generates more discriminative object representations, and thus greatly improves the model performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image information processing, in particular to a segmentation and tracking method based on attention and feature fusion. BACKGROUND

[0002] Single target tracking is one of the hotspots in the field of computer vision research and is widely used. Camera tracking focus, automatic target tracking of unmanned aerial vehicles, and tracking of specific objects such as human tracking, vehicle tracking in traffic monitoring systems, face tracking, and gesture tracking in intelligent interaction systems all require single target tracking technology. With the continuous in-depth research of researchers, visual target tracking has made breakthrough progress in the past decade, making visual tracking algorithms not only limited to traditional machine learning methods, but also combined with deep learning and related filters, which are the artificial intelligence boom in recent years, and have achieved robust and accurate results. With the great success of deep learning in speech recognition, image classification, and target detection, more and more research has applied deep learning frameworks to target tracking tasks, and target tracking technology has developed more comprehensively and has been widely used in various social fields. In particular, in the current social environment, the demand for high-tech tracking methods in various aspects of society is increasing, which further highlights the importance of target tracking technology.

[0003] Target tracking is to establish the positional relationship of the object to be tracked in a continuous video sequence and obtain the complete motion trajectory of the object. Given the target coordinate position in the first frame of the image, the exact position of the target in the next frame of the image is calculated. During the movement, the target may exhibit some changes in the image, such as changes in posture or shape, changes in scale, background occlusion, or changes in light brightness. The research of target tracking algorithms also focuses on solving these changes and specific applications. The key to the target tracking problem is the accuracy and speed of tracking, and with the continuous improvement of computer hardware performance, more and more things can be done through target tracking. Therefore, specific applications based on single target tracking have appeared in many fields.

[0004] With the development of target tracking, the accuracy of the shape and boundary of the target is increasingly required. Therefore, the method of combining image segmentation with target tracking can unify the two problems into one framework and more accurately locate the tracking target. Video object segmentation and tracking problems seem to be independent, but in fact they are complementary and inseparable. That is, the solution to one problem usually directly or indirectly solves the other problem. Many research works have noticed that simultaneously processing object segmentation and tracking problems can overcome their respective difficulties and improve their performance. Therefore, a segmentation and tracking method based on attention and feature fusion is proposed to solve the above problems. SUMMARY

[0005] (I) Technical problems solved

[0006] In view of the deficiencies in the prior art, the present application provides a segmentation and tracking method based on attention and feature fusion, which solves the problems raised in the background art.

[0007] (II) Technical solutions

[0008] In order to achieve the above-mentioned purpose, the present application specifically adopts the following technical solutions:

[0009] A segmentation and tracking method based on attention and feature fusion, comprising the following steps:

[0010] Step 1, constructing a basic segmentation and tracking framework based on SiamMask;

[0011] Step 2, adding a mixed attention module;

[0012] Step 3, adding feature fusion;

[0013] Step 4, model training, inputting sample pictures into a twin network for training, and the training process is offline;

[0014] Step 5, model testing.

[0015] Further, in step 1, the twin sub-network, the feature extraction network ResNet-50, the cross-correlation operation layer of the depth separable convolution, and the output three branches are included; wherein the twin sub-network layer is used to measure the similarity of two inputs, the ResNet-50 layer and the cross-correlation operation layer of the depth separable convolution are used to generate a plurality of candidate window response features, and the output three branches are a mask branch, a bounding box branch, and a score branch.

[0016] Further, the mixed attention module added in step 2 includes channel attention and spatial attention, wherein the two attention modules are connected in parallel, and the input is added after passing through the channel attention and spatial attention modules respectively, to obtain more accurate and calibrated feature maps; channel attention: the feature map is converted from [C, H, W] to [C, 1, 1] by using a global average pooling method, then two 1x1x1 convolutions are used for information processing to obtain a C-dimensional vector, then a sigmoid function is used for normalization to obtain the corresponding mask, and finally the channel is multiplied to obtain the feature map after information calibration; spatial attention: a 1x1x1 convolution is directly used on the feature map to convert [C, H, W] to [1, H, W] features, then a sigmoid is used for activation to obtain a spatial attention map, and finally the spatial attention map is directly applied to the original feature map to complete the information calibration of the space.

[0017] Further, in step 3, in order to strengthen the backbone network CNN feature expression, the model design is divided into three parts: bottom-up structure, top-down structure, and lateral connection structure. For the feature extraction network ResNet-50, only the first four stages are used, and a dilated convolution kernel with a dilated rate of 2 is used in the first layer of the fourth stage. The feature fusion strategy is improved, and the outputs of ResNet convolution blocks conv2, conv3, and conv4 are defined as {C2, C3, C4}. The layers with the same feature map size are classified as a stage, and the output of the last layer of each stage is extracted and down-sampled to be {1 / 4, 1 / 8, 1 / 16} times of the original image. Then, the features are fused through lateral connection with the features generated by up-sampling from top to bottom. Finally, 3*3 convolution is performed to obtain the outputs P2, P3, and P4.

[0018] Further, the model training in step 4 includes the following steps:

[0019] S1, input the sample picture into the twin network for training, and the training process is offline. Four data sets, COCO, ImageNet-DET 2015, ImageNet-VID 2015, and YouTube-VOS, are used for training.

[0020] S2, the twin network is used to measure the similarity of the input sample: the sample includes a target image and a search image. The target image refers to the image to be tracked, which is 127*127, and the search image refers to the image for tracking the target, which is 255*255. The twin neural network has two input branches, one branch inputting the target template Z and the other branch inputting the search region X. The two inputs are input into two weight-shared neural networks, which map the inputs to new spaces to form the representation of the inputs in the new spaces. The similarity of the two inputs is evaluated through loss calculation.

[0021] S3, feature map extraction network: the target template and the search region are put into the same feature extraction network, i.e. Resnet. Only the first four stages of RseNet are used, and a dilated convolution kernel with a dilated rate of 2 is used in the first layer of the fourth stage. The 3*3 convolution kernel is changed to a 7*7 convolution kernel, and the stride step is set to 1. Finally, two feature maps are obtained through the feature extraction network, which are the target image feature map and the search region feature map, respectively.

[0022] S4, generation of feature map: for the generation of the target feature map, the target is selected first, then preprocessed, and finally cropped with a cropping frame as the center, the center of the preprocessed picture is directly selected as the center, the size is 127, the coordinates of the cropping frame are obtained, and then random scaling, random translation of several pixels, random flipping and other operations are performed on the obtained cropping frame, and finally affine transformation is performed to obtain a picture with the target in the center of the picture;

[0023] For the generation of the search region feature map, similar to the generation of the target feature map, in addition to cropping search on the original picture, a mask is also cropped on the mask picture, and then random blurring, flipping and other data enhancement operations are performed on the picture and the mask. After that, a depth separable convolution operation is performed to obtain a response that maintains the number of channels unchanged, which is 256 here, the response in the middle is called RoW, which is the response of the candidate window, and then three branches are divided on the basis of the RoW, respectively for segmentation, regression and classification.

[0024] S5, pre-training: the network backbone is pre-trained on ImageNet-1k. SGD and a pre-training stage are used, that is, the learning rate is increased from 0.001 to 0.005 in the first five epochs, and then reduced to 0.0003 in the next 15 epochs.

[0025] Further, the step 5 model test comprises the following steps:

[0026] S1, test the trained model in the VOT dataset to obtain the tracking effect.

[0027] S2, track the tracking effect in a new video sequence, manually annotate a rectangular frame in the first frame, which contains the center position and size of the target to be tracked, and then the tracking algorithm needs to track the frame and perform segmentation, and the algorithm calculates the position offset and size change of the target in the subsequent frames.

[0028] (Three) beneficial effects

[0029] Compared with the prior art, the present application provides a segmentation and tracking method based on attention and feature fusion, which has the following beneficial effects:

[0030] The present application increases the mixed attention template, enhances the feature learning ability of the network through the mutual dependence of channel features and spatial features, generates more discriminative object representations, and greatly improves the performance of the model.

[0031] This invention addresses the problem of low model accuracy in tracking by introducing feature fusion. By utilizing feature fusion, the final output features can better represent the information of the input image in various dimensions, effectively improving the model's adaptability.

[0032] This invention introduces multiple hybrid attention to learn feature weights through the network based on the loss, so that effective feature maps have greater weights and ineffective or less effective feature maps have smaller weights, thus enabling the training model to achieve better results. It also employs feature fusion, which combines low-level detailed information and high-level semantic information, increasing the receptive field of the low-level layer. This allows the low-level layer to obtain more contextual information when tracking small targets, thereby improving robustness. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is a schematic diagram of the SiamMask network framework structure of the present invention;

[0035] Figure 3 This is a schematic diagram of the hybrid attention structure of the present invention;

[0036] Figure 4 This is a schematic diagram of the ResNet+ feature fusion structure of the present invention;

[0037] Figure 5 This is a schematic diagram of the training structure of the present invention;

[0038] Figure 6 This is an evaluation graph of the VOT2018 test dataset based on the present invention;

[0039] Figure 7 This is a schematic diagram of the structure resulting from the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example

[0042] like Figures 1-7 As shown, an embodiment of the present invention proposes a segmentation and tracking method based on attention and feature fusion, comprising the following steps:

[0043] like Figure 2As shown, step one, build a basic segmentation and tracking framework based on SiamMask; the core idea of SiamMask target tracking is to first frame the object image that needs to be tracked through the initial frame, which is used as the basis for searching in subsequent frames; secondly, input the target and search into the twin network at the same time, output two feature maps, and do cross-correlation on the two feature maps to get the feature map of RoW; then, a simple two-layer 1*1 convolution kernel convolution operation is performed on the output feature map to get the output, and finally the mask of the target is generated, the input of SiamMask refers to the twin network structure, which has two inputs, target image and search image, wherein the target image refers to the image to be tracked, for example, 127*127*3, 127*127 represents the size of the template image, and 3 represents the number of channels. The search image refers to the image in which the target is tracked, for example, 255*255*3, 255*255 represents the size of the search image, and 3 represents the number of channels;

[0044] The backbone network for extracting features is a CNN structure shared by two branches, one branch is the target template Z as input, and the other branch is the search region X as input, which will be input into the twin network at the same time, f θ ResNet is used for sharing weights, f θ The first four stages of RseNet are used, and a dilated convolution kernel with an expansion rate of 2 is used in the first layer of convolution in the fourth stage, that is, the 3*3 convolution kernel becomes 7*7, and the stage step is set to 1, which causes the fourth stage not to reduce the resolution of the feature map. Finally, two feature mappings are obtained through the backbone network, which are the target image feature map and the search region feature map, respectively. F1 and F2 are the generated feature maps. Since the size of Z is smaller than that of X, the feature map F1 is also smaller than the feature map F2.

[0045] Here *d is a depth level operation, that is, a cross-correlation operation of depth separable convolution, to obtain multiple candidate window response features, that is, to calculate the correlation of two feature maps to obtain a candidate frame response feature map with unchanged channel number, for example, 17*17*256. The response in the middle of the candidate window response feature map is called RoW, that is, the response of the candidate window, for example, 1*1*256.

[0046] Then three branches are divided on the basis of this RoW, which are segmentation, regression and classification respectively;

[0047] mask branch: h θThe network consists of two 1x1 convolutional layers, one with 256 channels and one with 63*63 channels. The 256-channel RoW is extracted from the cross-correlation result and passed through two fully connected layers to obtain a 63*63-channel RoW. Finally, the RoW is unfolded into a 63*63 response map. In this process, the information within the entire RoW is utilized for the classification of each pixel, eliminating the influence of objects similar to the target object. At the same time, low-level features and high-level features are fused to produce a more accurate mask.

[0048] box branch: the framework of the entire algorithm is to input the current frame and the previous frame into the network, and output the position of the bounding box in the current frame. After cropping the regions in the current frame and the previous frame, the features are extracted and concatenated into a fully connected layer. The purpose is to compare the features of the target and the current frame to find where the target has moved. The fully connected layer learns a complex feature comparison function, which outputs the relative motion of the target. Then, the output of the fully connected layer is connected to a 4-node layer representing the coordinates of the two corners of the bounding box to output the position of the target. The RoW is passed through a convolutional layer to obtain a 4K-channel response map, which represents the x, y, w, and h bias of each of the k anchor boxes.

[0049] score branch: the features of X and Z are extracted and subjected to a deep cross-correlation operation to obtain a confidence map score, which is calculated as follows:

[0050] score = φ(z) * φ(x) + b

[0051] where z is the template image, x is the search image, φ is the transformation function, * represents the cross-correlation operation, and b is a learnable number with an initial value of 0.

[0052] During training, the loss function is a binary logistic regression loss function for the responses of all candidate windows, which is calculated as follows:

[0053]

[0054] where y n ∈±1 is the true label of the real pixel-wise mask graph with size w*h, is the prediction of each pixel of the response of the nth candidate window;

[0055] The weighted loss function of the entire system is calculated as follows:

[0056] L 3B = λ1·L mask + λ2·L score + λ3·L box

[0057] The mask branch takes the DeepMask as reference, extracts a blue 256-channel RoW from the cross-correlation result, and obtains a 63*63-channel RoW through two fully connected layers. Each column is expanded into a 63*63 response map, representing the coincidence degree or similarity degree score of the target template and the search area at a certain position. Then, the resolution is expanded through upsampling to obtain a mask map. The box branch obtains a 4K-channel response map through convolution of the RoW, that is, the x, y, w, h bias of each of the k anchor boxes. The score branch outputs a 2k-channel response map, which is the foreground and background confidence of the k anchor boxes. The two branches use smooth L1 loss and cross-entropy loss, respectively.

[0058] As shown in Figure 3 Step two, add a mixed attention module. The channel attention and spatial attention modules are connected in parallel. After the input is respectively processed by the channel attention and spatial attention modules, the two modules are added to obtain a more accurate and calibrated feature map.

[0059] The channel attention first extracts image features through a convolution layer, then compresses the feature matrix into a feature vector, and then obtains the relationship between the features and the classification through a trained fully connected layer. Finally, the probability of all classifications corresponding to the target is calculated. The input feature map is U=[u1, u2,..., uc], where each channel ui, ui∈R H*W , obtains a vector z after passing through a global pooling layer, z∈R 1*1*C , and the formula for each position k is as follows:

[0060]

[0061] The obtained vector is trained through two fully connected layers to obtain the importance of each channel, then passes through ReLU, which enhances the independence between channels, and finally passes through a sigmod layer to normalize to 0 to 1 to obtain σ z' The formula for the whole process is as follows:

[0062] U1=[σ z'1 u1, σ z'2 u2, σ z'3 u3,..., σ z'n un]

[0063] Where z' i is the importance of the i-th channel ui,

[0064] The spatial attention first directly convolves the feature map, so that the feature map changes from [C, H, W] to [1, H, W] features, and then uses a sigmoid function to map the variable to [0, 1] for activation to obtain a spatial attention map. The spatial attention map is directly added to the original feature map to complete the spatial information calibration. The input feature map is U = [u 1,1 , u 1,2 ,..., u i ,j ,..., u H,W ], H and W are the sizes of the feature map, (i, j) is the spatial position of the feature map, and the spatial squeezing operation is realized by convolution. The convolution here is a 1*1 convolution, and the number of output channels is 1. The formula is as follows:

[0065] q = W sq ★U

[0066] Where q is a feature map with a channel number of 1, and then normalized to [0-1] by sigmod. The formula is as follows:

[0067]

[0068] Where represents the importance of the spatial position coordinate (i, j) in the feature map,

[0069] As shown in Figure 4 , step three, feature fusion, feature fusion will not change the original backbone network. The side feature maps of each layer of the original backbone network are taken out for operation. The model design is divided into three parts: bottom-up structure, top-down structure, and horizontal connection structure. Bottom-up: as the network in the left half part deepens, the size of the feature map will become smaller and the semantic information will become richer. Here, the last feature map of each stage is used to form a feature pyramid. Top-down: through the method of upsampling, the feature map is enlarged, so that the low-level features also contain rich semantic information. Horizontal connection: the results of upsampling and the feature maps of the same size generated by bottom-up are fused. That is, the feature map from the left is first subjected to a 1*1 convolution operation, then added to the feature map from above, and then subjected to a 3*3 convolution to obtain the feature output of the current layer.

[0070] When the input is [320*320*64], the output of the layer1 layer of resnet50 is [160*160*256], wherein the layer1 output of resnet and the input dimension are the same, layer1 does not perform W and H downsampling in all resnets, and is only adjusted for channels, the output of the Layer2 layer is [80*80*512], the output of the Layer3 layer is [40*40*1024], H and W are reduced by two times at the Conv1 layer and the MaxPool layer before the layer1 layer, so the height and width at the layer1 are reduced by 4 times, layer1 does not modify the height and width, layer2-layer3 each layer reduces the height and width by two times, P2-P4 are used for predicting the bbox of an object, box regression and mask in the future;

[0071] Step four, input the sample picture into the network for training, and the training process is offline; wherein the sample picture is 127*127, the search picture is 255*255, and the network backbone is pre-trained on ImageNet-1k. A total of 20 cycles are trained, wherein the learning rate is increased from 0.001 to 0.005 in the first five epochs, and is reduced to 0.0003 in the last 15 epochs;

[0072] Step five, model testing; the training effect of the method is tested according to the evaluation index of the VOT dataset provided by the VOT official website, from Figure 6 It can be found that the single target tracking algorithm proposed in the application has better performance than the original basic algorithm.

[0073] As Figure 2 shown, in some embodiments, the basic network framework is a basic tracking framework based on SiamMask, including a twin subnetwork, a feature extraction network ResNet-50, a cross-correlation operation layer of a deep separable convolution, and three output branches; wherein the twin subnetwork layer is used to measure the similarity of two inputs, the ResNet-50 layer and the cross-correlation operation layer of the deep separable convolution are used to generate a plurality of candidate window response features, and the three output branches are a mask branch, a box branch and a score branch.

[0074] The mask branch: is used for predicting a target mask, and the classification of each pixel utilizes information in the entire RoW, eliminates the influence of objects similar to the target object, and fuses low-level features and high-level features to generate a more accurate mask;

[0075] box branch: is a deviation value of the anchor frame and the actual position of the box, which is used to correct the position of the final frame. The framework of the whole algorithm is to input the current frame and the previous frame into the network, and output the position of the bounding box in the current frame. After the current frame and the previous frame are respectively cropped into regions, they are sent into the network for feature extraction. These features are concatenated and input into the fully connected layer. The purpose is to compare the features of the target and the features of the current frame to find where the target has moved. The fully connected layer learns a complex feature comparison function, which outputs the relative motion of the target. Then the output of the fully connected layer is connected to a 4-node layer representing the coordinates of the two corners of the bounding box to output the position of the target. The RoW obtains a 4K channel response map through the convolutional layer, which is the x, y, w, h bias of k anchor frames respectively;

[0076] score branch: used to distinguish whether the anchor frame is foreground or background, extract the features of the template image and the search image, and perform a deep cross-correlation operation on them to obtain a confidence map score.

[0077] As shown in Figure 3 In some embodiments, the SiamMask network introduces a new hybrid attention mechanism, which includes channel attention and spatial attention. The two attention modules are connected in parallel, and the input is added after passing through the channel attention and spatial attention modules respectively to obtain a more accurate and calibrated feature map. Channel attention: the feature map is converted from [C, H, W] to [C, 1, 1] by global average pooling method, then two 1x1x1 convolutions are used for information processing to obtain a C-dimensional vector, then a sigmoid function is used for normalization to obtain the corresponding mask, and finally the channel is multiplied to obtain the feature map after information calibration.

[0078] Spatial attention: directly using 1x1x1 convolution on the feature map, converting [C, H, W] to [1, H, W] feature, then using sigmoid for activation to obtain a spatial attention map, and normalizing to [0-1] to obtain a 1*H*W-dimensional spatial attention map feature. Then the feature is re-calibrated and multiplied with the original U in space to complete the information calibration of space.

[0079] As shown in Figure 4In some embodiments, the SiamMask network introduces feature fusion, strengthens the backbone network CNN feature expression, and the model design is divided into three parts: bottom-up structure, top-down structure, and horizontally connected structure. For the feature extraction network ResNet-50, only the first four stages are used, and a dilated convolution kernel with a dilated rate of 2 is used in the first layer of convolution in the fourth stage. The feature fusion strategy is improved. The outputs of ResNet convolution blocks conv2, conv3, and conv4 are defined as {C2, C3, C4}. The layers with the same size of feature map output are classified as a stage. The output of the last layer of each stage is extracted, down-sampled to {1 / 4, 1 / 8, 1 / 16} times of the original image, and then fused with the feature map generated by up-sampling from top to bottom through horizontal connection. Finally, 3*3 convolution is performed to obtain the output P2, P3, and P4.

[0080] C4 is operated by 256 1*1 convolution kernels to obtain 64*64*256, denoted as P4. P4 is up-sampled by a step of 2, and then added to the result obtained by operating C3 by 256 1*1 convolution kernels, denoted as P3. P3 is up-sampled by a step of 2, and then added to the result obtained by operating C2 by 256 1*1 convolution kernels, denoted as P2.

[0081] As Figure 5 In some embodiments, the model training includes the following steps:

[0082] First, obtain samples and input the samples into the network model for training. The samples include target images and search images. The target image refers to an image to be tracked with a size of 127*127, and the search image refers to an image for performing tracking on the target with a size of 255*255. The backbone network is a CNN structure shared by two branches. One branch takes a target template Z as input, and the other branch takes a search region X as input. The target template and the search region are put into the same feature extraction network, i.e., Resnet. Only the first four stages of RseNet are used, and a dilated convolution kernel with a dilated rate of 2 is used in the first layer of convolution in the fourth stage. The 3*3 convolution kernel is changed to a 7*7 convolution kernel, and the stride step is set to 1. Finally, two feature maps are obtained through the feature extraction network, which are the target image feature map and the search region feature map.

[0083] For the generation of the target feature map, the target is first selected and then preprocessed. Finally, the center of the cropped frame is used for center cropping. The center of the preprocessed picture is directly selected as the center, and the size is 127. The coordinates of the cropped frame are obtained. Then, random scaling, random translation of several pixels, random flipping, and other operations are performed on the obtained cropped frame. Finally, the target is obtained in the picture center through affine transformation.

[0084] The generation of the search region feature map is similar to that of the target feature map. In addition to cropping the search area from the original image, a mask needs to be cropped from the mask image. Then, random blurring, flipping, and other data augmentation operations need to be performed on the image and mask simultaneously. After that, a depthwise separable convolution operation is performed, and the number of channels in the obtained response remains unchanged, which is 256 in this case. The intermediate response is called the RoW, which is the response of the candidate window. Then, three branches are divided on this RoW for segmentation, regression, and classification.

[0085] Segmentation is used to predict the target mask; regression is used to distinguish whether the anchor box is foreground or background; and classification is a deviation value between the anchor box position and the actual position, used to correct the final box position.

[0086] The network backbone was pre-trained on ImageNet-1k. SGD and a pre-training phase were used, where the learning rate was increased from 0.001 to 0.005 for the first five epochs and then decreased to 0.0003 for the following 15 epochs.

[0087] like Figure 6 As shown, in some embodiments, the model testing includes the following steps: testing the tracking performance of the trained model in a new video sequence;

[0088] like Figure 7 As shown, in single-target tracking, a rectangular box is usually given in the first frame, containing the center position and size of the target to be tracked. This box is usually manually annotated. Then, the tracking algorithm needs to follow this box in subsequent frames and calculate the target's position offset and size changes in subsequent frames. In order to have a direct visual perception, the test results on the VOT dataset will be displayed on the video sequence.

[0089] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An attention and feature fusion based segmentation and tracking method, characterized in that: The method comprises the following steps: Step 1, constructing a SiamMask-based basic segmentation and tracking framework; In step 1, a twin subnetwork, a feature extraction network ResNet-50, a cross-correlation operation layer of a depth separable convolution, and three output branches are included. The twin subnetwork layer is used to measure the similarity of two inputs, the ResNet-50 layer and the cross-correlation operation layer of the depth separable convolution are used to generate a plurality of candidate window response features, and the three output branches are a mask branch, a bounding box branch, and a score branch. Step 2, adding a hybrid attention module; In step 2, the hybrid attention module includes channel attention and spatial attention, wherein the two attention modules are connected in parallel, and the input is added after passing through the channel attention and spatial attention modules to obtain a more accurate and calibrated feature map; the channel attention: the feature map is converted from [C, H, W] to [C, 1, 1] through global average pooling, then two 1x1x1 convolutions are used for information processing to obtain a C-dimensional vector, then a sigmoid function is used for normalization to obtain a corresponding mask, and finally the information-calibrated feature map is obtained through channel multiplication; the spatial attention: a 1x1x1 convolution is directly used on the feature map to convert [C, H, W] into a feature of [1, H, W], then a sigmoid function is used for activation to obtain a spatial attention map, and finally the spatial attention map is directly applied to the original feature map to complete the information calibration of the space. Step 3, adding feature fusion; In step 3, in order to strengthen the feature expression of the backbone network CNN, the model design is divided into three parts: a bottom-up structure, a top-down structure, and a horizontal connection structure; for the feature extraction network ResNet-50, only the first four stages are used, and a dilation rate of 2 is used in the first layer convolution of the fourth stage; the feature fusion strategy is improved, the outputs of the ResNet convolution blocks conv2, conv3, and conv4 are defined as {C2, C3, C4}, wherein the layers with the same output feature map size are classified into a stage, the output feature of the last layer of each stage is extracted, down-sampling is performed to make the outputs be {1 / 4, 1 / 8, 1 / 16} times of the original image, then the feature maps generated by the top-down up-sampling are fused through horizontal connection, and finally a 3*3 convolution is performed to obtain the outputs P2, P3, and P4. Step 4, model training, inputting sample pictures into the twin network for training, and the training process is offline; Step 5, model testing. 2.The segmentation and tracking method based on attention and feature fusion of claim 1, characterized in that: The model training in step 4 comprises the following steps: S1, inputting sample pictures into the twin network for training, and the training process is offline; four data sets of COCO, ImageNet-DET 2015, ImageNet-VID 2015, and YouTube-VOS are used for training; S2, the twin network is used to measure the similarity of the input sample: the sample includes a target image and a search image, the target image refers to an image to be tracked, which is 127*127, and the search image refers to an image for tracking the target, which is 255*255; the twin neural network has two input branches, one branch of the target template Z as input, and the other branch of the search region X as input; the two inputs enter two weight-shared neural networks, which respectively map the inputs to a new space to form the representation of the inputs in the new space; the similarity of the two inputs is evaluated through the calculation of the loss; S3, the feature map extraction network: the target template and the search region are put into the same feature extraction network, i.e. Resnet, only the first four stages of RseNet are used, and a convolution kernel with an expansion rate of 2 is used in the first layer of convolution in the fourth stage, the 3*3 convolution kernel is changed into a 7*7 convolution kernel, and the stride step is set to 1, finally two feature maps are obtained through the feature extraction network, which are target image feature map and search region feature map; S4, generation of feature map: for the generation of target feature map, the target is selected first, then preprocessed, and finally cropped in the center of the cropping box, the center of the preprocessed picture is directly selected as the center, the size is 127, the coordinates of the cropping box are obtained, then random scaling, random translation of several pixels, and random flipping operations are performed on the obtained cropping box, and finally affine transformation is performed to obtain a picture with the target in the center of the picture; For the generation of the search region feature map, compared with the generation of the target feature map, in addition to cropping the search from the original picture, a mask is also cropped from the mask picture, and then random blur and flipping data enhancement operations are performed on the picture and the mask synchronously; then the depth separable convolution operation is performed, and the response obtained keeps the channel number unchanged, which is 256 here, the response in the middle is called RoW, which is the response of the candidate window, and then three branches are divided on the basis of the RoW, which are segmentation, regression and classification respectively; S5, pre-training: the network backbone is pre-trained on ImageNet-1k; SGD and a pre-training stage are used, that is, the learning rate is increased from 0.001 to 0.005 in the first five epochs, and then reduced to 0.0003 in the next 15 epochs. 3.The segmentation and tracking method based on attention and feature fusion of claim 1, characterized in that: The step 5 model test comprises the following steps: S1, test the trained model in the VOT dataset to obtain the tracking effect; S2, track the tracking effect in a new video sequence, manually annotate a rectangular box in the first frame, which contains the center position and size of the target to be tracked, then the tracking algorithm needs to track the box in the subsequent frames while performing segmentation, and the algorithm calculates the position offset and size change of the target in the subsequent frames.

Images