A data augmentation method for marine fish tracking

Abstract

The present invention relates to the field of computer vision technology, and specifically to a data enhancement method for marine fish tracking. It is mainly used to solve the problems of tracking drift, target occlusion, target deformation, interference of similar targets, etc. in marine fish tracking. The present invention generates new marine fish image data by combining the tracking target and targets in the image that are similar or dissimilar to the tracking target. First, the tracking target and other potential targets in the image are extracted using a mature semantic segmentation model to obtain a target set similar to the tracking target and a target set dissimilar to the tracking target; then the tracking target and the potential targets are enhanced and fused with the original image to generate a new tracking image. Finally, during the tracking process, images are randomly selected in the tracking frame for image enhancement.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a data enhancement method for marine fish tracking. Background Art

[0002] Marine fish tracking is of great significance to the scientific research of marine ecosystems and sustainable fishery management, and has important applications in marine ecosystem health research, fishery protection, marine species protection, and marine ecological chain research.

[0003] In marine fish tracking, the initial position of the target to be tracked is framed in the video image, and key information such as the position and size of the target can be accurately predicted in subsequent frames. Existing target tracking algorithms use the original marine fish motion images to extract image depth features based on deep models such as convolutional neural networks and Transformer models, and then find the position of the tracked target in subsequent motion through processes such as feature matching. Therefore, the target tracking model needs to deal with problems such as target occlusion, target deformation, motion blur, and interference from similar targets. In fish tracking, the above problems will become more difficult due to the marine environment, marine fish movement characteristics, etc. It is difficult to meet the marine fish tracking needs simply based on the original marine motion image frames. Therefore, the present invention proposes a data image enhancement method for marine fish tracking.

[0004] In computer vision, image enhancement techniques can enhance the generalization ability of the model, reduce model overfitting, and enhance data diversity. Image enhancement techniques include geometric transformations (such as rotation, scaling, shearing, flipping) and photometric transformations (such as saturation, grayscale, color jitter). In addition, existing image enhancement techniques may also be enhanced by mixing images, such as MixUp pixel-by-pixel mixing of images. CutMix replaces the content of random areas with patches cut from another image and other data enhancement techniques. These image enhancement methods have been proven to be effective in tasks such as object detection, semantic segmentation, and video classification, but there are the following problems when applied to marine fish movement: First, existing image enhancement methods enhance images at a global scale, but target tracking requires careful processing of the relationship between the target area and other background areas. Second, compared with other target tracking tasks, fish tracking is more prone to target occlusion and interference from similar targets. Finally, the problem of tracking model performance enhancement. Summary of the invention

[0005] In view of the deficiencies of the above-mentioned prior art, the present invention provides a data enhancement method for marine fish tracking. The method proposes to enhance the diversity of tracking targets and enhance the distinguishability of tracking targets from background interference in marine fish images, which can well solve the problems of similar background interference, rapid motion and occlusion that occur in marine fish tracking. Specifically, the present invention proposes to extract potential moving targets in marine fish images, and divide these targets into difficult confusing targets and simple targets, and then enhance the tracking images based on the moving targets and confusing targets. The generated images can simulate various deformations of fish and enhance the robustness of the model to target occlusion and similar background interference. Finally, the present invention randomly selects images from all training frames for enhancement based on the tracking process, which further improves the marine fish tracking performance.

[0006] In order to make up for the deficiencies of the prior art, the utility model provides a data enhancement method for marine fish tracking.

[0007] The present invention is achieved through the following technical solutions:

[0008] A data enhancement method for marine fish tracking, characterized in that it comprises the following steps:

[0009] Step 1: Extract image objects using mature semantic segmentation models

[0010] Given a target to be tracked in the first frame image of a fish motion video, the first frame image is input into an existing mature semantic segmentation model, N groups of background targets are extracted using the existing mature semantic segmentation model, the spatial Euclidean distance of the center position between the background target and the tracked target is calculated, the characteristic cosine distance between the background target and the tracked target is calculated, the two distances are weighted to generate a confusion distance matrix between the tracked target and the image background target, and the background target is distinguished into background similar targets and background distinguishable targets according to the confusion matrix;

[0011] Step 2: Tracking target data enhancement

[0012] Perform data enhancement on the tracked target, and randomly select an enhancement method to enhance the tracked target, and perform weighted addition of the enhanced image pixels and the tracked target pixels at the original position;

[0013] Step 3: Background target data enhancement

[0014] A sample is randomly selected from the background similar target for data enhancement and the enhanced pixel value is randomly added to the position farther from the tracking target; a sample is randomly selected from the background distinguishable target for data enhancement and the enhanced pixel value is randomly added to the position closer to the tracking target;

[0015] Step 4: Tracking using target tracking technology based on the enhanced image

[0016] Use a deep convolutional neural network to extract deep features of the search image and template image, calculate the similarity between the features of each area or window in the search image and the features of the template image; use cosine similarity or Euclidean distance to measure the similarity between the two; predict the new position of the target in the search image based on the results of the similarity calculation; select the position with the highest similarity as the final predicted position of the target, and finally update the target update model based on the online learning strategy;

[0017] Step 5: After a certain number of frames, repeat steps 1-4 to generate an enhanced image in a new frame.

[0018] Furthermore, in step 1, the segmentation model is a SAM model.

[0019] Furthermore, in step 1, the specific steps are as follows: in the initial frame of the video sequence, a 127×127×3 template frame image z is cropped with the tracking target as the center, where the position of the template frame in the image is p z =(x l ,y l ,x r ,y r ), where x l ,y l is the coordinate of the lower left corner, x r ,y r ) is the coordinate of the upper right corner;

[0020] Then the original frame image and the template frame image are input into the SAM model; after the original frame image is input into the SAM model, the SAM model outputs the potential target, and after eliminating the tracking target, the position coordinates p corresponding to the potential target in the background information are given. i =(x l ,y l ,x r ,y r ) and the corresponding deep features

[0021] After the template frame image is input into the SAM model, the corresponding depth features are given Calculate the cosine distance between the tracking target feature and the background target feature, is the position coordinate p corresponding to the potential target in the background information i =(x l ,y l ,x r ,y r )’s corresponding depth features;

[0022] Calculate the Euclidean distance between the tracking target position and the background target position, and calculate Where (x1, y1) is the center coordinate of the tracking target frame, and (x2, y2) is the center coordinate of the background target frame;

[0023] Taking into account the cosine distance and Euclidean distance, the difference distance value between the background target and the tracked target is generated. max(d p ) is the maximum value of the distances between multiple background targets and the tracking target;

[0024] Select a background target argmin(d) with the smallest difference from the tracking target from all background targets, denoted as O min , whose location area in the image is p min =(x l ,y l ,x r ,y r );Select a background target argmax(d) with the largest difference from the tracking target from all background targets, denoted as O max , whose location area in the image is p max =(x l ,y l ,x r ,y r ).

[0025] Furthermore, through step 1, three image targets in the image are obtained: the tracking target and two background targets;

[0026] First, we use image enhancement technology to enhance the target data. The enhancement methods include: color jitter, brightness change, image flip, motion blur, etc. By enhancing the target data, we simulate the appearance changes that may occur in marine fish.

[0027] The enhanced image of the tracking target z is denoted as z arg , z arg The pixel values ​​are mixed with the tracking target area in the original image. The fusion method is: mix =w*z+(1-w)*z arg , where w is the weight controlling the mixing of the two, and z is the unenhanced tracking target area image in the original image; by mixing with each other, new appearance changes are added on the basis of the original fish appearance representation, making the model more robust to fish movement changes.

[0028] Furthermore, in step 3, the background target data enhancement steps are as follows:

[0029] First, based on the background target O with the smallest difference from the tracking target min , it is regarded as a confusing target that is similar to the tracking target in representation and position; therefore, for the background target O min After data enhancement, the enhanced background target is recorded as Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a larger threshold, when the distance between the random position and the tracked target is greater than the threshold, mix the enhanced background target with the original image pixel value, the fusion method is

[0030] Afterwards, based on the background target O which is significantly different from the tracking target max , which is regarded as a confusing target with a large difference in representation and position from the tracking target. max After data enhancement, the enhanced background target is recorded as

[0031] Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a small threshold, when the distance between the random position and the tracked target is less than the threshold, mix the enhanced background target with the original image pixel value, the fusion method is

[0032] Furthermore, in step 4, the background target data enhancement steps are as follows:

[0033] First, the template frame image is cropped out from the enhanced image, and then the template frame image and the image are sent to the template branch and the search branch for feature extraction respectively. In the subsequent tracking task, they are sent to the common embedding space for similarity learning, and the tracking box position is generated. The regression loss is used to minimize the distance between the tracking box and the real box, and the back-propagation algorithm is used to update the target tracking model.

[0034] Compared with the prior art, the beneficial effect of the present invention is that the present invention provides a data enhancement method for marine fish tracking, which is performed by enhancing different targets in marine fish tracking images. First, the tracking target is enhanced to simulate changes such as target deformation and rapid motion that may occur in marine fish, and then other target elements in the image are enhanced to simulate background interference, target occlusion and other problems that occur in target tracking in an ocean environment. Through the image enhancement method proposed by the present invention, the marine fish tracking model learns more generalized tracking targets, more discriminative background knowledge, and better handles problems such as target deformation, rapid motion, and target occlusion that occur in fish motion tracking, further improving the positioning accuracy and tracking robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings.

[0036] Figure 1 The overall process architecture diagram of the method of the present invention;

[0037] Figure 2 This is a flow chart of Example 2 of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example 1

[0040] The main steps of the present invention are as follows:

[0041] Step 1: Given the first frame image (search image) of a fish motion video and the target to be tracked (template image), input the search image into the existing mature semantic segmentation model (Segment Anything Model, SAM) model, and use the existing SAM model to extract N groups of background targets. Calculate the spatial Euclidean distance between the center position of the background target and the tracked target, calculate the characteristic cosine distance between the background target and the tracked target, and generate the confusion distance matrix between the tracked target and the image background target after weighting the cosine distance and the Euclidean distance. According to the confusion matrix, the background target is distinguished into background similar targets and background distinguishable targets.

[0042] Step 2: perform data enhancement (color jitter, brightness change, image flipping, motion blur, etc.) on the tracked target, and randomly select an enhancement method to enhance the tracked target, and perform weighted addition of the enhanced image pixels and the tracked target pixels at the original position (this step of the present embodiment is weighted fusion, and the weighted fusion method in step 2 of embodiment 2 can be used, which will not be repeated).

[0043] Step 3. Randomly select a sample from the background similar target for data enhancement and randomly add the enhanced pixel value to the position that is farther away from the tracking target (here "farther" refers to the larger value in the spatial Euclidean distance, and "farther" here is a threshold. The pixel value is randomly added to the position where the distance from the tracking target is greater than the threshold); randomly select a sample from the background distinguishable target for data enhancement and randomly add the enhanced pixel value to the position that is closer to the tracking target (here closer is also a threshold. The pixel value is randomly added to the position where the distance from the tracking target is less than the threshold).

[0044] Step 4: Use target tracking technology to track based on the enhanced image. Use a deep convolutional neural network to extract the deep features of the search image and the template image, and calculate the similarity between the features of each area or window in the search image and the features of the template image. Use cosine similarity or Euclidean distance to measure the similarity between the two. Based on the results of the similarity calculation, predict the new position of the target in the search image. Select the position with the highest similarity as the final predicted position of the target, and finally update the target update model based on the online learning strategy.

[0045] Step 5: After a certain number of frames, repeat steps 1-4 to generate an enhanced image in a new frame.

[0046] Example 2

[0047] The present invention provides a data enhancement method for marine fish tracking, and its overall process architecture is as follows: Figure 1 As shown, there are 5 steps in total.

[0048] 1 Extract image targets using SAM model

[0049] In the initial frame of the video sequence, a 127×127×3 template frame image z is cropped with the tracking target as the center, where the position of the template frame in the image is p z =(x l ,y l ,x r ,y r ), where x l ,y l is the coordinate of the lower left corner, x r ,y r is the coordinate of the upper right corner. Then the original frame image and the template frame image are input into the SAM model. The SAM model is an advanced image segmentation model. Based on the training of massive data, the SAM model has a strong generalization ability, and the SAM model supports the segmentation of targets in the image without prompts. After the original frame image is input into the SAM model, the model outputs the potential target, and after eliminating the tracking target, the position coordinates p corresponding to the potential target in the background information are given. i =(x l ,y l ,x r ,y r ) and the corresponding deep features The corresponding depth features after the template frame image is input into the SAM model Calculate the cosine distance between the tracking target feature and the background target feature, Calculate the Euclidean distance between the tracking target position and the background target position, and calculate Where (x1, y1) is the center coordinate of the tracking target frame, and (x2, y2) is the center coordinate of the background target frame. Considering the two distances, the difference distance value between the background target and the tracking target is generated. Select a background target argmin(d) with the smallest difference from the tracking target from all background targets, denoted as O min , whose location area in the image is p min =(x l ,y l ,x r ,y r ). Select a background target argmax(d) with the largest difference from the tracking target from all background targets, denoted as O max , whose location area in the image is p max =(x l ,y l ,x r ,y r ).

[0050] 2. Tracking target data enhancement

[0051] After step 1, the present invention obtains three image targets in the image: the tracking target and two background targets. First, for the tracking target, the image enhancement technology is used to enhance the target data. The enhancement means include: color jitter, brightness change, image flipping, motion blur, etc. By enhancing the target data, the appearance change that may occur in marine fish can be simulated. The image after the tracking target z is enhanced is denoted as z arg , z arg The pixel values ​​are mixed with the tracking target area in the original image. The fusion method is: mix =w*z+(1-w)*z arg , where w is the weight controlling the mixture of the two. By mixing them together, new appearance changes are added on the basis of the original fish appearance representation, making the model more robust to fish movement changes.

[0052] 3 Background target data enhancement

[0053] In order to alleviate the problems of background interference, target occlusion, etc. in the movement of marine fish, the present invention uses the interference targets appearing in the background to enhance the image data. First, based on the background target O with the smallest difference from the tracking target min , we regard it as a confusing target that is similar to the tracking target in representation and position, and we hope that it appears far away from the tracking target. min After data enhancement, the enhanced background target is recorded as Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a larger threshold, when the distance between the random position and the tracked target is greater than the threshold, mix the enhanced background target with the original image pixel value, the fusion method is Afterwards, based on the background target O which is significantly different from the tracking target max , we regard it as a confusing target that is quite different from the tracking target in representation and position, and we hope that it will appear close to the tracking target. max After data enhancement, the enhanced background target is recorded as Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a small threshold, when the distance between the random position and the tracked target is less than the threshold, mix the enhanced background target with the original image pixel value, the fusion method is

[0054] In this step, when the larger threshold is set to 2 / 3 of the image size, the Euclidean distance between the random position and the center of the tracked target is greater than 2 / 3d; when the smaller threshold is set to 1 / 3 of the image size, the Euclidean distance between the random position and the center of the tracked target is less than 1 / 3d.

[0055]

[0056] Where (H, W) is the image size.

[0057] Step 4: Target tracking based on enhanced images

[0058] After hybrid enhancement, the tracking target area in the original image data is enhanced. A target different from the tracking target is used for enhancement at a position close to the tracking target, and a target similar to the tracking target is used for enhancement at a position far from the tracking target. Based on the enhanced image, the target tracking model is trained. First, the template frame image is cropped from the enhanced image, and then the template frame image and the image are sent to the template branch and the search branch for feature extraction, and are sent to the common embedding space for similarity learning in the subsequent tracking task to generate the tracking box position, use regression loss to minimize the distance between the tracking box and the real box, and use the back propagation algorithm to update the target tracking model.

[0059] Step 5: Random enhancement

[0060] In the subsequent training process, image frames at a certain interval are randomly selected from the training frames for target enhancement.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data enhancement method for marine fish tracking, characterized in that: The following steps are involved: Step 1: Extract image objects using mature semantic segmentation models Given a target to be tracked in the first frame of a fish motion video, the first frame is input into an existing mature semantic segmentation model, N groups of background targets are extracted, the spatial Euclidean distance between the center position of the background target and the tracked target is calculated, the characteristic cosine distance between the background target and the tracked target is calculated, the two distances are weighted to generate a confusion distance matrix between the tracked target and the image background target, and the background target is distinguished into background similar targets and background distinguishable targets according to the confusion matrix; Step 2: Tracking target data enhancement Perform data enhancement on the tracked target, and randomly select an enhancement method to enhance the tracked target, and perform weighted addition of the enhanced image pixels and the tracked target pixels at the original position; Step 3: Background target data enhancement Randomly select a sample from the background similar to the target for data augmentation and randomly add the enhanced pixel value to the position far away from the tracked target; After randomly selecting a sample from the background distinguishing target for data enhancement, the enhanced pixel value is randomly added to the position close to the tracking target; "far" and "near" here are thresholds, and the pixel value is randomly added to the position where the distance to the tracking target is greater than the threshold; the pixel value is randomly added to the position where the distance to the tracking target is less than the threshold; The steps for background target data enhancement are as follows: First, based on the background target O with the smallest difference from the tracking target min , it is regarded as a confusing target that is similar to the tracking target in representation and position; therefore, for the background target O min After data enhancement, the enhanced background target is recorded as Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a threshold, which is 2 / 3 of the image size. When the distance between the random position and the tracked target is greater than the threshold, mix the enhanced background target with the original image pixel value. The fusion method is Where w is the weight controlling the mixture of the two. Then, based on the background target O which is significantly different from the tracked target, max , which is regarded as a confusing target with a large difference in representation and position from the tracking target. max After data enhancement, the enhanced background target is recorded as Will Add to the random position of the image, calculate the distance between the random position and the tracked target, design a threshold, which is 1 / 3 of the image size. When the distance between the random position and the tracked target is less than the threshold, mix the enhanced background target with the original image pixel value. The fusion method is Step 4: Tracking using target tracking technology based on the enhanced image Use a deep convolutional neural network to extract deep features of the search image and template image, calculate the similarity between the features of each area or window in the search image and the features of the template image; use cosine similarity or Euclidean distance to measure the similarity between the two; Based on the results of similarity calculation, predict the new position of the target in the search image; select the position with the highest similarity as the final predicted position of the target, and finally update the target update model based on the online learning strategy; Step 5: After a certain number of frames, repeat steps 1-4 to generate an enhanced image in a new frame.

2. The method according to claim 1, characterized in that In step 1, the segmentation model is the SAM model.

3. The method according to claim 1, characterized in that In step 1, the specific steps are as follows: in the initial frame of the video sequence, a 127×127×3 template frame image z is cropped with the tracking target as the center, where the position of the template frame in the image is p z =(x l ,y l , x r ,y r ), where x l ,y l is the coordinate of the lower left corner, x r ,y r is the coordinate of the upper right corner; Then the original frame image and the template frame image are input into the SAM model; after the original frame image is input into the SAM model, the SAM model outputs the potential target, and after eliminating the tracking target, the position coordinates p corresponding to the potential target in the background information are given. i =(x l ,y l , x r ,y r ) and the corresponding deep features After the template frame image is input into the SAM model, the corresponding depth features are given Calculate the cosine distance between the tracking target feature and the background target feature, Calculate the Euclidean distance between the tracking target position and the background target position, and calculate Where (x1, y1) is the center coordinate of the tracking target frame, and (x2, y2) is the center coordinate of the background target frame; Taking into account the cosine distance and Euclidean distance, the difference distance value between the background target and the tracked target is generated. max(d p ) is the maximum value of the distances between multiple background targets and the tracking target; Select a background target argmin(d) with the smallest difference from the tracking target from all background targets, denoted as O min , whose location area in the image is p min =(x l ,y l , x r ,y r );Select a background target argmax(d) with the largest difference from the tracking target from all background targets, denoted as O max , whose location area in the image is P max =(x l ,y l , x r ,y r ).

4. The method according to claim 1, characterized in that In step 2, the specific steps are as follows: through step 1, three image targets in the image are obtained: the tracking target and two background targets; First, we use image enhancement technology to enhance the target data. The enhancement methods include: color jitter, brightness change, image flip, motion blur, etc. By enhancing the target data, we simulate the appearance changes that may occur in marine fish. The enhanced image of the tracking target z is denoted as z arg , z arg The pixel values ​​are mixed with the tracking target area in the original image. The fusion method is: mix =w*z+(1-w)*z arg , where w is the weight controlling the mixing of the two, and z is the unenhanced tracking target area image in the original image; by mixing with each other, new appearance changes are added on the basis of the original fish appearance representation, making the model more robust to fish movement changes.

5. The method according to claim 1, characterized in that In step 4, the background target data enhancement steps are as follows: First, the template frame image is cropped out from the enhanced image, and then the template frame image and the image are sent to the template branch and the search branch for feature extraction respectively. In the subsequent tracking task, they are sent to the common embedding space for similarity learning, and the tracking box position is generated. The regression loss is used to minimize the distance between the tracking box and the real box, and the back-propagation algorithm is used to update the target tracking model.

6. The method according to claim 1, characterized in that In step 3, when the threshold is set to 2 / 3 of the image size, that is, the Euclidean distance between the random position and the center of the tracked target is greater than 2 / 3d, Where (H, W) is the image size.

7. The method according to claim 1, characterized in that In step 3, when the threshold is set to 1 / 3 of the image size, that is, the Euclidean distance between the random position and the center of the tracked target is less than 1 / 3d, Where (H, W) is the image size.

Images