Target tracking method and target tracking device

Abstract

The present disclosure provides a target tracking method, including: obtaining a target image and an image to be tracked, the target image including a target object; dividing the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks; extracting a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; and determining the position of the target object from the image to be tracked by a prediction function according to the first feature map and the second feature map. The present disclosure also provides a target tracking device.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a target tracking method and a target tracking device. Background Art

[0002] Object tracking is an important research problem in the field of computer vision. Its purpose is to find the target object in the subsequent video frames based on the position of the target object in the initial image frame in the video sequence. Current tracking algorithms usually have good tracking accuracy, but there are bottlenecks in the deployment of tracking algorithms, making it difficult to deploy them on small, low-power edge computing devices. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a target tracking method and a target tracking device.

[0004] According to a first aspect of the present disclosure, a target tracking method is provided, comprising: acquiring a target image and an image to be tracked, the target image comprising a target object; dividing the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks; extracting a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; and determining a position of the target object from the image to be tracked by a prediction function based on the first feature map and the second feature map.

[0005] According to an embodiment of the present disclosure, extracting a first feature map from the multiple first image blocks includes: performing local reasoning on the multiple first image blocks in sequence to obtain multiple first reasoning results; stitching the multiple first reasoning results to obtain a first stitching object; and performing global reasoning on the first stitching object to obtain a first feature map.

[0006] According to an embodiment of the present disclosure, the multiple first image blocks include N first image blocks, where N is an integer greater than or equal to 2; the local reasoning is performed on the multiple first image blocks in sequence to obtain multiple first reasoning results, including: performing local reasoning on the i-th first image block among the N first image blocks in a preset storage space, i∈[1,N); and when it is determined that the local reasoning of the i-th first image block is completed, releasing the preset storage space to perform local reasoning on the i+1-th first image block in the preset storage space.

[0007] According to an embodiment of the present disclosure, extracting the first feature map from the multiple first image blocks includes: inferring the multiple first image blocks through a neural network to obtain the first feature map, and the multiple first image blocks have the same neural network weights.

[0008] According to an embodiment of the present disclosure, determining the position of the target object from the image to be tracked through a prediction function based on the first feature map and the second feature map includes: fusing the first feature map and the second feature map through a prediction function to obtain a fused feature map; and determining the position of the target object in the image to be tracked by performing a regression operation on the fused feature map.

[0009] According to an embodiment of the present disclosure, determining the position of the target object in the image to be tracked by performing a regression operation on the fused feature map includes: calculating a classification output loss function, an overall confidence loss function, and a boundary offset loss function based on the fused feature map; constructing a joint loss function based on the classification output loss function, the overall confidence loss function, and the boundary offset loss function; and determining the position of the target object in the image to be tracked through the joint loss function.

[0010] According to an embodiment of the present disclosure, determining the position of the target object in the image to be tracked through the joint loss function includes: outputting a plurality of candidate boxes through the prediction function, wherein the candidate boxes are six-dimensional vectors, and the six-dimensional vectors are described by parameters related to the joint loss function; calculating prediction scores of the plurality of candidate boxes through the joint loss function; determining M prediction boxes from the plurality of candidate boxes, wherein the prediction scores of the M prediction boxes are the top M prediction scores of the plurality of candidates, where M is a positive integer; and determining the position of the target object in the image to be tracked based on the M prediction boxes.

[0011] According to an embodiment of the present disclosure, determining the position of the target object in the image to be tracked based on the M prediction boxes includes: obtaining M coordinates of the M prediction boxes in the image to be tracked; calculating the average value of the M coordinates to obtain the coordinates of the target object in the image to be tracked.

[0012] According to an embodiment of the present disclosure, the image to be tracked is a continuous multi-frame image, and the method further includes: defining a search area in the next frame image with the coordinates of the target object in the previous frame image as the center, and the area of ​​the search area is 4 times the area occupied by the target object in the previous frame image.

[0013] A second aspect of the present disclosure provides a target tracking device, including: an acquisition module, used to acquire a target image and an image to be tracked, the target image including a target object; a division module, used to divide the target image and the image to be tracked, respectively, to obtain a plurality of first image blocks and a plurality of second image blocks; an extraction module, used to extract a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; and a determination module, used to determine the position of the target object from the image to be tracked by a prediction function based on the first feature map and the second feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 A schematic diagram schematically shows a target tracking method according to an embodiment of the present disclosure;

[0016] Figure 2 A flowchart of a target tracking method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 A schematic diagram schematically shows a target tracking method according to an embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram of an inference network according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5 A schematic diagram schematically shows a hybrid of local reasoning and global reasoning according to an embodiment of the present disclosure;

[0020] Figure 6 A schematic diagram schematically shows a target tracking result according to an embodiment of the present disclosure; and

[0021] Figure 7 The structural block diagram of the target tracking device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] An embodiment of the present disclosure provides a target tracking method, including: acquiring a target image and an image to be tracked, the target image including a target object; dividing the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks; extracting a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; and determining the position of the target object from the image to be tracked by a prediction function based on the first feature map and the second feature map.

[0027] Figure 1 A schematic diagram of a target tracking algorithm according to an embodiment of the present disclosure is schematically shown.

[0028] like Figure 1 As shown in the figure, the target object in the continuous video sequence is tracked based on the twin network. Two neural networks are used as feature extraction networks to extract features in the search area of ​​the template image and the search image. The two neural networks share parameters. Then, the prediction function is used to determine the position of the target object in the template image in the search area based on the extracted features. The tracking method based on the twin network can be expressed by the following formula:

[0029] Loc = g(f(x), f(z))

[0030] Among them, f(x) is the feature extraction function, g(x) is the prediction function, x is the template image, and z is the search image.

[0031] The template image can be the first frame image of the video sequence or any frame image in the video sequence. The target object is any mark in the template image. Taking the frame image (the first frame image or any frame image) as the starting point, the target object in the video frame image behind the frame image in the video sequence is tracked.

[0032] Figure 2 The following is a flowchart schematically showing a target tracking method according to an embodiment of the present disclosure. Figure 3 A schematic diagram of a target tracking method according to an embodiment of the present disclosure is schematically shown.

[0033] like Figure 2 As shown, the target tracking method of this embodiment includes operations S210 to S240.

[0034] In operation S210 , a target image and an image to be tracked are acquired, and the target image includes a target object.

[0035] The target image is the template image in the target tracking algorithm, and the image to be tracked is the search area of ​​the search image in the target tracking algorithm. The target image can be the first frame of the video sequence, or any frame in the video sequence that contains the object of interest to the user.

[0036] The target image is not updated during target tracking.

[0037] In operation S220, the target image and the image to be tracked are divided respectively to obtain a plurality of first image blocks and a plurality of second image blocks.

[0038] like Figure 3 As shown, the resolution of the target image is 128*128, and the resolution of the image to be tracked is 256*256. The target image and the image to be tracked are both divided into image blocks of size 16*16. The multiple first image blocks are multiple target image blocks obtained by dividing the target image, and the multiple second image blocks are multiple image blocks to be tracked obtained by dividing the image to be tracked.

[0039] In operation S230, a first feature map and a second feature map are extracted from the plurality of first image blocks and the plurality of second image blocks.

[0040] In the disclosed embodiment, the first feature map is a target feature map of a target image block, and the second feature image block is a to-be-tracked feature map of a to-be-tracked image block.

[0041] Figure 4 The figure schematically shows a schematic diagram of an inference network according to an embodiment of the present disclosure.

[0042] The reasoning process for the target image block and the image block to be tracked is the same. Taking the target image block as an example, extracting a target feature map from multiple target image blocks includes: performing local reasoning on multiple target image blocks in sequence to obtain multiple first reasoning results; splicing multiple first reasoning results to obtain a first spliced ​​object; and performing global reasoning on the first spliced ​​object to obtain a target feature map.

[0043] In the disclosed embodiment, a neural network is used to infer multiple image blocks to obtain a feature map, and each image block has the same neural network weight. The weight overhead of the neural network is reduced during the image block-based reasoning, and the total number of neural network parameters is reduced.

[0044] In the shallow layers of the neural network, local reasoning based on image patches is used, and in the deep layers of the neural network, global reasoning is used. The local reasoning network based on image patches uses the deep separation convolutional network structure in MobileNetV2. Figure 2 As shown in the figure, MB45*5 means that the expansion ratio of the MobileNet network structure is 4, and the convolution kernel size is 5*5. The global inference network uses the MLPMixer network structure.

[0045] Figure 5 The figure schematically shows a hybrid of local reasoning and global reasoning according to an embodiment of the present disclosure.

[0046] like Figure 5 As shown in the figure, based on the local reasoning of the image block, the receptive field of the reasoning process is limited, and the input of the convolution layer of the neural network only includes a limited number of pixels of the image block. Combining local reasoning with global reasoning can provide a global receptive field, so that the input of the convolution layer includes every pixel of the image block, thereby ensuring tracking accuracy.

[0047] In the disclosed embodiments, feature maps may be extracted under a limited storage budget.

[0048] Assume that the target image is divided into N target image blocks, where N is an integer greater than or equal to 2. Performing local reasoning on the N target image blocks in sequence in a limited storage space includes: performing local reasoning on the i-th target image block among the N target image blocks in a preset storage space, i∈[1,N); and when it is determined that the local reasoning on the i-th target image block is completed, releasing the preset storage space to perform local reasoning on the i+1-th first image block in the preset storage space.

[0049] With a limited storage budget, only one image block is processed at a time. After the first image block is processed, the storage space occupied by the first image block is released and the second image block is processed, thereby reducing the storage overhead required to generate feature maps during the calculation process and achieving lightweight neural networks.

[0050] In operation S240 , a position of the target object is determined from the image to be tracked by using a prediction function according to the first feature map and the second feature map.

[0051] In the embodiment of the present disclosure, operation S240 determines the position of the target object from the image to be tracked through a prediction function based on the first feature map and the second feature map, including: fusing the first feature map and the second feature map through the prediction function to obtain a fused feature map; and determining the position of the target object in the image to be tracked by performing a regression operation on the fused feature map.

[0052] When determining the location of the target object in the image to be tracked through regression operation, predictions can be made from three branches: target classification results, overall confidence, and target boundary offset.

[0053] Construct the loss functions of the three prediction branches, then combine the three loss functions to construct a joint loss function, and use the stochastic gradient descent method as the parameter optimizer of the neural network.

[0054] The joint loss function L is as follows:

[0055] L=L CLS +λ1L C +λ2L reg

[0056] Among them, L CLS is the loss function of the target classification result, using the cross entropy loss function; L C is the loss function of the overall confidence, using the cross entropy loss function; L reg As the loss function for target boundary offset, an IOU loss function is used; preferably, weighting parameters λ1 and λ2 are determined empirically as λ1=1 and λ2=3.

[0057] In the process of determining the position of the target object in the image to be tracked, multiple candidate boxes are output through the prediction function; the prediction scores of the multiple candidate boxes are calculated through the joint loss function; M prediction boxes are determined from the multiple candidate boxes, and the prediction scores of the M prediction boxes are the top M prediction scores of the multiple candidates; the position of the target object in the image to be tracked is determined according to the M prediction boxes. Among them, the candidate box is a six-dimensional vector, and the six-dimensional vector is described by parameters related to the joint loss function, and M is a positive integer.

[0058] Determining the position of the target object in the image to be tracked according to the M prediction frames includes: obtaining M coordinates of the M prediction frames in the image to be tracked; and calculating an average value of the M coordinates to obtain the coordinates of the target object in the image to be tracked.

[0059] In the embodiment of the present disclosure, the prediction function outputs 25*25, a total of 625 candidate boxes. The candidate box can be represented as a six-dimensional vector (CLS p ,CLS n ,C,l,t,r,b). Among them, CLS p , CLS n CLS represents the probability that the candidate box is the target object and the background respectively, C represents the confidence of the candidate box tracking result, l, t, r and b represent the distance between the candidate box and the left, top, right and bottom boundaries of the image to be searched respectively. p and CLS n is the loss function L of the target classification result CLS The corresponding parameter, C is the loss function L of the overall confidence C The corresponding parameters, l, t, r and b, are the loss function L of the target boundary offset reg The corresponding parameters.

[0060] In the training of the neural network, the true value of the confidence C is determined by the following formula:

[0061]

[0062] The overall confidence output in the prediction function represents the overall confidence of the candidate box, and the prediction score of the candidate box can be calculated by the function q = (1-λ)cls*C+λH. Among them, cls is the target classification result score of the candidate box, λ is the balance weight, and H is the cosine window function. To ensure that the candidate box can move smoothly in adjacent image frames. Preferably, the 8 candidate boxes with the largest final score q are selected as the prediction boxes. The average value of the coordinates of these 8 prediction boxes is calculated as the final tracking result output.

[0063] Figure 6 A schematic diagram schematically shows a target tracking result according to an embodiment of the present disclosure.

[0064] like Figure 6 As shown, the first frame image is used as the target template, and target tracking is performed in the subsequent 50th frame, 100th frame, and 200th frame images.

[0065] The present invention performs target tracking based on a twin network, and utilizes an image block-based reasoning algorithm to reduce both the weight overhead of the neural network and the storage overhead required to generate feature maps. At the same time, by combining local reasoning based on image blocks with global reasoning, a lightweight and efficient backbone network is constructed. The backbone network in the twin network determines the performance, speed, and parameter amount of the twin network tracking. Combining the optimized backbone network with the prediction algorithm can ensure the accuracy of the tracking algorithm while ensuring the low parameter amount and storage overhead characteristics, thereby achieving higher parameter utilization.

[0066] The present disclosure also provides another embodiment of a target tracking method. The image to be tracked is a continuous multi-frame image, and the target tracking method further includes: defining a search area in a next frame of image with the coordinates of the target object in the previous frame of image as the center, and the area of ​​the search area is 4 times the area of ​​the area occupied by the target object in the previous frame of image.

[0067] In the disclosed embodiment, the coordinates of the final prediction box in the previous frame image are obtained, and the search area of ​​the next frame image is delineated with the center of the final prediction box. Usually, an area with a length and width twice that of the target object is delineated with the center as a new search area, so as to further improve the tracking efficiency.

[0068] Based on the above target tracking method, the present disclosure also provides a target tracking device. Figure 7 The device is described in detail.

[0069] Figure 7 The structural block diagram of the target tracking device according to an embodiment of the present disclosure is schematically shown.

[0070] like Figure 7 As shown, the target tracking device 700 of this embodiment includes an acquisition module 710 , a division module 720 , an extraction module 730 and a determination module 740 .

[0071] The acquisition module 710 is used to acquire a target image and an image to be tracked, wherein the target image includes a target object. In one embodiment, the acquisition module 710 may be used to perform the operation S210 described above, which will not be described in detail herein.

[0072] The division module 720 is used to divide the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks. In one embodiment, the division module 720 can be used to perform the operation S220 described above, which will not be described in detail herein.

[0073] The extraction module 730 is used to extract the first feature map and the second feature map from the plurality of first image blocks and the plurality of second image blocks. In one embodiment, the extraction module 730 may be used to perform the operation S230 described above, which will not be described in detail herein.

[0074] The determination module 740 is used to determine the position of the target object from the image to be tracked by using a prediction function according to the first feature map and the second feature map. In one embodiment, the determination module 740 can be used to perform the operation S240 described above, which will not be described in detail here.

[0075] According to an embodiment of the present disclosure, any multiple modules in the acquisition module 710, the extraction module 720, the division module 730 and the determination module 740 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 710, the extraction module 720, the division module 730 and the determination module 740 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or encapsulating the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 710, the extraction module 720, the division module 730 and the determination module 740 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0076] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0077] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0078] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A target tracking method, comprising: Acquire a target image and an image to be tracked, wherein the target image includes a target object; Dividing the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks; Extracting a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; as well as Determining the position of the target object from the image to be tracked by using a prediction function according to the first feature map and the second feature map; Wherein, determining the position of the target object from the image to be tracked by using a prediction function according to the first feature map and the second feature map includes: By using a prediction function, the first feature map and the second feature map are fused to obtain a fused feature map; Based on the fused feature map, calculating a classification output loss function, an overall confidence loss function, and a boundary shift loss function; Constructing a joint loss function according to the classification output loss function, the overall confidence loss function and the boundary shift loss function; Outputting a plurality of candidate boxes through the prediction function, wherein the candidate boxes are six-dimensional vectors, and the six-dimensional vectors are described by parameters related to the joint loss function; Calculating the prediction scores of the multiple candidate boxes by using the joint loss function; Determine M prediction frames from the multiple candidate frames, the prediction scores of the M prediction frames being the top M of the prediction scores of the multiple candidates, where M is a positive integer; and According to the M prediction boxes, a position of the target object in the image to be tracked is determined.

2. The target tracking method according to claim 1, wherein: Extracting a first feature map from the plurality of first image blocks comprises: Performing local reasoning on the multiple first image blocks in sequence to obtain multiple first reasoning results; combining the plurality of first inference results to obtain a first combined object; and Perform global reasoning on the first stitched object to obtain a first feature map.

3. The target tracking method according to claim 2, wherein: The multiple first image blocks include N first image blocks, where N is an integer greater than or equal to 2; The sequentially performing local reasoning on the multiple first image blocks to obtain multiple first reasoning results includes: Performing local reasoning on the i-th first image block among the N first image blocks in a preset storage space, i∈[1,N); as well as When it is determined that the local reasoning on the i-th first image block is completed, the preset storage space is released to perform local reasoning on the i+1-th first image block in the preset storage space.

4. The target tracking method according to claim 2, wherein: The extracting a first feature map from the plurality of first image blocks comprises: The first feature map is obtained by inferring the multiple first image blocks through a neural network, and the multiple first image blocks have the same neural network weights.

5. The target tracking method according to claim 1, wherein: Determining the position of the target object in the image to be tracked according to the M prediction frames includes: Obtaining M coordinates of the M prediction boxes in the image to be tracked; The average value of the M coordinates is calculated to obtain the coordinates of the target object in the image to be tracked.

6. The target tracking method according to claim 1, wherein: The image to be tracked is a continuous multi-frame image, and the method further includes: A search area in a next frame of image is defined with the coordinates of the target object in the previous frame of image as the center, and the area of ​​the search area is 4 times the area of ​​the area occupied by the target object in the previous frame of image.

7. A target tracking device, comprising: An acquisition module, used for acquiring a target image and an image to be tracked, wherein the target image includes a target object; A division module is used to divide the target image and the image to be tracked respectively to obtain a plurality of first image blocks and a plurality of second image blocks. An extraction module, configured to extract a first feature map and a second feature map from the plurality of first image blocks and the plurality of second image blocks; as well as A determination module, configured to determine the position of the target object from the image to be tracked by using a prediction function according to the first feature map and the second feature map; Wherein, determining the position of the target object from the image to be tracked by using a prediction function according to the first feature map and the second feature map includes: By using a prediction function, the first feature map and the second feature map are fused to obtain a fused feature map; Based on the fused feature map, calculating a classification output loss function, an overall confidence loss function, and a boundary shift loss function; Constructing a joint loss function according to the classification output loss function, the overall confidence loss function and the boundary shift loss function; Outputting a plurality of candidate boxes through the prediction function, wherein the candidate boxes are six-dimensional vectors, and the six-dimensional vectors are described by parameters related to the joint loss function; Calculating the prediction scores of the multiple candidate boxes by using the joint loss function; Determine M prediction frames from the multiple candidate frames, the prediction scores of the M prediction frames being the top M of the prediction scores of the multiple candidates, where M is a positive integer; and According to the M prediction boxes, a position of the target object in the image to be tracked is determined.

Images