Target tracking method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a target tracking method and device, electronic equipment and computer readable storage medium, comprising: acquiring a field of view range and a corresponding target tracking state adopted for target tracking of a first image frame; determining a field of view range adopted for target tracking of a second image frame based on the field of view range and the corresponding target tracking state adopted for the first image frame; and performing target tracking based on the field of view range adopted for the second image frame to obtain a target tracking result corresponding to the second image frame. The step of performing target tracking can be processed by an artificial intelligence module. In the process of performing target tracking on the current image frame, the field of view range adopted for target tracking of the current image frame is dynamically adjusted according to the field of view range and the target tracking state of the previous image frame, thereby avoiding the cumulative error, reduced robustness and wasted computing power caused by the fixed field of view range in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular, the present application relates to a target tracking method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] Visual Object Tracking is an important research technology in computer vision technology. The system usually collects a sequence of images through a collection device, for example, a video is shot through a visible light camera. The user marks the object to be tracked in the first frame (also known as the initial frame) of the video, and the target tracking algorithm needs to complete the continuous tracking of the object in the subsequent frames of the video and provide the position information of the object. The core idea of the target tracking technology is to extract the template information according to the target marked in the first frame, calculate the matching degree of different candidate positions and the template in the search region of the subsequent video frames, and select the position with the highest matching degree to determine the target position. The selected search region range is also the field of view (FoV) of the target tracking technology. However, the size of the field of view (i.e. the search region range) used in the prior art is fixed, which causes the target tracking process to have problems such as error accumulation, reduced robustness, and waste of computing power, so it is necessary to improve the existing target tracking method. SUMMARY

[0003] The purpose of the present application is to at least solve one of the above technical defects. The technical solutions provided by the embodiments of the present application are as follows:

[0004] In a first aspect, the embodiments of the present application provide a target tracking method, comprising:

[0005] obtaining a field of view range used for target tracking on a first image frame and a corresponding target tracking state;

[0006] determining a field of view range used for target tracking on a second image frame based on the field of view range used for the first image frame and the corresponding target tracking state;

[0007] performing target tracking based on the field of view range used for the second image frame to obtain a target tracking result corresponding to the second image frame.

[0008] In an optional embodiment of the present application, the first image frame and the second image frame are collected by different image collection devices of the same intelligent device.

[0009] In an optional embodiment of the present application, the field of view range used for the first image frame or the second image frame is one of at least two field of view ranges.

[0010] In an optional embodiment of the present application, the target tracking state corresponding to the first image frame is acquired, including:

[0011] The target tracking result corresponding to the first image frame is acquired.

[0012] The target tracking state corresponding to the first image frame is determined based on the target tracking result.

[0013] In an optional embodiment of the present application, the tracking result includes a prediction confidence, and the target tracking state corresponding to the first image frame is determined based on the target tracking result, including:

[0014] According to a comparison result between the prediction confidence and a preset threshold, the target tracking state corresponding to the first image frame is determined.

[0015] In an optional embodiment of the present application, the preset threshold includes a first preset threshold and a second preset threshold, and according to a comparison result between the prediction confidence and the preset threshold, the target tracking state corresponding to the first image frame is determined, including at least one of the following:

[0016] If the prediction confidence is not less than the first preset threshold, the target tracking state corresponding to the first image frame is determined as a first state.

[0017] If the prediction confidence is not greater than the second preset threshold, the target tracking state corresponding to the first image frame is determined as a second state.

[0018] If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, the target tracking state corresponding to the first image frame is determined as a third state.

[0019] In an optional embodiment of the present application, the tracking result includes at least one of a target position and a target size, and a prediction confidence, and the target tracking state corresponding to the first image frame is determined based on the target tracking result, including:

[0020] At least one of a target relative displacement corresponding to the first image frame and a ratio between the target size in the first image frame and a global size is acquired based on at least one of the target position and the target size.

[0021] The target tracking state corresponding to the first image frame is determined based on at least one of a comparison result between the target relative displacement and a predetermined displacement, and a comparison result between the ratio and a preset ratio, and a comparison result between the prediction confidence and a predetermined threshold.

[0022] In an optional embodiment of the present application, the determination of the corresponding target tracking state based on at least one of the comparison result between the target relative displacement and the predetermined displacement, and the comparison result between the ratio and the preset ratio, and the comparison result between the prediction confidence and the predetermined threshold comprises at least one of the following:

[0023] If the prediction confidence is not less than the first preset threshold and the target relative displacement is not greater than the preset displacement, it is determined that the corresponding target tracking state is the first state.

[0024] If the prediction confidence is not greater than the second preset threshold and the ratio is not greater than the preset ratio, it is determined that the corresponding target tracking state is the second state.

[0025] If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, it is determined that the corresponding target tracking state is the third state.

[0026] In an optional embodiment of the present application, the at least two field of view ranges include a first field of view range, a second field of view range and a third field of view range, wherein the second field of view range is greater than the first field of view range, and the third field of view range is less than the first field of view range.

[0027] Based on the field of view range adopted by the first image frame and the corresponding target tracking state, the field of view range adopted for target tracking of the second image frame is determined, comprising at least one of the following:

[0028] When the field of view range adopted by the first image frame is the first field of view range, if the target tracking state is the second state, it is determined that the field of view range adopted by the second image frame is the second field of view range, if the target tracking state is the first state, it is determined that the field of view range adopted by the second image frame is the third field of view range, and if the target tracking state is the third state, it is determined that the field of view range adopted by the second image frame is the first field of view range.

[0029] When the field of view range adopted by the first image frame is the second field of view range, if the target tracking state is the second state, it is determined that the field of view range adopted by the second image frame is the second field of view range, if the target tracking state is the first state, it is determined that the field of view range adopted by the second image frame is the first field of view range, and if the target tracking state is the third state, it is determined that the field of view range adopted by the second image frame is the second field of view range.

[0030] When the field of view range adopted by the first image frame is the third field of view range, if the target tracking state is the second state, it is determined that the field of view range adopted by the second image frame is the first field of view range, if the target tracking state is the first state, it is determined that the field of view range adopted by the second image frame is the third field of view range, and if the target tracking state is the third state, it is determined that the field of view range adopted by the second image frame is the third field of view range.

[0031] In an optional embodiment of the present application, the target tracking is performed based on the field of view range adopted by the second image frame to obtain a target tracking result corresponding to the second image frame, including:

[0032] Based on an initial image frame in a sequence image in which the second image frame is located, a corresponding template feature is obtained;

[0033] Based on the field of view range adopted by the second image frame and the target position corresponding to the first image frame, a search region corresponding to the second image frame is obtained, and a corresponding search feature is obtained based on the search region;

[0034] Based on the template feature and the search feature, a target tracking result corresponding to the second image frame is obtained.

[0035] In an optional embodiment of the present application, the target tracking result corresponding to the second image frame is obtained based on the template feature and the search feature, including:

[0036] If the field of view range adopted by the second image frame is the second field of view range or the third field of view range, the template feature is scaled to obtain a scaled template feature;

[0037] Based on the scaled template feature and the search feature, a target tracking result corresponding to the second image frame is obtained.

[0038] In an optional embodiment of the present application, the template feature is scaled to obtain a scaled template feature, including:

[0039] If the field of view range adopted by the second image frame is the second field of view range, the template feature is scaled down to obtain the scaled template feature, and if the field of view range of the second image frame is the third field of view range, the template feature is scaled up to obtain the scaled template feature.

[0040] In an optional embodiment of the present application, the target tracking result corresponding to the second image frame is obtained based on the scaled template feature and the search feature, including:

[0041] The scaled template feature is taken as a convolution kernel to perform a convolution operation on the search feature to obtain a corresponding feature map;

[0042] Based on the feature map, a corresponding target tracking result is obtained.

[0043] In a second aspect, the embodiments of the present application provide a target tracking device, including:

[0044] A target tracking state acquisition module is configured to acquire a field of view range adopted for target tracking of a first image frame and a corresponding target tracking state;

[0045] The field-of-view range dynamic determination module is configured to determine a field-of-view range for target tracking on the second image frame based on the field-of-view range adopted by the first image frame and the corresponding target tracking state.

[0046] The tracking module is configured to perform target tracking based on the field-of-view range adopted by the second image frame to obtain a target tracking result corresponding to the second image frame.

[0047] In a third aspect, an electronic device is provided, including a memory and a processor.

[0048] The memory stores a computer program.

[0049] The processor is configured to execute the computer program to implement the method provided in the first aspect or any optional embodiment of the first aspect.

[0050] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the method provided in the first aspect or any optional embodiment of the first aspect is implemented.

[0051] The technical solution provided by the present application has the following beneficial effects:

[0052] In the process of target tracking on the current image frame, the field-of-view range adopted for target tracking on the current image frame is dynamically adjusted according to the field-of-view range and the target tracking state of the previous image frame, thereby avoiding the cumulative error, reduced robustness and waste of computing power caused by the fixed field-of-view range in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0054] Figure 1 The basic structure of the twin network in one example of the embodiments of the present application is shown in the schematic diagram.

[0055] Figure 2 The generation process of the search area in one example of the embodiments of the present application is shown in the schematic diagram.

[0056] Figure 3a The schematic diagram of the target exceeding the field-of-view range in one example of the embodiments of the present application is shown in the schematic diagram.

[0057] Figure 3b The schematic diagram of the unnecessary expansion of the field-of-view range in one example of the embodiments of the present application is shown in the schematic diagram.

[0058] Figure 4A flowchart of a target tracking method provided by an embodiment of the present application;

[0059] Figure 5 A flowchart of a target tracking method provided by an embodiment of the present application;

[0060] Figure 6a A flowchart of a target tracking method provided by an embodiment of the present application;

[0061] Figure 6b A flowchart of a target tracking method provided by an embodiment of the present application;

[0062] Figure 6c A flowchart of a target tracking method provided by an embodiment of the present application;

[0063] Figure 7a A flowchart of a target tracking method provided by an embodiment of the present application;

[0064] Figure 7b A flowchart of a target tracking method provided by an embodiment of the present application;

[0065] Figure 8 A flowchart of a target tracking method provided by an embodiment of the present application;

[0066] Figure 9 A flowchart of a target tracking method provided by an embodiment of the present application;

[0067] Figure 10a A flowchart of a target tracking method provided by an embodiment of the present application;

[0068] Figure 10b A flowchart of a target tracking method provided by an embodiment of the present application;

[0069] Figure 10c A flowchart of a target tracking method provided by an embodiment of the present application;

[0070] Figure 10d A flowchart of a target tracking method provided by an embodiment of the present application;

[0071] Figure 10eA schematic diagram of a template feature zoom-in process in one example of the embodiment of the present application;

[0072] Figure 11a A comparison diagram of using normal-FoV and using wide-FoV in the target tracking method based on the twin network in one example of the embodiment of the present application;

[0073] Figure 11b A comparison diagram of using normal-FoV and using narrow-FoV in the target tracking method based on the twin network in one example of the embodiment of the present application;

[0074] Figure 12 An effect diagram of the target tracking method provided by the embodiment of the present application applied to a smart phone with two cameras;

[0075] Figure 13 A structural block diagram of a target tracking device provided by the embodiment of the present application;

[0076] Figure 14 A structural diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0077] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation of the present application.

[0078] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0080] In traditional object tracking technology, a user labels the form of an object in a first frame as a bounding box, which covers the object to the maximum extent and covers the background to the minimum extent. Then, in subsequent video frames, the algorithm should provide the position information of the object while providing such a bounding box. In recent years, some research problems have changed the form of object tracking labeling to the segmentation mask of the object, which finely labels the object at the pixel level and distinguishes it from the background. Thus, in subsequent video frames, the algorithm provides the position information of the target object while providing such a segmentation mask. Compared with the traditional bounding box tracking, the mask tracking has many similarities with the video object segmentation technology; on the basis of the bounding box tracking technology, the mask tracking usually adds an additional segmentation branch, which integrates the object segmentation technology, and outputs the segmentation mask of the object while outputting the center position of the object. The tracking technology provided in the embodiments of the present application is mainly described as a bounding box tracking technology, but can also be extended to a mask tracking technology.

[0081] Further, a target tracking algorithm based on a twin network can be used to convert the modeling of the target into general similarity learning. Specifically, by comparing the template of the first frame and the candidate region of the current frame, the similarity is calculated, and then the target tracking result is obtained according to the matching value. As shown in Figure 1 The basic structure of the twin network is shown in FIG. 1. The feature vectors of image block one and image block two are extracted by two networks respectively, and then the dot product value of the two feature vectors is calculated to obtain the similarity score of the two image blocks.

[0082] When the target tracking algorithm based on the twin network calculates the similarity of the template feature kernel zf and the search region feature (or search feature) xf, it is necessary to ensure that the target in the template and the target in the search region are roughly at the same scale, so that there is a corresponding meaning when calculating the convolution. Therefore, as shown in Figure 2 The target tracking method usually takes the position of the target predicted in the previous frame (i.e., the previous image frame) as the center, and takes a fixed multiple (such as 4 times) of the size of the target in the previous frame (which can be the square root of the product of the width and height of the target) as the size, and crops the current frame (i.e., the current image frame) to obtain the search region. Considering that the change of the target between two frames is within the normal physical motion range, the scale of the target in the search region obtained by the above method can be basically consistent with the scale of the target in the template.

[0083] The fixed search region size can be regarded as a fixed field of view, which can cause the following problems:

[0084] (1) Accumulation of errors: Since the search area of each frame comes from the information of the previous frame, if an error occurs, the error will gradually accumulate to the next frame, and eventually lead to the decrease of target tracking accuracy.

[0085] (2) Reduced robustness: When the error causes the real target object to gradually move out of the field of view range, since the search range is fixed, the probability of finding the real target object again is zero. As shown in Figure 3a , due to the accumulation of errors, the search area of the current frame clipped according to the target information of the previous frame, the real target has exceeded the field of view range, and the prediction of the current frame cannot be correct.

[0086] (3) Waste of computing power: When fixed multiple clipping is used, if the size exceeds the full image, the average value is usually used to fill in. When the current prediction is relatively certain (high confidence), a large range of filling wastes computing power. As shown in Figure 3b , the target is obvious in the scene, and the prediction confidence is continuously high, but since the algorithm fixes the search area, a large area of filling wastes computing power.

[0087] To solve the above problems, the embodiment of the present application provides a target tracking method, which will be described in detail below.

[0088] Figure 4 The flowchart of the target tracking method provided by the embodiment of the present application is shown in Figure 4 , which can include:

[0089] Step S401, acquiring the field of view range and the corresponding target tracking state used for target tracking of the first image frame.

[0090] Step S402, determining the field of view range used for target tracking of the second image frame based on the field of view range and the corresponding target tracking state used for the first image frame.

[0091] Wherein, the first image frame and the second image frame come from the same sequence image, the image frames in the same sequence are arranged in order, and the first image frame is the image frame before the second image frame. In order to facilitate the description of the depth estimation scheme provided by the embodiment of the present application, the current image frame is selected as the second image frame, and the previous image frame of the current image frame is selected as the first image frame. It can be understood that the present application is not limited thereto. The current image frame and the previous image frame come from the same sequence image, and the previous image frame is the previous frame of the current image frame. In the target tracking process, the target tracking result is obtained by first tracking the previous image frame.

[0092] Specifically, a field of view range used for target tracking of the previous image frame is acquired, and a corresponding target tracking state is acquired according to the acquired target tracking result. The field of view range used for target tracking of the current image frame is determined according to the field of view range used for target tracking of the previous image frame and the corresponding target tracking state.

[0093] In step S403, target tracking is performed based on the field of view range used for the second image frame, and a target tracking result corresponding to the second image frame is obtained.

[0094] Specifically, target tracking is performed on the current image frame based on the field of view range determined in the previous step, and a corresponding target tracking result is obtained. It can be understood that, before target tracking is performed on the current image frame, the field of view range corresponding to the current image frame is determined based on the field of view range and the target tracking state of the previous image frame. Obviously, the search area corresponding to the current image frame is not of a fixed size, but is determined according to the search area size of the previous image frame and the obtained target tracking state, that is, the field of view range of the current image frame is dynamically adjusted. Specifically, when the target tracking state of the previous image frame is relatively stable, the field of view range corresponding to the current image frame can be appropriately reduced to reduce the amount of calculation, and when the target tracking state of the previous image frame is target loss, the field of view range corresponding to the current image frame can be enlarged to improve the possibility of recovering the target and the robustness of target tracking. At the same time, since the field of view range corresponding to the current image frame is dynamically adjusted, the cumulative error can be reduced, and the accuracy of target tracking can be improved.

[0095] The method provided in the present application dynamically adjusts the field of view range used for target tracking of the current image frame according to the field of view range and the target tracking state of the previous image frame in the process of target tracking of the current image frame, thereby avoiding the cumulative error, reduced robustness, and waste of computing power caused by the fixed field of view range in the prior art.

[0096] In an optional embodiment of the present application, the field of view range includes at least two of a first field of view range, a second field of view range, and a third field of view range, and the field of view range used by the first image frame or the second image frame is one of the at least two, wherein the second field of view range is obtained by enlarging the first field of view range by a first multiple, and the third field of view range is obtained by reducing the first field of view range by a second multiple.

[0097] The first field of view range can be understood as a normal field of view range, and the second field of view range and the third field of view range are a wide field of view range and a narrow field of view range obtained by zooming in and zooming out on the basis of the first field of view range, wherein the first zooming factor corresponding to zooming in and the second zooming factor corresponding to zooming out can be set according to actual needs. It can be understood that the present application only provides three field of view ranges, which also correspond to three preset target tracking states of each image frame. In specific practice, two or more field of view ranges and more corresponding target tracking states can be set, which are not limited by the present application.

[0098] As shown in Figure 5 The target tracking process of the video sequence by using the target tracking method provided in the present application can include the following steps:

[0099] (1) inputting a first frame (i.e., an initial frame or an initial image frame) of a video sequence;

[0100] (2) a user performs target labeling;

[0101] (3) inputting a second frame of the video sequence, and starting target position prediction (i.e., tracking) from this frame;

[0102] (4) for the second frame of the video sequence, using a normal field of view range (i.e., normal-FoV hereinafter) to predict the target position;

[0103] (5) outputting the target tracking result of the second frame, including the target position, the prediction confidence, etc.;

[0104] (6) judging the tracking state of the second frame according to the target tracking result of the second frame;

[0105] (7) inputting a third frame, and selecting a dynamic-FoV for the third frame according to the tracking state of the second frame;

[0106] (8) performing target tracking on the third frame according to the selected dynamic-FoV, and outputting the target tracking result of the third frame; the subsequent frames are processed in the same way.

[0107] In the target tracking scheme provided in the present application, the field of view range corresponding to the current image frame is determined according to the field of view range and the target tracking state corresponding to the previous image frame, wherein the field of view range corresponding to the previous image frame has been determined in the process of tracking the target in the previous image frame. Therefore, the key to determining the field of view range corresponding to the current image frame is to obtain the target tracking state corresponding to the previous image frame, which will be described in detail below.

[0108] In an optional embodiment of the present application, the target tracking state corresponding to the previous image frame of the current image frame is obtained by:

[0109] obtaining a target tracking result corresponding to the previous image frame;

[0110] determining a target tracking state corresponding to the previous image frame based on the target tracking result corresponding to the previous image frame.

[0111] Specifically, after completing the target tracking of the previous image frame, the target tracking state corresponding to the previous image frame is evaluated according to the obtained target tracking result. Specifically, the target tracking result can include a target position (such as the center coordinates of the target), a target size (such as the width and height of the target), and a prediction confidence, wherein a bounding box corresponding to the target is determined according to the target position and the target size, and the prediction confidence can indicate the credibility of the target position and the target size.

[0112] In an optional embodiment of the present application, the tracking result includes a corresponding prediction confidence, and determining the target tracking state corresponding to the previous image frame based on the target tracking result corresponding to the previous image frame includes:

[0113] if the prediction confidence is not less than a first preset threshold, determining that the target tracking state corresponding to the previous image frame is target prediction stable (i.e. the first state);

[0114] if the prediction confidence is not greater than a second preset threshold, determining that the target tracking state corresponding to the previous image frame is target loss (i.e. the second state);

[0115] if the prediction confidence is greater than the second preset threshold and less than the first preset threshold, determining that the target tracking state corresponding to the previous image frame is between target loss and target prediction stable (i.e. the third state).

[0116] Specifically, the prediction confidence can be used to evaluate the target tracking state corresponding to the previous image frame. Specifically, first, two thresholds are preset as a first preset threshold and a second preset threshold, wherein the first preset threshold is greater than the second preset threshold; then the prediction confidence corresponding to the previous image frame is compared with the two preset thresholds, and then the corresponding target prediction state is determined according to the comparison result.

[0117] Further, if the prediction confidence is not less than the first preset threshold, the confidence of the target position and the target size in the target tracking result is high, i.e., the target tracking result is stable, and the corresponding target tracking state can be determined as target prediction stable (or target tracking stable); if the prediction confidence is not greater than the second preset threshold, the confidence of the target position and the target size in the target tracking result is low, i.e., the target tracking result is unstable, and the corresponding target tracking state can be determined as target loss; if the prediction confidence is greater than the second preset threshold and less than the first preset threshold, the confidence of the target position and the target size in the target tracking result can be accepted, i.e., the stability of the target tracking result can be accepted, and the corresponding target tracking state can be determined as between target loss and target prediction stable.

[0118] In an optional embodiment of the present application, the tracking result includes corresponding prediction confidence, target position and target size, the corresponding target tracking state is determined based on the target tracking result of the previous image frame, and includes:

[0119] Based on the target position and the target size, the target relative displacement of the previous image frame and the ratio between the target size and the global size in the previous image frame are obtained;

[0120] If the prediction confidence is not less than the first preset threshold and the target relative displacement is not greater than the preset displacement, the corresponding target tracking state is determined as target prediction stable;

[0121] If the prediction confidence is not greater than the second preset threshold and the ratio is not greater than the preset ratio, the corresponding target tracking state is determined as target loss;

[0122] If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, the corresponding target tracking state is determined as between target loss and target prediction stable.

[0123] Specifically, the target tracking state of the previous image frame can be evaluated based on the prediction confidence in combination with the target position and the target size. Specifically, first, based on the preset first preset threshold and the second preset threshold, the preset displacement (for evaluating the target relative displacement) and the preset ratio (for evaluating the ratio between the target size and the global size) are further added; then, the prediction confidence of the previous image frame is compared with the two preset thresholds, the target relative displacement of the previous image frame is compared with the preset displacement, and the ratio between the target size and the global size in the previous image frame is compared with the preset ratio, and then the corresponding target prediction state is determined according to the comparison result.

[0124] The target relative displacement corresponding to the previous image frame refers to the relative displacement of the target from the previous frame to the previous image frame. For example, the previous image frame is denoted as T-1 frame, and the previous frame of the previous image frame is denoted as T-2 frame. Then, the absolute value of the difference between the target positions corresponding to the T-1 frame and the T-2 frame is calculated, and the average value of the target sizes corresponding to the T-1 frame and the T-2 frame is calculated. Then, the ratio of the absolute value to the average value is obtained, that is, the corresponding target relative displacement is obtained. The global size corresponding to the previous image frame is the size of the previous image frame.

[0125] Further, if the prediction confidence is not less than the first preset threshold and the target relative displacement is not greater than the preset displacement, the confidence of the target position and the target size in the target tracking result is high, that is, the target tracking result is very stable, and the corresponding target tracking state can be determined as target prediction stable (or target tracking stable); if the prediction confidence is not greater than the second preset threshold and the proportion is not greater than the preset proportion, the confidence of the target position and the target size in the target tracking result is low, that is, the target tracking result is unstable, and the corresponding target tracking state can be determined as target loss; if the prediction confidence is greater than the second preset threshold and less than the first preset threshold, the confidence of the target position and the target size in the target tracking result can be accepted, that is, the stability of the target tracking result can be accepted, and the corresponding target tracking state can be determined as between target loss and target prediction stable. It can be understood that in the determination of target prediction stable and target loss, the parallel judgment conditions are added, the accuracy of the determined target tracking state is improved, and then the accuracy of subsequent target tracking can be improved.

[0126] After the target tracking state corresponding to the previous image frame is determined in the above manner, the field of view range corresponding to the current image frame can be determined in combination with the field of view range corresponding to the previous image frame. The process is described in detail below.

[0127] In an optional embodiment of the present application, based on the field of view range and the target tracking state corresponding to the previous image frame, the field of view range used for target tracking of the current image frame is determined, comprising:

[0128] When the field of view range corresponding to the previous image frame is the first field of view range, if the target tracking state is target loss, the field of view range corresponding to the current image frame is determined as the second field of view range, if the target tracking state is target prediction stable, the field of view range corresponding to the current image frame is determined as the third field of view range, and if the target tracking state is between target loss and target prediction stable, the field of view range corresponding to the current image frame is determined as the first field of view range.

[0129] When the field of view corresponding to the current image frame is the second field of view, if the target tracking state is target loss, it is determined that the field of view corresponding to the current image frame is the second field of view; if the target tracking state is target prediction stability, it is determined that the field of view corresponding to the current image frame is the first field of view; and if the target tracking state is between target loss and target prediction stability, it is determined that the field of view corresponding to the current image frame is the second field of view.

[0130] When the field of view corresponding to the current image frame is the third field of view, if the target tracking state is target loss, it is determined that the field of view corresponding to the current image frame is the first field of view; if the target tracking state is target prediction stability, it is determined that the field of view corresponding to the current image frame is the third field of view; and if the target tracking state is between target loss and target prediction stability, it is determined that the field of view corresponding to the current image frame is the third field of view.

[0131] Specifically, the current image frame is denoted as image frame T (corresponding to time T), the previous image frame is denoted as image frame T-1 (corresponding to time T-1), the first field of view is denoted as normal-Fov, the second field of view is denoted as wide-Fov, and the third field of view is denoted as narrow-Fov. Then, when the field of view corresponding to the image frame T-1 is normal-Fov, the determination process of the field of view corresponding to the image frame T is as shown in Figure 6a When the field of view corresponding to the image frame T-1 is wide-Fov, the determination process of the field of view corresponding to the image frame T is as shown in Figure 6b When the field of view corresponding to the image frame T-1 is narrow-Fov, the determination process of the field of view corresponding to the image frame T is as shown in Figure 6c .

[0132] As shown in Figure 7a , the field of view corresponding to the image frame T-1 is normal-Fov, after outputting the target position, target size and prediction confidence corresponding to time T-1 and the like, the tracking state at this time is checked: if the prediction confidence at time T-1 is less than a threshold TH_1, it is considered that the target prediction uncertainty at this time is large, the target is lost, and the next time (i.e. time T) is converted to wide-Fov; if the tracking confidence at time T-1 is greater than a threshold TH_2, it is considered that the prediction at this time is very stable, and the next time (i.e. time T) is converted to narrow-Fov. Wherein, TH_1 and TH_2 are algorithm hyperparameters (i.e. the two aforementioned preset thresholds), for example, TH_1 = 0.55, TH_2 = 0.95. If the target state is between the two (i.e. between target loss and target prediction stability), the existing FoV is kept unchanged.

[0133] As shown in Figure 7bAs shown, the field of view (FoV) corresponding to image frame T-1 is normal. After outputting the target position, target size, and prediction confidence at time T-1, the tracking status at that time is checked: If the prediction confidence at time T-1 is less than the threshold THC_1, and the ratio of the target size to the global size is less than the preset ratio THR, then the target prediction uncertainty at that time is considered large, the target is lost, and it may exist in the region after the FoV is expanded. In this case, the next time step, T, is converted to wide-view FoV. If the prediction confidence at time T-1 is greater than the threshold THC_2, and the relative displacement of the target is less than the preset displacement THD, then the prediction at that time is considered extremely stable, and it may exist in the region after the FoV is reduced. In this case, the next time step, T, is converted to narrow-view FoV. Similarly, the thresholds THC_1, THC_2, THR, and THS are algorithm hyperparameters (i.e., the aforementioned two preset thresholds, preset ratios, and preset displacements). For example, THC_1 = 0.55, THC_2 = 0.95, THS = 0.2, and THD = 0.5. If the target tracking state is between the two (i.e., between target loss and target prediction stability), then the existing FoV remains unchanged.

[0134] In one optional embodiment of this application, obtaining the target tracking result corresponding to the current image frame based on the field of view corresponding to the current image frame includes:

[0135] Based on the initial image frame in the sequence of images containing the current image frame, obtain the corresponding template features;

[0136] Based on the field of view corresponding to the current image frame and the target position corresponding to the previous image frame, the search region corresponding to the current image frame is obtained, and the corresponding search features are obtained based on the search region.

[0137] Based on template features and search features, the target tracking result corresponding to the current image frame is obtained.

[0138] Specifically, target tracking can be achieved using Siamese fully-convolutional networks based on Siamese networks. For example, ... Figure 8 As shown, the process of obtaining target tracking results based on a Siamese fully convolutional network, as provided in this application, may include:

[0139] (1) In the initial image frame, based on the target information annotated by the user, the template z region (127*127*3) is extracted;

[0140] (2) Input the template z into the backbone network to extract template features zf(6*6*128);

[0141] (3) In the current image frame, the target position corresponding to the previous image frame is taken as the center of the search region, and the field of view range determined according to the foregoing manner is taken as the size of the search region, to determine the search region x (255*255*3) corresponding to the current image frame;

[0142] (4) The search region x is input into the backbone network to extract the search feature xf corresponding to the search region;

[0143] (5) The obtained feature vectors zf and xf of different sizes are subjected to full convolution operation, that is, dot product summation operation, to obtain a similarity heat map (score map) related to the sizes of zf and xf;

[0144] (6) The position with the maximum similarity value is located on the similarity heat map, and the position in the original search region is mapped back, that is, the target position corresponding to the current image frame.

[0145] Further, the twin region recommendation network is combined into the twin network after the twin full convolution network. The region recommendation network is combined after the twin network, and k anchors are generated when the correlation of each position corresponding to the template feature zf and the search feature xf is calculated each time. The k anchors represent k candidate frames (multiple frames in the region recommendation network in the figure), and each candidate frame has the position as the center but has different size, that is, width and height. As shown in the figure, when the 4*4 template feature kernel zf is subjected to convolution operation on the 20*20 search region feature xf, a 17*17 size map is generated, and if k anchors are generated at each position, a total of 17*17*k anchors are generated. The twin region recommendation network has two branches, that is, classification and regression: the classification branch represents the predicted positive sample and negative sample probability values; and the regression branch represents the predicted center coordinates (x, y) and size (w, h) of the frame. Figure 9

[0146] In an optional embodiment of the present application, the target tracking result corresponding to the current image frame is obtained based on the template feature and the search feature, and the method comprises the following steps:

[0147] If the field of view range of the current image frame is the second field of view range or the third field of view range, the template feature is subjected to scaling processing to obtain a scaled template feature;

[0148] The target tracking result corresponding to the current image frame is obtained based on the scaled template feature and the search feature.

[0149] ​Specifically, since the template features are extracted from the initial image frame (i.e. the first frame of the sequence image) in the target tracking process, the field of view range corresponding to the initial image frame is by default normal-FoV. When the subsequent sequence image frame uses dynamic-FoV (wide-FoV or narrow-FoV), in order to fuse the feature scales extracted under different FoVs, the template features need to be scaled, that is, the application provides a template feature kernel rescaling technology, which is as follows: when the wide-FoV is r times larger than the normal-FoV, the template features are scaled to 1 / r times of the original size (from normal-FoV), that is, the template features are reduced by r times; when the narrow-FoV is 1 / r times smaller than the normal-FoV, the template features are scaled to r times of the original size (from normal-FoV), that is, the template features are enlarged by r times. It can be understood that the first multiple and the second multiple in the above scheme are the same, both of which are r.

[0150] It should be noted that the target tracking scheme based on the twin network or the twin region recommendation network provided in the embodiments of the application does not change the setting of the backbone network, that is, the size of the input image of the backbone network is fixed and does not change. The reason for such design is as follows: (1) the backbone network does not need to be retrained; (2) fixing the calculation amount of the backbone network is beneficial to the subsequent development of the software system based on the algorithm; (3) different backbone network structures can be quickly replaced without affecting the settings of other modules.

[0151] In an optional embodiment of the application, the template features are scaled to obtain scaled template features, including:

[0152] If the field of view range of the current image frame is the second field of view range, the template features are reduced by the first multiple to obtain the scaled template features, and if the field of view range of the current image frame is the third field of view range, the template features are enlarged by the second multiple to obtain the scaled template features.

[0153] Specifically, for example, as shown in Figure 10a If the current image frame uses normal FoV, the search region is centered on the target position corresponding to the previous image frame, and the image block is cropped with s times the size of the target size, and the cropped image block is scaled according to the input requirement of the backbone network, such as 255*255 size, and the search region features W*H*C are obtained after the backbone network, and the correlation calculation is performed with the template features w*h*C.

[0154] As shown in Figure 10bAs shown in FIG. 6, if the current image frame adopts a wide view-FoV, for example, the FoV is enlarged to r times (r is greater than 1, for example, r = 2) of the normal-FoV, that is, the search region is centered on the target position corresponding to the previous image frame, and the image block is cropped at s * r times the size of the target in the previous image frame, and the cropped image block is scaled to meet the input requirements of the backbone network, such as 255*255, then the target on the image is actually scaled down to 1 / r times the original size. In order to match the search feature and the template feature when calculating the correlation, the template feature needs to be scaled to the corresponding scale, that is, reduced to r times the original size.

[0155] As shown in FIG. 6, if the current frame adopts a narrow view-FoV, for example, the FoV is reduced to 1 / r times (r is greater than 1, for example, r = 2, 1 / r = 1 / 2) of the normal-FoV, that is, the search region is cropped to s * 1 / r times the size of the target in the previous image frame, and then scaled to 255*255, it can be imagined that the object on the image is actually scaled up to r times the original size. Therefore, the template feature needs to be scaled to the corresponding scale, that is, enlarged to r times the original size. Figure 10c

[0156] Further, the template feature kernel rescaling network is a single-layer or fewer-layer (such as 2 to 3 layers) neural network, which is used to scale the template feature of w*h*C to (w / r)*(h / r)*C or (rw)*(rh)*C, where w / r, h / r, rw, and rh are integers. For example, as shown in FIG. 7, when w = h = 4, r = 2, and C = 256, a convolutional neural network with a 1-layer convolution kernel of size 3*3 and a padding size of 0 can be used to scale the feature vector of 4*4*256 to 2*2*256; or a 1-layer maximum pooling layer (Max-Pooling) can also achieve the effect. As shown in FIG. 8, a feature vector of 4*4*256 can be scaled to 8*8*256 by using an up-sampling layer and a 1-layer convolution kernel of size 1*1, which can achieve the effect. Figure 10d Figure 10e

[0157] In an optional embodiment of the present application, based on the scaled template feature and the search feature, a target tracking result corresponding to the current image frame is obtained, including:

[0158] The scaled template feature is used as a convolution kernel to perform convolution operation on the search feature to obtain a corresponding feature map;

[0159] The corresponding target tracking result is obtained based on the feature map.

[0160] ​​​Specifically, after the scaled template features are used as convolution kernels to perform convolution operation on the search features, corresponding feature maps are obtained, and then corresponding target tracking results are obtained based on the feature maps. Specifically, the obtaining of the corresponding target tracking results based on the feature maps can be realized by a region recommendation algorithm.

[0161] For example, as shown in Figure 11a and 11b , it is a schematic diagram of a dynamic FoV target tracking method based on a twin network structure. The first frame template is cropped and scaled to an image block with a size of 127*127, which is input into the backbone network to obtain template features with a size of 4*4*256. The current frame is cropped and scaled to an image block with a size of 255*255, which is input into the backbone network to obtain search region features with a size of 20*20*256. If a normal-FoV tracker is used, the 4*4*256 template features are convolved on the 20*20*256 search region features to calculate the correlation, and then a region recommendation network is used to output 17*17*K (K is the number of anchor points per unit) anchor points, each of which includes a positive sample probability value (classification branch) and candidate box position information (regression branch).

[0162] As shown in Figure 11a , if a wide-FoV tracker is used, the search image is an image block obtained by scaling an image block with a search range expanded to r=2 times to 255*255, and the object is reduced to 1 / 2 times, so the template features need to be scaled to 2*2*256 through a template rescaling network, and then the correlation is calculated with the search features. Through the region recommendation network, 19*19*K anchor points are output, each of which includes a positive sample probability value (classification branch) and candidate box position information (regression branch).

[0163] As shown in Figure 11b , if a narrow-FoV tracker is used, the search image is an image block obtained by scaling an image block with a search range reduced to r=1 / 2 times to 255*255, and the object is expanded to 2 times, so the template features need to be scaled to 8*8*256 through a template rescaling network, and then the correlation is calculated with the search features. Through the region recommendation network, 12*12*K anchor points are output, each of which includes a positive sample probability value (classification branch) and candidate box position information (regression branch).

[0164] The target tracking method provided in this application is applicable not only to different search area sizes in single-camera systems but also to camera combinations with different field-of-view ranges in multi-camera systems. To meet diverse photography needs, most smartphones currently feature two or more visible light cameras. The parameters of each camera, such as aperture range and equivalent focal length, vary. The following explanation uses the target tracking function in a smartphone with dual cameras to further illustrate the beneficial effects of this application. The main camera uses an f / 1.8 lens with a 26mm equivalent focal length, while the ultra-wide-angle lens uses an f / 2.2 lens with a 13mm equivalent focal length. In the target tracking scheme provided in this application embodiment, the first and second image frames can be acquired by different image acquisition devices of the same smart device; that is, the current image frame and the previous image frame can be acquired by different cameras on the smartphone, for example, the previous image frame is acquired by the main camera, and the current image frame is acquired by the ultra-wide-angle camera.

[0165] like Figure 12 As shown in Figure a, the main camera uses normal-FoV to track the target. Because the target ship (in the left bounding box in Figure a) is close to the boundary and part of it is outside the main camera's field of view, the matching value with the template is very low, resulting in an incorrect prediction of the ship as the one in the right bounding box of Figure a. Knowing the low prediction confidence, according to the field of view determination scheme provided in this application, the next frame is converted to wide-view-FoV tracking on the ultra-wide-angle camera as shown in Figure b. In the ultra-wide-angle camera image, the field of view image patch cropped to this frame based on the predicted target position (in the left bounding box in Figure a) is shown in Figure c, and the correct predicted target ship position is output as shown in the bounding box in Figure d.

[0166] Figure 13 A structural block diagram of a target tracking device provided in an embodiment of this application is shown below. Figure 13 As shown, the device 1300 may include: a target tracking status acquisition module 1301, a field of view dynamic determination module 1302, and a tracking module 1303, wherein:

[0167] The target tracking status acquisition module 1301 is used to acquire the field of view and the corresponding target tracking status used for target tracking of the first image frame;

[0168] The field of view dynamic determination module 1302 is used to determine the field of view used for target tracking in the second image frame based on the field of view used in the first image frame and the corresponding target tracking state.

[0169] The tracking module 1303 is used to perform target tracking based on the field of view adopted by the second image frame, and obtain the target tracking result corresponding to the second image frame.

[0170] The scheme provided in the application, in the process of target tracking on the current image frame, dynamically adjusts the field of view range used for target tracking on the current image frame according to the field of view range and the target tracking state of target tracking on the previous image frame, thereby avoiding the cumulative error, reduced robustness and waste of computing power caused by the fixed field of view range in the prior art.

[0171] In an optional embodiment of the application, the first image frame and the second image frame are collected by different image collection devices of the same intelligent device.

[0172] In an optional embodiment of the application, the field of view range used by the first image frame or the second image frame is one of at least two field of view ranges.

[0173] In an optional embodiment of the application, the target tracking state acquisition module is specifically configured to:

[0174] acquire the target tracking result corresponding to the first image frame;

[0175] determine the corresponding target tracking state based on the target tracking result corresponding to the first image frame.

[0176] In an optional embodiment of the application, the target tracking state acquisition module is specifically further configured to:

[0177] determine the target tracking state corresponding to the first image frame according to the comparison result between the prediction confidence and the preset threshold.

[0178] In an optional embodiment of the application, the preset threshold includes a first preset threshold and a second preset threshold, and according to the comparison result between the prediction confidence and the preset threshold, the target tracking state acquisition module is specifically further configured to:

[0179] if the prediction confidence is not less than the first preset threshold, determine that the corresponding target tracking state is a first state;

[0180] if the prediction confidence is not greater than the second preset threshold, determine that the corresponding target tracking state is a second state;

[0181] if the prediction confidence is greater than the second preset threshold and less than the first preset threshold, determine that the corresponding target tracking state is a third state.

[0182] In an optional embodiment of the application, the tracking result includes at least one of the target position and the target size, and the prediction confidence, and the target tracking state acquisition module is specifically further configured to:

[0183] The target tracking state is determined based on at least one of a comparison result between the target relative displacement and the predetermined displacement, a comparison result between the ratio and the preset ratio, and a comparison result between the prediction confidence and the predetermined threshold.

[0184] In an optional embodiment of the present application, the target tracking state acquisition module is specifically further configured to:

[0185] If the prediction confidence is not less than the first preset threshold and the target relative displacement is not greater than the preset displacement, the corresponding target tracking state is determined as the first state.

[0186] If the prediction confidence is not greater than the second preset threshold and the ratio is not greater than the preset ratio, the corresponding target tracking state is determined as the second state.

[0187] If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, the corresponding target tracking state is determined as the third state.

[0188] In an optional embodiment of the present application, the at least two field of view ranges include a first field of view range, a second field of view range and a third field of view range, wherein the second field of view range is greater than the first field of view range, and the third field of view range is less than the first field of view range; and the field of view range dynamic determination module is specifically configured to:

[0189] When the field of view range adopted by the first image frame is the first field of view range, if the target tracking state is the second state, the field of view range adopted by the second image frame is determined as the second field of view range; if the target tracking state is the first state, the field of view range adopted by the second image frame is determined as the third field of view range; and if the target tracking state is the third state, the field of view range adopted by the second image frame is determined as the first field of view range.

[0190] When the field of view range adopted by the first image frame is the second field of view range, if the target tracking state is the second state, the field of view range adopted by the second image frame is determined as the second field of view range; if the target tracking state is the first state, the field of view range adopted by the second image frame is determined as the first field of view range; and if the target tracking state is the third state, the field of view range adopted by the second image frame is determined as the second field of view range.

[0191] When the field of view range adopted by the first image frame is the third field of view range, if the target tracking state is the second state, the field of view range adopted by the second image frame is determined as the first field of view range; if the target tracking state is the first state, the field of view range adopted by the second image frame is determined as the third field of view range; and if the target tracking state is the third state, the field of view range adopted by the second image frame is determined as the third field of view range.

[0192] In an optional embodiment of the present application, the tracking result acquisition module comprises a template feature acquisition submodule, a search feature acquisition submodule, and a tracking result acquisition submodule, wherein:

[0193] The template feature acquisition submodule is configured to acquire corresponding template features based on an initial image frame in a sequence image in which the second image frame is located.

[0194] The search feature acquisition submodule is configured to acquire a search region corresponding to the second image frame based on a field of view range adopted by the second image frame and a target position corresponding to the first image frame, and acquire corresponding search features based on the search region.

[0195] The tracking result acquisition submodule is configured to acquire a target tracking result corresponding to the second image frame based on the template features and the search features.

[0196] In an optional embodiment of the present application, the tracking result acquisition submodule is specifically configured to:

[0197] If the field of view range adopted by the second image frame is the second field of view range or the third field of view range, the template features are scaled to obtain scaled template features.

[0198] The target tracking result corresponding to the second image frame is acquired based on the scaled template features and the search features.

[0199] In an optional embodiment of the present application, the tracking result acquisition submodule is further configured to:

[0200] If the field of view range adopted by the second image frame is the second field of view range, the template features are scaled to obtain scaled template features, and if the field of view range of the second image frame is the third field of view range, the template features are scaled to obtain scaled template features.

[0201] In an optional embodiment of the present application, the tracking result acquisition submodule is further configured to:

[0202] The scaled template features are taken as a convolution kernel to perform convolution operation on the search features to obtain a corresponding feature map.

[0203] The target tracking result corresponding to the second image frame is acquired based on the feature map.

[0204] Reference will be made to the following description Figure 14 which shows an electronic device (e.g., an electronic device 100) suitable for implementing embodiments of the present application. Figure 4The diagram illustrates the structure of the terminal device or server 1400 of the method shown. The electronic devices in the embodiments of this application may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 14 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0205] The electronic device includes a memory and a processor. The memory stores a program for executing the methods described in the various method embodiments above. The processor is configured to execute the program stored in the memory. The processor may be referred to as processing device 1401 as described below. The memory may include at least one of read-only memory (ROM) 1402, random access memory (RAM) 1403, and storage device 1408 as described below, as specifically shown below:

[0206] like Figure 14 As shown, electronic device 1400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 1401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1402 or a program loaded from storage device 1408 into random access memory (RAM) 1403. RAM 1403 also stores various programs and data required for the operation of electronic device 1400. Processing device 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0207] Typically, the following devices can be connected to I / O interface 1405: input devices 1406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1409. Communication device 1409 allows electronic device 1400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 14 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0208] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication device 1409, or installed from the storage device 1408, or installed from the ROM 1402. When the computer program is executed by the processing device 1401, the above-mentioned functions defined in the methods of embodiments of the present application are executed.

[0209] It should be noted that the computer-readable storage medium described above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical fiber, an RF (radio frequency) or the like, or any suitable combination of the above.

[0210] In some embodiments, the client, server, or both can communicate using any known or later developed network protocols, such as the HyperText Transfer Protocol (HTTP), and can be interconnected with any form or medium of digital data communication (for example, a communication network) and any additional devices associated therewith.

[0211] The computer-readable medium described above can be included in the electronic device described above; alternatively, the computer-readable medium can exist as a standalone entity.

[0212] The computer-readable medium described above can be included in the electronic device described above; alternatively, the computer-readable medium can exist as a standalone entity.

[0213] Obtaining a field of view range and a corresponding target tracking state used for target tracking on the first image frame; determining a field of view range used for target tracking on the second image frame based on the field of view range and the corresponding target tracking state used for target tracking on the first image frame; and performing target tracking based on the field of view range used for target tracking on the second image frame to obtain a target tracking result corresponding to the second image frame.

[0214] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages, including object oriented programming languages such as Java, Smalltalk, C++, or conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0215] The computer program product of the first aspect of the present application can include a computer readable storage medium. The computer readable storage medium can be a tangible medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device. The computer readable storage medium can be a machine-readable signal medium or a machine-readable storage medium.

[0216] The modules or units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. In some cases, the names of the modules or units do not constitute a limitation on the modules or units themselves. For example, the target tracking state obtaining module can also be described as a "module that obtains a target tracking state".

[0217] The functions described in the above description can be performed by one or more hardware logic components. For example, non-limiting examples of hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0218] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store the program for use by or in connection with the instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the machine-readable storage medium will include one or more of an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0219] The apparatus provided in the embodiments of the present application can implement at least one of the plurality of modules through an AI model. The functions associated with AI can be performed through the non-volatile memory, the volatile memory, and the processor.

[0220] The processor can include one or more processors. At this time, the one or more processors can be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), etc., or a pure graphics processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0221] The one or more processors control the processing of the input data according to a predefined operation rule or an artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory. The predefined operation rule or the artificial intelligence model is provided through training or learning.

[0222] Here, the provision through learning means that a predefined operation rule or an AI model having a desired characteristic is obtained by applying a learning algorithm to a plurality of learning data. The learning can be performed in the apparatus itself according to the embodiments in which AI is implemented, and / or can be implemented through a separate server / system.

[0223] The AI model can include a plurality of neural network layers. Each layer has a plurality of weight values, and the calculation of one layer is performed through the calculation result of the previous layer and the plurality of weights of the current layer. Examples of the neural network include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q-network.

[0224] The learning algorithm is a method of training a predetermined target apparatus (e.g., a robot) using a plurality of learning data so as to allow or control the target apparatus to determine or predict. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0225] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific method implemented by the computer readable medium described above when executed by the electronic device can refer to the corresponding process in the foregoing method embodiments, which will not be described here again.

Claims

1. A target tracking method, characterized in that, include: The first target tracking state is determined by tracking the first image frame using the field of view associated with the first image frame. Based on the field of view associated with the first image frame and the first target tracking state, the field of view associated with the second image frame is determined. By using the field of view associated with the second image frame to perform target tracking on the second image frame, a second target tracking result is generated; The determination of the first target tracking state includes: By performing target tracking on the first image frame, a first target tracking result is generated; The first target tracking status is determined based on the first target tracking result; The first target tracking result includes at least one of target position and target size, and a prediction confidence level. Determining the first target tracking state based on the first target tracking result includes: Based on at least one of the target position and the target size, obtain at least one of the target relative displacement corresponding to the first image frame and the ratio between the target size in the first image frame and the global size of the first image frame; The first target tracking state is determined based on at least one of the comparison results between the target relative displacement and the reference displacement, and the comparison results between the ratio and the reference ratio, and the comparison results between the prediction confidence and the predetermined threshold.

2. The method according to claim 1, characterized in that, The first image frame and the second image frame were acquired by different image acquisition devices of the same electronic device.

3. The method according to claim 1, characterized in that, The field of view associated with the first image frame or the second image frame is one of at least two field of view ranges.

4. The method according to claim 1, characterized in that, The field of view associated with the first image frame and the field of view associated with the second image frame are selected from a predetermined field of view.

5. The method according to claim 1, characterized in that, Determining the first target tracking state based on the first target tracking result includes: The first target tracking state is determined based on the comparison between the predicted confidence level and the preset threshold.

6. The method according to claim 5, characterized in that, The preset threshold includes a first preset threshold and a second preset threshold. Based on the comparison between the prediction confidence and the preset threshold, the first target tracking state is determined to include at least one of the following: If the prediction confidence is not less than the first preset threshold, then the first target tracking state is determined to be the first state; If the prediction confidence is not greater than the second preset threshold, then the first target tracking state is determined to be the second state; If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, then the first target tracking state is determined to be the third state.

7. The method according to claim 1, characterized in that, Based on at least one of the comparison results between the target relative displacement and the reference displacement, and the comparison results between the ratio and the reference ratio, and the comparison results between the prediction confidence and the predetermined threshold, the first target tracking state is determined to include at least one of the following: In response to the prediction confidence being no less than a first preset threshold and the target relative displacement being no greater than a reference displacement, the first target tracking state is determined to be the first state; If the prediction confidence is not greater than a second preset threshold and the ratio is not greater than a reference ratio, then the first target tracking state is determined to be the second state. If the prediction confidence is greater than the second preset threshold and less than the first preset threshold, then the corresponding target tracking state is determined to be the third state.

8. The method according to any one of claims 1-7, characterized in that, The field of view associated with the first image frame or the second image frame is selected from a first field of view, a second field of view, and a third field of view, wherein the second field of view is larger than the first field of view, and the third field of view is smaller than the first field of view; Based on the field of view associated with the first image frame and the first target tracking state, the field of view associated with the second image frame is determined, including at least one of the following: When the field of view associated with the first image frame is the first field of view, in response to the first target tracking state being the second state, the field of view associated with the second image frame is determined to be the second field of view; in response to the first target tracking state being the first state, the field of view associated with the second image frame is determined to be the third field of view; in response to the first target tracking state being the third state, the field of view associated with the second image frame is determined to be the first field of view. When the field of view associated with the first image frame is the second field of view, in response to the first target tracking state being the second state, the field of view associated with the second image frame is determined to be the second field of view; in response to the first target tracking state being the first state, the field of view associated with the second image frame is determined to be the first field of view; in response to the first target tracking state being the third state, the field of view associated with the second image frame is determined to be the second field of view. When the field of view associated with the first image frame is the third field of view, in response to the first target tracking state being the second state, the field of view associated with the second image frame is determined to be the first field of view; in response to the first target tracking state being the first state, the field of view associated with the second image frame is determined to be the third field of view; in response to the first target tracking state being the third state, the field of view associated with the second image frame is determined to be the third field of view.

9. The method according to claim 8, characterized in that, The generation of the second target tracking result includes: Based on the initial image frame in the sequence of images containing the second image frame, obtain the corresponding template features; Based on the field of view associated with the second image frame and the position of the target in the first image frame, the search area of ​​the second image frame is determined, and search features are obtained from the search area. Based on the template features and the search features, a second target tracking result is generated.

10. The method according to claim 9, characterized in that, The step of obtaining the second target tracking result based on the template features and the search features includes: If the field of view associated with the second image frame is the second field of view or the third field of view, then the template feature is scaled to obtain the scaled template feature. Based on the scaled template features and the search features, the second target tracking result is obtained.

11. The method according to claim 10, characterized in that, The template features are scaled to obtain scaled template features, including: If the field of view associated with the second image frame is the second field of view, then the template feature is reduced to obtain the scaled template feature; if the field of view associated with the second image frame is the third field of view, then the template feature is enlarged to obtain the scaled template feature.

12. The method according to claim 10, characterized in that, The step of generating a second target tracking result based on the scaled template features and the search features includes: The scaled template features are used as convolution kernels to perform convolution operations on the search features to obtain the corresponding feature map. The second target tracking result is obtained based on the feature map.

13. An electronic device, characterized in that, Including memory and processor; The memory contains computer programs; The processor is configured to execute the computer program to implement the method of any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

Images