Target tracking method, device, equipment and storage medium

Abstract

The application discloses a target tracking method, device and equipment and a storage medium, and comprises the following steps: inputting a target template image, a search area image and natural language description into a visual semantic mutual learning network model; performing feature processing on the target template image, the search area image and the natural language description according to a linear mapping layer and an attention encoder; performing feature alignment according to a visual semantic cascaded encoder; and performing target prediction and tracking on the visual features of the search area in the aligned semantic visual pair. Since the visual semantic cascaded encoder is added in the visual semantic mutual learning network model to align the visual features and the semantic features, progressive mutual learning between the semantic features and the visual features can be performed, and the multi-modal features can be generated after alignment, so that semantic ambiguity can be eliminated and knowledge can be transferred. Target prediction is performed according to the aligned semantic visual pair, so that visual language tracking with high precision and high robustness can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target tracking, and in particular to a target tracking method, device, equipment and storage medium. BACKGROUND

[0002] In recent years, visual language tracking has received extensive attention from researchers at home and abroad. Early visual language tracking methods integrate natural language prompts into detection networks or twin networks to assist target tracking. Existing visual language tracking methods use language information to guide visual information to produce semantically relevant visual representations.

[0003] When different visual pixels correspond to the same word, the word may introduce semantic ambiguity, hindering accurate semantic visual alignment. For example, for the semantic word "ear", there is a significant visual difference between the "ear" of a cat and the "ear" of a dog. Without considering semantic visual feature alignment, these methods are difficult to achieve advanced tracking performance. The main challenge of this task is to perform visual language feature fusion in an effective manner to enhance the tracked target object. Existing target tracking methods introduce a multi-modal fusion module into the twin network to release the potential of visual language tracking, or perform multi-modal feature fusion to predict the target bounding box. However, the above methods are insufficient to produce aligned semantic visual pairs and are prone to produce ambiguous semantic representations. Semantic ambiguity hinders cross-modal interaction between mismatched semantic-visual pairs, which is not conducive to knowledge transfer for zero-shot visual language tracking. SUMMARY

[0004] The main purpose of the present application is to provide a target tracking method, device, equipment and storage medium, which aims to solve the technical problems of poor visual language tracking performance and low accuracy caused by the inability to align semantic and visual features in the prior art.

[0005] To achieve the above purpose, the present application provides a target tracking method, which comprises the following steps:

[0006] inputting a target template image, a search region image and a natural language description into a visual semantic reciprocity learning network model, wherein the visual semantic reciprocity learning network model comprises a linear mapping layer, an attention encoder, a visual semantic cascaded encoder and a target prediction head;

[0007] performing feature processing on the target template image, the search region image and the natural language description according to the linear mapping layer and the attention encoder to obtain encoded semantic features and encoded visual features;

[0008] align the encoded visual features and the encoded semantic features according to the visual-semantic cascaded encoder to obtain an aligned semantic-visual pair;

[0009] perform target prediction on visual features of a search region in the aligned semantic-visual pair according to the target prediction head to obtain a target center and a target box, and perform target tracking according to the target center and the target box.

[0010] Optionally, the linear mapping layer comprises a visual linear mapping layer and a semantic linear mapping layer, and the attention encoder comprises a visual attention encoder and a semantic attention encoder;

[0011] the feature processing of the target template image, the search region image and the natural language description according to the linear mapping layer and the attention encoder to obtain encoded visual features and encoded semantic features comprises:

[0012] mapping the target template image and the search region image into initial visual features according to the visual linear mapping layer, and mapping the natural language description into initial semantic features according to the semantic linear mapping layer;

[0013] encoding the initial visual features according to the visual attention encoder to output encoded visual features, and encoding the initial semantic features according to the semantic attention encoder to output encoded semantic features.

[0014] Optionally, the visual-semantic cascaded encoder comprises an instance-aware semantic encoder, the instance-aware semantic encoder comprises an instance-aware semantic attention module and a semantic interaction and activation module, and the aligned semantic-visual pair is an interaction-activated semantic feature;

[0015] the aligning of the encoded visual features and the encoded semantic features according to the visual-semantic cascaded encoder to obtain an aligned semantic-visual pair comprises:

[0016] performing first feature alignment on the encoded visual features and the encoded semantic features according to the instance-aware semantic attention module to generate an instance-aware semantic feature;

[0017] processing the instance-aware semantic feature according to the semantic interaction and activation module to generate an interaction-activated semantic feature.

[0018] Optionally, the instance-aware semantic attention module comprises a query mapping function, a key mapping function, a value mapping function and a softmax activation function;

[0019] The semantic attention module according to the instance perception performs first feature alignment on the encoded visual features and the encoded semantic features, and generates instance-perceived semantic features, including:

[0020] The semantic features are query-mapped according to the query mapping function, and query mapping results are obtained;

[0021] The visual features are key-mapped and value-mapped according to the key mapping function and the value mapping function, respectively, and key mapping results and value mapping results are obtained;

[0022] The query mapping results, the key mapping results, and the value mapping results are aggregated according to the softmax activation function, and instance-perceived semantic features are generated.

[0023] Optionally, the semantic interaction and activation module includes a global average pooling function, a GELU activation function, a sigmoid activation function, and a multi-layer perceptron function;

[0024] The instance-perceived semantic features are processed according to the semantic interaction and activation module, and interaction-activated semantic features are generated, including:

[0025] The instance-perceived semantic features are activated according to the global average pooling function, the GELU activation function, and the sigmoid activation function, and activated semantic features are obtained;

[0026] The activated semantic features are point-multiplied with the instance-perceived semantic features, and the result of the point multiplication is summed with the instance-perceived semantic features, and semantic features after semantic interaction are obtained;

[0027] The semantic features after semantic interaction are twice activated according to the multi-layer perceptron function, and interaction-activated semantic features are obtained.

[0028] Optionally, the visual semantic cascading encoder includes a semantic-perceived instance decoder, the semantic-perceived instance decoder includes a semantic-perceived instance attention module and a visual interaction and activation module, and the aligned semantic visual pair is interaction-activated visual features;

[0029] After the semantic features after semantic interaction are twice activated according to the multi-layer perceptron function, and interaction-activated semantic features are obtained, the method further includes:

[0030] The encoded visual features and the interaction-activated semantic features are secondly aligned according to the semantic-perceived instance attention module, and semantic-perceived instance features are generated;

[0031] According to the visual interaction and activation module, the instance features of the semantic perception are processed, and visual features after interaction activation are generated.

[0032] Optionally, the target prediction head is used to perform target prediction on the visual features of the search region in the aligned semantic visual pair, target centers and target boxes are obtained, and target tracking is performed according to the target centers and the target boxes, including:

[0033] The target prediction head is used to perform target prediction on the visual features after interaction activation, and a classification score map, an offset coordinate map and a regression scale map are obtained;

[0034] The position with the highest score in the classification score map is determined as a target position, and target centers and target boxes are predicted in the target position according to offset information in the offset coordinate map and scale information in the regression scale map;

[0035] Target tracking is performed according to the target centers and the target boxes.

[0036] In addition, to achieve the above-mentioned purpose, the application further provides a target tracking device, which comprises:

[0037] A data input module is configured to input a target template image, a search region image and a natural language description into a visual semantic reciprocal learning network model, wherein the visual semantic reciprocal learning network model comprises a linear mapping layer, an attention encoder, a visual semantic cascaded encoder and a target prediction head.

[0038] A feature processing module is configured to perform feature processing on the target template image, the search region image and the natural language description according to the linear mapping layer and the attention encoder, and obtain encoded semantic features and encoded visual features.

[0039] A feature alignment module is configured to perform feature alignment on the encoded visual features and the encoded semantic features according to the visual semantic cascaded encoder, and obtain an aligned semantic visual pair.

[0040] A target prediction module is configured to perform target prediction on the visual features of the search region in the aligned semantic visual pair according to the target prediction head, obtain target centers and target boxes, and perform target tracking according to the target centers and the target boxes.

[0041] In addition, to achieve the above-mentioned purpose, the application further provides a target tracking device, which comprises a memory, a processor and a target tracking program stored in the memory and executable on the processor, wherein the target tracking program is configured to implement the steps of the target tracking method as described above.

[0042] In addition, to achieve the above object, the application further provides a storage medium, which stores a target tracking program, and the target tracking program realizes the steps of the target tracking method when executed by a processor.

[0043] In the application, the target template image, the search region image and the natural language description are input into the visual semantic mutual learning network model, the target template image, the search region image and the natural language description are processed by the linear mapping layer and the attention encoder, the feature alignment is performed by the visual semantic cascade encoder, the visual feature of the search region in the aligned semantic visual pair is predicted, and the target tracking is performed. Since the visual semantic cascade encoder is added in the visual semantic mutual learning network model to perform the feature alignment of the visual feature and the semantic feature, the progressive mutual learning between the semantic feature and the visual feature can be performed, the multi-modal feature with perfect alignment is generated, so that the semantic ambiguity can be eliminated and the knowledge can be transferred. The target is predicted according to the aligned semantic visual pair, so that the visual language tracking with high precision and high robustness can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a structural schematic diagram of a target tracking device related to a hardware running environment of an embodiment of the application;

[0045] Figure 2 is a flowchart of a first embodiment of the target tracking method of the application;

[0046] Figure 3 is a flowchart of a second embodiment of the target tracking method of the application;

[0047] Figure 4 is a flowchart of a third embodiment of the target tracking method of the application;

[0048] Figure 5 is a structural block diagram of a first embodiment of the target tracking device of the application.

[0049] The realization of the object, the functional characteristics and the advantages of the application will be further explained by combining the embodiments with the drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0051] Reference Figure 1 , Figure 1 is a structural schematic diagram of a target tracking device related to a hardware running environment of an embodiment of the application.

[0052] As Figure 1As shown, the target tracking device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is configured to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can further include a standard wired interface and a wireless interface. The network interface 1004 can optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0053] Those skilled in the art can understand that the structure shown in the above embodiment is not a limitation on the target tracking device, and the target tracking device can include more or fewer components than those shown in the figure, or some components can be combined, or different components can be arranged. Figure 1

[0054] As shown in the above embodiment, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a target tracking program. Figure 1

[0055] In the target tracking device shown in the above embodiment, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the target tracking device of the present application can be arranged in the target tracking device, and the target tracking device calls the target tracking program stored in the memory 1005 through the processor 1001, and executes the target tracking method provided by the embodiment of the present application. Figure 1 The embodiment of the present application provides a target tracking method, which will be described in detail with reference to the above embodiment.

[0056] Figure 2 Figure 2 The flowchart of the first embodiment of the target tracking method of the present application is shown in the above embodiment.

[0057] In the embodiment, the target tracking method includes the following steps:

[0058] ​​​​Step S10: input the target template image, the search region image, and the natural language description into a visual semantic reciprocal learning network model, the visual semantic reciprocal learning network model comprising a linear mapping layer, an attention encoder, a visual semantic cascaded encoder, and a target prediction head.

[0059] It should be noted that the execution subject of the method of the present embodiment can be a computing service device with target tracking, network communication, and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc.; or a target tracking device with the same or similar functions. The present embodiment and the following embodiments will be described by taking the target tracking device as an example.

[0060] It can be understood that the visual semantic reciprocal learning network (PSVRL) model proposed in the present embodiment can include a linear mapping layer, an attention encoder, a visual semantic cascaded encoder, and a target prediction head, etc. The target template image, the search region image, and the natural language description are input into the visual semantic reciprocal learning network model, and the target frame information of the search region image can be output, which is used for target tracking.

[0061] Step S20: perform feature processing on the target template image, the search region image, and the natural language description according to the linear mapping layer and the attention encoder, to obtain encoded semantic features and encoded visual features.

[0062] It should be understood that the target template image and the search region image can be sliced and spliced, and then the spliced slices are mapped into initial visual features by using a visual linear mapping layer; at the same time, the natural language description is mapped into initial semantic features by using a semantic linear mapping layer. Then in the encoding stage, the initial visual features are encoded by using a visual attention encoder to output encoded visual features; at the same time, the initial semantic features are encoded by using a semantic attention encoder to output encoded semantic features.

[0063] Further, in order to more accurately detect the target, the initial visual features and the initial semantic features can be mapped through a linear mapping layer during feature processing, and then encoded through an attention encoder to obtain encoded visual features and encoded semantic features, so as to facilitate subsequent better feature alignment processing. Therefore, the linear mapping layer includes a visual linear mapping layer and a semantic linear mapping layer, and the attention encoder includes a visual attention encoder and a semantic attention encoder; the step S20 includes: mapping the target template image and the search area image into initial visual features according to the visual linear mapping layer, and mapping the natural language description into initial semantic features according to the semantic linear mapping layer; encoding the initial visual features to output encoded visual features according to the visual attention encoder, and encoding the initial semantic features to output encoded semantic features according to the semantic attention encoder.

[0064] It can be understood that before encoding, the semantic linear mapping layer and the visual linear mapping layer are specifically implemented as follows: the PSVRL model adopts a semantic linear mapping layer P l The natural language description is encoded into initial semantic features S0∈R C×T ; the PSVRL model adopts a visual linear mapping layer P v The spliced target template and search area visual slices are encoded into initial visual features I0∈R C×N .

[0065] It should be understood that in the encoding stage, the PSVRL model adopts a pre-trained semantic attention encoder (BERT) to encode the initial semantic features S0, to generate encoded semantic features S∈R C×T , where C and T are the number of channels and words, respectively. At the same time, the PSVRL model adopts a pre-trained visual attention encoder (ViT) to encode the initial visual features I0, to generate encoded visual features I∈R C×N , where C and N are the number of channels and slices, respectively. Then, the PSVRL model adopts a visual-semantic cascaded encoder Θ to perform reciprocal learning on the encoded visual-semantic pair (S, I) composed of the instance-aware semantic encoder Θ enc and the semantic-aware instance decoder Θ dec , to obtain an aligned semantic-visual pair (S″′, I″′).

[0066] Step S30: performing feature alignment on the encoded visual features and the encoded semantic features according to the visual-semantic cascaded encoder, to obtain an aligned semantic-visual pair.

[0067] It should be understood that the encoded visual features and the encoded semantic features can be learned reciprocally by a visual-semantic cascaded encoder, which is composed of a semantic encoder aware of instances and an instance decoder aware of semantics. The instance-aware semantic encoder can be used to perform a first alignment of the encoded visual features and the encoded semantic features, and the semantic-aware instance decoder can be used to perform a second alignment of the interactively activated semantic features and the encoded visual features, so as to obtain a well-aligned semantic-visual pair.

[0068] Step S40: performing target prediction on the visual features of the search region in the aligned semantic-visual pair according to the target prediction head, obtaining a target center and a target box, and performing target tracking according to the target center and the target box.

[0069] It is easy to understand that after encoding, the visual features of the search region in the aligned semantic-visual pair corresponding to the search region image are predicted for target center and bounding box by using the target prediction head, and target tracking is performed according to the target center and the target box.

[0070] In this embodiment, the target template image, the search region image, and the natural language description are input into the visual-semantic reciprocal learning network model, the target template image, the search region image, and the natural language description are processed for features according to a linear mapping layer and an attention encoder, the features are aligned according to a visual-semantic cascaded encoder, the visual features of the search region in the aligned semantic-visual pair are predicted for target, and target tracking is performed. Since the visual-semantic cascaded encoder is added in the visual-semantic reciprocal learning network model to align the visual features and the semantic features in this embodiment, progressive reciprocal learning between the semantic features and the visual features can be performed, and well-aligned multi-modal features can be generated, so as to eliminate semantic ambiguity and transfer knowledge. The target is predicted from the aligned semantic-visual pair, so as to realize high-precision and high-robustness visual language tracking.

[0071] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the target tracking method of the present application is shown.

[0072] Further, through the first alignment of the visual features and the semantic features, the visual features related to the semantic features can be effectively encoded in the semantic features and the semantic features related to a specific target can be effectively activated. Therefore, based on the first embodiment, in this embodiment, the visual-semantic cascaded encoder includes an instance-aware semantic encoder, the instance-aware semantic encoder includes an instance-aware semantic attention module and a semantic interaction and activation module, and the aligned semantic-visual pair is an interactively activated semantic feature; the step S30 includes:

[0073] Step S301: performing first feature alignment on the encoded visual feature and the encoded semantic feature according to the instance-aware semantic attention module to generate instance-aware semantic feature.

[0074] It should be noted that the instance-aware semantic encoder (IASE) includes an instance-aware semantic attention module and a semantic interaction and activation module. The instance-aware semantic encoder (IASE) performs first feature alignment on the encoded visual feature I∈R C×N and the encoded semantic feature S∈R C×T by using the instance-aware semantic attention module.

[0075] Further, in order to generate semantic features more discriminative than original semantic features, the instance-aware semantic attention module includes a query mapping function, a key mapping function, a value mapping function and a softmax activation function; and the step S301 includes: performing query mapping on the encoded semantic feature according to the query mapping function to obtain a query mapping result; performing key mapping and value mapping on the encoded visual feature according to the key mapping function and the value mapping function respectively to obtain a key mapping result and a value mapping result; and performing aggregation on the query mapping result, the key mapping result and the value mapping result according to the softmax activation function to generate instance-aware semantic feature.

[0076] It should be noted that the first feature alignment can be performed by the following formulas (1) and (2):

[0077] S q =q(S),I k =k(I),I v =v(I), (1)

[0078]

[0079] Wherein, q is the query mapping function of S. k and v are the key mapping function and the value mapping function of I respectively. softmax represents the softmax activation function. Under the guidance of instance features, the generated instance-aware semantic feature S' is more discriminative than the original semantic feature S.

[0080] Step S302: processing the instance-aware semantic feature according to the semantic interaction and activation module to generate interaction-activated semantic feature.

[0081] Further, since the semantic features of different words are interdependent, the instance-aware semantic encoder (IASE) adopts a semantic word interaction and activation module for semantic interaction, which can compress relevant words and disperse irrelevant words. After semantic interaction, a multi-layer perceptron can further activate important feature channels and remove redundant feature channels. Therefore, the semantic interaction and activation module includes a global average pooling function, a GELU activation function, a sigmoid activation function, and a multi-layer perceptron function; the step S301 further includes: activating the instance-aware semantic features according to the global average pooling function, the GELU activation function, and the sigmoid activation function to obtain activated semantic features; performing a dot product operation on the activated semantic features and the instance-aware semantic features, and performing a summation operation on the result of the dot product operation and the instance-aware semantic features to obtain semantic features after semantic interaction; and performing secondary activation on the semantic features after semantic interaction according to the multi-layer perceptron function to obtain semantic features after interaction activation.

[0082] It should be noted that the instance-aware semantic encoder (IASE) can adopt a semantic word interaction and activation module for semantic interaction to compress relevant words and disperse irrelevant words. The specific steps are as shown in the following formulas (3) and (4):

[0083]

[0084] S"=f gc (S′)·S′+S′ (4)

[0085] where · denotes a dot product operation by word, represents a global average pooling function, σ(·) represents a GELU activation function, sigmoid represents a sigmoid activation function, W p1 and W p2 respectively represent parameters of two fully connected layers. After semantic interaction, a multi-layer perceptron is used to further activate important feature channels and remove redundant feature channels, as shown in the following formula (5):

[0086] S″′=mlp(S″)+S″+S′, (5)

[0087] where mlp(·) represents a multi-layer perceptron function of S″.

[0088] In the embodiment, the semantic attention module according to instance perception is used to perform first feature alignment on the coded visual feature and the coded semantic feature, to generate instance-perception semantic feature; and the semantic interaction and activation module is used to process the instance-perception semantic feature, to generate interactive-activated semantic feature. Since the visual feature and the semantic feature are aligned for the first time by the instance-perception semantic attention module, the visual feature related to the semantic feature can be effectively coded in the semantic feature, and the semantic feature channel related to a specific target can be effectively activated.

[0089] Reference Figure 4 , Figure 4 The flowchart of the third embodiment of the object tracking method is shown.

[0090] Further, through the second alignment of the visual feature and the semantic feature, the semantic feature related to the visual feature can be effectively coded in the visual feature, and the visual feature slice related to a specific target can be effectively activated.

[0091] Therefore, based on the first embodiment, in the embodiment, the visual semantic cascaded encoder comprises a semantic-perception instance decoder, the semantic-perception instance decoder comprises a semantic-perception instance attention module and a visual interaction and activation module, and the aligned semantic visual pair is interactive-activated visual feature; the step S30 further comprises:

[0092] Step S303: performing second feature alignment on the coded visual feature and the interactive-activated semantic feature according to the semantic-perception instance attention module, to generate semantic-perception instance feature.

[0093] It is worth noting that the visual semantic cascaded encoder further comprises a semantic-perception instance decoder, and the semantic-perception instance decoder (SAID) comprises a semantic-perception instance attention module and a visual block interaction and activation module. First, the semantic-perception instance decoder (SAID) uses the semantic-perception instance attention module to further align the coded visual feature I and the aligned semantic feature S'''. The specific alignment steps are shown in the following formulas (6) and (7):

[0094] I q = q'(I), S k '''= k'(S'''), S v '''= v'(S'''), (6)

[0095]

[0096] Here, q′ is the query mapping function of I. k′ and v′ are the key mapping function and value mapping function of S″′, respectively. softmax represents the softmax activation function. By employing semantically aware instance attention, SAID can perform more accurate semantic visual alignment and aggregate aligned semantic information in I′.

[0097] Step S304: Process the instance features of the semantic perception according to the visual interaction and activation module to generate the visual features after interaction activation.

[0098] It should be noted that, since the detailed information between different visual blocks is beneficial for fine-grained target tracking, the Semantic Aware Instance Decoder (SAID) uses visual block interaction and activation module to perform visual block interaction, and enhances the correlation between visual blocks by using inverse residuals and linear mapping. The specific steps are shown in the following formula (8):

[0099] I e ′=f e (I′)=σ((I′) T ·W e ),

[0100] I s ′=f s (I e ′)=σ(I e ′·W s ),

[0101] I n ′=f n (I s ′)=(I s ′·W n ) T (8)

[0102] Among them, f e (·) indicates an extended linear function implemented by the FC layer (with parameter W). e ), f s (·) indicates a selection function implemented by the FC layer (with parameter W). s ), f n (·) is a compressed linear function implemented by the FC layer (with parameter W). n Subsequently, the Semantic Aware Instance Decoder (SAID) uses a multilayer perceptron with residual connections to activate fine features in the visual blocks as shown in equations (9) and (10):

[0103] I″=I n ′+I′, (9)

[0104] I″′=mlp(I″)+I″, (10)

[0105] where mlp(·) denotes a multi-layer perceptron function of I".

[0106] Further, in order to more accurately detect the target, the visual features after interaction activation can be used for target prediction, and then the target position is determined according to the classification score map, and then the target center and the target frame are predicted in the target position according to the offset information and the scale information, and then the target tracking is performed according to the target center and the target frame. Therefore, the step S40 comprises: performing target prediction on the visual features after interaction activation according to the target prediction head, to obtain a classification score map, an offset coordinate map and a regression scale map; determining the position with the highest score in the classification score map as the target position, and predicting the target center and the target frame in the target position according to the offset information in the offset coordinate map and the scale information in the regression scale map; and performing target tracking according to the target center and the target frame.

[0107] It can be understood that after the encoding stage, the target prediction head is specifically implemented as follows: the search region features output by the visual semantic cascaded encoder are input into a fully convolutional network (FCN) for target center and bounding box prediction. The FCN is composed of 5 stacked Conv-BN-ReLU layers, which outputs three branches to predict the classification score map, the offset coordinate map and the regression scale map respectively. The position with the highest score in the classification score map is regarded as the target position, and the offset coordinates and the regression scale corresponding to the position are used to predict the target center and the target frame.

[0108] In the training stage, the focal loss weighted by the Gaussian kernel is used for target center classification; the L1 loss and the GIoU loss are used for target frame regression. The total loss of the prediction head is composed of the focal loss L cls of the target center classification, the GIoU loss L iou of the target frame coordinate regression and the loss L1, as shown in the following formula (11):

[0109]

[0110] where μ iou and are regularization parameters, which are set to 2 and 5 respectively.

[0111] In this embodiment, the second feature alignment of the encoded visual features and the interaction-activated semantic features is performed by the semantic-aware instance attention module, to generate semantic-aware instance features; and the semantic-aware instance features are processed by the visual interaction and activation module, to generate interaction-activated visual features. Due to the second alignment of the visual features and the semantic features in this embodiment, the semantic features related to the visual features can be effectively encoded in the visual features, and the visual feature slices related to a specific target can be effectively activated.

[0112] In addition, the embodiment of the present application further provides a storage medium, wherein the storage medium stores a target tracking program, and the target tracking program is executed by a processor to realize the steps of the target tracking method as described above.

[0113] Reference Figure 5 , Figure 5 is a structural block diagram of a first embodiment of the target tracking device of the present application.

[0114] As Figure 5 shown, the target tracking device provided by the embodiment of the present application comprises:

[0115] A data input module 501 is configured to input a target template image, a search area image and a natural language description into a visual semantic mutual learning network model, wherein the visual semantic mutual learning network model comprises a linear mapping layer, an attention encoder, a visual semantic cascaded encoder and a target prediction head.

[0116] A feature processing module 502 is configured to perform feature processing on the target template image, the search area image and the natural language description according to the linear mapping layer and the attention encoder, to obtain encoded semantic features and encoded visual features.

[0117] A feature alignment module 503 is configured to perform feature alignment on the encoded visual features and the encoded semantic features according to the visual semantic cascaded encoder, to obtain an aligned semantic visual pair.

[0118] A target prediction module 504 is configured to perform target prediction on the visual features of the search area in the aligned semantic visual pair according to the target prediction head, to obtain a target center and a target frame, and perform target tracking according to the target center and the target frame.

[0119] The embodiment inputs a target template image, a search region image and a natural language description into a visual semantic mutual learning network model, performs feature processing on the target template image, the search region image and the natural language description according to a linear mapping layer and an attention encoder, performs feature alignment according to a visual semantic cascaded encoder, performs target prediction on the visual features of the search region in the aligned semantic visual pair, and performs target tracking. Since the visual semantic cascaded encoder is added in the visual semantic mutual learning network model to perform feature alignment on the visual features and the semantic features, progressive mutual learning between the semantic features and the visual features can be performed, and well-aligned multi-modal features can be generated, so that semantic ambiguity can be eliminated and knowledge can be transferred. Target prediction is performed on the aligned semantic visual pair, so that visual language tracking with high precision and high robustness can be realized.

[0120] Based on the first embodiment of the target tracking device, a second embodiment of the target tracking device is provided.

[0121] In the embodiment, the feature processing module 502 is further configured to map the target template image and the search region image into initial visual features according to the visual linear mapping layer, and map the natural language description into initial semantic features according to the semantic linear mapping layer; encode the initial visual features according to the visual attention encoder to output encoded visual features, and encode the initial semantic features according to the semantic attention encoder to output encoded semantic features.

[0122] As an implementation manner, the feature alignment module 503 is further configured to perform first feature alignment on the encoded visual features and the encoded semantic features according to the instance-aware semantic attention module to generate instance-aware semantic features; and process the instance-aware semantic features according to the semantic interaction and activation module to generate interactively activated semantic features.

[0123] As an implementation manner, the feature alignment module 503 is further configured to perform query mapping on the encoded semantic features according to the query mapping function to obtain a query mapping result; perform key mapping and value mapping on the encoded visual features according to the key mapping function and the value mapping function respectively to obtain a key mapping result and a value mapping result; and aggregate the query mapping result, the key mapping result and the value mapping result according to the softmax activation function to generate instance-aware semantic features.

[0124] As an implementation form, the feature alignment module 503 is further configured to perform activation on the instance-aware semantic feature according to the global average pooling function, the GELU activation function and the sigmoid activation function, to obtain an activated semantic feature; perform point multiplication operation on the activated semantic feature and the instance-aware semantic feature, and perform summation operation on the result of the point multiplication operation and the instance-aware semantic feature, to obtain a semantic-interacted semantic feature; and perform secondary activation on the semantic-interacted semantic feature according to the multi-layer perceptron function, to obtain an interaction-activated semantic feature.

[0125] As an implementation form, the feature alignment module 503 is further configured to perform second feature alignment on the encoded visual feature and the interaction-activated semantic feature according to the semantic-aware instance attention module, to generate a semantic-aware instance feature; and perform processing on the semantic-aware instance feature according to the visual interaction and activation module, to generate an interaction-activated visual feature.

[0126] As an implementation form, the target prediction module 504 is further configured to perform target prediction on the interaction-activated visual feature according to the target prediction head, to obtain a classification score map, an offset coordinate map and a regression scale map; determine a position with the highest score in the classification score map as a target position, and predict a target center and a target frame in the target position according to offset information in the offset coordinate map and scale information in the regression scale map; and perform target tracking according to the target center and the target frame.

[0127] Other embodiments or specific implementations of the target tracking device of the present application can refer to the above-mentioned method embodiments, which will not be described here.

[0128] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0129] The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0130] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be through hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art contribution can be embodied in the form of software products, the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a number of instructions to make a terminal device (may be a mobile phone, computer, server, air conditioner, or network equipment, etc.) executes the method described in various embodiments of the present application.

[0131] The above is only the preferred embodiment of the present application, not therefore limit the patent scope of the present application, any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target tracking method, characterized in that, The target tracking method includes: The target template image, the search region image, and the natural language description are input into the visual semantic reciprocal learning network model, which includes a linear mapping layer, an attention encoder, a visual semantic cascade encoder, and a target prediction head. Based on the linear mapping layer and the attention encoder, feature processing is performed on the target template image, the search region image, and the natural language description to obtain encoded semantic features and encoded visual features. The encoded visual features and the encoded semantic features are aligned according to the visual-semantic concatenated encoder to obtain an aligned semantic-visual pair. Based on the visual features of the aligned semantic visual centering search region, the target prediction head is used to predict the target center and the target bounding box, and the target is tracked based on the target center and the target bounding box. The visual semantic cascade encoder includes an instance-aware semantic encoder, which includes an instance-aware semantic attention module and a semantic interaction and activation module. The aligned semantic visual pair is the semantic feature after interaction activation. The step of aligning the encoded visual features and the encoded semantic features according to the visual-semantic concatenated encoder to obtain aligned semantic-visual pairs includes: The instance-aware semantic attention module performs a first feature alignment on the encoded visual features and the encoded semantic features to generate instance-aware semantic features. The semantic interaction and activation module processes the semantic features perceived by the instance to generate semantic features after interaction activation.

2. The target tracking method as described in claim 1, characterized in that, The linear mapping layer includes a visual linear mapping layer and a semantic linear mapping layer, and the attention encoder includes a visual attention encoder and a semantic attention encoder; The step of performing feature processing on the target template image, the search region image, and the natural language description based on the linear mapping layer and the attention encoder to obtain encoded semantic features and encoded visual features includes: The target template image and the search region image are mapped to initial visual features according to the visual linear mapping layer, and the natural language description is mapped to initial semantic features according to the semantic linear mapping layer. The initial visual features are encoded according to the visual attention encoder to output encoded visual features, and the initial semantic features are encoded according to the semantic attention encoder to output encoded semantic features.

3. The target tracking method as described in claim 1, characterized in that, The instance-aware semantic attention module includes a query mapping function, a key mapping function, a value mapping function, and a softmax activation function; The step of performing a first feature alignment between the encoded visual features and the encoded semantic features based on the instance-aware semantic attention module to generate instance-aware semantic features includes: The encoded semantic features are queried and mapped according to the query mapping function to obtain the query mapping result; The encoded visual features are key-mapped and value-mapped according to the key mapping function and the value mapping function, respectively, to obtain key mapping results and value mapping results; The query mapping result, the key mapping result, and the value mapping result are aggregated based on the softmax activation function to generate instance-aware semantic features.

4. The target tracking method as described in claim 3, characterized in that, The semantic interaction and activation module includes a global average pooling function, a GELU activation function, a sigmoid activation function, and a multilayer perceptron function; The step of processing the semantic features perceived by the instance based on the semantic interaction and activation module to generate semantic features after interaction activation includes: The instance-aware semantic features are activated using the global average pooling function, the GELU activation function, and the sigmoid activation function to obtain the activated semantic features. The activated semantic features are multiplied by the instance-aware semantic features, and the result of the multiplication is summed with the instance-aware semantic features to obtain the semantic features after semantic interaction. The semantic features after semantic interaction are reactivated based on the multilayer perceptron function to obtain the semantic features after interaction activation.

5. The target tracking method as described in claim 4, characterized in that, The visual semantic cascade encoder includes a semantically aware instance decoder, which includes a semantically aware instance attention module and a visual interaction and activation module. The aligned semantic visual pair is the visual feature after interaction activation. After performing secondary activation on the semantic features after semantic interaction based on the multilayer perceptron function to obtain the semantic features after interaction activation, the method further includes: The semantically aware instance attention module performs a second feature alignment on the encoded visual features and the semantically activated features after interaction to generate semantically aware instance features. The visual interaction and activation module processes the instance features of the semantic perception to generate visual features after interaction activation.

6. The target tracking method as described in claim 5, characterized in that, The step of predicting the target based on the visual features of the aligned semantic visual centering search region using the target prediction head, obtaining the target center and the target bounding box, and performing target tracking based on the target center and the target bounding box includes: Based on the target prediction head, target prediction is performed on the visual features after the interaction is activated to obtain a classification score map, an offset coordinate map, and a regression scale map. The position with the highest score in the classification score map is determined as the target position, and the target center and target box are predicted at the target position based on the offset information in the offset coordinate map and the scale information in the regression scale map. Target tracking is performed based on the target center and the target bounding box.

7. A target tracking device, characterized in that, The target tracking device includes: The data input module is used to input the target template image, the search region image, and the natural language description into the visual semantic reciprocal learning network model, which includes a linear mapping layer, an attention encoder, a visual semantic cascade encoder, and a target prediction head. The feature processing module is used to perform feature processing on the target template image, the search region image and the natural language description based on the linear mapping layer and the attention encoder to obtain encoded semantic features and encoded visual features; The feature alignment module is used to align the encoded visual features and the encoded semantic features according to the visual semantic cascade encoder to obtain aligned semantic visual pairs. The visual semantic cascade encoder includes an instance-aware semantic encoder, which includes an instance-aware semantic attention module and a semantic interaction and activation module. The aligned semantic visual pairs are semantic features after interaction activation. The target prediction module is used to predict the target based on the visual features of the aligned semantic visual centering search area according to the target prediction head, obtain the target center and the target bounding box, and perform target tracking based on the target center and the target bounding box; The feature alignment module is further configured to perform a first feature alignment on the encoded visual features and the encoded semantic features according to the instance-aware semantic attention module to generate instance-aware semantic features; and to process the instance-aware semantic features according to the semantic interaction and activation module to generate interactively activated semantic features.

8. A target tracking device, characterized in that, The device includes: a memory, a processor, and a target tracking program stored in the memory and executable on the processor, the target tracking program being configured to implement the steps of the target tracking method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a target tracking program, which, when executed by a processor, implements the steps of the target tracking method as described in any one of claims 1 to 6.

Images