Target tracking method, device, equipment and storage medium

Abstract

The application discloses a target tracking method and device, equipment and a storage medium, and the method comprises the following steps: acquiring initial language information and region information in a sampling video, wherein the region information comprises an initial target template and current search region information; inputting the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and boundary box information of the search region, wherein the progressive joint visual language attention network model comprises a linear mapping layer, a joint visual language coding layer and a target prediction head, and each joint visual language coding layer comprises an attention coding layer, a visual language alignment layer and a visual language correction layer. Since each joint visual language coding layer in the progressive joint visual language attention network model comprises the attention coding layer, the visual language alignment layer and the visual language correction layer, progressive joint coding of language features and visual features is realized, and the accuracy and robustness of visual language tracking are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a target tracking method, device and equipment and storage medium. BACKGROUND

[0002] In recent years, visual language tracking has been widely concerned by researchers at home and abroad, and has become a research hotspot in the field of target tracking. For example, visual language tracking is performed through a language tracking paradigm; visual language tracking is performed by introducing natural language information into a framework for tracking through detection; and language visual tracking is performed by introducing natural language information into a twin network through a twin natural language region proposal network. However, these early visual language tracking methods regard visual and language as independent clues before the final fusion stage, and it is difficult to achieve good tracking performance.

[0003] Although subsequent methods learn a unified visual language representation by embedding a modal mixing module into a convolutional neural network, align the visual and language modalities, and achieve good tracking performance. However, it is difficult to fully utilize the multi-level design of the self-attention network to perform progressive feature alignment at various levels (e.g., pixels, semantics, and categories), and the tracking performance needs to be further improved. In the architecture of the attention network, multi-source and multi-modal information interaction is performed, and visual target positioning and tracking are simultaneously performed in a unified framework. Although good tracking performance is achieved, the method is difficult to effectively utilize the encoding layer of the self-attention network to mine rich visual language context information, and the tracking performance needs to be further improved.

[0004] Therefore, there is an urgent need for a target tracking method that can effectively improve the accuracy and robustness of visual language tracking. SUMMARY

[0005] The main purpose of the present application is to provide a target tracking method, device and equipment and storage medium, which aims to solve the technical problems of low accuracy and robustness of visual language tracking in the prior art.

[0006] To achieve the above-mentioned purpose, the present application provides a target tracking method, which comprises the following steps:

[0007] Obtaining initial language information and region information in a sampled video, the region information comprising initial target template and current search region information;

[0008] input the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of a search region, the progressive joint visual language attention network model comprising a linear mapping layer, joint visual language encoding layers, and a target prediction head, each joint visual language encoding layer comprising an attention encoding layer, a visual language alignment layer, and a visual language correction layer.

[0009] Optionally, the step of inputting the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of a search region comprises:

[0010] the initial language information and the region information are respectively converted into initial semantic features and initial visual features by the linear mapping layer;

[0011] the initial semantic features and the initial visual features are extracted by the attention encoding layer to obtain semantic features and visual features;

[0012] the semantic features and the visual features are fused by the visual language alignment layer to obtain initial multi-modal features;

[0013] the initial multi-modal features are corrected by the visual language correction layer to obtain target multi-modal features;

[0014] the target multi-modal features are input into a full convolution network based on the target prediction head for prediction to obtain target center and bounding box information of a search region.

[0015] Optionally, the initial language information is natural language information, the linear mapping layer comprises a language linear mapping layer and a visual linear mapping layer, and the step of converting the initial language information and the region information into initial semantic features and initial visual features by the linear mapping layer comprises:

[0016] the natural language information is converted into initial semantic features by the language linear mapping layer;

[0017] the initial target template and the current search region information are converted into initial visual features by the visual linear mapping layer.

[0018] Optionally, the attention encoding layer comprises a language attention encoding layer and a visual attention encoding layer, and the step of extracting the initial semantic features and the initial visual features by the attention encoding layer to obtain semantic features and visual features comprises:

[0019] the initial semantic features of each joint visual language encoding layer are extracted by each language attention encoding layer to obtain semantic features;

[0020] The initial visual features of each joint visual language encoding layer are extracted by each visual attention encoding layer to obtain visual features.

[0021] Optionally, the step of fusing the semantic features and the visual features through the visual language alignment layer to obtain initial multi-modal features comprises:

[0022] determining the similarity between each visual slice in the visual features and each semantic sentence in the semantic features;

[0023] aligning the semantic features and the visual features in space based on the similarity to generate spatially-aware semantic features;

[0024] performing splicing and mapping based on the spatially-aware semantic features and the visual features to generate initial multi-modal features.

[0025] Optionally, the step of performing correction processing on the initial multi-modal features through the visual language correction layer to obtain target multi-modal features comprises:

[0026] based on the visual language correction layer, performing global average pooling on the initial multi-modal features in space to determine the importance of each feature channel;

[0027] performing a weighting operation on the initial multi-modal features according to the importance of each feature channel to obtain a weighted multi-modal feature;

[0028] superimposing the weighted multi-modal feature and the initial multi-modal feature to generate a target multi-modal feature.

[0029] Optionally, the step of inputting the target multi-modal feature into a fully convolutional network for prediction based on the target prediction head to obtain target center and bounding box information of the search region comprises:

[0030] based on the target prediction head, obtaining search region slice features of the target multi-modal feature;

[0031] converting the search region slice features into a feature map, and inputting the feature map into a fully convolutional network for prediction to obtain a classification score map, an offset score map and a scale score map;

[0032] taking the position with the highest score in the classification score map as the target center of the search region;

[0033] determining the bounding box information of the search region based on the offset score map and the scale score map.

[0034] In addition, to achieve the above object, the present application also provides a target tracking device, which comprises:

[0035] An information acquisition module is configured to acquire initial language information and region information in a sample video, wherein the region information comprises initial target template and current search region information.

[0036] A result output module is configured to input the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of a search region, wherein the progressive joint visual language attention network model comprises a linear mapping layer, joint visual language encoding layers and a target prediction head, and each joint visual language encoding layer comprises an attention encoding layer, a visual language alignment layer and a visual language correction layer.

[0037] In addition, to achieve the above object, the present application also provides a target tracking device, which comprises a memory, a processor and a target tracking program stored in the memory and executable on the processor, wherein the target tracking program is configured to implement the steps of the target tracking method as described above.

[0038] In addition, to achieve the above object, the present application also provides a storage medium, wherein the storage medium stores a target tracking program, and the target tracking program is executable on a processor to implement the steps of the target tracking method as described above.

[0039] The present application discloses acquiring initial language information and region information in a sample video, wherein the region information comprises initial target template and current search region information; inputting the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of a search region, wherein the progressive joint visual language attention network model comprises a linear mapping layer, joint visual language encoding layers and a target prediction head, and each joint visual language encoding layer comprises an attention encoding layer, a visual language alignment layer and a visual language correction layer. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 FIG. 1 is a structural schematic diagram of a target tracking device of a hardware running environment related to an embodiment scheme of the present application;

[0041] Figure 2 A flowchart of a first embodiment of the object tracking method of the present application;

[0042] Figure 3 A flowchart of a second embodiment of the object tracking method of the present application;

[0043] Figure 4 A flowchart of a third embodiment of the object tracking method of the present application;

[0044] Figure 5 A structural block diagram of a first embodiment of the object tracking device of the present application.

[0045] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein merely exemplify the application and do not limit the application.

[0047] Reference Figure 1 , Figure 1 A structural diagram of an object tracking device related to the hardware running environment of the embodiment of the present application.

[0048] As shown in Figure 1 , the object tracking device can include a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication between the components. The user interface 1003 can include a display, an input unit such as a keyboard, and can also include a standard wired interface, a wireless interface. The network interface 1004 can optionally include a standard wired interface, a wireless interface (such as a wireless fidelity (WI-FI) interface). The memory 1005 can be a high-speed random access memory (RAM), or a stable non-volatile memory (NVM), such as a disk memory. The memory 1005 can also be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art can understand that Figure 1 the structure shown in the foregoing embodiments does not constitute a limitation on the object tracking device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements.

[0050] As Figure 1 shown, the memory 1005 as a storage medium can include an operating system, a network communication module, a user interface module, and a target tracking program.

[0051] In Figure 1 the target tracking device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the target tracking device of the application can be arranged in the target tracking device, and the target tracking device calls the target tracking program stored in the memory 1005 through the processor 1001, and executes the target tracking method provided by the embodiments of the application.

[0052] The embodiments of the application provide a target tracking method, referring to Figure 2 , Figure 2 the flowchart of the first embodiment of the target tracking method of the application.

[0053] In this embodiment, the target tracking method comprises the following steps:

[0054] Step S10: obtaining initial language information and region information in a sample video, the region information comprising initial target template and current search region information.

[0055] It should be noted that the execution subject of the embodiment can be a computer server device with data processing, network communication and program running functions, such as a server, a tablet computer, a personal computer, an ipa, or an electronic device capable of realizing the above functions, a target tracking device, etc. The target tracking device can include a target tracking system, which can be applied in target tracking and the like. Hereinafter, the target tracking system (hereinafter referred to as system) is taken as an example to illustrate the embodiment and the following embodiments.

[0056] It should be understood that the above-mentioned initial language information and region information can be language information and region information in the sample video in the search region, and the above-mentioned region information comprises initial target template and current search region information, the above-mentioned initial target template can be a target tracking object framed in the first video frame in the sample video, and the above-mentioned current search region information is the region information of the current video frame in the sample video.

[0057] It should be explained that before obtaining the initial language information and the region information in the sample video, the region information comprising the initial target template and the current search region information, it further comprises:

[0058] Step S01: obtaining initial target template, current search region information and language information in each sample video sequence.

[0059] Step S02: Construct an initial progressive joint visual language attention network model, which includes a linear mapping layer, a joint visual language encoding layer, and a target prediction head. Each joint visual language encoding layer includes an attention encoding layer, a visual language alignment layer, and a visual language correction layer.

[0060] Step S03: Train the initial progressive joint visual-language attention network model based on the initial target template, current search region information and language information in each video sequence to obtain the training results.

[0061] Step S04: Based on the training results, determine the focus loss for target center classification, the GIoU loss for target bounding box coordinate regression, and the L1 loss.

[0062] Step S05: Optimize the initial progressive joint visual-language attention network model using the focus loss of the target center classification, the GIoU loss of the target box coordinate regression, and the L1 loss to obtain the progressive joint visual-language attention network model.

[0063] It should be noted that during the model training phase, Gaussian kernel-weighted focus loss is used for target center classification, where L1 loss and GIoU loss are used for target bounding box regression. The total loss of the target prediction head is composed of the focus loss from target center classification. GIoU loss for bounding box coordinate regression and L1 loss The composition is as shown in equation (1):

[0064] (1)

[0065] In the formula, and These are regularization parameters, for example, =2, =5.

[0066] Step S20: Input the initial language information and the region information into the progressive joint visual language attention network model to obtain the target center and bounding box information of the search region. The progressive joint visual language attention network model includes a linear mapping layer, a joint visual language encoding layer and a target prediction head. Each joint visual language encoding layer includes an attention encoding layer, a visual language alignment layer and a visual language correction layer.

[0067] It should be noted that each joint visual language coding layer consists of a visual attention coding layer. (Composed of a multi-head attention layer and a feedforward layer), Language Attention Encoding Layer (Composed of a multi-head attention layer and a feedforward layer), visual language alignment layer and visual language correction layer Composition. In the first The progressive joint visual-language attention network model employs three steps to generate, fuse, and refine visual and semantic features. First, using the first layer... Visual features of the layer as a visual attention encoding layer The input is used to generate the visual features of the j-th layer. Simultaneously, the semantic features of layer j-1 are used as the language attention encoding layer. The input is used to generate the visual features of the j-th layer. Then, visual features and semantic features Input to visual language alignment layer To generate aligned multimodal features Finally, aligned multimodal features As a visual language correction layer The input is used to generate corrected multimodal features. .

[0068] It needs to be explained that step S20 includes:

[0069] Step S201: Through the linear mapping layer, the initial language information and the region information are converted into initial semantic features and initial visual features, respectively.

[0070] It should be noted that the initial language information is natural language information, the region information includes the initial target template and the current search region information, and the linear mapping layer includes the language linear mapping layer and the visual linear mapping layer.

[0071] It should be explained that the natural language information can be converted into initial semantic features through the language linear mapping layer; and the initial target template and the current search area information can be converted into initial visual features through the visual linear mapping layer.

[0072] In practical implementation, a language linear mapping layer can be used. Convert natural language descriptions into initial semantic features Where C and T are the number of channels and the number of words, respectively; a visual linear mapping layer is used. The stitched initial target template and visual slices of the search region are converted into initial visual features. Where C and N are the number of channels and slices, respectively.

[0073] Step S202: extracting the initial semantic features and the initial visual features by using the attention encoding layers to obtain semantic features and visual features.

[0074] It should be noted that the attention encoding layer includes a language attention encoding layer and a visual attention encoding layer. The initial semantic features of each joint visual language encoding layer are extracted by each language attention encoding layer to obtain semantic features; the initial visual features of each joint visual language encoding layer are extracted by each visual attention encoding layer to obtain visual features.

[0075] It should be explained that in order to convert the initial visual features into high-dimensional semantic embeddings, the initial semantic features are extracted by the language attention encoding layer to obtain semantic features, and the semantic features of the jth layer are denoted as where C and T are the number of channels and words, respectively.

[0076] In order to convert the initial visual features into high-dimensional visual embeddings, the initial visual features are extracted by the visual attention encoding layer to obtain visual features, and the visual features of the jth layer are denoted as where C and N are the number of channels and slices, respectively.

[0077] Step S203: fusing the semantic features and the visual features by the visual language alignment layer to obtain initial multi-modal features.

[0078] It should be explained that the above-mentioned visual language alignment layer takes the visual features and the semantic features output by the language attention encoding layer and the visual attention encoding layer as input, and aligns the visual slices in the visual features and the semantic sentences in the semantic features.

[0079] It can be understood that through visual language alignment, the common semantics in visual and language information can be effectively enhanced, thereby effectively enhancing the specific target information tracked.

[0080] It should be understood that by fusing the semantic features and the visual features by the visual language alignment layer to obtain initial multi-modal features, different levels of language information can be effectively integrated into different levels of visual information, and the finally encoded multi-modal features can more effectively predict the target in the search area.

[0081] In a specific implementation, the above step S203 includes:

[0082] Step S2031: determining the similarity between each visual slice in the visual features and each semantic sentence in the semantic features.

[0083] Step S2032: based on the similarity, aligning the semantic features and the visual features in space to generate spatial-aware semantic features.

[0084] Step S2033: based on the spatial-aware semantic features and the visual features, performing stitching and mapping to generate initial multi-modal features.

[0085] It should be noted that the visual language alignment layer adopted in this embodiment allows the slices in the visual features and the sentences in the semantic features to gradually interact in the encoding stage, which is effective for fully utilizing the encoding layer of the attention network to mine visual language context information.

[0086] The visual language alignment layer takes the visual features (where H and W represent the height and width of the feature map) and the semantic features as input, and aligns the visual slices in the visual features and the semantic sentences in the semantic features in the following steps. First, for each visual slice, the visual language alignment layer aggregates the semantic features in the sentence dimension to produce spatial-aware semantic features that can effectively encode the semantics corresponding to the local neighborhood of the visual slice. The specific encoding process is as follows:

[0087] ;

[0088] ; (2)

[0089] ; ; .

[0090] wherein, is a query mapping function of . and are the key mapping function and the value mapping function of , respectively. is the output mapping function of . After obtaining the spatial-aware semantic features , the features are concatenated along the feature channel to generate aligned visual semantic features , represents a concatenation operation, is a mapping function of the visual features , and is a mapping function of the concatenated features.

[0091] Step S204: performing correction processing on the initial multi-modal features by the visual language correction layer to obtain target multi-modal features.

[0092] It should be noted that the visual language correction layer takes the initial multi-modal feature as input, performs weighting and superposition processing on the initial multi-modal feature, and obtains the target multi-modal feature.

[0093] It can be understood that through visual language correction, the multi-modal feature channel related to the target can be effectively enhanced, thereby effectively performing visual language tracking of target perception.

[0094] Step S205: inputting the target multi-modal feature into a full convolution network based on the target prediction head for prediction to obtain target center and bounding box information of the search area.

[0095] In a specific implementation, based on the target prediction head, a search area slice feature of the target multi-modal feature is obtained; the search area slice feature is converted into a feature map, and the feature map is input into a full convolution network for prediction to obtain target center and bounding box information of the search area.

[0096] The embodiment discloses obtaining initial language information and region information in a sample video, the region information including an initial target template and current search region information; converting the initial language information and the region information into initial semantic features and initial visual features through a linear mapping layer; extracting the initial semantic features and the initial visual features by using an attention encoding layer to obtain semantic features and visual features; fusing the semantic features and the visual features through a visual language alignment layer to obtain initial multi-modal features; performing correction processing on the initial multi-modal features through a visual language correction layer to obtain target multi-modal features; inputting the target multi-modal features into a full convolution network based on a target prediction head for prediction to obtain target center and bounding box information of the search area. Since the embodiment inputs the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of the search area, the progressive joint visual language attention network model includes a linear mapping layer, a joint visual language encoding layer, and a target prediction head, each joint visual language encoding layer includes an attention encoding layer, a visual language alignment layer, and a visual language correction layer, and compared with the prior art, the embodiment realizes progressive joint encoding of language features and visual features, and effectively improves the accuracy and robustness of visual language tracking.

[0097] Reference Figure 3 , Figure 3 The flowchart of the second embodiment of the target tracking method of the present application is shown.

[0098] Based on the above first embodiment, in the present embodiment, the step S204 comprises:

[0099] Step S2041: Based on the visual language correction layer, the initial multi-modal feature is globally averaged and pooled in space to determine the importance of each feature channel.

[0100] Step S2042: The initial multi-modal feature is weighted according to the importance of each feature channel to obtain a weighted multi-modal feature.

[0101] Step S2043: The weighted multi-modal feature and the initial multi-modal feature are superimposed to generate a target multi-modal feature.

[0102] It should be noted that the visual language alignment layer can gradually align the visual feature and the semantic feature in space. However, different channels in the aligned multi-modal feature (i.e., the initial multi-modal feature) usually correspond to different meanings, and the feature channel related to the tracked target object needs to be enhanced. Therefore, the visual language correction layer is inserted in each joint visual language encoding layer to estimate the channel weight of the initial multi-modal feature, so as to realize the multi-modal target tracking of target perception.

[0103] It should be explained that the visual language correction layer uses the aligned multi-modal feature (i.e., the initial multi-modal feature) to predict each channel weight, and weights the multi-modal feature in the channel. The specific calculation process of the visual language correction layer is as follows:

[0104] ;

[0105] ⊙ .(3)

[0106] In the formula, ⊙ represents a channel-wise multiplication operation, represents a global average pooling operation, represents a multi-layer perceptron (MLP) function. Through the visual language correction layer, first, the initial multi-modal feature is globally averaged and pooled in space to estimate the importance of each channel, so as to weight the initial multi-modal feature. Then, the weighted multi-modal feature and the initial multi-modal feature are superimposed to generate a corrected multi-modal feature (i.e., a target multi-modal feature). The visual language correction layer can activate the feature channel of target perception for cross-modal representation learning.

[0107] The embodiment discloses a visual language correction layer, and performs global average pooling on initial multi-modal features output by a visual language alignment layer in space to determine the importance of each feature channel; performs weighting operation on the initial multi-modal features according to the importance of each feature channel to obtain weighted multi-modal features; and superimposes the weighted multi-modal features and the initial multi-modal features to generate target multi-modal features. Compared with the prior art, the visual language correction layer is used in the embodiment, first, the initial multi-modal features are subjected to global average pooling in space to estimate the importance of each channel, so that the initial multi-modal features are weighted, then the weighted multi-modal features are superimposed with the initial multi-modal features to generate target multi-modal features, and the visual language correction layer can activate the feature channels of target perception to perform cross-modal representation learning.

[0108] Reference Figure 4 , Figure 4 A flowchart of a third embodiment of the target tracking method of the present application.

[0109] Based on the above embodiments, in the present embodiment, the step S205 comprises:

[0110] Step S2051: Based on the target prediction head, the search area slice feature of the target multi-modal feature is obtained.

[0111] Step S2052: The search area slice feature is converted into a feature map, and the feature map is input into a full convolutional network for prediction to obtain a classification score map, an offset score map and a scale score map.

[0112] It should be noted that the full convolutional network (FCN) described above is composed of a plurality of stacked Conv-BN-ReLU layers, which outputs three branches to predict the classification score map, the offset score map and the scale score map respectively.

[0113] Step S2053: The position with the highest score in the classification score map is taken as the target center of the search area.

[0114] Step S2054: The boundary box information of the search area is determined based on the offset score map and the scale score map.

[0115] The embodiment discloses that based on the target prediction head, a search area slice feature of the target multi-modal feature is acquired; the search area slice feature is converted into a feature map, and the feature map is input into a full convolution network for prediction to obtain a classification score map, an offset score map and a scale score map; a position with the highest score in the classification score map is taken as a target center of the search area; and boundary box information of the search area is determined based on the offset score map and the scale score map. Compared with the prior art, for prediction of the target center and the boundary box information, the embodiment first converts the search area slice feature of the multi-modal feature output by the progressive joint visual language encoder into a feature map, and then inputs the feature map into the full convolution network for prediction, so that the accuracy of prediction of the target center of the search area and the boundary box information of the search area is effectively improved.

[0116] In addition, the embodiment of the application further provides a storage medium, wherein the storage medium stores a target tracking program, and the target tracking program is executed by a processor to implement the steps of the target tracking method.

[0117] Reference Figure 5 , Figure 5 FIG. 1 is a structural block diagram of a first embodiment of a target tracking device according to the application.

[0118] As Figure 5 shown, the target tracking device provided by the embodiment of the application comprises an information acquisition module 501 and a result output module 502.

[0119] The information acquisition module 501 is configured to acquire initial language information and region information in a sampling video, wherein the region information comprises initial target template information and current search area information.

[0120] The result output module 502 is configured to input the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and boundary box information of a search area, wherein the progressive joint visual language attention network model comprises a linear mapping layer, joint visual language encoding layers and a target prediction head, and each joint visual language encoding layer comprises an attention encoding layer, a visual language alignment layer and a visual language correction layer.

[0121] The result output module 502 is further configured to convert the initial language information and the region information into initial semantic features and initial visual features respectively by the linear mapping layer; extract the initial semantic features and the initial visual features by the attention encoding layer to obtain semantic features and visual features; fuse the semantic features and the visual features by the visual language alignment layer to obtain initial multi-modal features; perform correction processing on the initial multi-modal features by the visual language correction layer to obtain target multi-modal features; and input the target multi-modal features into a full convolution network based on the target prediction head to perform prediction, and obtain target center and bounding box information of a search region.

[0122] The result output module 502 is further configured to convert the natural language information into initial semantic features by the language linear mapping layer; and convert the initial target template and the current search region information into initial visual features by the visual linear mapping layer.

[0123] The result output module 502 is further configured to extract the initial semantic features of each joint visual language encoding layer by each language attention encoding layer to obtain semantic features; and extract the initial visual features of each joint visual language encoding layer by each visual attention encoding layer to obtain visual features.

[0124] The result output module 502 is further configured to determine a similarity between each visual slice in the visual features and each semantic sentence in the semantic features; align the semantic features and the visual features in space based on the similarity to generate spatial perception semantic features; and perform splicing and mapping based on the spatial perception semantic features and the visual features to generate initial multi-modal features.

[0125] The embodiment discloses obtaining initial language information and region information in a sampled video, the region information including initial target template and current search region information; inputting the initial language information and the region information into a progressive joint visual language attention network model to obtain target center and bounding box information of a search region, the progressive joint visual language attention network model including a linear mapping layer, joint visual language coding layers and a target prediction head, and each joint visual language coding layer including an attention coding layer, a visual language alignment layer and a visual language correction layer. Since the initial language information and the region information are input into the progressive joint visual language attention network model to obtain the target center and the bounding box information of the search region, the progressive joint visual language attention network model includes the linear mapping layer, the joint visual language coding layers and the target prediction head, and each joint visual language coding layer includes the attention coding layer, the visual language alignment layer and the visual language correction layer, compared with the prior art, the embodiment realizes progressive joint coding of language features and visual features, and effectively improves the accuracy and robustness of visual language tracking.

[0126] Based on the above-mentioned first embodiment of the target tracking device, a second embodiment of the target tracking device is provided.

[0127] In the embodiment, the result output module 502 is further configured to perform global average pooling on the initial multi-modal features in space based on the visual language correction layer to determine the importance of each feature channel, perform weighting operation on the initial multi-modal features according to the importance of each feature channel to obtain weighted multi-modal features, and superimpose the weighted multi-modal features and the initial multi-modal features to generate target multi-modal features.

[0128] Other embodiments or specific implementation manners of the target tracking device can refer to the above-mentioned method embodiments, which will not be described here.

[0129] It should be noted that, in this document, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0130] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example method can be realized by means of software and a necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk) and includes a number of instructions for causing an end device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method described in each embodiment of the present application.

[0132] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A target tracking method, characterized in that, The target tracking method includes the following steps: Acquire initial language information and region information from the sampled video, wherein the region information includes the initial target template and the current search region information; The initial language information and the region information are input into the progressive joint visual language attention network model to obtain the target center and bounding box information of the search region. The progressive joint visual language attention network model includes a linear mapping layer, a joint visual language encoding layer and a target prediction head. Each joint visual language encoding layer includes an attention encoding layer, a visual language alignment layer and a visual language correction layer. The step of inputting the initial language information and the region information into the progressive joint visual-language attention network model to obtain the target center and bounding box information of the search region includes: Through the linear mapping layer, the initial language information and the region information are respectively converted into initial semantic features and initial visual features; The initial semantic features and the initial visual features are extracted using the attention encoding layer to obtain semantic features and visual features; The semantic features and the visual features are fused through the visual language alignment layer to obtain initial multimodal features; The initial multimodal features are corrected by the visual language correction layer to obtain the target multimodal features; Based on the target prediction head, the multimodal features of the target are input into a fully convolutional network for prediction to obtain the target center and bounding box information of the search region; The step of fusing the semantic features and the visual features through the visual language alignment layer to obtain initial multimodal features includes: Determine the similarity between each visual slice in the visual features and each semantic statement in the semantic features; Based on the similarity, the semantic features and the visual features are aligned in space to generate spatially perceived semantic features; Initial multimodal features are generated by splicing and mapping the spatially perceived semantic features and the visual features.

2. The target tracking method as described in claim 1, characterized in that, The initial language information is natural language information. The linear mapping layer includes a language linear mapping layer and a visual linear mapping layer. The step of converting the initial language information and the region information into initial semantic features and initial visual features respectively through the linear mapping layer includes: The natural language information is converted into initial semantic features through the language linear mapping layer; The visual linear mapping layer converts the initial target template and the current search area information into initial visual features.

3. The target tracking method as described in claim 1, characterized in that, The attention encoding layer includes a language attention encoding layer and a visual attention encoding layer. The step of extracting the initial semantic features and the initial visual features using the attention encoding layer to obtain semantic features and visual features includes: Semantic features are obtained by extracting the initial semantic features of each of the joint visual language coding layers through each of the language attention coding layers; Visual features are obtained by extracting the initial visual features of each of the joint visual language coding layers through each of the visual attention coding layers.

4. The target tracking method as described in claim 1, characterized in that, The step of correcting the initial multimodal features through the visual language correction layer to obtain the target multimodal features includes: Based on the visual language correction layer, the initial multimodal features are subjected to global average pooling in space to determine the importance of each feature channel; The initial multimodal features are weighted according to the importance of each feature channel to obtain weighted multimodal features; The weighted multimodal features and the initial multimodal features are superimposed to generate the target multimodal features.

5. The target tracking method as described in claim 1, characterized in that, The step of inputting the multimodal features of the target into a fully convolutional network for prediction based on the target prediction head to obtain the target center and bounding box information of the search region includes: Based on the target prediction head, the search region slice features of the target multimodal features are obtained; The slice features of the search region are converted into feature maps, and the feature maps are input into a fully convolutional network for prediction to obtain classification score maps, offset score maps, and scale score maps. The position with the highest score in the classification score map is used as the target center of the search area; The bounding box information of the search region is determined based on the offset score map and the scale score map.

6. A target tracking device, characterized in that, The target tracking device includes: The information acquisition module is used to acquire initial language information and region information in the sampled video, wherein the region information includes the initial target template and the current search region information; The result output module is used to input the initial language information and the region information into the progressive joint visual language attention network model to obtain the target center and bounding box information of the search region. The progressive joint visual language attention network model includes a linear mapping layer, a joint visual language encoding layer and a target prediction head. Each joint visual language encoding layer includes an attention encoding layer, a visual language alignment layer and a visual language correction layer. The result output module is further configured to: convert the initial language information and the region information into initial semantic features and initial visual features respectively through the linear mapping layer; extract the initial semantic features and the initial visual features using the attention encoding layer to obtain semantic features and visual features; fuse the semantic features and the visual features through the visual language alignment layer to obtain initial multimodal features; correct the initial multimodal features through the visual language correction layer to obtain target multimodal features; and input the target multimodal features into a fully convolutional network for prediction based on the target prediction head to obtain the target center and bounding box information of the search region. The result output module is further configured to determine the similarity between each visual slice in the visual features and each semantic statement in the semantic features; based on the similarity, the semantic features and the visual features are aligned spatially to generate spatially perceived semantic features; and based on the spatially perceived semantic features and the visual features, they are spliced ​​and mapped to generate initial multimodal features.

7. A target tracking device, characterized in that, The device includes: a memory, a processor, and a target tracking program stored in the memory and executable on the processor, the target tracking program being configured to implement the steps of the target tracking method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a target tracking program, which, when executed by a processor, implements the steps of the target tracking method as described in any one of claims 1 to 5.

Images