Target tracking processing method and device, storage medium and electronic equipment

Abstract

The application discloses a target tracking processing method and device, a storage medium and an electronic device, and relates to the technical field of computer vision. The method comprises the following steps: extracting a multi-modal target feature from multi-modal sensing data; performing cross-modal feature fusion processing on the multi-modal target feature to obtain a multi-modal fusion feature; performing aggregation processing on a first frame template feature and a dynamic template feature to obtain an aggregated template feature; and performing fusion processing on the multi-modal fusion feature and the aggregated template feature to obtain a comprehensive feature, which is used for predicting a target position. The application can realize real-time target tracking with high performance under low computing resource conditions.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a target tracking processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Target tracking performance can be improved by combining multimodal sensing data from ordinary cameras (RGB cameras) and other sensors (such as thermal imaging, depth cameras, event cameras, sonar, etc.). However, current target tracking using multimodal sensing data typically requires high computational resources, making it difficult to perform on resource-constrained devices such as drones and embedded systems. Moreover, tracking performance degrades significantly in complex scenarios (such as nighttime, occlusion, high-speed motion, etc.), limiting its practicality. Summary of the Invention

[0003] This application provides a target tracking processing scheme that can achieve high-performance target tracking in real time under low computing resource conditions.

[0004] The embodiments of this application provide the following technical solutions: According to one embodiment of this application, a target tracking processing method includes: extracting multimodal target features from multimodal sensing data; performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; aggregating first-frame template features and dynamic template features to obtain aggregated template features; and fusing the multimodal fused features and the aggregated template features to obtain comprehensive features, wherein the comprehensive features are used to predict the target position.

[0005] In some embodiments of this application, the step of performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features includes: performing cross-modal cross-attention calculation on the multimodal target features to obtain cross-modal features; and performing fusion enhancement processing on the cross-modal features based on joint spatial dimension and channel dimension to obtain the multimodal fused features.

[0006] In some embodiments of this application, the step of performing joint spatial and channel dimension fusion enhancement processing on the cross-modal features to obtain the multimodal fusion features includes: performing global average pooling on the cross-modal features to obtain compressed features; performing convolution processing on the compressed features to obtain channel weights; calculating local neighborhood pixel association weights based on the spatial coordinates of the cross-modal features to obtain spatial weights; multiplying the channel weights and spatial weights point by point and then superimposing them on the cross-modal features to obtain superimposed features; and obtaining the multimodal fusion features based on the superimposed features.

[0007] In some embodiments of this application, obtaining the multimodal fusion feature based on the superimposed feature includes: performing convolution processing on the superimposed feature to obtain a first feature; adding the convolved feature to the multimodal target feature element by element to obtain a second feature; and obtaining the multimodal fusion feature based on the second feature.

[0008] In some embodiments of this application, the step of aggregating the first frame template features and the dynamic template features to obtain aggregated template features includes: weighted fusing the first frame template features and the dynamic template features to obtain fused template features; and performing noise filtering on the fused template features to obtain the aggregated template features.

[0009] In some embodiments of this application, the step of fusing the multimodal fusion feature and the aggregated template feature to obtain a comprehensive feature includes: concatenating the multimodal fusion feature and the aggregated template feature to obtain a concatenated feature; and performing convolution processing on the concatenated feature to obtain the comprehensive feature.

[0010] In some embodiments of this application, after fusing the multimodal fusion features and the aggregated template features to obtain comprehensive features, the method further includes: performing region-by-region similarity matching based on the comprehensive features to obtain the similarity between the regional features of different candidate regions in the comprehensive features and the target matching features; and obtaining the predicted target location based on the regional location of the candidate region corresponding to the largest similarity.

[0011] According to one embodiment of this application, a target tracking processing apparatus includes: a feature extraction module, configured to: extract multimodal target features from multimodal sensing data; a feature fusion module, configured to: perform cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; a feature aggregation module, configured to: aggregate first-frame template features and dynamic template features to obtain aggregated template features; and a feature synthesis module, configured to: fuse the multimodal fused features and the aggregated template features to obtain synthesized features, wherein the synthesized features are used to predict the target position.

[0012] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the methods described in the embodiments of this application.

[0013] According to another embodiment of this application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0014] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0015] In this embodiment, multimodal target features are extracted from multimodal sensing data; the multimodal target features are subjected to cross-modal feature fusion processing to obtain multimodal fused features; the first frame template features and dynamic template features are aggregated to obtain aggregated template features; the multimodal fused features and the aggregated template features are fused to obtain comprehensive features, which are used to predict the target location.

[0016] In this embodiment of the application, when tracking a target using multimodal sensing data, multimodal target features are fused across modalities to obtain multimodal fusion features representing multimodal semantics. First-frame template features and dynamic template features are aggregated to obtain aggregated template features representing stable spatiotemporal appearance information of the tracked target. Multimodal fusion features and aggregated template features are fused to obtain comprehensive features that simultaneously contain multimodal semantics and stable spatiotemporal appearance information. Based on these comprehensive features, high-performance target tracking can be achieved. Moreover, cross-modal feature fusion processing and template feature aggregation processing can be achieved through lightweight feature fusion and aggregation modules, with very low computational resource requirements. Therefore, high-performance target tracking can be achieved in real time under low computational resource conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a target tracking processing method according to an embodiment of this application is shown.

[0019] Figure 2 A flowchart of cross-modal feature fusion processing according to an embodiment of this application is shown.

[0020] Figure 3 A flowchart of the aggregation process according to an embodiment of this application is shown.

[0021] Figure 4A block diagram of a target tracking processing apparatus according to an embodiment of this application is shown.

[0022] Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0024] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus; for example, a unit may be a portion of circuitry, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus that includes that element.

[0025] For example, the target tracking processing method provided in this disclosure includes a series of steps, but the target tracking processing method provided in this disclosure is not limited to the steps described. Similarly, the target tracking processing apparatus provided in this disclosure includes a series of units, but the apparatus provided in this disclosure is not limited to the units explicitly described, and may also include units that need to be set up for obtaining relevant information or processing based on information.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0027] It is understood that in the specific implementation of this application, relevant data is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0028] Target tracking performance can be improved by combining multimodal sensing data from ordinary cameras (RGB cameras) and other sensors (such as thermal imaging, depth cameras, event cameras, sonar, etc.). However, current target tracking using multimodal sensing data typically requires high computational resources, making it difficult to perform on resource-constrained devices such as drones and embedded systems. Moreover, tracking performance degrades significantly in complex scenarios (such as nighttime, occlusion, high-speed motion, etc.), limiting its practicality.

[0029] To address these issues, this application provides a target tracking processing scheme that enables real-time, high-performance target tracking under low computing resource conditions.

[0030] The following is a detailed description of relevant embodiments of the target tracking processing scheme provided in this application.

[0031] Figure 1 A flowchart illustrating a target tracking processing method according to an embodiment of this application is shown. The entity executing this target tracking processing method can be an electronic device. This electronic device can be a drone, vehicle-mounted terminal, monitoring equipment, robot, mobile phone, computer, smartwatch, or other electronic device.

[0032] like Figure 1 As shown, the target tracking processing method may include steps S110 to S140.

[0033] Step S110: Extract multimodal target features from multimodal sensing data; Step S120: Perform cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; Step S130: Aggregate the first frame template features and dynamic template features to obtain aggregated template features; Step S140: The multimodal fusion features and aggregated template features are fused to obtain comprehensive features, which are used to predict the target location.

[0034] During target tracking, sensing data can be continuously acquired from multiple modal sensors, such as ordinary cameras (RGB cameras) and other sensors (e.g., thermal imaging, depth cameras, event cameras, sonar, etc.), to obtain a sequence of multimodal sensing data. In this sequence, each frame of multimodal sensing data includes sensing data from multiple modalities. For example, in one example, each frame of multimodal sensing data may include a color image captured by an ordinary camera and a depth image captured by a depth camera.

[0035] Multimodal target features of frame i can be extracted from the multimodal sensing data of frame i. Specifically, target features are extracted from the sensing data of each modality in the multimodal sensing data. Thus, the multimodal target features can include target features of multiple modalities. The target features are feature maps or feature vectors extracted from the search area of ​​the target tracking data in the sensing data.

[0036] For example, in one instance, a lightweight convolutional neural network can be used to extract feature maps or feature vectors (i.e., target features of the first modality) from the search region of the target in a color image, and to extract feature maps or feature vectors (i.e., target features of the second modality) from the search region of the target in a depth image. In addition, target features of other modalities can also be extracted.

[0037] The multimodal target features of the i-th frame are subjected to cross-modal feature fusion processing, that is, the target features of multiple modalities included in the multimodal target features of the i-th frame are fused together to obtain multimodal fusion features that fuse target features of multiple modalities, which are in turn multimodal fusion features that represent multimodal semantics.

[0038] In the aforementioned sequence, the multimodal sensing data of the first frame containing the tracked target can be used as the first frame template data. Target features can be extracted from the target region where the tracked target is located from the sensing data of each modality in the first frame template data. The multimodal target features corresponding to the first frame template data are then subjected to cross-modal feature fusion processing to obtain the first frame template features.

[0039] In the aforementioned sequence, the multimodal sensing data of the tracked target in the other frames after the first frame can be used as dynamic template data. Target features can be extracted from the target region where the tracked target is located from the sensing data of each modality in the dynamic template data. The multimodal target features corresponding to the dynamic template data are then subjected to cross-modal feature fusion processing to obtain dynamic template features.

[0040] By aggregating the first frame template features and dynamic template features, we can combine multi-frame temporal information and single-frame spatial information to obtain aggregated template features that can characterize the stable spatiotemporal appearance information of the tracked target. These aggregated template features can compensate for the feature loss of single template features in complex scenarios such as nighttime, occlusion, and target deformation.

[0041] Furthermore, by fusing the multimodal fusion features and aggregated template features of the i-th frame, a comprehensive feature of the i-th frame that simultaneously contains multimodal semantics and stable spatiotemporal appearance information can be obtained. Based on this comprehensive feature, the position of the tracking target (i.e., the target position) can be accurately predicted.

[0042] In this embodiment of the application, when tracking targets using multimodal sensing data, multimodal target features are fused across modalities to obtain multimodal fused features representing multimodal semantics. First-frame template features and dynamic template features are aggregated to obtain aggregated template features representing stable spatiotemporal appearance information of the tracked target. The multimodal fused features and aggregated template features are then fused to obtain a comprehensive feature that simultaneously contains multimodal semantics and stable spatiotemporal appearance information. Based on this comprehensive feature, high-performance target tracking can be achieved. Furthermore, cross-modal feature fusion processing and template feature aggregation processing can be implemented through lightweight feature fusion and aggregation modules, requiring very low computational resources. Therefore, high-performance target tracking can be achieved in real-time under low computational resource conditions. The target tracking processing scheme of this application can achieve high-performance target tracking in a lightweight multimodal target tracking system and is suitable for resource-constrained devices.

[0043] The following description Figure 1 Further optional specific embodiments for each step performed during target tracking processing in the example.

[0044] See Figure 2 In one embodiment, step S120 involves performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features, including: step S210, performing cross-modal cross-attention calculation on the multimodal target features to obtain cross-modal features; and step S220, performing fusion enhancement processing on the cross-modal features using joint spatial dimension and channel dimension to obtain multimodal fused features.

[0045] Lightweight cross-modal cross-attention computation is performed on multimodal target features to obtain cross-modal features. ,in, ; It is the feature obtained by multiplying the target feature of the first modality by the first weight matrix. It is the feature obtained by multiplying the target feature of the second modality by the second weight matrix. It is the feature obtained by multiplying the target feature of the second mode by the third weight matrix.

[0046] Cross-modal cross-attention computation yields cross-modal features that can characterize multimodal semantics. Further fusion enhancement processing of these cross-modal features, combining spatial and channel dimensions, results in multimodal fusion features with enhanced key semantic channels and key locations. This multimodal fusion feature, combined with the proposed scheme, can further improve target tracking performance.

[0047] Furthermore, in one embodiment, the cross-modal features are subjected to joint spatial and channel dimension fusion enhancement processing to obtain multimodal fusion features, which may include: performing global average pooling processing on the cross-modal features to obtain compressed features; performing convolution processing on the compressed features to obtain channel weights; calculating local neighborhood pixel association weights based on the spatial coordinates of the cross-modal features to obtain spatial weights; multiplying the channel weights and spatial weights point by point and then superimposing them on the cross-modal features to obtain superimposed features; and obtaining multimodal fusion features based on the superimposed features.

[0048] In this embodiment, the process of fusing and enhancing cross-modal features using both spatial and channel dimensions first involves global average pooling to compress the feature dimensions, resulting in compressed features. Then, channel dimension weights are generated: the compressed features are subjected to lightweight convolution (e.g., using a 1×1 small convolutional layer) to weaken semantic channels of ineffective modalities and strengthen semantic channels of effective modalities, resulting in channel weights used to enhance key semantic channels.

[0049] Then, spatial dimension weights are generated: local neighborhood pixel association weights are calculated based on the spatial coordinates of cross-modal features to obtain spatial weights that can be used to enhance key locations in complex scenes. Specifically, instead of using complex global convolution, a region-by-region spatial weighting method is adopted. Taking each location in the cross-modal features as the center point, the association weights (such as distance) between the center point and each location in its neighborhood are calculated and normalized to obtain the normalized weight of the center point. That is, the normalized weight of each location can be obtained. The spatial weights are composed of the normalized weights of each location in the cross-modal features.

[0050] Then, the channel weights and spatial weights are multiplied point by point and superimposed onto the original cross-modal features to obtain the superimposed features. These superimposed features achieve key semantic channel enhancement and key position enhancement. Based on the superimposed features, further multimodal fusion features with key semantic channel enhancement and key position enhancement can be obtained. Moreover, the overall fusion enhancement process can be implemented lightweightly, ensuring low computational overhead.

[0051] In one embodiment, obtaining multimodal fusion features based on superimposed features may include: using superimposed features as multimodal fusion features.

[0052] Furthermore, in one embodiment, obtaining multimodal fusion features based on superimposed features may include: performing convolution processing on the superimposed features to obtain a first feature; adding the convolutional feature to the multimodal target feature element by element to obtain a second feature; and obtaining the multimodal fusion feature based on the second feature.

[0053] In this implementation, the superimposed features are further subjected to lightweight convolution processing (e.g., through a 1×1 small convolutional layer) to complete feature channel mapping and semantic enhancement, resulting in a first feature. Further, the convolutional feature is added element-wise to the multimodal target feature to obtain a second feature, preserving the original effective multimodal target feature and avoiding gradient vanishing. Based on this second feature, a multimodal fusion feature is obtained. Overall, multimodal feature optimization and refinement are achieved with low computational power, balancing performance and speed. Specifically, obtaining the multimodal fusion feature based on the second feature can be achieved by normalizing the second feature.

[0054] See Figure 3 In one embodiment, step S130, which aggregates the first frame template features and dynamic template features to obtain aggregated template features, may include: step S310, weighted fusion of the first frame template features and dynamic template features to obtain fused template features; and step S320, noise filtering processing of the fused template features to obtain aggregated template features.

[0055] The first-frame template features and dynamic template features are weighted and fused to obtain fused template features, which compensates for the feature loss of single template features in complex scenarios such as nighttime, occlusion, and target deformation. Further noise filtering (such as filtering background noise and sensor noise) is applied to the fused template features, and the resulting aggregated template features, when used in the proposed scheme, can further improve the robustness of target tracking.

[0056] Before weighted fusion of the first-frame template features and dynamic template features, the first-frame template features and dynamic template features can be aligned spatially. When weighted fusion of the first-frame template features and dynamic template features, weights can be preset using fixed weights (e.g., giving the first-frame template features a larger weight and subsequent dynamic template features a smaller weight) or similarity weights (template features with greater similarity to the multimodal fusion features have higher weights). Noise filtering of the fused template features can be achieved by performing lightweight convolution processing (e.g., using a 1×1 small convolutional layer).

[0057] Optionally, in other embodiments, step S130, which involves aggregating the first frame template features and dynamic template features to obtain aggregated template features, may include: weighted fusion of the first frame template features and dynamic template features to obtain fused template features; and using the fused template features as aggregated template features.

[0058] In one embodiment, step S140, fusing the multimodal fusion features and the aggregated template features to obtain comprehensive features, may include: concatenating the multimodal fusion features and the aggregated template features to obtain concatenated features; and performing convolution processing on the concatenated features to obtain comprehensive features.

[0059] By concatenating the multimodal fusion features and the aggregated template features, we obtain concatenated features. Then, we perform convolution processing on the concatenated features through lightweight convolution processing (such as convolution processing through a 1×1 small convolutional layer). This ensures that we obtain comprehensive features that simultaneously contain multimodal semantics and stable spatiotemporal appearance information with low computational cost and high speed.

[0060] In one embodiment, after fusing the multimodal fusion features and the aggregated template features to obtain the comprehensive features, the method further includes: performing region-by-region similarity matching based on the comprehensive features to obtain the similarity between the regional features of different candidate regions in the comprehensive features and the target matching features; and obtaining the predicted target location based on the regional location of the candidate region corresponding to the largest similarity.

[0061] Based on comprehensive features, a region matching strategy is adopted. This involves traversing the region features of candidate regions within the comprehensive features, calculating the similarity between the region features and the target matching features, and then accurately obtaining the predicted target location (such as target coordinates and bounding box information) based on the region location corresponding to the candidate region with the highest similarity. Furthermore, after obtaining the predicted target position, the predicted target position can be corrected by combining the motion trajectory of the tracked target in the previous few frames, so as to avoid losing the tracked target.

[0062] Furthermore, with the solution proposed in this application, when parameter adjustments are required, lightweight parameter fine-tuning can be achieved without having to update and adjust the parameters of the entire processing process. The backbone feature extraction network can be frozen, and only a small number of parameters used for convolution and normalization during cross-modal feature fusion processing, aggregation processing, and fusion processing need to be fine-tuned. The computational load of a single fine-tuning is small, ensuring real-time operation of resource-constrained devices such as embedded devices.

[0063] To facilitate better implementation of the target tracking processing method provided in the embodiments of this application, the embodiments of this application also provide a target tracking processing apparatus based on the above-described target tracking processing method. The meanings of the terms used are the same as in the target tracking processing method described above, and specific implementation details can be found in the descriptions in the method embodiments. Figure 4 A block diagram of a target tracking processing apparatus according to an embodiment of this application is shown.

[0064] like Figure 4As shown, the target tracking processing device 400 may include: a feature extraction module 410 for extracting multimodal target features from multimodal sensing data; a feature fusion module 420 for performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; a feature aggregation module 430 for aggregating the first frame template features and dynamic template features to obtain aggregated template features; and a feature synthesis module 440 for fusing the multimodal fused features and the aggregated template features to obtain synthesized features, which are used to predict the target position.

[0065] In some embodiments of this application, when performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features, the feature fusion module 420 can be used to: perform cross-modal cross-attention calculation on the multimodal target features to obtain cross-modal features; and perform joint spatial dimension and channel dimension fusion enhancement processing on the cross-modal features to obtain the multimodal fused features.

[0066] In some embodiments of this application, when performing joint spatial and channel dimension fusion enhancement processing on the cross-modal features to obtain the multimodal fused features, the feature fusion module 420 can be used to: perform global average pooling processing on the cross-modal features to obtain compressed features; perform convolution processing on the compressed features to obtain channel weights; calculate local neighborhood pixel association weights based on the spatial coordinates of the cross-modal features to obtain spatial weights; multiply the channel weights and the spatial weights point by point and then superimpose them onto the cross-modal features to obtain superimposed features; and obtain the multimodal fused features based on the superimposed features.

[0067] In some embodiments of this application, when obtaining the multimodal fusion feature based on the superimposed feature, the feature fusion module 420 can be used to: perform convolution processing on the superimposed feature to obtain a first feature; add the convolutional feature to the multimodal target feature element by element to obtain a second feature; and obtain the multimodal fusion feature based on the second feature.

[0068] In some embodiments of this application, when aggregating the first frame template features and dynamic template features to obtain aggregated template features, the feature aggregation module 430 can be used to: perform weighted fusion of the first frame template features and the dynamic template features to obtain fused template features; and perform noise filtering on the fused template features to obtain the aggregated template features.

[0069] In some embodiments of this application, when fusing the multimodal fusion features and the aggregated template features to obtain comprehensive features, the feature synthesis module 440 can be used to: concatenate the multimodal fusion features and the aggregated template features to obtain concatenated features; and perform convolution processing on the concatenated features to obtain the comprehensive features.

[0070] In some embodiments of this application, after fusing the multimodal fusion features and the aggregated template features to obtain comprehensive features, the device further includes a location prediction module for: performing region-by-region similarity matching based on the comprehensive features to obtain the similarity between the regional features of different candidate regions in the comprehensive features and the target matching features; and obtaining the predicted target location based on the regional location of the candidate region corresponding to the largest similarity.

[0071] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0072] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown, specifically: The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 501 is the control center of the electronic device, connecting various parts of the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory 502, to perform various functions of the computer device and process data. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0073] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0074] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0075] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0076] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device can load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application.

[0077] For example, processor 501 can perform the following: extract multimodal target features from multimodal sensing data; perform cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; aggregate the first frame template features and dynamic template features to obtain aggregated template features; and fuse the multimodal fused features and the aggregated template features to obtain comprehensive features, which are used to predict the target location.

[0078] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0079] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0080] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0081] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0082] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0083] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0084] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A target tracking processing method, characterized in that, include: Multimodal target features are extracted from multimodal sensing data; The multimodal target features are subjected to cross-modal feature fusion processing to obtain multimodal fused features; The first frame template features and dynamic template features are aggregated to obtain aggregated template features; The multimodal fusion features and the aggregated template features are fused to obtain comprehensive features, which are used to predict the target location.

2. The method according to claim 1, characterized in that, The step of performing cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features includes: Cross-modal cross-attention calculation is performed on the multimodal target features to obtain cross-modal features; The cross-modal features are subjected to joint spatial dimension and channel dimension fusion enhancement processing to obtain the multimodal fusion features.

3. The method according to claim 2, characterized in that, The process of fusing and enhancing the cross-modal features using both spatial and channel dimensions to obtain the multimodal fused features includes: The cross-modal features are subjected to global average pooling to obtain compressed features; The compressed features are then subjected to convolution to obtain the channel weights; The spatial weights are obtained by calculating the local neighbor pixel association weights based on the spatial coordinates of the cross-modal features. The channel weights and spatial weights are multiplied point by point and then superimposed onto the cross-modal features to obtain the superimposed features; The multimodal fusion feature is obtained based on the superposition feature.

4. The method according to claim 3, characterized in that, The process of obtaining the multimodal fusion feature based on the superposition feature includes: The superimposed features are convolved to obtain the first feature; The convolutional features are added element-wise to the multimodal target features to obtain the second feature; The multimodal fusion feature is obtained based on the second feature.

5. The method according to claim 1, characterized in that, The process of aggregating the first frame template features and dynamic template features to obtain aggregated template features includes: The first frame template features and the dynamic template features are weighted and fused to obtain the fused template features; The fused template features are subjected to noise filtering to obtain the aggregated template features.

6. The method according to claim 1, characterized in that, The process of fusing the multimodal fusion features and the aggregated template features to obtain comprehensive features includes: The multimodal fusion feature and the aggregated template feature are concatenated to obtain the concatenated feature; The concatenated features are subjected to convolution processing to obtain the comprehensive features.

7. The method according to claim 1, characterized in that, After fusing the multimodal fusion features and the aggregated template features to obtain the comprehensive features, the method further includes: Based on the comprehensive features, region-by-region similarity matching is performed to obtain the similarity between the regional features of different candidate regions and the target matching features in the comprehensive features; The predicted target location is obtained based on the regional location of the candidate region corresponding to the largest similarity.

8. A target tracking processing device, characterized in that, include: The feature extraction module is used to extract multimodal target features from multimodal sensing data. The feature fusion module is used to: perform cross-modal feature fusion processing on the multimodal target features to obtain multimodal fused features; The feature aggregation module is used to aggregate the first frame template features and dynamic template features to obtain aggregated template features. The feature synthesis module is used to fuse the multimodal fusion features and the aggregated template features to obtain a comprehensive feature, which is used to predict the target location.

9. A storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to perform the method described in any one of claims 1 to 7.

Images