An Unreal Engine-based multi-level semantic interactive cross-modal tracking method

Abstract

The application provides a multilevel semantic interactive cross-modal tracking method based on an Unreal Engine. The method comprises the following steps: constructing a virtual simulation world by using the Unreal Engine 5; constructing pedestrian and vehicle virtual multi-target tracking data; constructing a text-track matching pair to generate multi-modal tracking data; constructing a multi-target tracking model for layer-by-layer fusion of multi-modal semantic features; enhancing the perception query feature by using the text feature; mapping the decoded perception feature to a semantic space by using a linear layer, and calculating the similarity with the encoded text feature; updating the target track information by using the perception query result; and the application not only solves the problem of missing track semantic data set, but also significantly improves the accuracy and recall rate of semantic target tracking in a complex dynamic environment by using the cross-modal alignment capability of the CLIP through the layer-by-layer semantic interaction module.

Description

Technical Field

[0001] This invention relates to the fields of signal processing technology and deep learning, specifically to a multi-level semantic interactive cross-modal tracking method based on Unreal Engine. Background Technology

[0002] In recent years, the field of visual tasks has witnessed a trend towards the integration of natural language description, giving rise to a series of innovative technologies. Semantic object tracking is one such task, aiming to open a new chapter in human-computer interaction: users communicate directly with the tracking system through natural language commands to accurately identify and track specified target objects in images or videos. This task presents a dual challenge: on the one hand, the system must possess the ability to deeply understand complex natural language; on the other hand, it also needs to be equipped with efficient image processing capabilities. Therefore, visual object tracking methods capable of fully understanding semantic commands are needed, which presents a significant challenge.

[0003] Despite its innovative nature, the availability of relevant datasets remains relatively limited. Typically, existing benchmark datasets are re-annotated versions of publicly available multi-object tracking benchmarks. This approach results in the accuracy of the new benchmark being affected by both the inherent limitations of the original benchmark and subjective biases inherent in manual annotation, thus impacting the performance of downstream tasks. To address this issue, this invention proposes a multi-level semantic interactive cross-modal tracking method based on Unreal Engine. This method overcomes the shortcomings of existing datasets by constructing a high-precision, low-cost benchmark dataset. Specifically, by leveraging Unreal Engine to generate a highly realistic virtual simulation environment, the reliance on manual annotation is eliminated, enabling large-scale dataset production. Furthermore, addressing the recall problem caused by insufficient semantic and textual fusion in existing technologies, this invention introduces an end-to-end multi-level guidance framework. This framework ensures effective integration of semantic information at every level from the autoencoder stage to the prediction head, significantly improving model performance in practical applications. Summary of the Invention

[0004] This invention provides a multi-level semantic interactive cross-modal tracking method based on Unreal Engine, characterized in that the method includes:

[0005] Step 1: Build a virtual simulation world using Unreal Engine 5, collect the trajectories of pedestrians and vehicles in the world, and generate multi-target tracking data;

[0006] Step 2 involves cleaning, filtering, labeling, and quality assessment of the target trajectory to ensure data accuracy and usability.

[0007] Step 3: Using a large language model, the appearance and motion descriptions of the markers are combined according to certain rules and associated with the target trajectory segments to obtain a high-quality, low-cost semantic tracking dataset.

[0008] Step 4: Construct a multi-target tracking model that fuses multimodal semantic features layer by layer;

[0009] Step 5: Video frames are cut and fed into the model. Two backbone networks are used to extract features from the image and text respectively, and an extended modal encoder is used to achieve feature fusion.

[0010] Step 6: First, initialize a set of detection queries and combine them with the previously output tracking queries to form a perception query. Then, apply text features to guide the perception query and input it together with the fused features into the decoder to capture semantic targets in the video.

[0011] Step 7: Use a linear layer to map the decoded query features to the semantic space, and compare their similarity with the encoded text features to filter out query results with high similarity.

[0012] Step 8: Use a multilayer perceptron to perform category classification and coordinate regression processing on the filtered query results.

[0013] Step 9: Update the trajectory tracking results, retain the perception query in the next frame and change it to a tracking query. Repeat steps 5 to 9 until tracking of all image sequences in the video is completed.

[0014] Specifically, in step 1, using publicly available resources in Unreal Engine 5, and modifying the appearance (color and size) of the 3D models, vehicle and crowd instances are generated within the system. These are then placed in the virtual world, and the internal vehicle and pedestrian movement rules are used to simulate an urban traffic system. Finally, cameras are placed in the world to record video footage, directly obtaining the coordinates and appearance information of the targets within the footage. By using this method, continuously modifying the 3D models of objects and camera positions, multi-target tracking data can be generated in batches.

[0015] Specifically, in step 2, the multi-target tracking data obtained in step 1 undergoes cleaning, filtering, description labeling, and quality assessment. This includes: firstly, filtering out trajectories severely obscured by buildings and scenes with few targets collected in step 1; secondly, describing and recording the motion of target trajectory segments; and finally, performing a quality assessment on the processed label data to select descriptions with longer durations and more obvious behaviors.

[0016] Specifically, in step 3, the motion descriptions annotated in step 2 are first combined with the appearance descriptions automatically generated by 3D modeling. Then, a large language model is used to derive multiple sets of synonyms from the combined words, and some common behavioral description phrases are manually selected. Finally, the corresponding description phrases are associated with trajectory segments, the start and end times of the trajectory are recorded, and text-trajectory matching pairs are generated. This method significantly reduces the cost of manual annotation and ensures the accuracy and objectivity of the dataset.

[0017] Specifically, in step 4, the model includes an image encoder, a frozen text encoder, a fusion encoder, a semantic guidance module, a tracking decoder, and a semantic relevance branch prediction module. The text encoder uses RoBERTa and CLIP, while both the fusion encoder and decoder employ a DETR structure containing multiple Transformer layers. The semantic guidance module takes a learnable query as its input to perceive different semantic features, with a size of M×N. dim , where M is the number of learnable queries;

[0018] Specifically, in step 5, firstly, the video generated in step 3 is segmented into frames to obtain an image sequence, and then a CNN-based backbone network is used to extract features from the image at time t to obtain the image pyramid feature I. t Next, the text instructions are encoded into a vector S using a frozen text encoder. Then, I is encoded into a vector S using a cross-modal fusion encoder. t Each layer is fused with S to generate cross-modal features E t ;

[0019] Specifically, in step 6, a set of detection queries is first initialized with learnable variables, denoted as... And combine it with the tracking query from the output of the previous frame. Generate a perceptual query, denoted as Q. t The semantic guidance module is used to interact with perceptual queries and semantic commands: Q... t It is added to its positional encoding and fed into the self-attention module to integrate Q. t The deep features; linearly mapping text features to the same dimension, the calculation formula is: S emb =Linear(S); then use cross-attention to apply it to Q. t Provide text guidance to generate perceptual queries with text prompts. The calculation formula is as follows: Finally, a fusion decoder is used to enable the perceptual query to capture the high-dimensional information of the target in the fused features, and the output is denoted as...

[0020] Specifically, in step 7, the result obtained in step 6 is processed through a linear layer. Mapped to the semantic feature space, the calculation formula is: The text instruction is re-encoded using the CLIP text encoder, and the calculation formula is: S clip =Encoder clip (S). And leveraging CLIP's powerful cross-modal alignment capabilities, calculations are performed. With S clip The similarity between the two modalities ultimately yields the relevance between each query and the text instruction, denoted as R. cos The calculation formula is: Furthermore, query results with a similarity greater than 0.5 are filtered out;

[0021] Specifically, in step 8, a multilayer perceptron is used to perform category classification and coordinate regression processing on the filtered query results to obtain the category information and coordinate information of the semantic target in the current frame;

[0022] Specifically, in step 9, the trajectory queue is updated using the results if the target is... If captured, a new trajectory is created; if by If captured, its corresponding old trajectory is updated. Finally, the perception query is performed. Record and update the tracking query for the next frame. Repeat steps 5 through 9 until all image sequences in the video have been tracked.

[0023] This invention addresses the challenges of multimodal tracking in real-world scenarios due to the lack of semantic descriptions of trajectories, and the limitations of existing semantic tracking methods that rely solely on a single cross-modal interaction layer to fuse textual and visual features, resulting in insufficient perception of different semantic targets and difficulty in tracking complex dynamic targets. To address these issues, a multi-layered semantic interactive cross-modal tracking method based on Unreal Engine 5 is proposed. By leveraging the 3D modeling capabilities of Unreal Engine 5, a simulated world is generated. A semi-automatic annotation tool enables high-quality, low-cost trajectory-semantic pairs, filling a gap in datasets in this domain. Simultaneously, a semantic guidance module facilitates interaction between perceptual queries and text, and CLIP's powerful cross-modal alignment capabilities align the fused features of the perceptual query with textual features. This layer-by-layer semantic fusion framework significantly improves the model's recall and tracking accuracy for semantic targets. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the steps of a multi-level semantic interactive cross-modal tracking method based on Unreal Engine. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] While innovative research in semantic object tracking is constantly emerging, the availability of datasets and the limitations of current technologies remain significant challenges. Traditionally, researchers create new benchmark datasets by adding annotations to existing public benchmarks. However, the effectiveness of this method is often limited by the limitations of the original dataset and subjective errors during manual annotation, thus affecting model performance. Furthermore, existing technologies often utilize only a single interaction layer to integrate textual and visual features, resulting in insufficient perception of semantic objects and difficulty in effectively tracking dynamic and complex scenes. This invention proposes a multi-level semantic interactive cross-modal tracking method based on Unreal Engine. First, Unreal Engine is used to generate a highly accurate and cost-effective benchmark dataset. An end-to-end multi-level guidance framework is also proposed. This framework significantly improves the model's ability to recognize and predict real-world scenes by effectively integrating semantic information across all levels from the autoencoder stage to the prediction head, thereby improving the overall system performance and recall. The specific steps are as follows:

[0028] S101 uses Unreal Engine 5 to generate a simulated world;

[0029] Specifically, various 3D models are selected from the Unreal Engine 5 public resource library as base materials. These models are then customized, altering their appearance attributes, such as adjusting the color and size of vehicles and people, to generate instances that meet system requirements. Vehicle models are configured in diverse ways, with different body colors and vehicle specifications set to match real-world vehicle types; character models have their clothing colors and body sizes changed to generate diverse character targets. Finally, these customized vehicle and people models are placed into a virtual city environment built using Unreal Engine 5 to generate a virtual simulation world.

[0030] S102 collects the trajectories of pedestrians and vehicles in the world and generates multi-target tracking data;

[0031] Specifically, select key observation points in the virtual world and place cameras there, ensuring that the cameras cover key traffic nodes and hotspot areas to capture sufficient dynamic information. Start video recording to collect the coordinates and appearance data of the target as it moves within the frame. Continue to change the camera's position, such as raising or lowering its shooting angle, or moving the camera to a new viewpoint. Repeat the above steps to expand the recording range, increase scenario diversity, and systematically collect different trajectory data for multi-target tracking.

[0032] S103, 3D modeling generates text descriptions, and human-calibrated trajectory motion descriptions;

[0033] Specifically, for the multi-target tracking trajectory label description text obtained in S102, a preliminary appearance text description containing basic information such as target type, color, and size is first constructed based on the target 3D model. Then, the motion mode of each target in the trajectory, such as straight-line movement, turning, stopping, and acceleration, is manually labeled to generate a preliminary dynamic text description.

[0034] S104 performs cleaning, filtering, and quality assessment on text-trajectory matching pairs.

[0035] Specifically, the text-trajectory matching data obtained from S103 is cleaned, filtered, and quality assessed. This includes: firstly, filtering and removing video data with incomplete trajectory information due to building obstruction; secondly, eliminating scenes with few targets or unclear behavioral patterns; and then, conducting a quality assessment on the remaining data, eliminating trajectories with unclear descriptions or short durations, and retaining only data with longer durations and clear behavioral patterns in rich contexts.

[0036] S105, using a large language model, combines the descriptions and associates them with trajectory fragments to obtain semantic tracking data;

[0037] Specifically, the appearance text descriptions and dynamic text descriptions annotated in step S103 are first randomly combined. For example, selecting "red" and pairing it with "straight ahead" yields the description "red straight-going vehicle." Then, a large language model is used to generate a series of synonyms for these combinations, and common behavioral description phrases are manually selected. Next, the selected description phrases are associated with the trajectory segments filtered in S104, and the start and end times of the trajectories are recorded to ensure the generation of accurate and diverse text-trajectory matching pairs, ultimately generating semantic tracking data. This method significantly reduces the cost of manual annotation and ensures the accuracy and objectivity of the dataset.

[0038] S106 utilizes two backbone networks to extract features and employs a cross-modal encoder to fuse image and text features;

[0039] Specifically, in the application, semantic instructions and image sequences are input into the model to track specific semantic targets within the image sequence. First, the image at time t is read from the image sequence, and then a CNN-based backbone network is used to extract features from the image at time t to obtain the image pyramid feature I. t Next, the text instructions are encoded into a vector S using a frozen text encoder. Then, I is encoded into a vector S using a cross-modal fusion encoder. t Each layer is fused with S to generate cross-modal features E t ;

[0040] S107 uses text to guide the detection and tracking of queries; semantically aware queries capture semantic targets in fused features through a decoder.

[0041] Specifically, we first initialize a set of detection queries with learnable variables, denoted as . And combine it with the tracking query from the output of the previous frame. Generate a perceptual query, denoted as Q. t The semantic guidance module is used to interact with perceptual queries and semantic commands: Q... t It is added to its positional encoding and fed into the self-attention module to integrate Q. t The deep features; linearly mapping text features to the same dimension, the calculation formula is: S emb =Linear(S); then use cross-attention to apply it to Q. t Provide text guidance to generate perceptual queries with text prompts. The calculation formula is as follows: Finally, a fusion decoder is used to enable the perceptual query to capture the high-dimensional information of the target in the fused features, and the output is denoted as...

[0042] S108 uses a linear layer to map the decoded features to the semantic space and calculates the similarity with the text features.

[0043] Specifically, the S107 obtained through a linear layer Mapped to the semantic feature space, the calculation formula is: The text instruction is re-encoded using the CLIP text encoder, and the calculation formula is: S clip =Encoder clip (S). And leveraging CLIP's powerful cross-modal alignment capabilities, calculations are performed. With S clip The similarity between the two modalities ultimately yields the relevance between each query and the text instruction, denoted as R. cos The calculation formula is: Furthermore, query results with a similarity greater than 0.5 are filtered out;

[0044] S109: For decoding categories and coordinates with high similarity, detect queries that generate new trajectories and track queries to update old trajectories;

[0045] Specifically, a multilayer perceptron is used to perform category classification and coordinate regression processing on the filtered query results to obtain the category and coordinate information of the semantic target in the current frame; the results are used to update the trajectory queue, if the target is... If captured, a new trajectory is created; if by If captured, its corresponding old trajectory is updated. Finally, the perception query is performed. Record and update the tracking query for the next frame.

[0046] S110, track until the end of the sequence and record the position information of all trajectories;

[0047] Specifically, repeat steps S106 to S110 until all image sequences in the video are tracked; the system will eventually record the trajectory information of all semantic targets in the video, including their movement paths and position changes over time.

[0048] This invention proposes a multi-layered semantic interactive cross-modal tracking method based on Unreal Engine. In application, it utilizes Unreal Engine 5 to construct a highly realistic simulation environment and leverages semi-automated annotation tools to generate accurate trajectory-semantic data pairs at low cost and high efficiency, effectively overcoming the limitations of existing technologies in terms of datasets. Secondly, semantic features guide the query, serving as a core hub to promote deep interaction between the perceptual query and the text. Furthermore, it combines the CLIP model's superior cross-modal alignment capabilities, ensuring high consistency between the fused features extracted from the perceptual query and the text features. This unique layer-by-layer semantic interaction mechanism significantly enhances the system's sensitivity and recall for detecting various semantic targets, while also improving overall tracking accuracy, enabling the model to more accurately complete target tracking tasks in complex dynamic scenarios.

Claims

1. A multi-level semantic interactive cross-modal tracking method based on Unreal Engine, characterized in that, The method includes the following steps: Step 1: Build a virtual simulation world using Unreal Engine 5, collect the trajectories of pedestrians and vehicles in the world, and generate multi-target tracking data; Step 2 involves cleaning, filtering, labeling, and quality assessment of the target trajectory to ensure data accuracy and usability. Step 3: Using a large language model, the appearance and motion descriptions of the markers are combined according to certain rules and associated with the target trajectory segments to obtain a high-quality, low-cost semantic tracking dataset. Step 4: Construct a multi-target tracking model that fuses multimodal semantic features layer by layer; Step 5: Video frames are cut and fed into the model. Two backbone networks are used to extract features from the image and text respectively, and an extended modal encoder is used to achieve feature fusion. Step 6: First, initialize a set of detection queries and merge them with the previously output tracking queries into a perception query. Then, apply text features to guide the perception query and input it together with the fused features into the decoder to capture semantic targets in the video. Step 7: Use a linear layer to map the decoded query features to the semantic space, and compare their similarity with the encoded text features to filter out query results with high similarity. Step 8: Use a multilayer perceptron to perform category classification and coordinate regression processing on the filtered query results; Step 9: Update the trajectory tracking results, retain the perception query in the next frame and change it to a tracking query. Repeat steps 5 to 9 until tracking of all image sequences in the video is completed.

2. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, in step 1, a virtual simulation world is built using Unreal Engine 5, and the trajectories of pedestrians and vehicles in the world are collected to generate multi-target tracking data. Specifically, this involves: using publicly available resources in Unreal Engine 5 and changing the appearance of the 3D models, such as color and size, to generate vehicle and crowd instances in the system; then placing them in the virtual world and using internal vehicle and pedestrian movement rules to simulate the urban traffic system; finally, placing cameras in the world to record video footage, and the coordinate and appearance information of the targets in the footage can be directly obtained; by using this method, the 3D modeling of objects and camera positions are continuously changed to generate multi-target tracking data in batches.

3. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 2, the target trajectory is cleaned, filtered, labeled, and quality assessed to ensure the accuracy and usability of the data. Specifically, the following steps are taken: First, the trajectories that are severely obscured by buildings and scenes with few targets collected in step 1 are filtered out. Second, the motion of the target trajectory segments is described and recorded. Finally, the quality of the processed label data is assessed, and descriptions with longer durations and obvious behaviors are selected.

4. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 3, the motion description labeled in step 2 is combined with the appearance description automatically generated by 3D modeling; then, a large language model is used to derive multiple sets of synonyms from the combined words, and some common behavioral description phrases are manually selected; finally, the corresponding description phrases are associated with the trajectory fragments, the start and end times of the trajectory are recorded, and text-trajectory matching pairs are generated.

5. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 4, a multi-target tracking model that fuses multimodal semantic features layer by layer is constructed. Specifically, it includes an image encoder, a frozen text encoder, a fusion encoder, a semantic guidance module, a tracking decoder, and a semantic relevance branch prediction module. The text encoders selected are RoBERTa and CLIP. Both the fusion encoder and fusion decoder adopt a DETR structure, which contains multiple Transformer layers. The semantic guidance module's input is set with a learnable query to perceive different semantic features, with a size of [size missing]. ,in The number of queries that can be learned.

6. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 5, the video generated in step 3 is first segmented into frames to obtain an image sequence, and then a CNN-based backbone network is used to process the sequence. Image pyramid features are obtained by extracting features from images at different times. Next, the text instructions are encoded into vectors using a frozen text encoder. Then, the cross-modal fusion encoder is used to... Each layer and Fusion to generate cross-modal features .

7. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 6, a set of detection queries is first initialized with learnable variables, denoted as . And combine it with the tracking query from the output of the previous frame. Generate a perceptual query, denoted as ; The semantic guidance module facilitates interaction between perceptual queries and semantic commands: It is added to its positional encoding and fed into the self-attention module for integration. The deep features; linearly mapping text features to the same dimension, the calculation formula is: Then use cross-attention to... Provide text guidance to generate perceptual queries with text prompts. The calculation formula is as follows: Finally, a fusion decoder is used to allow the perceptual query to capture the high-dimensional information of the target in the fused features, and the output is denoted as... .

8. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 7, the solution obtained in step 6 is processed through a linear layer. Mapped to the semantic feature space, the calculation formula is: The CLIP text encoder is used to re-encode text instructions. The calculation formula is as follows: And leveraging CLIP's powerful cross-modal alignment capabilities, it calculates... and The similarity between the two modalities ultimately yields the relevance between each query and the text instruction, denoted as . The calculation formula is: Furthermore, query results with a similarity greater than 0.5 are filtered out.

9. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 8, a multilayer perceptron is used to perform category classification and coordinate regression on the filtered query results to obtain the category information and coordinate information of the semantic target in the current frame.

10. The multi-level semantic interactive cross-modal tracking method based on Unreal Engine as described in claim 1, characterized in that, In step 9, the trajectory tracking results are updated, and the perception query is retained until the next frame, becoming a tracking query. Steps 5 to 9 are repeated until tracking of all image sequences in the video is completed. Specifically, the trajectory queue is updated using the results. If the target is... If captured, a new trajectory is created; if by If captured, its corresponding old trajectory is updated; finally, the perception query is performed. Record and update the tracking query for the next frame. Repeat steps 5 through 9 until all image sequences in the video have been tracked.

Images