Cross-attention mechanism optimization method and system for multi-modal feature fine-grained alignment

Abstract

The invention relates to a cross-attention mechanism optimization method and a cross-attention mechanism optimization system for multi-modal feature fine-grained alignment. The method comprises the following steps: constructing a space-time topological graph of a dynamic scene, and representing a space-time relationship among intelligent agents through nodes and various connecting edges; coding the heterogeneous spatio-temporal information by using a plurality of groups of parallel recurrent neural networks, and converting the heterogeneous spatio-temporal information into uniform dimension feature representation; establishing a multi-modal cross attention alignment component, and quantifying feature quality from multiple dimensions through a multi-criterion evaluation unit; an adaptive weight fusion system is adopted to dynamically integrate evaluation results, and a unified quality score is generated; constructing a progressive optimization architecture based on the quality score, and performing multi-task cooperative training on a double-layer attention mechanism by combining real-time sampling and a directional optimization strategy; and finally generating a movement decision. According to the method, fine-grained alignment and optimization of multi-modal features are realized, and the accuracy and adaptability of intelligent agent navigation and trajectory prediction in a complex dynamic environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and robot navigation technology, and in particular to a multimodal feature alignment and optimization method based on deep reinforcement learning. Specifically, it is a cross-attention mechanism optimization method and system that integrates spatiotemporal graph modeling, attention mechanism and intrinsic reward for fine-grained alignment of multimodal features. Background Technology

[0002] With the rapid development of artificial intelligence technology, mobile robots are facing unprecedented challenges in navigating crowded environments. In complex and dynamic scenarios such as shopping malls, hospitals, and airports, robots need to avoid obstacles while considering human-computer interaction, adhering to social paradigms in the human world, maintaining an appropriate social distance from humans, and adopting suitable movement strategies.

[0003] Traditional robot navigation methods are primarily based on reactive obstacle avoidance algorithms, such as A* and Dijkstra's algorithm. These methods perform well in static or low-density environments, but often fail to effectively handle the dynamics and interactivity of human behavior in high-density crowd environments. Physical model-based methods, such as sociomechanical models, can describe interactions between pedestrians, but their performance is limited when dealing with complex interactions.

[0004] In recent years, deep reinforcement learning (DRL) methods have provided new solutions for robot navigation. By optimizing policies through interaction with the environment, DRL methods have demonstrated stronger environmental adaptability. However, existing methods still have the following limitations: First, they lack the ability to perceive pedestrian intentions, making it difficult to accurately predict the future trajectory of pedestrians; second, they are inefficient in exploring sparse reward environments and are prone to getting trapped in local optima; in addition, the alignment of multimodal features is not refined enough, making it difficult to fully utilize the implicit patterns in spatiotemporal information.

[0005] In pedestrian trajectory prediction, deep learning-based methods are gradually becoming mainstream. For example, Social-LSTM and graph neural networks improve prediction accuracy by modeling social interactions between pedestrians. However, existing prediction models still have limited ability to model indirect interaction relationships and are difficult to effectively coordinate with navigation decision modules.

[0006] Therefore, there is an urgent need for a robot navigation method that can effectively integrate spatiotemporal information, finely align multimodal features, and possess efficient exploration capabilities, in order to improve navigation performance and social compliance in complex dynamic environments. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a robot navigation method that can effectively integrate spatiotemporal information, achieve fine-grained alignment of multimodal features, and have efficient exploration capabilities.

[0008] In a first aspect, embodiments of this application provide a cross-attention mechanism optimization method for fine-grained alignment of multimodal features, the method comprising: Construct a spatiotemporal topology graph for a dynamic scene, wherein the nodes of the spatiotemporal topology graph represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges characterize the temporal evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities. By using multiple sets of parallel recurrent neural networks in the edge information encoder, different types of connection relationships are processed respectively, and heterogeneous spatiotemporal information is converted into a feature representation of a unified dimension. The connection relationships include at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges. Based on the unified dimension feature representation, a multimodal cross-attention alignment component is established, wherein the multimodal cross-attention alignment component includes a multi-criteria evaluation unit to perform quantitative analysis of feature quality from multiple perspectives; An adaptive weight fusion system is used to integrate the evaluation results of multiple criteria. A context-aware selection network is used to dynamically adjust the contribution of each evaluation criterion to generate a unified quality score. Based on the quality score, a progressive optimization architecture is established, and a multi-task collaborative training of the two-layer attention mechanism is performed by combining real-time sampling and targeted optimization strategies. The dual-layer attention mechanism includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction. The temporal feature integration module performs sequence modeling on the optimized multimodal features to generate the final mobility decision or path prediction.

[0009] Secondly, embodiments of this application provide a cross-attention mechanism optimization system for fine-grained alignment of multimodal features, applied to the cross-attention mechanism optimization method for fine-grained alignment of multimodal features as described in the first aspect, the system comprising: The spatiotemporal topology construction module is used to construct a spatiotemporal topology graph for dynamic scenes. In the spatiotemporal topology graph, the nodes represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges depict the time evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities. The edge information encoding module processes different types of connection relationships through multiple sets of parallel recurrent neural networks, converting heterogeneous spatiotemporal information into a unified dimension feature representation. The connection relationships include at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges. The multimodal alignment module establishes a multimodal cross-attention alignment component based on the unified dimension feature representation. The multimodal cross-attention alignment component includes a multi-criteria evaluation unit to quantitatively analyze feature quality from multiple perspectives. The weight fusion module uses an adaptive weight fusion system to integrate the evaluation results of multiple criteria. It dynamically adjusts the contribution of each evaluation criterion through a context-aware selection network to generate a unified quality score. The training module is optimized by establishing a progressive optimization architecture based on the quality score and combining real-time sampling and targeted optimization strategies to perform multi-task collaborative training on the two-layer attention mechanism. The dual-layer attention module includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction. The decision output module performs sequence modeling on the optimized multimodal features through the temporal feature integration module to generate the final movement decision or path prediction.

[0010] Thirdly, embodiments of this application provide an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to implement the cross-attention mechanism optimization method for fine-grained alignment of multimodal features as described in the first aspect when executing the instructions.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to perform the cross-attention mechanism optimization method for fine-grained alignment of multimodal features as described in the first aspect.

[0012] Beneficial effects:

[0013] 1. Improved accuracy and foresight in trajectory prediction. By constructing a spatiotemporal topological graph that integrates implicit edge factors and introducing a sparse attention mechanism based on manifold geometry and optimal transport theory, the complex and indirect spatial interaction relationships between pedestrians can be effectively captured, thereby significantly improving trajectory prediction accuracy and providing robots with more reliable pedestrian intent perception.

[0014] 2. Enhanced navigation decision-making capabilities in complex environments. By employing a hierarchical attention mechanism (combining a social attention module and an intent-aware attention module) and multi-task collaborative training, the robot can simultaneously understand macro-level group behavior and micro-level individual intents, making safer and more socially compliant navigation decisions in dense and dynamic scenarios.

[0015] 3. Solved the exploration challenge in sparse reward environments. An innovative intrinsic reward mechanism based on State Feature Encoder (SFE) was designed to incentivize the robot to actively explore unknown regions by quantifying the differences between adjacent states, effectively overcoming the problems of low exploration efficiency and easy getting trapped in local optima in traditional reinforcement learning in sparse reward scenarios.

[0016] 4. Fine-grained alignment and adaptive fusion of multimodal features were achieved. Through multi-criteria evaluation units (semantic, topological, informative, and redundancy) and a context-aware weight fusion system, refined evaluation and dynamic weighting of multimodal features were realized, improving the quality and robustness of feature alignment.

[0017] 5. An end-to-end joint optimization framework was constructed. The trajectory prediction and navigation decision modules are tightly coupled through a progressive optimization architecture to form a closed-loop optimization system from perception, prediction to decision-making, which significantly improves the coordination and efficiency of overall navigation performance. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a cross-attention mechanism optimization method for fine-grained alignment of multimodal features provided in an embodiment of this application.

[0019] Figure 2 The system architecture diagram for cross-attention mechanism optimization of fine-grained alignment of multimodal features provided in this application is shown.

[0020] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0022] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0023] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Example 1

[0025] Figure 1 This is a schematic diagram of a cross-attention mechanism optimization method for fine-grained alignment of multimodal features provided in an embodiment of this application. Figure 1 As shown, a cross-attention mechanism optimization method for fine-grained alignment of multimodal features includes: S110, Construct a spatiotemporal topology graph of the dynamic scene, wherein the nodes of the spatiotemporal topology graph represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges characterize the time evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities.

[0026] S120, through multiple sets of parallel recurrent neural networks in the edge information encoder, different types of connection relationships are processed respectively, and heterogeneous spatiotemporal information is converted into a feature representation of a unified dimension. The connection relationship includes at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges.

[0027] S130, Based on the unified dimension feature representation, a multimodal cross-attention alignment component is established, wherein the multimodal cross-attention alignment component includes a multi-criteria evaluation unit to perform quantitative analysis of feature quality from multiple perspectives.

[0028] S140 employs an adaptive weight fusion system to integrate the evaluation results of multiple criteria. It dynamically adjusts the contribution of each evaluation criterion through a context-aware selection network to generate a unified quality score.

[0029] S150. Based on the quality score, a progressive optimization architecture is established, and a multi-task collaborative training of the two-layer attention mechanism is performed by combining real-time sampling and targeted optimization strategies.

[0030] S160, the dual-layer attention mechanism includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction.

[0031] S170 uses a temporal feature integration module to perform sequence modeling on the optimized multimodal features, generating the final mobility decision or path prediction.

[0032] Specifically, in step S110, the spatiotemporal topology graph of the dynamic scene is a graph structure used to describe the relationships between intelligent agents (such as pedestrians and robots) in the spatial and temporal dimensions in the dynamic environment. It abstracts the complex dynamic environment into structured graph data, providing a foundation for subsequent feature extraction and relationship modeling.

[0033] Specifically, the implementation of constructing the spatiotemporal topology graph of the dynamic scene includes mapping the state information of each agent at time t to nodes in the topology graph. The state information includes at least position coordinates, velocity, direction of motion, and acceleration. Specifically, mapping the state information of each agent at time t to nodes in the topology graph realizes a graph structure representation of the agent's state, transforming the real-time state (position, velocity, direction, acceleration) of each agent in the dynamic environment into node features that can be processed by a graph neural network, providing basic data units for subsequent graph computation. For example, in an autonomous driving simulation scenario, at time t = 1 second, there is a car A, a car B, and a pedestrian C on the road. Then, the topology graph will have three nodes, representing the states of A, B, and C at 1 second. The data for node A might be a vector [x-coordinate, y-coordinate, velocity magnitude, direction of motion angle, acceleration].

[0034] Proximity edges are constructed to connect pairs of agents whose spatial distance within the same time step is less than a preset threshold. The weight of the proximity edge is related to the relative position and relative velocity between the agents. Specifically, proximity edges are constructed to connect pairs of agents whose spatial distance within the same time step is less than a preset threshold, thus establishing a spatial proximity relationship model. This is specifically used to capture real-time spatial interactions between agents, providing a basis for modeling local obstacle avoidance and close-range interactions, and controlling the computational complexity through the distance threshold. For example, if car A and car B are driving side by side on the road at t=1 second, with a very close distance (less than the preset 10-meter threshold), then a proximity edge is established between node A and node B. The weight of this edge can be calculated as 1 / (a scalar of relative distance + relative velocity), indicating that there is a high probability of mutual influence between them.

[0035] Temporal connection edges are established to connect the node states of the same agent at adjacent time steps. The features of the temporal connection edges include the displacement and velocity change of the agent per unit time. Specifically, establishing temporal connection edges to connect the node states of the same agent at adjacent time steps is used to construct a temporal continuity model. This is specifically used to maintain the temporal consistency of individual motion trajectories, capture the evolution of motion patterns, and provide historical context information for trajectory prediction. The edges connect the node states of the same agent at adjacent times (e.g., t and t+1). The features of the edges describe the changes in the agent's own state. For example, for car A, a temporal connection edge is established between the node at t=1 and the node at t=2. The feature vector of this edge can contain [displacement Δx, Δy, velocity change Δv from t=1 to t=2], which describes the motion evolution of car A itself.

[0036] By designing implicit connection edges and employing multi-hop spatial relationship propagation and temporal attention mechanisms, this approach captures potential interactions between non-directly adjacent agents. The weights of these implicit connection edges are calculated through inter-layer propagation in the graph convolutional network. This step is used to mine indirect interaction relationships, discover and model the mutual influence between non-directly adjacent agents, and expand the perception range through multi-hop propagation, thereby enhancing the understanding of complex group behavior. For example, at an intersection, car A is preparing to turn right, while pedestrian C is waiting on the crosswalk across the intersection. Their spatial distance may exceed 10 meters (not forming a proximity connection), but there is a potential interaction between car A's turning intention and pedestrian C's crossing intention. By analyzing the structure of the entire graph (e.g., car A -> traffic light -> pedestrian C, which is a multi-hop path), the graph convolutional network can calculate an implicit connection edge between A and C and assign a weight to it, representing their potential mutual influence.

[0037] A dynamic topology update mechanism is introduced to adaptively adjust the existence and weight coefficients of various connection edges based on the real-time motion state of agents in the scene, ensuring that the spatiotemporal topology graph accurately reflects the evolution of the dynamic scene. This step is used to achieve dynamic adaptation of the topology structure, reflect environmental changes in real time, automatically optimize the information content of the graph structure, and ensure that the spatiotemporal topology graph always accurately represents the current scene state. The entire graph is not static but is updated in real time with the passage of time and the movement of agents. The existence and weight of edges are adjusted according to the latest scene state. For example, initially, cars A and B are very close and have a strong proximity connection edge. Subsequently, A accelerates away from B, and when the distance exceeds a threshold, this edge may be removed or its weight reduced to a very low value. At the same time, when pedestrian C starts crossing the road, the weight of the implicit connection edge with car A may increase significantly because the risk of interaction increases.

[0038] Specifically, in step S120, the multiple parallel recurrent neural networks in the edge information encoder are encoding modules composed of multiple independent and parallel recurrent neural networks (such as GRU and LSTM), used to process different types of connection edge information. Heterogeneous spatiotemporal interaction information (proximity, temporal sequence, implicit association) is encoded into a feature representation of a unified dimension, which facilitates subsequent fusion and alignment.

[0039] Furthermore, in this embodiment, the specific implementation of processing connection relationships through multiple sets of parallel recurrent neural networks in the edge information encoder includes: setting up multiple sets of independent gated recurrent network units, each specifically processing the feature flows generated by adjacent connection edges, temporal connection edges, and implicit association edges. Each set of network units contains dedicated weight parameters that are not shared between them. Here, the multiple sets of independent gated recurrent network units refer to three sets of GRU (Gated Recurrent Unit) or LSTM networks with identical structures but non-shared parameters, specializing in processing different types of spatiotemporal relationships. For example, three independent GRU units: GRU(hidden_size) processes adjacent connection edges, GRU(hidden_size) processes temporal connection edges, and GRU(hidden_size) processes implicit association edges.

[0040] For the proximity connection edge feature, a spatial aggregation recurrent unit is employed. Its input is the weighted sum of the states of all neighboring nodes of the target node at the current time, and the hidden states are used to encode local spatial interaction patterns. Specifically, the spatial aggregation recurrent unit is used in a GRU network to specifically handle spatial proximity relationships and encode local spatial interaction patterns.

[0041] For temporal connection edge features, a Temporal Evolutionary Recurrent Unit (GRU) is employed. Its input includes the difference features between the current node state and the node states at historical time points, while the hidden states characterize the temporal continuity of individual motion. The Temporal Evolutionary Recurrent Unit is a GRU network specifically designed to handle temporal continuity, used to model the temporal evolution of individual motion. For example, it can predict the motion trend at the next moment based on the position changes of a pedestrian over the past five time steps.

[0042] For latent association edge features, a multi-hop propagation recurrent unit is employed. Its input is sampled and aggregated from node features within the multi-hop neighborhood through a graph attention mechanism, and the hidden state is used to represent potential long-range spatial dependencies. The multi-hop propagation recurrent unit refers to a GRU network that processes latent associations, combined with a graph attention mechanism, to capture long-range spatial dependencies.

[0043] The hidden states output by each recurrent unit are mapped to a unified feature space through a shared projection layer, forming a fixed-dimensional edge-type feature vector. The shared projection layer refers to a fully connected neural network layer whose weights are shared among the three GRUs, used to map the outputs of different GRUs to a unified feature space. This unifies spatial, temporal, and latent features into a 256-dimensional vector space.

[0044] A feature normalization layer is introduced to normalize the feature vectors of multiple edge classes, ensuring that features with different physical meanings are comparable in numerical distribution, thus providing a stable input foundation for subsequent multimodal alignment. The feature normalization layer, also known as LayerNorm or batch normalization, is used to standardize the distribution of features with different physical meanings. For example, it ensures that position features (unit: meters) and velocity features (unit: meters per second) are comparable in numerical range.

[0045] Specifically, in step S130, the multimodal cross-attention alignment component is an attention mechanism module used to interact and align features from different modalities (such as trajectory, vision, and sensor data). This enables fine-grained interaction between multimodal features, enhancing the complementarity and consistency of feature representations. The multi-criteria evaluation unit includes analysis modules for multiple evaluation dimensions, specifically including semantic association strength, topological matching degree, information content level, and data redundancy. It quantifies feature quality from multiple perspectives, providing fine-grained evaluation criteria for feature alignment.

[0046] The specific implementation of establishing a multimodal cross-attention alignment component and performing multi-criteria quantitative analysis includes: the multi-criteria evaluation unit contains multiple parallel feature analysis paths, which correspond to the evaluation dimensions of semantic association strength, topological matching degree, information content level and data redundancy degree, respectively.

[0047] The semantic association strength analysis pathway uses a cross-modal semantic projection layer to map features from different modalities to a shared semantic space, calculating the similarity and association metrics between feature vectors as a semantic consistency assessment. For example, in an autonomous driving scenario, there are visual modal features (pedestrian images seen by a camera) and motion modal features (pedestrian movement speed detected by radar). The cross-modal semantic projection layer maps these two different sources of features to the same semantic space. It calculates their cosine similarity in the shared space. A high similarity indicates semantic consistency between the visually identified pedestrian and the slow-moving object detected by the motion features, resulting in a high score. Conversely, if the visually identified pedestrian is a stationary vehicle but the motion features indicate high-speed movement, there is semantic inconsistency, leading to a low score.

[0048] The topology matching analysis pathway uses a graph structure comparison module to calculate the consistency between the topological adjacency relationships of features in the latent space and the connecting edges in the original spatiotemporal topology graph, thus assessing the ability to preserve structure. For example, in the original spatiotemporal topology graph, car A and car B have a strong connection (adjacent edge), while car A and pedestrian C have a weak connection (latent association edge). The graph structure comparison module checks whether the feature vectors of A and B remain similar in the latent space, and whether the feature vectors of A and C remain relatively far apart. If the adjacency relationships in the latent space are highly consistent with the original graph, it indicates that the features have well preserved the scene structure, resulting in a high score. If originally unrelated features become unusually close in the latent space, it indicates a loss of structural information, resulting in a low score.

[0049] The information content level analysis pathway analyzes the information entropy of feature distribution and the activation intensity of feature dimensions through the feature entropy calculation module, quantifying the total amount of information carried by the features. For example, given a 64-dimensional feature vector representing the state of an agent: Information entropy calculation: If the vector's values ​​are evenly distributed (high entropy value), it indicates rich information content. Activation intensity analysis: Examines which dimensions are significantly activated. For example: Dimensions 1-10: location information (high activation intensity), Dimensions 11-20: speed information (medium activation intensity), Dimensions 21-30: interaction intent (low activation intensity), Dimensions 31-64: almost zero (low information content). Combining entropy value and activation pattern, an information content score is given.

[0050] The data redundancy analysis pathway uses an autocorrelation analysis module to calculate the correlation and covariance between different dimensions of a feature, assessing the information redundancy within the feature. For example, analyzing an 8-dimensional feature vector: the correlation coefficient between dimension 1 and dimension 2 reaches 0.95 → high redundancy; the correlation coefficient between dimension 3 and dimension 4 is 0.1 → information independence; the correlation coefficients between dimensions 5, 6, and 7 are all >0.8 → severe redundancy. Through covariance matrix analysis, it is found that the 8 dimensions can only provide 4 independent pieces of information. The higher the redundancy, the lower the score.

[0051] Each analysis pathway outputs a standardized score, and multiple scores are weighted and fused to form the final feature quality assessment result. The weights of each pathway are dynamically adjusted according to the current task type and scenario. Each analysis pathway outputs a standardized score between 0 and 1, and the four scores are weighted and summed to form the final feature quality assessment result. This multi-criteria evaluation is equivalent to establishing a comprehensive assessment report for feature quality, ensuring from different dimensions that the extracted features are semantically accurate, structurally sound, information-rich, and free of redundancy, providing a high-quality feature foundation for subsequent decision-making.

[0052] Specifically, in step S140, the adaptive weight fusion system is a dynamic weight allocation mechanism that automatically adjusts the contribution of each evaluation criterion based on the context. This avoids the limitations of manually setting weights and improves the robustness and scene adaptability of feature fusion. The context-aware selection network is a neural network capable of perceiving the current environmental context information and is used to dynamically adjust the weights of the evaluation criteria. This ensures that the weight allocation matches the current task and scene features, enhancing the system's intelligence and flexibility.

[0053] Furthermore, the specific implementation of integrating the multi-criteria evaluation results using the adaptive weight fusion system includes: the context-aware selection network takes the joint representation of the current multimodal features as input, and generates a multi-dimensional weight vector through a multi-layer neural network, corresponding to the contribution of each evaluation criterion. Specifically, based on the specific situation of the current features, the importance of each of the four criteria (semantics, topology, information content, and redundancy) is dynamically determined. For example, in scenario A: the input features of a dense urban intersection include complex interactions of 10 cars and 5 pedestrians, and the network may output weights: [topology matching: 0.4, semantic association: 0.3, information content: 0.2, redundancy: 0.1], because structural relationships are most important in complex scenarios. In scenario B: the input features of a single vehicle tracking on a highway are mainly the visual and motion data of a single vehicle, and the network may output weights: [semantic association: 0.5, redundancy: 0.2, information content: 0.2, topology matching: 0.1], because semantic accuracy is more important in simple scenarios.

[0054] The calculation process of the weight vector incorporates an attention mechanism, dynamically capturing the relative importance of each evaluation criterion in the current feature context through query-key-value matching. For example, query (Q): the global feature density of the current scene, such as intersections, rainy days, and nighttime. Key (K): the semantic representation of the four evaluation criteria [structural relationship, semantic consistency, information richness, feature simplicity], and value (V): the basic importance of the four criteria. Through QK matching, it is found that structural relationship and semantic consistency are most relevant to the current scene, and the output is: assign higher weights to topological matching and semantic association.

[0055] A weight smoothing constraint module is set up to apply regularization constraints to the magnitude of weight changes between adjacent time steps, ensuring the stability of the contribution of the evaluation criteria over time. For example, the weight at time t is [0.4, 0.3, 0.2, 0.1], and the weight at time t+1 is [0.1, 0.5, 0.3, 0.1] ← such abrupt changes are not allowed. After smoothing constraints, it becomes [0.35, 0.35, 0.2, 0.1] ← a smooth transition.

[0056] A task-adaptive adjustment mechanism is established, preloading different weight bias parameters according to different task types in the input scenario to achieve task-driven evaluation strategy switching. For example, Task 1: Trajectory Prediction. Preloaded bias: +0.2 topology matching, +0.1 temporal consistency. Final weight bias: more focused on structural relationships. Task 2: Behavior Recognition. Preloaded bias: +0.3 semantic association, +0.1 information content. Final weight bias: more focused on semantic meaning. Task 3: Anomaly Detection. Preloaded bias: +0.2 redundancy, +0.2 information content. Final weight bias: more focused on feature anomaly patterns.

[0057] A confidence calibration layer is designed to dynamically adjust the final weights of each evaluation criterion based on the variance of the score distribution. Criteria with large score fluctuations have their weight contributions appropriately reduced. For example, semantic association strength score: [0.8, 0.75, 0.82, 0.79] ← small variance, stable and reliable; topological matching score: [0.9, 0.3, 0.85, 0.2] ← large variance, unstable. After confidence calibration: semantic association weight: ×1.2 (enhanced); topological matching weight: ×0.6 (weakened).

[0058] The scores of multiple evaluation criteria are integrated with their corresponding dynamic weights using a weighted fusion method to generate a unified quality score, which serves as the input signal for subsequent incremental optimization of the architecture. For example, the original scores are: Semantic Relevance: 0.8, Topological Matching: 0.6, Information Content: 0.9, Redundancy: 0.7. The dynamic weights are: Semantic Relevance: 0.3, Topological Matching: 0.4, Information Content: 0.2, Redundancy: 0.1. The final quality score is calculated as follows: 0.8 × 0.3 + 0.6 × 0.4 + 0.9 × 0.2 + 0.7 × 0.1 = 0.24 + 0.24 + 0.18 + 0.07 = 0.7.

[0059] Specifically, in step S150, the incremental optimization architecture is a phased, step-by-step optimization training framework, typically including pre-training, fine-tuning, and joint optimization stages. This improves training stability and efficiency, and avoids gradient conflicts and pattern collapse in multi-task collaborative training.

[0060] Specifically, the establishment of a progressively optimized architecture, combined with real-time sampling and targeted optimization strategies, enables multi-task collaborative training of the two-layer attention mechanism. The specific implementation methods include: The progressive optimization architecture employs a phased training strategy. In the first phase, the parameters of the group behavior perception unit are fixed, and individual intent parsing units are trained separately. In the second phase, the parameters of the group behavior perception unit are unfrozen, allowing for end-to-end joint fine-tuning. Specifically, in the first phase, the parameters of the group behavior perception unit are fixed (frozen), preventing it from participating in gradient updates and allowing only the individual intent parsing units to be trained independently. This allows the individual intent parsing units to learn to predict within a static or given group context, laying a solid foundation and avoiding overly complex joint optimization problems from the outset. In the second phase, the parameters of the group behavior perception unit are unfrozen, and it is jointly fine-tuned end-to-end with the pre-trained individual intent parsing units. This allows the two units to adapt to each other and work collaboratively. The group perception unit can adjust the accuracy of its macroscopic representation based on feedback from individual predictions, thereby optimizing overall performance.

[0061] The real-time sampling strategy dynamically adjusts the sample selection probability based on the unified quality score, prioritizing difficult samples in the medium quality score range for training, while storing a number of high-quality and low-quality samples. The unified quality score refers to the comprehensive score of the input feature quality generated by the aforementioned adaptive weight fusion system. Difficult samples are those with a medium quality score. These samples have feature quality that is neither extremely good nor extremely poor, and typically contain more complex, harder-to-learn, but more information-rich scene patterns. Specifically, in each training batch, samples are not randomly selected, but their selection probability is dynamically calculated based on their unified quality score. Difficult samples with medium quality scores are prioritized because they represent the focus and challenge of model learning, effectively improving model performance. A small number of high-quality and low-quality samples are retained to ensure the diversity of training data, prevent overfitting of the model to difficult samples, and maintain the stability of the training process.

[0062] The targeted optimization strategy designs differentiated loss functions for different task objectives. For navigation decision-making tasks, policy gradient optimization based on temporal difference error is used, while for trajectory prediction tasks, a weighted combination of negative log-likelihood loss and dynamic regularization distance is used. Targeted optimization strategy refers to designing different optimization objectives (loss functions) specifically for different downstream tasks. Navigation decision-making tasks refer to the task of an agent (such as an autonomous vehicle) deciding its next action (such as acceleration or steering). Trajectory prediction tasks refer to the task of predicting the future movement paths of other agents (such as surrounding vehicles and pedestrians). Specifically, for navigation decision-making tasks, policy gradient optimization based on temporal difference error is used. This is a reinforcement learning paradigm. The model output action is the policy, and the policy is updated by calculating the difference between the actual reward and the expected reward after executing the action (temporal difference error), enabling the agent to learn to make decisions that maximize long-term gains. For trajectory prediction tasks, a weighted combination of negative log-likelihood loss and dynamic regularization distance is used. Negative log-likelihood loss ensures that the predicted trajectory distribution is as close as possible to the true trajectory in a probabilistic sense. Dynamic regularization distance is a metric for aligning two sequences that may occur at different velocities. Here, it is used to directly measure the overall error between the predicted and actual trajectories in terms of shape and timing. Combining the two optimizes both prediction accuracy and temporal alignment.

[0063] The targeted optimization strategy employs a multi-scale trajectory alignment method based on the Wasserstein-Fisher-Rao metric for the trajectory prediction task as the loss function, specifically constructed as follows: Define the multi-scale temporal alignment loss function for trajectory prediction: Let the true trajectory sequence be... Predicting trajectory sequences ,in , Let be the position coordinates at time t; construct a trajectory distribution difference calculation based on the Wasserstein-Fisher-Rao (WFR) metric: , in, The cost function is a composite cost function, which includes position error and velocity error. The divergence is Kullback-Leibler. Wasserstein-Fisher-Rao loss value : Transmission plan, representing the quality transmission scheme from the predicted distribution to the actual distribution. The feasible domain of the transmission plan. : Probability distribution of predicted trajectory : The probability distribution of the true trajectory : These are the KL divergence regularization coefficient and the Fisher information regularization coefficient, respectively.

[0064] Multi-task collaborative training is achieved through a hard parameter sharing mechanism. Weights are shared at the encoder layer, while task-specificity is maintained at the decoder layer. Gradient clipping and weight normalization are used to avoid gradient conflicts between tasks. Specifically, weights are shared in the encoder layer (e.g., low-level feature extraction parts such as spatiotemporal topology construction and edge information encoding), allowing different tasks to learn common low-level feature representations. Task-specificity is maintained in the decoder layer (the part that outputs the final decision or prediction), with each task having its own dedicated network layer. Gradient clipping and weight normalization are used to avoid gradient conflicts or cancellations between different tasks, ensuring training stability.

[0065] A dynamic curriculum learning scheduler is established to automatically adjust the task difficulty coefficient based on the improvement curve of the quality score during training. Initially, single-task training is emphasized, gradually increasing the complexity of multi-task collaborative training. Dynamic curriculum learning is a training strategy that simulates the human learning process, learning from easy to difficult. The quality score improvement curve represents the trend of the model's performance on the validation set as it changes over the training period. Specifically, the scheduler monitors the improvement of the uniform quality score during training. Initial stage: Set a low task difficulty coefficient, perhaps focusing on only one main task or using simpler samples. As training progresses: When the model's performance improvement tends to plateau, the task difficulty coefficient is automatically increased, for example, by introducing more tasks for collaborative training or increasing the proportion of difficult samples, gradually increasing the complexity of training.

[0066] A meta-learning update mechanism is introduced. After each training cycle, the hyperparameters of the real-time sampling strategy and the targeted optimization strategy are rapidly and adaptively adjusted based on the validation set performance, enabling the optimization strategy to self-improve. Meta-learning, in this context, refers to the paradigm of learning how to learn, specifically the optimization training strategy itself. Specifically, after each training cycle, the system reflects on and adjusts itself based on the validation set performance. It rapidly and adaptively adjusts the hyperparameters of the real-time sampling strategy (e.g., the threshold for selecting difficult samples) and the targeted optimization strategy (e.g., the weights of each term in the loss function). This allows the entire training strategy to self-improve and become increasingly efficient.

[0067] An early stopping module is designed to automatically save the optimal model parameters and terminate the training process when the improvement in the quality score falls below a preset threshold over multiple consecutive training epochs. Specifically, the module continuously monitors changes in the uniform quality score on the validation set. If it finds that the improvement in the score is below the preset threshold (e.g., 0.1%) for multiple consecutive training epochs (e.g., 5 epochs), it determines that the model has converged and will no longer provide significant improvement. At this point, the system automatically saves the currently obtained optimal model parameters and terminates the training process early to save computational resources and avoid overfitting.

[0068] The steps described above depict a highly automated and intelligent modern deep learning training pipeline. It mitigates complexity through phased training, focuses on key samples through real-time sampling, precisely addresses different tasks through targeted optimization, improves efficiency and generalization through multi-task sharing, and allows the training process itself to continuously evolve through meta-learning and dynamic courses. Finally, an early stopping mechanism ensures training efficiency and quality. This entire methodology is the key guarantee that its models can achieve excellent performance in complex and dynamic scenarios.

[0069] Specifically, in step S160, the dual-layer attention mechanism is a dual-layer attention structure composed of a group behavior perception unit and an individual intent parsing unit. It models both macro-level group interaction and micro-level individual behavior, enhancing the ability to understand complex dynamic scenarios.

[0070] Furthermore, the specific implementation of the two-layer attention mechanism includes: The group behavior perception unit adopts a graph attention network architecture. Its input is the state features of all agents in the spatiotemporal topology graph. It calculates the interaction weights between nodes through a multi-head self-attention mechanism to generate a compact vector representation of group behavior. The group behavior perception unit is responsible for understanding the interaction relationships between all agents in the scene from a global perspective and generating a generalized scene representation. The graph attention network architecture is a neural network that processes graph-structured data and can calculate the importance weights between nodes in the graph. The spatiotemporal topology graph is the graph constructed as described in step 210 above, where nodes are agents and edges represent the relationships between them. The multi-head self-attention mechanism is a mechanism that allows the model to simultaneously pay attention to information from different representation subspaces, enabling richer computation of relationships between nodes. Specifically, the state features of all agents in the spatiotemporal topology graph are input into this unit. Internally, the unit uses a graph attention network to calculate the attention weight of each node (agent) to all other nodes (agents) through a multi-head self-attention mechanism. This weight represents the intensity of the interaction. For example, a car will pay more attention to the car in front of it than to the car on the roadside in the distance. Finally, the features of all nodes are weighted and aggregated according to the attention weights to generate a compact vector, which encodes the macroscopic interaction pattern (i.e., group behavior) of the entire scene.

[0071] The individual intent parsing unit takes the group behavior vector output by the previous unit and the historical trajectory of the current agent as input, and employs a causal masked attention mechanism to ensure that the prediction at the current moment depends only on historical information. Specifically, the individual intent parsing unit is responsible for predicting the future behavior or intent of a particular agent from an individual's perspective. The causal masked attention mechanism is a mask used in attention computation to ensure that when processing sequential data, the current moment can only see its previous historical information, not future information. This is crucial for prediction tasks. Specifically, this unit has two inputs: 1) the macro-scene context vector output by the group behavior perception unit; 2) the historical trajectory of the agent to be predicted. The unit internally uses a causal masked attention mechanism to process the historical trajectory. This means that when predicting the state at time t, the model can only be based on information from historical times t-1, t-2, ..., ensuring the causality and realism of the prediction. The output of this unit is a prediction of the individual's future state (such as position or intent).

[0072] A bidirectional information flow channel is established between the two units. The group behavior perception unit transmits the macro-scene context to the individual intent parsing unit, while the individual intent parsing unit feeds back the changes in the intent of key individuals to the group behavior perception unit. This bidirectional information flow channel serves as the data pathway connecting the group behavior perception unit and the individual intent parsing unit, allowing information to flow in both directions, forming a closed loop. Specifically, from top to bottom: the group behavior perception unit passes its output macro-scene context vector to the individual intent parsing unit. This allows individuals to consider the global environment when making decisions (e.g., the intersection is congested, so I should slow down). From bottom to top: the individual intent parsing unit feeds back the changes in the intent of key individuals it has parsed (e.g., detecting a pedestrian's intention to enter the lane) to the group behavior perception unit. This allows the group perception unit to dynamically adjust its macro-representation based on the abnormal behavior of key individuals.

[0073] A sparse attention mechanism is designed in the group behavior perception unit to reduce computational complexity by selecting and retaining the most important node connections. The sparse attention mechanism is an optimization technique that significantly reduces computation by filtering out most unimportant attention connections and retaining only the most critical ones. Specifically, in the group behavior perception unit, the calculated attention weight matrix is ​​usually dense (each node is connected to all other nodes). By using a sparse attention mechanism, for example, retaining only the top-K most important connections for each node, or discarding connections with weights below a certain threshold, a sparse attention matrix is ​​obtained. The computational complexity is significantly reduced from O(N²), enabling the model to handle scenarios with a large number of agents.

[0074] The sparse attention mechanism in the group behavior perception unit adopts an attention sparsity method based on manifold geometry constraints, which is implemented as follows: Define the attention metric function for node features on a Riemannian manifold: Suppose the features of nodes i and j are mapped to a Poincaré sphere model in d-dimensional hyperbolic space, and their attention scores are calculated based on the hyperbolic distance. , in, The original attention score of node i to node j. The query vector and key vector for nodes i and j in the Poincaré sphere model. Euclidean norm (vector magnitude). Feature dimension, used to scale the attention score. : Inverse hyperbolic cosine function, used to calculate hyperbolic distance.

[0075] Construct a curvature-adaptive sparsity-gated function: Define the manifold importance score of node i. Taking into account both local curvature changes and global topology preservation: , in, For hyperbolic distance, For the overall (global) curvature, The local curvature estimate for node i can be obtained by scalarization of the Gaussian curvature or Riemann curvature tensor of the local neighborhood. It is a hyperbolic cosine, used to map the distance to the positive number domain while taking into account the symmetry of the distance. is the feature vector of node i in the current layer (which can be regarded as the attention output or the aggregated representation). Let L1 be the L1 norm of the vector, which measures the sparsity of the features. Let Riemann gradient (gradient on manifold M) have Frobenius norm. Reflects the rate of change of features on the manifold. N: Total number of nodes in the graph.

[0076] Attention sparsity based on optimal transport theory is employed to construct the source distribution. and target distribution The Sinkhorn algorithm is used to solve for the regularized optimal transmission plan. : , in, : Optimal transmission plan matrix The feasible domain of the transmission plan. Dirac measure The amount of data transferred from node i to node j. Entropy regularization coefficient Entropy of the transmission plan , The final sparse attention weights are determined by the Hadamard product of the optimal transport plan and the original attention: , in, Attention weights after sparsification; Numerical stability constant to prevent division by zero errors; Adaptive threshold, used to determine which connections are preserved. Optimal transmission value.

[0077] Multi-scale temporal modeling is introduced into the individual intent parsing unit to capture short-term motion trends and long-term behavioral patterns through parallel temporal processing layers. Multi-scale temporal modeling utilizes parallel network layers with different time windows or parameters to simultaneously capture short-term and long-term patterns in the time series. Specifically, multiple temporal processing layers (such as temporal convolutional layers with different kernel sizes or GRU layers with different forget gate parameters) are connected in parallel within the individual intent parsing unit. One layer may focus on short-term motion trends (such as acceleration changes within the next 0.5 seconds), while another layer may focus on long-term behavioral patterns (such as the intention of overall movement direction within the past 5 seconds). Finally, these temporal features at different scales are fused to form a more comprehensive individual representation.

[0078] An inter-unit coordination controller is configured to dynamically adjust the output weights of two units based on the current scene complexity. In sparse scenes, the focus is on individual intent parsing, while in dense scenes, the focus is on group behavior perception. The inter-unit coordination controller is an intelligent switch or weighting module that dynamically determines the ratio of group behavior to individual intent information based on the current scene. Specifically, this controller takes the current scene complexity (such as the number of agents, average speed, and interaction density) as input. In sparse scenes (such as a highway with few cars): the output weight of the individual intent parsing unit is increased, emphasizing the individual's own movement patterns. In dense scenes (such as an urban intersection): the output weight of the group behavior perception unit is increased, emphasizing the impact of group interactions on individual behavior.

[0079] The hidden states of the two units are integrated through a gated fusion mechanism to generate the final two-layer attention feature representation. This gated fusion mechanism is similar to the gating mechanism in LSTM or GRU, used to control how information from different sources is fused. Hidden states are internal memory states retained and passed at each processing step in a recurrent neural network, encoding historical information about the sequence. Specifically, the hidden states of the two units are used as input. A gating signal (usually generated from the input data using a sigmoid function, with a value between 0 and 1) determines the weight of each state in the final output. The formula simplifies to: Final output = Gating signal * Group hidden states + (1 - Gating signal) * Individual hidden states. This method achieves adaptive and smooth fusion of the information flows from the two units.

[0080] Specifically, in step S170, the temporal feature integration module, which models and integrates time-series features, is typically based on recurrent neural networks or temporal convolutions. It captures temporal dependencies in dynamic scenes, generating smooth and consistent movement decisions or path predictions.

[0081] The input is optimized multimodal features. "Optimized" refers to features that have undergone all the aforementioned steps (spatiotemporal topology construction, edge information encoding, multimodal alignment, weight fusion, and two-layer attention mechanism optimization). These are high-quality, aligned, and context-rich refined features. Multimodal features refer to a unified feature representation that integrates spatiotemporal information from multiple sources and types. This may include: Geometric modality: the agent's precise position, velocity, and acceleration vectors. Interaction modality: vectors output from the group behavior perception unit, encoding interactions with other agents. Semantic modality: features output from the multimodal alignment component, carrying scene semantic information (e.g., avoiding obstacles, intending to turn left). Topological modality is the encoding of the agent's structural relationships within the spatiotemporal graph.

[0082] Core Processor: Time Series Feature Integration Module. The Time Series Feature Integration Module is a neural network module specifically designed for processing time series data. Its core task is to discover patterns in historical data and infer future states based on these patterns. Sequence Modeling: This refers to the way this module processes data. Instead of processing features at a single time point, it treats features from a series of time steps (t-τ, t-τ+1, ..., t) as a sequence to capture dynamic evolution. This module typically consists of a powerful sequence model. Common choices and implementations include: a) Variants based on Recurrent Neural Networks (RNNs): The model is a Gated Recurrent Unit (GRU) or a Long Short-Term Memory (LSTM) network. Optimized multimodal features from the past τ time steps are input into the GRU / LSTM in chronological order. The hidden states within the network act like memory units, accumulating and integrating historical information. The hidden state at the last time step contains condensed information about the entire sequence, used for final prediction. b) Temporal Convolutional Network (TCN): The model uses a one-dimensional convolutional network with causal convolutions and dilated convolutions. Treating the temporal feature sequence as a one-dimensional signal, the TCN slides across the time dimension through convolutional kernels, allowing for parallel computation and a long effective history length. c) Transformer-based Decoder: The model has a similar structure to GPT, but the input is a historical feature sequence. Causal masked self-attention ensures that when predicting time t, only information prior to time t is seen. It directly captures the dependencies between any two historical time points in the sequence through a self-attention mechanism. d) Hybrid Model: For example, an RNN is first used to pre-process the sequence, and then the output of the RNN is used as the input to a Transformer Decoder, combining the advantages of both.

[0083] The output is either a movement decision or a path prediction. Depending on the task, the module's end connects to different output headers, mapping the integrated timing information to specific outputs. For example, generating a movement decision (typically used for controlling autonomous agents) outputs a specific, discrete, or continuous action command. Or generating a path prediction (typically used for predicting the future trajectories of other agents) outputs a series of possible positions of the agent within a future timeframe.

[0084] Example 2

[0085] like Figure 2As shown, this application provides a cross-attention mechanism optimization system architecture diagram for fine-grained alignment of multimodal features, which is applied to the cross-attention mechanism optimization system for fine-grained alignment of multimodal features as described in Embodiment 1. It includes a spatiotemporal topology construction module 210, an edge information encoding module 220, a multimodal alignment module 230, a weight fusion module 240, an optimization training module 250, a two-layer attention module 260, and a decision output module 270.

[0086] The spatiotemporal topology construction module 210 is used to construct a spatiotemporal topology graph of a dynamic scene. The nodes of the spatiotemporal topology graph represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges depict the time evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities.

[0087] The edge information encoding module 220 processes different types of connection relationships through multiple sets of parallel recurrent neural networks, converting heterogeneous spatiotemporal information into a unified dimension feature representation. The connection relationships include at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges.

[0088] The multimodal alignment module 230 establishes a multimodal cross-attention alignment component based on the unified dimension feature representation. The multimodal cross-attention alignment component includes a multi-criteria evaluation unit to perform quantitative analysis of feature quality from multiple perspectives.

[0089] The weight fusion module 240 uses an adaptive weight fusion system to integrate the evaluation results of multiple criteria. It dynamically adjusts the contribution of each evaluation criterion through a context-aware selection network to generate a unified quality score.

[0090] The optimized training module 250 establishes a progressive optimization architecture based on the quality score, and combines real-time sampling and targeted optimization strategies to perform multi-task collaborative training on the two-layer attention mechanism.

[0091] The dual-layer attention module 260 includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction.

[0092] The decision output module 270 performs sequence modeling on the optimized multimodal features through the temporal feature integration module to generate the final movement decision or path prediction.

[0093] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0094] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0095] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0096] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0097] It should be noted that a portion of the electronic device described above can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0098] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0099] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0100] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0101] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A cross-attention mechanism optimization method for fine-grained alignment of multimodal features, characterized in that, Includes the following steps: Construct a spatiotemporal topology graph for a dynamic scene, wherein the nodes of the spatiotemporal topology graph represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges characterize the temporal evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities. By using multiple sets of parallel recurrent neural networks in the edge information encoder, different types of connection relationships are processed respectively, and heterogeneous spatiotemporal information is converted into a feature representation of a unified dimension. The connection relationships include at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges. Based on the unified dimension feature representation, a multimodal cross-attention alignment component is established, wherein the multimodal cross-attention alignment component includes a multi-criteria evaluation unit, which is used to perform quantitative analysis of feature quality from multiple perspectives; An adaptive weight fusion system is used to integrate the evaluation results of multiple criteria. A context-aware selection network is used to dynamically adjust the contribution of each evaluation criterion to generate a unified quality score. Based on the quality score, a progressive optimization architecture is established, and a multi-task collaborative training of the two-layer attention mechanism is performed by combining real-time sampling and targeted optimization strategies. The dual-layer attention mechanism includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction. The temporal feature integration module performs sequence modeling on the optimized multimodal features to generate the final mobility decision or path prediction.

2. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 1, characterized in that, The specific implementation methods for constructing the spatiotemporal topology graph of the dynamic scene include: The state information of each agent at time t is mapped to a node in the topology graph. The state information includes at least position coordinates, motion speed, motion direction and acceleration. Construct adjacent connection edges to connect agent pairs whose spatial distance is less than a preset threshold within the same time step. The weight of the adjacent connection edges is related to the relative position and relative velocity between the agents. Establish temporal connection edges to connect the node states of the same agent at adjacent time steps. The characteristics of the temporal connection edges include the displacement and velocity change of the agent per unit time. We design implicit connection edges and capture the potential interactions between non-directly adjacent agents through multi-hop spatial relationship propagation and temporal attention mechanisms. The weights of the multi-hop implicit connection edges are calculated through inter-layer propagation of the graph convolutional network. A dynamic topology update mechanism is introduced to adaptively adjust the existence and weight coefficients of various connection edges based on the real-time motion state of the agents in the scene.

3. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 1, characterized in that, The specific implementation method of processing different types of connection relationships through multiple sets of parallel recurrent neural networks in the side information encoder includes: Multiple independent gated recurrent network units are set up to specifically handle the feature flows generated by adjacent connection edges, temporal connection edges, and implicit association edges. Each network unit contains dedicated weight parameters that are not shared between them. For the adjacent connection edge feature, a spatial aggregation cyclic unit is used. Its input is the weighted sum of the states of all neighboring nodes of the target node at the current time. The hidden state is used to encode the local spatial interaction pattern. For temporal connection edge features, a temporal evolution recurrent unit is used. Its input includes the difference features between the current node state and the node state at historical moments. The hidden state is used to characterize the temporal continuity of individual motion. For latent association edge features, a multi-hop propagation recurrent unit is used. Its input is sampled and aggregated from the node features in the multi-hop neighborhood through a graph attention mechanism. The hidden state is used to represent potential long-distance spatial dependencies. The hidden states output by each recurrent unit are mapped to a unified feature space through a shared projection layer, forming a fixed-dimensional edge-type feature vector. A feature normalization layer is introduced to normalize the feature vectors of multiple edge classes.

4. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 1, characterized in that, The specific implementation methods for establishing a multimodal cross-attention alignment component and performing multi-criteria quantitative analysis include: The multi-criteria evaluation unit contains multiple parallel feature analysis paths, which correspond to the evaluation dimensions of semantic association strength, topological matching degree, information content level, and data redundancy degree, respectively. The semantic association strength analysis pathway maps features from different modalities to a shared semantic space through a cross-modal semantic projection layer, calculates the similarity and association measures between feature vectors, and uses this as an assessment of semantic consistency. The topology matching degree analysis pathway uses the graph structure comparison module to calculate the consistency between the topological adjacency relationship of features in the latent space and the connecting edges in the original spatiotemporal topology graph, and evaluates the structure preservation capability. The information content level analysis pathway analyzes the information entropy of feature distribution and the activation intensity of feature dimensions through the feature entropy calculation module, and quantifies the total amount of information carried by the feature. The data redundancy analysis pathway uses the autocorrelation analysis module to calculate the correlation and covariance between the dimensions of features and to assess the information redundancy within the features. Each analysis pathway outputs a standardized score, and multiple scores are weighted and fused to form the final feature quality assessment result, where the weight of each pathway is dynamically adjusted according to the current task type and scenario.

5. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 4, characterized in that, The specific implementation methods for integrating multi-criteria evaluation results using an adaptive weight fusion system include: The context-aware selection network takes the joint representation of the current multimodal features as input and generates a multidimensional weight vector through a multi-layer neural network, which corresponds to the contribution of each evaluation criterion. The calculation process of the weight vector introduces an attention mechanism, which dynamically captures the relative importance of each evaluation criterion in the current feature context through query-key value matching. A weight smoothing constraint module is set up to apply regularization constraints to the magnitude of weight changes between adjacent time steps, ensuring the stability of the contribution of the evaluation criteria over time. Establish a task adaptive adjustment mechanism to preload different weight bias parameters according to different task types in the input scenario, so as to perform task-driven evaluation strategy switching; Design a confidence calibration layer to dynamically adjust the final weight of each evaluation criterion based on the distribution variance of the scores, and appropriately reduce the weight contribution of criteria with large score fluctuations. By integrating the scores of multiple evaluation criteria with their corresponding dynamic weights through a weighted fusion method, a unified quality score is generated, which serves as the input signal for subsequent incremental optimization of the architecture.

6. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 5, characterized in that, The aforementioned progressive optimization architecture, combined with real-time sampling and targeted optimization strategies, enables multi-task collaborative training of the two-layer attention mechanism. Specific implementation methods include: The progressive optimization architecture adopts a phased training strategy. In the first phase, the parameters of the group behavior perception unit are fixed and the individual intention parsing unit is trained separately. In the second phase, the group behavior perception unit is unfrozen and end-to-end joint fine-tuning is performed. The real-time sampling strategy dynamically adjusts the sample selection probability based on the unified quality score, prioritizing the selection of difficult samples in the medium quality score range for training, while storing a number of high-quality and low-quality samples. The targeted optimization strategy designs differentiated loss functions for different task objectives. For navigation decision-making tasks, it adopts policy gradient optimization based on time difference error, and for trajectory prediction tasks, it adopts a weighted combination of negative log-likelihood loss and dynamic regularization distance. Multi-task collaborative training is performed through a hard parameter sharing mechanism, in which weights are shared in the encoder layer, task specificity is maintained in the decoder layer, and gradient conflicts between tasks are avoided through gradient clipping and weight normalization. Establish a dynamic course learning scheduler that automatically adjusts the task difficulty coefficient based on the improvement curve of the quality score during training. In the initial stage, focus on single-task training and gradually increase the complexity of multi-task collaborative training. A meta-learning update mechanism is introduced, and after each training cycle, the hyperparameters of the real-time sampling strategy and the directional optimization strategy are quickly and adaptively adjusted based on the performance of the validation set. The design includes an early stopping detection module that automatically saves the optimal model parameters and terminates the training process when the improvement in quality score is less than a preset threshold over multiple consecutive training cycles.

7. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 6, characterized in that, The specific implementation methods of the two-layer attention mechanism include: The group behavior perception unit adopts a graph attention network architecture. Its input is the state features of all agents in the spatiotemporal topology graph. It calculates the interaction weights between nodes through a multi-head self-attention mechanism to generate a compact vector representation of group behavior. The individual intent parsing unit takes the group behavior vector output by the previous unit and the historical trajectory of the current agent as input, and adopts a causal masking attention mechanism. A two-way information flow channel is established between the two units. The group behavior perception unit transmits the macro-scene context to the individual intention analysis unit, while the individual intention analysis unit feeds back the intention changes of key individuals to the group behavior perception unit. In the group behavior perception unit, a sparsity attention mechanism is designed to retain the most important node connections by filtering. Multi-scale temporal modeling is introduced into the individual intent parsing unit, and short-term motion trends and long-term behavioral patterns are captured through parallel temporal processing layers; Set up an inter-unit coordination controller to dynamically adjust the output weights of the two units based on the current scene complexity; The hidden states of the two units are integrated through a gating fusion mechanism to generate the final two-layer attention feature representation.

8. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 7, characterized in that, The design of a sparse attention mechanism in the group behavior perception unit employs an attention sparsity method based on manifold geometry constraints, specifically including: Map node features to a geometric model in hyperbolic space, and calculate attention scores between nodes based on hyperbolic distance; We construct a curvature-adaptive sparsity gating function and calculate the manifold importance score of nodes by comprehensively considering local curvature changes and global topology preservation. An attention sparsification method based on optimal transmission theory is used to construct the source and target distributions, and the transmission plan is solved by a regularized optimal transmission algorithm. Based on the optimal transmission plan and the original attention score, the final sparse attention weights are determined through a threshold filtering mechanism.

9. The method for cross-attention mechanism optimization of fine-grained alignment of multimodal features according to claim 6, characterized in that, The targeted optimization strategy employs a multi-scale trajectory alignment method based on the Wasserstein-Fisher-Rao metric for the loss function of the trajectory prediction task, specifically including: Define a multi-scale temporal alignment loss function for trajectory prediction, and construct a distribution difference assessment based on the real trajectory sequence and the predicted trajectory sequence; The Wasserstein-Fisher-Rao metric is used to calculate the trajectory distribution difference, combined with transmission plan optimization, divergence regularization, and information geometric constraints. A composite cost function is constructed that considers both position error and kinematic feature differences, and the velocity consistency requirement of the trajectory is incorporated into the measurement. The trajectory alignment quality is evaluated at different time scales through a multi-scale analysis mechanism.

10. A cross-attention mechanism optimization system for fine-grained alignment of multimodal features, applied to the cross-attention mechanism optimization method for fine-grained alignment of multimodal features as described in any one of claims 1 to 9, characterized in that, The system includes: The spatiotemporal topology construction module is used to construct a spatiotemporal topology graph for dynamic scenes. In the spatiotemporal topology graph, the nodes represent the state of the agent, the adjacent connection edges describe the spatial adjacency relationship, the temporal connection edges depict the time evolution law, and the implicit association edges explore the potential interaction between non-adjacent entities. The edge information encoding module processes different types of connection relationships through multiple sets of parallel recurrent neural networks, converting heterogeneous spatiotemporal information into a unified dimension feature representation. The connection relationships include at least one or more of the following: neighboring connection edges, temporal connection edges, and implicit association edges. The multimodal alignment module establishes a multimodal cross-attention alignment component based on the unified dimension feature representation. The multimodal cross-attention alignment component includes a multi-criteria evaluation unit to quantitatively analyze feature quality from multiple perspectives. The weight fusion module uses an adaptive weight fusion system to integrate the evaluation results of multiple criteria. It dynamically adjusts the contribution of each evaluation criterion through a context-aware selection network to generate a unified quality score. The training module is optimized by establishing a progressive optimization architecture based on the quality score and combining real-time sampling and targeted optimization strategies to perform multi-task collaborative training on the two-layer attention mechanism. The dual-layer attention module includes a group behavior perception unit and an individual intention parsing unit, which respectively handle macro-interaction patterns and micro-motion prediction. The decision output module performs sequence modeling on the optimized multimodal features through the temporal feature integration module to generate the final movement decision or path prediction.

Images