Scene graph robust representation learning method based on uncertainty-guided causal intervention

Abstract

The application discloses a scene graph robust representation learning method based on uncertainty guidance causal intervention. For a target image sequence, a trained scene graph robust representation model is used to obtain a risk prediction result, wherein the scene graph robust representation model comprises: a target feature construction module for converting the target image sequence into a structured entity feature representation; a variational scene graph generation module for performing entity layer variational modeling and relationship layer variational modeling on the structured entity feature representation to obtain a probabilistic scene graph; an uncertainty perception causal intervention module for performing soft intervention correction on an environment-related bias in a representation space to obtain a corrected relationship representation; a differentiable structure learning module for constructing a sparse scene graph through a sparsification strategy; and a time sequence aggregation and prediction module for obtaining a risk prediction result through graph-level feature aggregation and cyclic time sequence modeling based on the sparse scene graph. The application can improve the reliability of an automatic driving scene understanding system in a complex environment.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving and machine learning technology, and more specifically, to a scene graph robust representation learning method based on uncertainty-guided causal intervention. Background Technology

[0002] In autonomous driving scenario understanding and risk prediction tasks, scenario graphs, as a structured representation, depict spatial relationships and semantic interactions in traffic scenarios by representing traffic participants as nodes and interaction relationships as edges. They have good structural expressive power and interpretability.

[0003] In existing technologies, data-driven scene graph generation methods mainly rely on statistical correlation for relationship modeling. While they perform well when training data distribution is stable, they tend to over-rely on environmental features when environmental conditions change (such as weather, lighting, and map topology changes), leading to significant performance degradation in out-of-distribution scenarios. Some existing solutions introduce uncertainty modeling or causal inference mechanisms to improve robustness, but uncertainty is typically only used for smoothing prediction results or estimating confidence levels, not for guiding graph structure optimization and correction. Causal mechanisms often rely on fixed structures or external priors, lacking trainable and differentiable intervention mechanisms, making it difficult to effectively suppress spurious relationships driven by the environment during the representation learning stage. Furthermore, existing scene graphs typically construct relatively dense topologies, with numerous low-confidence relationship edges propagating environmental noise in the graph neural network, further weakening the model's cross-domain generalization ability.

[0004] In summary, it is necessary to propose a novel scene graph representation learning method that can identify and suppress spurious relationships related to the environment while maintaining prediction performance, and induce the formation of a more sparse, stable structural representation with cross-domain generalization ability, so as to improve the reliability of autonomous driving scene understanding systems in complex environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a robust representation learning method for scene graphs based on uncertainty-guided causal intervention. This method includes the following steps: Obtain the target image sequence; For the target image sequence, a risk prediction result is obtained using a scene graph robust representation model trained end-to-end. The robust scene graph representation model includes a target feature construction module, a variational scene graph generation module, an uncertainty-aware causal intervention module, a differentiable structure learning module, and a temporal aggregation and prediction module. The target feature construction module converts target image sequences into structured entity feature representations. The variational scene graph generation module performs entity-level variational modeling and relation-level variational modeling on the structured entity feature representations to obtain a probabilistic scene graph containing a relational representation of mean and uncertainty information. The uncertainty-aware causal intervention module uses a set of learnable prototypes to perform soft intervention correction on environment-related biases at the representation space level to obtain a corrected relational representation. The differentiable structure learning module constructs a sparse graph topology structure from the corrected relational representation using relation scoring and sparsification strategies to obtain a sparse scene graph. The temporal aggregation and prediction module obtains risk prediction results based on the sparse scene graph through graph-level feature aggregation and cyclic temporal modeling.

[0006] Compared with the prior art, the advantages of the present invention are that the scene graph robust representation learning method based on uncertainty-guided causal intervention provides a graph structure representation learning method for autonomous driving scene understanding. By combining variational uncertainty modeling with prototype-driven causal intervention for domain robust scene graph construction and reasoning, a sparser, stable structure representation with cross-domain generalization ability is formed while maintaining prediction performance, thereby improving the reliability of the autonomous driving scene understanding system in complex environments.

[0007] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0009] Figure 1 This is a flowchart of a scene graph robust representation learning method based on uncertainty-guided causal intervention according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a scenario graph robust representation learning method based on uncertainty-guided causal intervention according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the uncertainty perception causal intervention process according to an embodiment of the present invention. Detailed Implementation

[0010] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0011] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0012] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0013] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0014] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0015] See Figure 1 As shown, the proposed scene graph robust representation learning method based on uncertainty-guided causal intervention includes the following steps: Step S1: Construct a robust representation model of the scene graph.

[0016] Combination Figure 2 As shown, the robust scene graph representation model comprises a target feature construction module, a variational scene graph generation module, an uncertainty-aware causal intervention module, a differentiable structure learning module, and a temporal aggregation and prediction module. Overall, for the robust scene graph representation model architecture, the target feature construction module first converts the original image sequence into structured entity feature representations. Then, in the variational scene graph generation module, probabilistic modeling of entities and their relationships is performed to obtain a relationship representation containing mean and uncertainty information. Based on this, the uncertainty-aware causal intervention module introduces a set of learnable prototypes to perform soft intervention correction at the representation space level for environment-related biases. The corrected relationship representation is further input into the differentiable structure learning module, which constructs a stable sparse graph topology through relationship scoring and sparsification strategies to obtain a sparse scene graph. After obtaining the sparse scene graph, an uncertainty-weighted graph inference mechanism is used for relationship propagation and feature aggregation, and the temporal modeling module is combined to fuse information from multiple frames, ultimately outputting the risk prediction result and the corresponding uncertainty estimate. The entire model architecture is trained end-to-end through joint optimization to achieve synergistic optimization of structural stability, prediction accuracy, and uncertainty calibration.

[0017] The following section will detail the structure of each module in the scene graph robust representation model and the overall training process.

[0018] 1. Target Feature Construction Module.

[0019] First, the target feature construction module transforms raw image observations into structured target representations, providing the foundational input for subsequent scene graph construction. For the input image frame... The system obtains a set of target instances using an object detector, each instance containing bounding box coordinates and a class label. To ensure consistent feature scale across different resolutions, the geometric information is normalized. Each target is encoded as a fixed-dimensional feature vector. (1) in, For image width and height; These are the left, top, right, and bottom coordinates of the target bounding box, respectively. For target category The one-hot encoded vector. This feature vector contains both geometric structural information and semantic category information of the target, enabling subsequent relationship modeling to characterize both spatial location relationships and interaction features between different categories.

[0020] 2. Variational scene graph generation module.

[0021] Based on structured target features, a probabilistic scene graph representation is constructed. The variational scene graph generation module uses variational modeling to probabilistically model entities and their relationships, obtaining relationship representations that include mean and uncertainty information, thus generating a probabilistic scene graph. Unlike traditional deterministic embedding, variational modeling can simultaneously learn the mean and uncertainty representations of entities and relationships, enabling each node and edge to have the ability to express a distribution, thereby providing a computational basis for subsequent uncertainty perception interventions.

[0022] S21, Entity Layer Variational Modeling For each target entity Its structured feature vector is To characterize the latent semantic state of the entity under the current observation conditions, a diagonal Gaussian distribution is introduced as its posterior representation: (2) in, Representing entities The latent embedding vector, For entities Embedded mean, Let be the diagonal covariance vector. Indicates by parameters The control encoding network. The mean and log-variance are calculated using a neural network: (3) in and For a trainable mapping function, a three-layer stacked MLP (Multilayer Perceptron) can be used. Next, to achieve differentiability, a reparameterization technique is introduced: (4) in This represents element-wise multiplication. It is a standard normally distributed random noise, mainly used to transform the random sampling process into a differentiable form through reparameterization techniques, helping the model to perform backpropagation training. In this way, the entity representation not only includes the semantic feature mean but also the data dependency uncertainty at the observation level, providing richer information for subsequent relationship modeling.

[0023] Step S22, Relational Layer Variational Modeling After obtaining entity embedding and Then, for any entity pair Construct a probabilistic representation of its relation embedding. The relation posterior also adopts a diagonal Gaussian form. (5) in, Representing entities With entity Relationship embedding, The mean of the relationship, For the variance of the relationship, For parameters The relational encoding network. The mean and log-variance are calculated by concatenating entity embeddings: (6) in, This represents the function for calculating the mean of the relation embedding. The calculation functions representing the variance of relational embeddings are all implemented by a three-layer MLP neural network. This represents a vector concatenation operation. Subsequent reparameterization is then used for sampling. (7) in, It is a standard normally distributed random noise, mainly used to transform the random sampling process into a differentiable form through reparameterization techniques, aiding in backpropagation training of the model. Uncertainty in the relational layer. This reflects the degree of fluctuation in the expression of this relationship under the current observation conditions.

[0024] Unlike traditional graph representation methods, this invention explicitly models uncertainty, enabling relational representations to possess not only semantic expressive power but also stability measurement capabilities. Through two-layer variational modeling at the entity and relation layers, this module outputs a probabilistic scene graph representation containing mean and uncertainty information, providing a foundational input for the subsequent uncertainty-aware causal intervention module.

[0025] 3. Uncertainty perception causal intervention module.

[0026] The uncertainty-perceived causal intervention module uses the relational uncertainty obtained from variational modeling as an intervention intensity control signal, and then combines it with a set of learnable environmental prototypes to perform residual correction on the relational representation, thereby suppressing spurious relations related to the environment at the representation space level.

[0027] The uncertainty-perceiving causal intervention module is the core technology module of this invention, used to perform soft intervention correction on potential environment-driven biases at the relation representation level, such as... Figure 3 As shown. Unlike traditional methods that directly use relation embeddings for graph reasoning, this invention first uses the relation uncertainty obtained from variational modeling as an intervention intensity control signal, and then combines it with a set of learnable environmental prototypes to perform residual correction on the relation representation, thereby suppressing spurious relations related to the environment at the representation space level.

[0028] Step S31, Scaling up relational uncertainty In relational variational modeling, each relation Corresponding to an uncertainty parameter in vector form To facilitate subsequent calculations of intervention intensity, it needs to be converted into scalar form. Let the relation embedding dimension be... Then the scalar definition of the uncertainty of the relationship is: (8) in, Indicates the first The variance of the dimension, Equivalent to standard deviation, this operation averages across all dimensions to obtain a measure of overall uncertainty at the relational level. This scalar reflects the degree of volatility represented by the relation under current observation conditions; a larger value indicates that the relation is more unstable under different observational perturbations.

[0029] Step S32, Intervention intensity gating function To achieve adaptive intervention, a gating function is introduced. Mapping uncertainty to intervention intensity: ,in, , It uses a multilayer perceptron network and constrains through the sigmoid output. This design makes it suitable for situations with high uncertainty in the relationships. Larger, when the relationship is relatively stable The value is close to zero. Therefore, the intensity of intervention is positively correlated with the degree of relationship instability.

[0030] Step S33, Environment Prototype Space Construction Since real-world environments are difficult to model directly, this invention introduces a set of learnable prototypes to discretize and approximate potential environmental patterns. The prototype set is defined as follows: (9) in, , The preset number of prototypes, which can be 32, is used. Each prototype represents a potential environment-driven pattern. (Regarding the mean...) The degree of matching between the model and each prototype is calculated using an attention mechanism. First, the similarity weights are calculated: (10) in, Indicates the first A prototype.

[0031] Then, the environmental bias estimate is calculated: (11) This step maps continuous potential environmental impacts to a weighted combination of finite prototypes, enabling a computationally approximate approximation of environmental biases.

[0032] Step S34, Residual Causal Intervention Correction In obtaining intervention intensity Environmental bias estimation Then, residual correction is applied to the relational mean: (12) Then resampling was performed: (13) When the relationship is highly uncertain, intervention items This enhancement mechanism provides a stronger correction to environment-driven biases. When the relationship is stable, the intervention term approaches zero, without disrupting the original stable representation. The correction process is completed within the representation space, maintaining end-to-end differentiability. Through this uncertainty-aware causal intervention mechanism, this module can proactively weaken environment-driven spurious correlations at the relation representation level, providing a more stable input representation for subsequent structural sparsity and robust graph reasoning.

[0033] 4. Differentiable structure learning module.

[0034] The differentiable structure learning module is used to adaptively filter and sparsify the scene graph topology after the relation representation is corrected. For example, by using a learnable scoring function and uncertainty information in synergy, the scene graph topology is adaptively filtered and sparsified to obtain a sparse scene graph structure.

[0035] The core objective of the differentiable structure learning module is to select stable and discriminative relation edges from all candidate relations, suppressing unstable edges with low confidence or driven by the environment, thereby constructing a sparse and stable graph structure. Unlike traditional methods that use fixed thresholds or manual rules to construct graph structures, this module achieves an end-to-end differentiable structure learning process through the synergistic effect of a learnable scoring function and uncertainty information.

[0036] Step S41, Relationship Confidence Score The mean of the relationship after causal intervention correction First, calculate its corresponding confidence score: (14) in, It is a trainable scoring network, specifically composed of two MLP layers. Scalar, representing a relation The degree of importance. This scoring function is used to evaluate the contribution of the relation representation to the current task.

[0037] Step S42, Local sparsification based on Top-K To prevent the graph structure from becoming too dense, for each node Retain the highest-rated ones Relationship: (15) in, Represents a node The set of neighbors to be retained A preset neighbor limit is set. This mechanism ensures that each node retains only a limited number of high-confidence relationships, thereby limiting the complexity of the graph structure.

[0038] Step S43, Global Threshold Supplement Mechanism In addition to Top-K constraints, a global threshold can also be introduced. Relationships with confidence levels exceeding a threshold are retained: (16) in, The final set of sparse relations to be retained. A pre-set confidence threshold is used, for example, 0.6. Through a dual mechanism of Top-K and threshold, local sparsity is guaranteed while avoiding the omission of globally high-confidence relationships.

[0039] Step S44, Uncertainty-Weighted Message Propagation To further suppress the impact of highly uncertain relationships on the aggregation result during information propagation in graph neural networks, an uncertainty weighting coefficient is introduced for each retained edge: (17) in, Due to the uncertainty of the relationship, To prevent the use of tiny constants that divide by zero, this weight is used as an edge weight in the relation graph convolution calculation for information aggregation. This reduces the propagation weight of unstable relations and enhances the propagation weight of stable relations. Through the above scoring, filtering, and weighting mechanisms, this module achieves dynamic sparsification of the scene graph topology while maintaining structural learnability. This effectively reduces the propagation impact of environment-driven relations and provides a stable graph structure input for the subsequent temporal reasoning module.

[0040] 5. Time Series Aggregation and Prediction Module The temporal aggregation and prediction module is used to perform temporal modeling and risk prediction on multi-frame graph representations after constructing a sparse scene graph structure. For example, for a sparse scene graph structure, graph-level feature aggregation and cyclic temporal modeling mechanisms are used to fuse continuous frame scene graphs, and finally output the prediction results and their uncertainty estimates. Autonomous driving scenarios typically have significant temporal continuity; the risk state is not determined by a single frame but is formed by the dynamic evolution of multiple time steps. Therefore, this module uses graph-level feature aggregation and cyclic temporal modeling mechanisms to fuse continuous frame scene graphs, and finally output the prediction results and their uncertainty estimates.

[0041] Step S51, construct graph-level representation For each time step After completing the message propagation in the graph neural network, a set of node representations is obtained. To obtain the overall scene representation of the current frame, a global aggregation is performed on all nodes: (18) in, To replace the invariant aggregation function, global average pooling is employed in one implementation. Indicates the first The graph-level feature vector of the frame. This operation compresses node-level information into a scene-level representation while preserving global interaction structure features.

[0042] Step S52, Modeling Temporal Dynamics To characterize the dynamic evolution of risk states, a recurrent neural network is introduced to model the continuous graph-level representation: (19) in, For time step The hidden state, It is a long short-term memory network unit. To initialize the hidden state, a recursive update mechanism is used to capture the interaction change trends across time steps, avoiding decisions based solely on a single frame.

[0043] Step S53, output the prediction result After obtaining the final time step After determining the hidden state, the classification result is output through the prediction head: (20) in, It is a multilayer perceptron network. This represents the predicted probability vector, and the output is a binary risk probability, used to indicate whether a collision risk exists. Furthermore, to achieve risk perception and enhanced safety, data dependency uncertainty parameters can be simultaneously output in the prediction head, giving the final prediction result a confidence profile. Through this graph-level aggregation and temporal modeling mechanism, spatial structure information and temporal dynamic information can be comprehensively utilized to robustly predict the scene's risk state, while maintaining consistency with the aforementioned uncertainty mechanism.

[0044] Step S2: Use the set loss function to train the scene graph robust representation model end-to-end.

[0045] In one embodiment, a multi-objective loss function is designed to collaboratively optimize prediction performance, structural stability, uncertainty calibration capability, and prototype diversity, thereby ensuring end-to-end trainability of the entire model architecture and achieving collaborative convergence of structure and prediction. The overall training objective function is defined as: (twenty one) in, To predict losses, For the uncertainty calibration loss, Due to prototype diversity constraints, For variational regularization, These are the weighting coefficients, set to 1, 0.01, and 0.1 respectively.

[0046] To characterize the uncertainty of data dependencies, a Gaussian perturbation is introduced into the logits during the prediction phase. Let the mean vector of the prediction head output be... With log variance The standard deviation is: (twenty two) The perturbation logits are obtained through Monte Carlo sampling: (twenty three) in, It is random Gaussian noise sampled from a standard normal distribution.

[0047] The predicted loss is: (twenty four) in, The cross-entropy function, For real labels, This represents the number of samples. This method allows the model to learn the prediction mean and uncertainty parameters during the training phase.

[0048] To ensure the model has higher uncertainty when prediction errors occur, a ranking constraint is introduced. Definition: The sorting loss is: (25) in, This represents a measure of the model's uncertainty when the prediction is correct. This represents a measure of the model's uncertainty when a prediction is incorrect. This represents a constant hyperparameter used to control the tolerance in the ranking loss (i.e., the allowable difference in uncertainty between correct and incorrect predictions). It helps determine how much of this difference in uncertainty between correct and incorrect predictions needs to exceed before incurring a loss. This represents the model's prediction label, i.e., the prediction result output by the model. This represents the actual, true label, indicating the true category of the data sample.

[0049] To avoid uncertainty collapse caused by erroneous samples, a lower bound constraint is introduced: (26) in, This represents a lower bound for uncertainty. It is used to prevent uncertainty from collapsing excessively when predictions are incorrect (even if the model makes a wrong prediction, it should still maintain a certain level of uncertainty). This represents a measure of the model's uncertainty when a prediction is incorrect.

[0050] The final uncertainty loss is: (27) To prevent the set of environmental prototypes from degenerating into similar vectors, a prototype diversity constraint is further introduced. First, each prototype is normalized: (28) in, It is the first A prototype vector is an environment prototype learned by the model, used to represent certain features or patterns in the environment. This is the normalized prototype vector. (Based on the original prototype...) conduct Norm normalization results in vectors with unit length (i.e., their... (The norm is 1). This operation ensures that each prototype has a consistent scale, preventing prototypes with larger scales from affecting training.

[0051] Calculate the similarity matrix: (29) in, This represents the normalized prototype vector. Representing the prototype and prototype The similarity between them is measured by their dot product.

[0052] Diversity loss is defined as: (30) in, This represents the calculated similarity matrix. This represents the identity matrix. This constraint encourages orthogonality between different prototypes, thereby enhancing the expressive power of environmental patterns.

[0053] To prevent the variance of the variational distribution from increasing indefinitely, a KL regularization term is introduced for the posterior distribution of nodes and relations. For example, for any Gaussian distribution... Its KL divergence with the standard normal distribution is: (31) in, The first Gaussian distribution represents the... The square of the mean of a dimension. It indicates the center location of that dimension. The first Gaussian distribution represents the... Variance in this dimension represents the degree of diffusion or uncertainty in that dimension. It is the sum of the KL terms of all nodes and relationships. Through the aforementioned joint optimization mechanism, prediction accuracy, structural sparsity, uncertainty calibration, and representation stability can be ensured to converge collaboratively within the same training framework, thereby improving the model's generalization ability and reliability in out-of-distribution scenarios.

[0054] Step S3: For the target image sequence, the risk prediction result is obtained by using the trained scene graph robust representation model.

[0055] Once the scene graph robust representation model is trained, it can be applied to actual risk prediction. That is, the actual obtained target image sequence is input into the trained scene graph robust representation model to obtain risk prediction results or uncertainty estimation results.

[0056] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0057] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0058] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0059] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0060] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0062] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0064] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A scene graph robust representation learning method based on uncertainty-guided causal intervention, comprising the following steps: Obtain the target image sequence; For the target image sequence, a risk prediction result is obtained using a scene graph robust representation model trained end-to-end. The robust scene graph representation model includes a target feature construction module, a variational scene graph generation module, an uncertainty-aware causal intervention module, a differentiable structure learning module, and a temporal aggregation and prediction module. The target feature construction module converts target image sequences into structured entity feature representations. The variational scene graph generation module performs entity-level variational modeling and relation-level variational modeling on the structured entity feature representations to obtain a probabilistic scene graph containing a relational representation of mean and uncertainty information. The uncertainty-aware causal intervention module uses a set of learnable prototypes to perform soft intervention correction on environment-related biases at the representation space level to obtain a corrected relational representation. The differentiable structure learning module is used to construct a sparse graph topology structure from the modified relation representation through relation scoring and sparsification strategies to obtain a sparse scene graph; the temporal aggregation and prediction module is used to obtain risk prediction results based on the sparse scene graph through graph-level feature aggregation and cyclic temporal modeling.

2. The method according to claim 1, characterized in that, The variational scene graph generation module performs the following steps: Perform entity-layer variational modeling according to the following sub-steps: For each target entity The latent semantic state, characterized under the current observation conditions, is represented as: in, It is a physical entity The corresponding structured feature vectors, Representing entities The latent embedding vector, For entities Embedded mean, Let be the diagonal covariance vector. Indicates by parameters Controlled coding network, mean With variance Obtained through neural network calculation; Reparameterization is performed based on the following formula: in, This represents element-wise multiplication. It is random noise that follows a standard normal distribution; Perform relational layer variational modeling according to the following sub-steps: For entity pairs Construct a probabilistic representation of its relation embedding: in, It is the latent embedding vector of real j. Representing entities With entity Relationship embedding, The mean of the relationship, For the variance of the relationship, The parameter is Relational coding network, mean With relational variance Calculated by embedding and splicing entities: in, This represents a vector concatenation operation. This represents the function for calculating the mean of the relation embedding. A function to calculate the variance of relational embedding; Sampling is performed using reparameterization: in, It is random noise that follows a standard normal distribution.

3. The method according to claim 2, characterized in that, The uncertainty perception causal intervention module performs the following steps: The uncertainty scalar of the relation is defined according to the following formula: in, It is the relational embedding dimension. Indicates the first The variance of the dimension; Using gate functions Mapping uncertainty to intervention intensity is expressed as: in, It is the intensity of intervention. It is a multilayer perceptron network, constrained by the Sigmoid output; Construct the environment prototype space according to the following sub-steps: Define a prototype set: in, Indicates the first A prototype, The number of prototypes is preset, and each prototype represents a potential environment-driven pattern; The environmental bias estimate is calculated using the following formula. : in: in, Represents the similarity weight. Indicates the first One prototype; The residuals are corrected to obtain the corrected mean of the relation. : Resampling: in, It is random noise that follows a standard normal distribution.

4. The method according to claim 3, characterized in that, The differentiable structure learning module performs the following steps: For the corrected relational mean Calculate the corresponding confidence score : in, It is a trained scoring network. Indicates the degree of importance of the relationship; For each entity Retain the highest-rated ones Relationship: in, Representing entities The set of neighbors to be retained Set a preset neighbor limit; Retain confidence levels exceeding the threshold Relationship: in, The final set of sparse relations to be retained; When information propagates in a graph neural network, an uncertainty weighting coefficient is introduced for each retained edge: in, Due to the uncertainty of the relationship, It is a set constant.

5. The method according to claim 1, characterized in that, The time series aggregation and prediction module performs the following steps: Perform global aggregation on all nodes: in, It is a substitution-invariant aggregation function; Indicates the first The graph-level feature vector of a frame for each time step After completing the message propagation in the graph neural network, a set of node representations is obtained. ; Modeling continuous graph-level representations using recurrent neural networks: in, For time step The hidden state, It is a long short-term memory network unit. To initialize the hidden state; The classification result is output by the prediction head and is represented as follows: in, It is a multilayer perceptron network. This represents the predicted probability vector, and the output is a binary risk probability, used to indicate whether there is a collision risk.

6. The method according to claim 4, characterized in that, The scoring network is a multilayer perceptron.

7. The method according to claim 1, characterized in that, The objective function for training the robust representation model of the scene graph is set as follows: in: in, To predict losses, For the uncertainty calibration loss, Due to prototype diversity constraints, For variational regularization, These are the weighting coefficients of the corresponding terms. It is the sorting loss. It is a lower bound constraint. The cross-entropy function, For real labels, For the number of samples, It is the sorting loss. It is a lower bound constraint used to avoid the collapse of uncertainty caused by erroneous samples. This represents the calculated similarity matrix. Represents the identity matrix.

8. The method according to claim 7, characterized in that, and Set them to: in: in, When the prediction is correct, the scenario graph robustly represents a measure of the model's uncertainty. In the event of incorrect predictions, the scenario graph robustly represents a measure of the model's uncertainty. It is a constant hyperparameter. Indicates the predicted label, Indicates the actual, real label. The value represents the lower bound of uncertainty. It is the log-variance of the prediction head output of the scene graph robust representation model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images