A human-computer cooperation assembly cognitive reasoning method for spatio-temporal dynamic evolution

Abstract

The present application relates to a kind of human-computer cooperation assembly cognitive reasoning methods for spatio-temporal dynamic evolution, by extracting the visual feature in assembly scene, generate scene graph, construct time hyperedge, space hyperedge and task hyperedge, and fusion three kinds of hyperedge form hyperedge set, represent the time-varying non-paired relationship between human operator, robot, assembly operation and various assembly component nodes, define hyperedge association matrix, construct human-computer cooperation assembly knowledge spatio-temporal hypergraph, realize the hyperedge representation between assembly components;Design stacked graph neural network based on multi-event hox process with self-excitation characteristics, learn the high-order task association between each assembly node, and update its change with time, realize spatio-temporal hypergraph representation;Modeling the self-excitation process between different subtasks, while capturing individual characteristics and collective association, realize human-computer cooperation assembly.The problem that human-computer cooperation cognition is difficult to reason in time-varying task is solved, and human-computer cooperation assembly efficiency and initiative are improved.

Description

Technical Field

[0001] This invention relates to a knowledge graph reasoning and human-computer collaboration technology, and particularly to a human-computer collaborative assembly cognitive reasoning method oriented towards spatiotemporal dynamic evolution. Background Technology

[0002] Currently, mass personalized manufacturing has gradually become a popular production model. Personalized products are generally considered configurable products, meaning they can be customized according to customer requirements and options. These products allow for the generation of multiple variations through different combinations of parameters and configurations, resulting in multiple assembly paths. This requires highly flexible manufacturing capabilities and relies on operator adaptability to introduce dynamic characteristics into the assembly process. To address this challenge, human-robot collaboration has attracted attention from industry and academia. Most research has not endowed human-robot collaborative systems with the self-organizing ability of dynamic assembly task allocation, thus affecting the proactive collaboration between human operators and robots. To study the bidirectional autonomous cognitive capabilities of human operators and robots, the focus is mainly on transforming environmental perception cues into graph structures, focusing on pairwise relationships between operators, robots, and parts in discrete snapshots.

[0003] However, personalized product assembly involves a large number of diverse components, and the relationship between these components and the human-machine interface is not a fixed one-to-one relationship, but rather exhibits a many-to-many non-paired relationship that evolves over time. Therefore, current reasoning methods that focus on paired relationships cannot generate accurate assembly strategies. Summary of the Invention

[0004] To address the problem that methods focusing on the pairwise relationships of assembled components in discrete snapshots cannot characterize the time-varying higher-order associations of dynamic assembly, leading to inaccurate task allocation and affecting human-computer collaborative cognitive strategies, this invention proposes a human-computer collaborative assembly cognitive reasoning method oriented towards spatiotemporal dynamic evolution.

[0005] The technical solution of this invention is as follows:

[0006] A human-computer collaborative assembly cognitive reasoning method oriented towards spatiotemporal dynamic evolution includes the following steps:

[0007] Step S100: By extracting visual features from the assembly scene, a scene graph is generated, and temporal hyperedges, spatial hyperedges, and task hyperedges are constructed. These three types of hyperedges are then fused to form a hyperedge set, which represents the time-varying non-paired relationships between human operators, robots, assembly operations, and various assembly component nodes. A hyperedge association matrix is ​​defined, and a human-machine collaborative assembly knowledge spatiotemporal hypergraph is constructed to realize the hyperedge representation between assembly components.

[0008] Step S200: Design a stacked graph neural network with self-excitation characteristics based on a multi-event Hawkes process to learn the high-order task associations between assembly nodes and update their changes over time to achieve spatiotemporal hypergraph representation; model the self-excitation process between different sub-tasks, while capturing individual features and collective associations, revealing the evolution mechanism of time-varying assembly tasks, decoupling the operations that humans and machines need to complete, and achieving efficient human-machine collaborative assembly.

[0009] Furthermore, the specific steps include:

[0010] Step S100: Construct a spatiotemporal hypergraph of knowledge for human-machine collaborative assembly to represent time-varying unpaired relationships in the dynamic assembly process; perceive the assembly space through a depth camera, dynamically detect visual cues using Graph R-CNN and transform them into a graph structure to achieve structured processing of scene features; define three types of hyperedges—spatial hyperedges, temporal hyperedges, and task hyperedges—for operators, robots, assembly operations, and assembly component nodes in the assembly scene, and fuse them to form a hyperedge set: spatial hyperedges are based on Euclidean distance between nodes, representing higher-order associations of geographical proximity; temporal hyperedges are based on task timestamps and sliding time windows, representing temporal dependencies; task hyperedges are based on semantic information, representing collaborative dependencies; finally, the features of the three types of hyperedges—spatial proximity, temporal dependency, and task-related features—are spliced ​​and fused to generate hyperedge feature vectors, and the association between hyperedges and nodes is established through a hyperedge association matrix, completing the construction of the spatiotemporal hypergraph and realizing comprehensive modeling of time-varying unpaired relationships among multiple nodes in the dynamic assembly process;

[0011] Step S200: Design a stacked graph neural network with self-excitation characteristics based on a multi-event Hawkes process to learn high-order task associations between nodes and capture time-varying task evolution mechanisms; the multi-event Hawkes process estimates the probability of future events by defining a conditional strength function, and models the spatiotemporal evolution law between multiple dynamic events; the stacked graph neural network with self-excitation characteristics includes node embedding, hyperedge embedding, temporal fusion, and graph pooling modules, integrates historical information through recursive aggregation and mapping of multi-layer neighbor features, and designs a temporal representation network and Hawkes process fusion to achieve feature transfer under temporal dependence, realizing time-sensitive capture of historical neighbor messages and layer-by-layer feature stacking;

[0012] Step S300: Optimize cognitive reasoning performance by decoupling individual features and collective association features; the individual features convert prior experience of events into unique parameters through a learnable transformation model, and dynamically adjust network parameters in combination with a feature linear modulation model to ensure that prior experience quickly adapts to the uniqueness of events; the collective association features represent event features by connecting spatiotemporal hypergraph nodes, and use graph neural networks to explore the temporal propagation law to generate a new event predictor to capture systematic dynamic interaction features;

[0013] Step S400: Define loss functions to optimize network performance, including negative log-likelihood loss optimized for individual feature capture and smooth L1 loss optimized for collective association feature capture. By jointly optimizing the two types of losses, the consistency of hyperedge and node relationship transmission is ensured, and finally, dynamic cognitive reasoning of time-varying tasks in human-machine collaborative assembly is realized, decoupling human-machine operation and improving collaboration efficiency.

[0014] Furthermore, in step S100:

[0015] The human-machine collaborative assembly knowledge spatiotemporal hypergraph generates time-varying non-paired relationships for each node through scene feature structuring, representing the dynamic assembly process;

[0016] The scene features are structured, the assembly space is perceived by a depth camera, and as time goes by, visual cues are dynamically detected using Graph R-CNN, and the visual cues are transformed into a graph structure with assembly relationships.

[0017] The time-varying unpaired relation representation is achieved by using the hyperedges of each node in the human-machine collaborative assembly knowledge spatiotemporal hypergraph to represent the assembly relationship involving multiple entities, thus modeling the time-varying unpaired relation in dynamic assembly. At different points in time, the human-machine collaborative assembly knowledge spatiotemporal hypergraph dynamically represents the operator, robot, assembly operation, assembly components and their time-varying unpaired relations, associating the dynamic assembly state at different stages.

[0018] The time-varying unpaired relation represents a many-to-many higher-order relationship between multiple nodes under continuous time variation. It is different from the one-to-one paired relation and has the advantages of complex relation representation and reducing interference from redundant information.

[0019] Furthermore, in step S100:

[0020] The spatial hyperedge, i.e., the geographical proximity between nodes, represents the complex high-order associations between nodes in the same space; if the Euclidean distance between nodes is less than a specified threshold, they belong to the same spatial hyperedge; the formula is defined as follows:

[0021] ε space ={e|distance(v i ,v j )<θ}

[0022] Where ε spdce The set of spatial hyperedges represents the set of geographic proximity relationships between nodes; 'e' represents the hyperedge containing spatially related nodes; and 'distance' represents the distance between nodes v during the assembly process. i and v jThe Euclidean distance between nodes is θ, which is a predefined distance threshold used to determine whether nodes belong to the same spatial hyperedge; if the spatial distance between two nodes is less than the threshold θ, they are considered to belong to the same hyperedge.

[0023] The temporal hyperedge represents the temporal dependency between nodes in a time-varying task. Based on consecutive task timestamps, multiple nodes with temporal dependencies are grouped into the same hyperedge. A time window is set, and task instructions are used to construct the temporal hyperedge, as defined in the following formula:

[0024] ε time ={e'|time(v i ) <time(v j ), gap≤Δt}

[0025] Where ε time This represents the set of time-dependent hyperedges describing the temporal dependencies between nodes, where e' represents a hyperedge containing time-dependent nodes. i ) indicates section v i The timestamp of the point's occurrence, gap represents the time difference between two nodes, and Δt is the value of the time window; if the time difference between nodes is less than the time window value Δt, these nodes exhibit time dependence and are considered to belong to the same time hyperedge; if the time interval between these consecutive actions is less than the specified time window Δt, the nodes involved in these actions are classified as belonging to the same time hyperedge.

[0026] The task hyperedges represent the collaborative relationships between nodes in a human-computer collaborative task. These hyperedges rely on linguistic cues, task plans, and process information to extract task semantics in order to construct task hyperedges among multiple nodes. The formula is defined as follows:

[0027]

[0028] Where ε task The set of task superedges represents the task dependencies between nodes; e” represents the task superedges that characterize the relationships between nodes within the same human-computer collaborative task; v i and v j This indicates the nodes involved in the same human-machine collaborative task; Represents node v i and node v j They belong to the same human-computer collaboration task; these nodes are extracted from linguistic cues, and their relationship is defined as task hyperedges;

[0029] The hyperedge set represents the fusion and collection of spatial hyperedges, temporal hyperedges, and task hyperedges; this set comprehensively considers spatial, temporal, and task characteristics to establish relationships between nodes in human-computer collaboration; the formula is defined as follows:

[0030] H ε =concat(space features ,time features ,task features )

[0031] Where H ε Represents the eigenvectors of the hyperedge, space features Represents the proximity feature of the hyperedge space, time features Represents the time-dependent features of the hyperedge, task features The hyperedge features represent task-related features; concat represents the concatenation operation that combines multimodal features into a vector; in general, hyperedge features include task type, temporal dependency, priority information, and tool usage semantics.

[0032] The hyperedge correlation matrix represents the relationship between all hyperedges and each node, thus realizing the construction of a spatiotemporal hypergraph.

[0033] Furthermore, the hyperedge association matrix, taking the association of operators, robots, assembly operations, assembly components, and nodes, as well as spatial hyperedges, temporal hyperedges, and task hyperedges as examples, transforms the relationship between hyperedges and nodes into a hyperedge connection matrix composed of rows and columns. Simultaneously, matrix values ​​are defined to determine whether there is a relationship between hyperedges and nodes. The formula is defined as follows:

[0034] v={v operator ,v robot ,v operation ,v component}

[0035] ε={e Space ,e time ,e task}

[0036] e Space ={v operator ,v robot ,v operation}

[0037] e time ={v operator ,v operation ,v component}

[0038] e task ={v robot ,v operation ,v component}

[0039]

[0040] Where v represents the node set, v operator ,vrobot ,v operation ,v component These correspond to operator, robot, assembly operation, and part, respectively, where ε represents the hyperedge set, and e space ,e time ,e task Corresponding to spatial hyperedge, temporal hyperedge, and task hyperedge respectively, A ε This represents the hyperedge connection matrix, which describes the relationship between hyperedges and nodes. Each row represents a node, and each column represents a hyperedge.

[0041] Furthermore, in step S200:

[0042] The multi-event Hawkes process can determine whether multiple different nodes are related at a certain moment. By defining conditional strength, i.e., defining the conditional strength function of triples l, m, n forming a hyperedge at time t, it estimates the future relationship between multiple nodes, generates the probability of a certain event, models the evolution mechanism between multiple dynamic events in time-varying tasks, and reveals the spatiotemporal evolution law of events. The formula for the conditional strength function is defined as follows:

[0043]

[0044]

[0045] Here, μ l,m,n (t) represents the basic rate at which event nodes l, m, and n form a link at time t; S m′,n′ (t′)k(tt′) represents the influence of the historical neighbors m′ and n′ of event l at time t′ on the historical neighbors of the current event at time t; S l′,n′ (t′)k(tt′) represents the influence of the historical neighbors l′ and n′ of event m at time t′ on the historical neighbors of the current event at time t; S l′,m′ (t′)k(tt′) represents the influence of the historical neighbors l′ and m′ of event n at time t′ on the historical neighbors of the current event at time t; (l,m′,n′,t′)∈H l (t) represents the historical events of node l before time t; (l′,m,n′,t′)∈H m (t) represents the historical events of node m before time t; (l′,m′,n,t′)∈H n (t) represents the historical events of node n before time t; H l (t), H m (t) and H n(t) represents the set of historical events of nodes l, m, and n before time t, respectively; in addition, m′ and n′ are considered as historical neighbors of l; similarly, other combinations can be defined; k(tt′) represents the influence function, t′ represents a certain historical time, k represents the exponential decay kernel, and k decays over time;

[0046] The conditional strength describes the expected number of events to occur within a unit time interval at a future moment, influenced by historical events; if the sequence of past event occurrence times is represented as {t1,t2,…,t…} r Then, the conditional strength λ(t) at time t can be expressed as:

[0047]

[0048] Here, λ(t) represents the fundamental intensity function, indicating the event occurrence rate in the absence of any past events, while α(t) represents the event occurrence rate. i ) is an influence function, representing the influence of past events on the occurrence of future events.

[0049] Furthermore, in step S200:

[0050] The self-excited stacked graph neural network recursively receives, aggregates, and maps features of multi-layer neighbor nodes through node embedding, hyperedge embedding, temporal fusion, and graph pooling modules, integrating historical node features. Simultaneously, a temporal representation network of the graph neural network is designed and fused with a Hawkes process to achieve feature transfer between nodes under temporal dependencies; the formula is defined as follows:

[0051]

[0052] here, The event node l, located in layer p, is defined as the temporal representation of the event at time t. p Indicates the dimension of the embedding vector; Let m′ and n′, located in layer p-1, represent the feature representations of historical neighbors m′ and n′ at historical time t′, respectively. Let represent the aggregate representation of historical neighbors n′ and n′ at historical time t′ in layer p-1, and σ represent an activation function. and Both are learnable weight matrices; at the same time, utilizing Mapping the embeddings of its historical neighbors captures the temporal sensitivity of past events; furthermore, in the kernel function k s Within (tt′), SoftMax is used to represent the time decay effect; the temporal representation of a node is described by aggregating its own information and historical neighbor messages from the previous layer; layer-wise feature stacking is used for GNN.

[0053] Furthermore, in step S200:

[0054] The individual characteristics mentioned above decouple the non-paired relationships between nodes in the spatiotemporal hypergraph, ensuring the uniqueness of events. By encoding general relationships to connect knowledge, prior experience with time-varying characteristics is generated. A learnable transformation model is constructed, enabling the prior experience to quickly adapt to the uniqueness of each event. A transfer function is fitted to each event, capturing the individual characteristics of each event. The prior experience ex is transformed into a model with specific parameters δ. l,m,n,t The events (l, m, n, t) are as follows:

[0055]

[0056] in, Defined as event-specific time representations for nodes l, m, and n, where con() refers to the join operation; based on The prior event ex is transformed into the uniqueness parameter δ of the event through a learnable transformation model ψ. l,m,n,t This process is parameterized by η;

[0057] The learnable transformation model uses Feature Linear Modulation (FiLM), which performs a simple affine transformation on the features of the intermediate layers of the neural network, dynamically adjusting the network parameters to scale and shift prior experience, ensuring that the prior experience can quickly adapt to the uniqueness of each event; scaling and shifting are achieved through fully connected layers (FCL). C Scaling factor of expression scf (l,m,n,t) and movement factor shf (l,m,n) The implementation is as follows:

[0058]

[0059] Where ω scf and ω shf It is FCL C The learnable weight matrix, B scf and B shf It is FCL C The deviation vector;

[0060] The transfer function is an important part of generating conditional strength and is implemented using the softplus function.

[0061] Furthermore, in step S200:

[0062] Decouple the non-paired relationships of nodes in the spatiotemporal hypergraph, capture the collective correlation characteristics of all events, and ensure the systematic and dynamic interaction characteristics of events;

[0063] The collective association feature decouples the non-paired relationships between nodes in the spatiotemporal hypergraph, ensuring the systematic and dynamic interaction characteristics of events. By connecting the nodes in the spatiotemporal hypergraph, it characterizes the event features associated with the nodes. Over time, a graph neural network is used to explore the temporal propagation characteristics of event features, fusing the spatial features and temporal evolution of events to capture the collective association features of all events. The propagation and trends of all relevant nodes in the assembly process—operator, robot, operation, and component nodes—are essentially the same concept. For ease of discussion, a new event is defined here as... This represents the new events that occur at time t for nodes l, m, and n; the following uses a fully connected layer FCL. C A predictor for new events of nodes was built:

[0064]

[0065] in, δ is defined as the event-specific time representation of nodes l, m, and n. I Indicates a fully connected layer (FCL) C Required parameters.

[0066] Furthermore, in step S200:

[0067] Define a loss function, optimize network performance, ensure the consistency of relational propagation of all hyperedges and nodes in the spatiotemporal hypergraph, model the evolution of time-varying unpaired relations in the spatiotemporal hypergraph, and realize human-computer collaborative cognitive strategy reasoning.

[0068] The loss function includes simultaneous optimization of individual event features and collective correlation features. Individual feature capture optimization is defined by negative log-likelihood loss, which can provide appropriate conditional strength for the occurrence or non-occurrence of events. Collective correlation feature capture optimization is defined by smooth L1 loss, which increases error penalty and achieves better network performance.

[0069] The individual feature capture optimization assumes that the event (l,m,n,t)∈O has already occurred, and its loss L n (l,m,n,t) is as follows:

[0070]

[0071] Where, λ l,m,n (t) and λ l,m,q (t) denotes the conditional strength of the formation of hyperedges for the ternary pairs (l,m,n) and (l,m,q) at time t, respectively, and q represents the conditional strength based on the distribution d. n The nodes of the sampled negative samples, the distribution defines the events. Instances that do not occur; meanwhile, Q ne This represents the number of negative samples for each event that occurred;

[0072] The collective association feature capture optimization has a loss L C (l,m,n,t) is as follows:

[0073]

[0074] in, N represents the predicted node propagation value. l,m,n Represents the actual value of the node.

[0075] The beneficial effects of this invention are as follows:

[0076] 1. To address the redundant information caused by pairwise relationships, a temporal hypergraph is constructed to represent the complex non-pairwise relationships between time-varying tasks in human-computer collaboration.

[0077] 2. To address the dynamic and time-varying nature of human-computer collaborative tasks, we consider the self-excitation process of Hawkes processes (a mathematical model for modeling self-excitation processes) and introduce it into a stacked GNN architecture to simultaneously capture the individual and collective correlation features of assembly tasks, thereby revealing the dynamic triggering mechanism between time-varying tasks. Attached Figure Description

[0078] Figure 1 This invention provides an overall framework diagram for a human-computer collaborative assembly cognitive reasoning method oriented towards spatiotemporal dynamic evolution;

[0079] Figure 2 A flowchart for constructing a spatiotemporal hypergraph based on human-computer collaboration semantics provided by this invention;

[0080] Figure 3 A diagram illustrating the construction steps of a stacked graph neural network based on Hawkes processes, provided by this invention. Detailed Implementation

[0081] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0082] like Figure 1 As shown, this embodiment of the invention provides a human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution, including:

[0083] Step S100: As Figure 1 , 2As shown, by extracting visual features from the assembly scene, a scene graph is generated, and temporal hyperedges, spatial hyperedges, and task hyperedges are constructed. These three types of hyperedges are then fused to form a hyperedge set, which represents the time-varying non-paired relationships between human operators, robots, assembly operations, and various assembly component nodes. A hyperedge association matrix is ​​defined, and a human-machine collaborative assembly knowledge spatiotemporal hypergraph is constructed to realize the hyperedge representation between assembly components.

[0084] In step S100, the human-machine collaborative assembly knowledge spatiotemporal hypergraph generates time-varying unpaired relationships for each node through scene feature structuring, representing the dynamic assembly process.

[0085] The scene features are structured, the assembly space is perceived by a depth camera, and as time goes by, visual cues are dynamically detected using Graph R-CNN, and the visual cues are transformed into a graph structure with assembly relationships.

[0086] The time-varying unpaired relation representation uses the hyperedges of each node in the spatiotemporal hypergraph (short for the human-machine collaborative assembly knowledge spatiotemporal hypergraph) to represent the assembly relationships involving multiple entities, effectively modeling the time-varying unpaired relations in dynamic assembly. At different points in time, the human-machine collaborative assembly knowledge spatiotemporal hypergraph dynamically represents the operator, robot, assembly operation, assembly components, and their time-varying unpaired relations, associating the dynamic assembly states at different stages.

[0087] Among them, "dynamic assembly status" refers to the interaction between people, machines, and materials and the task execution status at a certain point in time during the assembly process.

[0088] "Associating the dynamic assembly states at different stages" means linking these temporal states together through a model to form a complete dynamic evolution representation of the assembly task.

[0089] The time-varying unpaired relation represents a many-to-many higher-order relationship between multiple nodes under continuous time variation. It is different from the one-to-one paired relation and has the advantages of complex relation representation and reducing interference from redundant information.

[0090] Note: These two terms are different; one describes a complex relationship, and the other describes how to learn and represent such a complex relationship.

[0091] The spatial hyperedge, or geographical proximity between nodes, characterizes the complex high-order relationships between nodes within the same space. If the Euclidean distance between nodes is below a specified threshold, they belong to the same spatial hyperedge. For example, components like nuts and bolts associated with a robot indicate that these components may require corresponding robot operations. The formula is defined as follows:

[0092] ε space ={e|distance(vi ,v j )<θ}

[0093] Where ε space This represents the set of spatial hyperedges describing the geographical proximity relationships between nodes. `e` represents a hyperedge containing spatially related nodes. `distance` represents the distance between nodes during the assembly process. i and v j The Euclidean distance between nodes is given by θ, which is a predefined distance threshold used to determine whether nodes belong to the same spatial hyperedge. If the spatial distance between two nodes is less than the threshold θ, they are considered to belong to the same hyperedge. For example, if a wrench and a nut are very close, and both are close to the robot, and the distance between them is less than the threshold, it can be concluded that these nodes belong to the same hyperedge, indicating that the robot is likely to use the wrench to tighten the nut.

[0094] The temporal hyperedge represents the temporal dependencies between nodes in a time-varying task. Based on consecutive task timestamps, multiple nodes with temporal dependencies are grouped into the same hyperedge. A time window is set, and task instructions are used to construct the temporal hyperedge, as defined in the following formula:

[0095] ε time ={e'|time(v i ) <time(v j ), gap≤Δt}

[0096] Where ε time This represents the set of time-dependent hyperedges describing the temporal dependencies between nodes, where e' represents a hyperedge containing time-dependent nodes. i ) indicates section v i The timestamp of a node's occurrence is used, gap represents the time difference between two nodes, and Δt is the value of the time window. If the time difference between nodes is less than the time window value Δt, these nodes exhibit temporal dependence and are considered to belong to the same temporal hyperedge. For example, an operator places a nut in the correct position, the robot moves a wrench to the nut's position, and then rotates the wrench to tighten the nut. If the time interval between these consecutive actions is very short (less than the specified time window Δt), the nodes involved in these actions are classified as belonging to the same temporal hyperedge.

[0097] The task hyperedges represent the collaborative relationships between nodes in a human-machine collaborative task. These hyperedges primarily rely on linguistic cues, task plans, and process information to extract task semantics (e.g., a robot transporting a reducer cover, an operator tightening a nut) to construct task hyperedges between multiple nodes, as defined by the following formula:

[0098]

[0099] Where ε task This represents the set of task superedges describing the task dependencies between nodes. "e" represents a task superedge representing the relationships between nodes within the same human-computer collaborative task. i and v j This refers to the nodes involved in the same human-machine collaborative task. Represents node v i and node v j These nodes belong to the same human-computer collaborative task. Typically, these nodes are extracted from linguistic cues, and their relationships are defined as task hyperedges. For example, in the task description "rotate the wrench to tighten the bolt and nut," the wrench, nut, and bolt are considered to belong to the same task hyperedge.

[0100] The hyperedge set represents the fusion and set of spatial hyperedges, temporal hyperedges, and task hyperedges. This set comprehensively considers spatial, temporal, and task characteristics to establish relationships between nodes in human-machine collaboration. It provides the foundation for the robot's optimal cognitive strategy, and the formula is defined as follows:

[0101] H ε =concat(space features ,time features ,task features )

[0102] Where H ε Represents the eigenvectors of the hyperedge, space features Represents the proximity feature of the hyperedge space, time features Represents the time-dependent features of the hyperedge, task features This represents the task-related features of the hyperedge. `concat` represents the concatenation operation that combines multimodal features into a vector. Overall, hyperedge features include task type, temporal dependencies, priority information, and tool usage semantics.

[0103] The hyperedge association matrix represents the relationship between all hyperedges and each node, realizing the construction of a spatiotemporal hypergraph. The following example uses the associations of nodes related to operators, robots, assembly operations, and assembly components, as well as spatial, temporal, and task hyperedges (where each node's association with a hyperedge is just one example to represent a general association). The relationship between hyperedges and nodes is transformed into a hyperedge connection matrix composed of rows and columns. Simultaneously, matrix values ​​are defined to determine whether there is a relationship between a hyperedge and a node. The formula is defined as follows:

[0104] v={v operator ,v robot ,v operation ,v component}

[0105] ε={e space ,e time ,etask}

[0106] e space ={v operator ,v robot ,v operation}

[0107] e time ={v operator ,v operation ,v component}

[0108] e task ={v robot ,v operation ,v component}

[0109]

[0110] Where v represents the node set, v operator ,v robot ,v operation ,v component These correspond to operator, robot, assembly operation, and part, respectively, where ε represents the hyperedge set, and e space ,e time ,e task Corresponding to spatial hyperedge, temporal hyperedge, and task hyperedge respectively, A ε This represents a hyperedge connectivity matrix describing the relationships between hyperedges and nodes, where each row represents a node and each column represents a hyperedge. For example, the first row represents node v. operator It belongs to the hyperedge e space and super edge e time Therefore, the first row will be [1, 1, 0]. If all nodes in a column belong to the same hyperedge, i.e., v ∈ e s Then A ε The value will be 1. (e.g., e) space ={v operator ,v robot ,v operation (Here, "}" refers generally to the hyperedges between different nodes.)

[0111] Step S200: As Figure 3 As shown, a stacked graph neural network with self-excitation characteristics based on a multi-event Hawkes process is designed to learn the high-order task associations between assembly nodes and update their changes over time, thereby realizing spatiotemporal hypergraph representation. The self-excitation process between different sub-tasks is modeled, while capturing individual features and collective associations, revealing the evolution mechanism of time-varying assembly tasks, decoupling the operations that humans and machines need to complete, and realizing efficient human-machine collaborative assembly.

[0112] "Spatiotemporal hypergraph representation" integrates three types of hyperedges: space, time, and task. It models the complex, time-varying, many-to-many relationships in the human-machine collaborative assembly process into a unified hypergraph representation to support cognitive reasoning and strategy generation.

[0113] A spatiotemporal hypergraph is a knowledge graph that represents human-computer collaboration relationships.

[0114] Spatiotemporal hypergraph representation is the learning and representation of this knowledge graph.

[0115] The multi-event Hawkes process can determine whether multiple different nodes are related at a certain moment. By defining conditional strength (defining the conditional strength function of triples l, m, n forming a hyperedge at time t), it estimates the future relationship between multiple nodes, generates the probability of a certain event occurring, models the evolution mechanism between multiple dynamic events in time-varying tasks, and reveals the spatiotemporal evolution law of events. The formula for the conditional strength function is defined as follows:

[0116]

[0117]

[0118] Here, μ l,m,n (t) represents the basic rate at which event nodes l, m, and n form a link at time t, independent of historical assembly events. m′,n′ (t′)k(tt′) represents the influence of the historical neighbors m′ and n′ of event l at time t′ on the historical neighbors of the current event at time t; S l′,n′ (t′)k(tt′) represents the influence of the historical neighbors l′ and n′ of event m at time t′ on the historical neighbors of the current event at time t; S l′,m′ (t′)k(tt′) represents the influence of the historical neighbors l′ and m′ of event n at time t′ on the historical neighbors of the current event at time t. (l,m′,n′,t′)∈H l (t) represents the historical events of node l before time t; (l′,m,n′,t′)∈H m (t) represents the historical events of node m before time t; (l′,m′,n,t′)∈H n (t) represents the historical events of node n before time t. H l (t), H m (t) and H n (t) represents the set of historical events for nodes l, m, and n before time t, respectively. Furthermore, m′ and n′ are considered historical neighbors of l. Similarly, combinations such as l′ and m′, l′ and n′, can be defined. k(tt′) represents the influence function, t′ represents a historical time, k represents the exponentially decaying kernel, and k decays over time.

[0119] The conditional strength, a key concept in Hawkes processes, represents the probability of an event occurring at a future moment given historical information. It describes the expected number of future events occurring within a unit time interval influenced by historical events. If we represent the sequence of past event occurrence times as {t1, t2, ..., t...} r Then, the conditional strength λ(t) at time t can be expressed as:

[0120]

[0121] Here, λ(t) represents the fundamental intensity function, indicating the event occurrence rate in the absence of any past events, while α(t) represents the event occurrence rate. i α(tt) is an influence function, representing the impact of past events on the occurrence of future events. Typically, α(tt) i The conditional strength is non-negative and decreases over time, indicating that the influence of past events on future events is diminishing. Essentially, conditional strength describes the rate at which a given past event influences the occurrence of a future event, a key concept in Hawkes processes.

[0122] The self-excited stacked graph neural network, through node embedding, hyperedge embedding, temporal fusion, and graph pooling modules, recursively receives, aggregates, and maps features from multiple layers (multiple layers refer to multiple vertically connected feature transformation layers in the stacked graph neural network; each layer performs nonlinear mapping and aggregation of input features through a weight matrix) of neighboring nodes, integrating historical node features. Simultaneously, a temporal representation network of the graph neural network is designed and fused with a Hawkes process to achieve feature transfer between nodes under temporal dependence.

[0123] The formula is defined as follows:

[0124]

[0125] here, d is defined as the temporal representation of an event node l (i.e., the node representation of an event) located in layer p at time t. p Indicates the dimension of the embedding vector. Let m′ and n′, located in layer p-1, represent the feature representations of historical neighbors m′ and n′ at historical time t′, respectively. Let represent the aggregate representation of historical neighbors m′ and n′ in layer p-1 at historical time t′, and σ represent an activation function. and Both are learnable weight matrices. Simultaneously, utilizing... Mapping the embeddings of its historical neighbors captures the temporal sensitivity of past events. Furthermore, in the kernel function k... sWithin (tt′), SoftMax is used to represent the time decay effect. To effectively capture the temporal sensitivity between base strength and historical neighbor information, the temporal representation of a node must be described by aggregating its own information and historical neighbor messages from the previous layer. Furthermore, due to the complexity of the temporal hypergraph, the model's representational power needs to be enhanced; therefore, layer-wise feature stacking is used for GNNs. More specifically, the input node features play a crucial role in initializing the node messages in the first layer of the network.

[0126] The individual characteristics primarily involve decoupling the unpaired relationships between nodes in the spatiotemporal hypergraph to ensure the uniqueness of events. By encoding general relationships to connect knowledge, prior experience with time-varying characteristics is generated. A learnable transformation model is constructed, enabling the prior experience to quickly adapt to the uniqueness of each event. A transfer function is fitted to each event, capturing its individual characteristics. The prior experience *ex* is transformed into a model with specific parameters δ. l,m,n,t The events (l, m, n, t) are as follows:

[0127]

[0128] in, Defined as event-specific time representations for nodes l, m, and n, their con() function refers to the join operation. Based on The prior event ex is transformed into the uniqueness parameter δ of the event through a learnable transformation model ψ. l,m,n,t This process is parameterized by η.

[0129] The learnable transformation model uses Feature Linear Modulation (FiLM), which involves performing a simple affine transformation on the features of the intermediate layers of the neural network, to dynamically adjust the network parameters. This scales and shifts the prior experience, ensuring that the prior experience can quickly adapt to the uniqueness of each event. The scaling and shifting are achieved through a fully connected layer, FCL. C Scaling factor of expression scf (l,m,n,t) and movement factor shf (l,m,n) The implementation is as follows:

[0130]

[0131] Where ω scf and ω shf It is FCL C The learnable weight matrix, B scf and B shf It is FCL C The deviation vector.

[0132] The transfer function is an important part of generating conditional strength and is usually implemented using the softplus function.

[0133] Decouple the non-paired relationships of nodes in the spatiotemporal hypergraph, capture the collective correlation characteristics of all events, and ensure the systematic and dynamic interaction characteristics of events;

[0134] The collective association feature primarily decouples the unpaired relationships between nodes in the spatiotemporal hypergraph, ensuring the systematic and dynamic interaction characteristics of events. By connecting the nodes in the spatiotemporal hypergraph, it characterizes the event features associated with those nodes. Over time, a graph neural network is used to explore the temporal propagation characteristics of these event features, fusing the spatial features and temporal evolution of events to capture the collective association features of all events. In the assembly process described above, the propagation and trends of all relevant nodes—operator, robot, operation, and component nodes—are essentially the same concept. For ease of discussion, a new event is defined here as... This represents a new event occurring at time t for nodes l, m, and n. The following uses a fully connected layer (FCL). C A predictor for new events of nodes was built:

[0135]

[0136] in, δ is defined as the event-specific time representation of nodes l, m, and n. I Indicates a fully connected layer (FCL) C Required parameters.

[0137] Define a loss function, optimize network performance, ensure the consistency of relational propagation of all hyperedges and nodes in the spatiotemporal hypergraph, model the evolution of time-varying unpaired relations in the spatiotemporal hypergraph, and realize human-computer collaborative cognitive strategy reasoning.

[0138] The loss function includes simultaneous optimization of individual event features and collective correlation features. Individual feature capture optimization is defined by negative log-likelihood, which can provide appropriate conditional strength for the occurrence or non-occurrence of events. Collective correlation feature capture optimization is defined by smooth L1 loss, which increases error penalty and achieves better network performance.

[0139] The individual feature capture optimization assumes that the event (l,m,n,t)∈O has already occurred, and its loss L n (l,m,n,t) is as follows:

[0140]

[0141] Where, λ l,m,n (t) and λ l,m,q (t) denotes the conditional strength of the formation of hyperedges for the ternary pairs (l,m,n) and (l,m,q) at time t, respectively, and q represents the conditional strength based on the distribution d. n The nodes of the sampled negative samples, the distribution defines the events. Instances that do not occur. Also, N ne This represents the number of negative samples for each event that occurred.

[0142] The collective association feature capture optimization has a loss L C (l,m,n,t) is as follows:

[0143]

[0144] in, N represents the predicted node propagation value. l,m,n Represents the actual value of the node.

[0145] In summary, to address the difficulty of reasoning in human-machine collaborative assembly in time-varying tasks, this invention proposes a cognitive reasoning method for human-machine collaborative assembly oriented towards spatiotemporal dynamic evolution. It constructs a spatiotemporal hypergraph to represent time-varying unpaired relationships, reducing redundant interference from one-to-one paired relationships. A stacked graph neural network with Hawkes processes is designed to learn event correlation representations in the spatiotemporal hypergraph, revealing individual characteristics and collective associations of subtasks, thereby enabling robot cognitive reasoning and improving the efficiency and initiative of human-machine collaborative assembly. This method has significant theoretical and practical value.

[0146] The above-described embodiments are merely one implementation of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention should be determined by the appended claims.

Claims

1. A human-computer collaborative assembly cognitive reasoning method oriented towards spatiotemporal dynamic evolution, characterized in that, Includes the following steps: Step S100: Construct a spatiotemporal hypergraph of knowledge for human-machine collaborative assembly to represent time-varying unpaired relationships in the dynamic assembly process; perceive the assembly space through a depth camera, dynamically detect visual cues using Graph R-CNN and transform them into a graph structure to achieve structured processing of scene features; define three types of hyperedges—spatial hyperedges, temporal hyperedges, and task hyperedges—for operators, robots, assembly operations, and assembly component nodes in the assembly scene and fuse them to form a hyperedge set; The spatial hyperedge, i.e., the geographical proximity between nodes, represents the complex high-order associations between nodes in the same space; if the Euclidean distance between nodes is less than a specified threshold, they belong to the same spatial hyperedge; the formula is defined as follows: in Represents a spatial hyperedge set that describes the geographical proximity relationships between nodes; This represents a hyperedge that contains spatially related nodes; Representing each node in the assembly process and The Euclidean distance between them, and This is a predefined distance threshold used to determine whether nodes belong to the same spatial hyperedge; if the spatial distance between two nodes is less than the threshold... If they belong to the same superedge, then they are considered to belong to the same superedge. The time hyperedge represents the temporal dependency relationship between nodes in a time-varying task; Based on consecutive task timestamps, multiple nodes with time dependencies are grouped into the same hyperedge; Set a time window and use task instructions to construct a time hyperedge, as defined in the following formula: in This represents the set of temporal superedges that describe the temporal dependencies between nodes. This represents a superedge that contains time-related nodes. Indicates section The timestamp of the point appears. This represents the time difference between two nodes. It is the value of the time window; if the time difference between nodes is less than the time window value. These nodes exhibit temporal dependencies and are considered to belong to the same temporal hyperedge; if the time interval between these consecutive actions is less than the specified time window. Then the nodes involved in these actions are classified as belonging to the same time superedge; The task hyperedges represent the collaborative relationships between nodes in a human-computer collaborative task. These hyperedges rely on linguistic cues, task plans, and process information to extract task semantics in order to construct task hyperedges among multiple nodes. The formula is defined as follows: in This represents a set of task superedges that describe the task dependencies between nodes. A task hyperedge represents the relationship between nodes within the same human-computer collaborative task; and This indicates the nodes involved in the same human-machine collaborative task; Represents a node and nodes Belonging to the same human-machine collaborative task; These nodes are extracted from linguistic cues, and their relationships are defined as task hyperedges; The hyperedge set represents the fusion and collection of spatial hyperedges, temporal hyperedges, and task hyperedges; this set comprehensively considers spatial, temporal, and task characteristics to establish relationships between nodes in human-computer collaboration; the formula is defined as follows: in Represents the eigenvectors of the hyperedge. This represents the proximity characteristic of the hyperedge space. This indicates the time dependency feature of the hyperedge. Indicates features related to hyperedge tasks; This represents a concatenation operation that combines multimodal features into a vector; hyperedge features include task type, temporal dependency, priority information, and tool usage semantics. The hyperedge association matrix represents the relationship between all hyperedges and each node, enabling the construction of a spatiotemporal hypergraph. This hyperedge association matrix associates nodes of operators, robots, assembly operations, and assembly components with spatial, temporal, and task hyperedges, transforming the relationships between hyperedges and nodes into a row-and-column hyperedge association matrix. Simultaneously, matrix values ​​are defined to determine whether a relationship exists between a hyperedge and a node. The formula is defined as follows: in Represents a set of nodes. These correspond to operator, robot, assembly operation, and part, respectively. Represents a superedge set. These correspond to spatial hyperedge, temporal hyperedge, and task hyperedge, respectively. This represents the hyperedge incidence matrix, which describes the relationship between hyperedges and nodes, where each row represents a node and each column represents a hyperedge. Finally, the features of three types of hyperedges—spatial proximity, temporal dependence, and task relevance—are spliced ​​and fused to generate hyperedge feature vectors. The association between hyperedges and nodes is established through the hyperedge association matrix, thus completing the construction of the spatiotemporal hypergraph and realizing comprehensive modeling of time-varying unpaired relationships among multiple nodes during dynamic assembly. Step S200: Design a stacked graph neural network with self-excitation characteristics based on a multi-event Hawkes process to learn high-order task associations between nodes and capture time-varying task evolution mechanisms; the multi-event Hawkes process estimates the probability of future events by defining a conditional strength function and models the spatiotemporal evolution law between multiple dynamic events. The stacked graph neural network with self-excitation characteristics includes node embedding, hyperedge embedding, temporal fusion, and graph pooling modules. It integrates historical information through recursive aggregation and mapping of multi-layer neighbor features, and designs a temporal representation network and Hawkes process to achieve feature transfer under temporal dependence, thereby realizing the time-sensitive capture of historical neighbor messages and layer-by-layer feature stacking. Step S300: Optimize cognitive reasoning performance by decoupling individual features and collective association features; the individual features convert prior experience of events into unique parameters through a learnable transformation model, and dynamically adjust network parameters in combination with a feature linear modulation model to ensure that prior experience quickly adapts to the uniqueness of events; the collective association features represent event features by connecting spatiotemporal hypergraph nodes, and use graph neural networks to explore the temporal propagation law to generate a new event predictor to capture systematic dynamic interaction features; Step S400: Define loss functions to optimize network performance, including negative log-likelihood loss optimized for individual feature capture and smooth L1 loss optimized for collective association feature capture. By jointly optimizing the two types of losses, the consistency of hyperedge and node relationship transmission is ensured, and finally, dynamic cognitive reasoning of time-varying tasks in human-machine collaborative assembly is realized, decoupling human-machine operation and improving collaboration efficiency.

2. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S100: The human-machine collaborative assembly knowledge spatiotemporal hypergraph generates time-varying non-paired relationships for each node through scene feature structuring, representing the dynamic assembly process; The scene features are structured, the assembly space is perceived by a depth camera, and as time goes by, visual cues are dynamically detected using Graph R-CNN, and the visual cues are transformed into a graph structure with assembly relationships. The time-varying unpaired relation representation is achieved by using the hyperedges of each node in the human-machine collaborative assembly knowledge spatiotemporal hypergraph to represent the assembly relationship involving multiple entities, thus modeling the time-varying unpaired relation in dynamic assembly. At different points in time, the human-machine collaborative assembly knowledge spatiotemporal hypergraph dynamically represents the operator, robot, assembly operation, assembly components and their time-varying unpaired relations, associating the dynamic assembly state at different stages. The time-varying unpaired relationship refers to a many-to-many higher-order relationship between multiple nodes under continuous time variation.

3. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S200: The multi-event Hawkes process determines whether there is a correlation between multiple different nodes at a certain moment by defining conditional strength, i.e., defining triples. In time The conditional strength function that forms the hyperedge estimates the future relationships between multiple nodes, generates the probability of a certain event occurring, models the evolution mechanism between multiple dynamic events in time-varying tasks, and reveals the spatiotemporal evolution law of events. The formula for the conditional strength function is defined as follows: here, Represents event nodes , and In time The basic rate at which links are formed; Indicates an event In time Historical neighbors and Regarding time The historical neighbor influence of the current event; Indicates an event In time Historical neighbors and Regarding time The historical neighbor influence of the current event; Indicates an event In time Historical neighbors and Regarding time The historical neighbor influence of the current event; Represents a node In time Previous historical events; Represents a node In time Previous historical events; Represents a node In time Previous historical events; , and Representing nodes respectively , and In time Previous historical events; in addition, and regarded as The historical neighbors; similarly, define other combinations; Indicates the influence function. Indicates a certain historical time. Indicates an exponentially decaying kernel. Decay over time; The conditional strength describes the expected number of events to occur within a unit time interval at future moments influenced by historical events; if the sequence of past event occurrence times is represented as... So time Conditional strength Represented as: here, The fundamental intensity function represents the event occurrence rate in the absence of any past events. It is an influence function, representing the influence of past events on the occurrence of future events.

4. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S200: The self-excited stacked graph neural network recursively receives, aggregates, and maps features of multi-layer neighbor nodes through node embedding, hyperedge embedding, temporal fusion, and graph pooling modules, integrating historical node features. Simultaneously, a temporal representation network of the graph neural network is designed and fused with a Hawkes process to achieve feature transfer between nodes under temporal dependencies; the formula is defined as follows: here, Defined as an event node located in layer p That is, the node of the event is represented in time. Time representation, Indicates the dimension of the embedding vector; They respectively represent the locations Historical neighbors in the floor and In historical time The characteristic representation, express Historical neighbors in the floor and In historical time The aggregation characterization, This represents an activation function, while and Both are learnable weight matrices; at the same time, utilizing Mapping the embeddings of its historical neighbors captures the temporal sensitivity of past events; furthermore, in the kernel function... Within the GNN, SoftMax is used to represent the time decay effect; the temporal representation of a node is described by aggregating its own information and historical neighbor messages from the previous layer; and layer-wise feature stacking is used for the GNN.

5. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S300: The individual features are decoupled from the non-paired relationships of nodes in the spatiotemporal hypergraph to ensure the uniqueness of events. By encoding general relationships to connect knowledge, prior experience with time-varying characteristics is generated, and a learnable transformation model is constructed to enable prior experience to quickly adapt to the uniqueness of each event. A transfer function is fitted for each event to capture the individual features of each event. Prior experience Convert to have specific parameters The incident The details are as follows: in, Defined as a node , and The event is represented at a specific time, where This refers to the connection operation; based on Prior events Through learnable transformation models Transformed into unique parameters of the event This process is by Parameterization; The learnable transformation model uses Feature Linear Modulation (FiLM), which involves performing a simple affine transformation on the intermediate layer features of the neural network to dynamically adjust network parameters. This scales and shifts prior experience, ensuring that the prior experience quickly adapts to the uniqueness of each event. The scaling and shifting are achieved through fully connected layers. scaling factor of expression and movement factor The implementation is as follows: in and yes The learnable weight matrix and yes The deviation vector; The transfer function is an important part of generating conditional strength and is implemented using the softplus function.

6. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S300: Decouple the non-paired relationships of nodes in the spatiotemporal hypergraph, capture the collective correlation characteristics of all events, and ensure the systematic and dynamic interaction characteristics of events; The collective association feature is to decouple the non-paired relationships of nodes in the spatiotemporal hypergraph, ensure the systematic and dynamic interaction features of events, and characterize the event features associated with nodes by connecting each node in the spatiotemporal hypergraph. As time goes by, the temporal propagation features of event features are explored through graph neural networks, and the spatial features and temporal evolution of events are integrated to capture the collective association features of all events. Define new events as , representing a node , and In time New events have occurred; the following uses a fully connected layer. A predictor for new events of nodes was built: in, Defined as a node , and The event is represented at a specific time. Represents a fully connected layer Required parameters.

7. The human-computer collaborative assembly cognitive reasoning method for spatiotemporal dynamic evolution as described in claim 1, characterized in that, In step S400: Define a loss function, optimize network performance, ensure the consistency of relational propagation of all hyperedges and nodes in the spatiotemporal hypergraph, model the evolution of time-varying unpaired relations in the spatiotemporal hypergraph, and realize human-computer collaborative cognitive strategy reasoning. The loss function includes simultaneous optimization of individual event features and collective correlation features. Individual feature capture optimization is defined by negative log-likelihood loss, which provides appropriate conditional strength for the occurrence or non-occurrence of events. Collective correlation feature capture optimization is defined by smooth L1 loss, which increases error penalty and achieves better network performance. The individual feature capture optimization assumes events The loss has already occurred. as follows: in, and They represent three elements ( )and( In time t The conditional strength for the formation of a hyperedge. Indicates distribution The nodes of the sampled negative samples, the distribution defines the event ( Instances that do not occur; at the same time, This represents the number of negative samples for each event that occurred; The optimization of collective association feature capture, its loss as follows: in, Represents the predicted node propagation value. Represents the actual value of the node.

Images