Methods and equipment for automatically detecting eye contact

Abstract

This invention provides a method and device for automatically detecting eye contact. The method includes: Step 1, extracting features from a node image to obtain a feature image; Step 2, updating the structure of the feature image; Step 3, updating the network nodes of the structure-updated feature image; Step 4, classifying and outputting eye contact behaviors. This invention improves the accuracy of predicting eye contact social behavior by extracting context-specific information about a person's gaze behavior, network topology information, and synchronization timing information between gaze events, and assigning different attention coefficients to the message propagation path of the graph network.

Description

Technical Field

[0001] This invention relates to the field of graph neural network technology, and in particular to a method and device for automatically detecting eye contact. Background Technology

[0002] Humans possess a unique ability to infer others' intentions from their eyes. Eye contact is a primitive form of human communication, expressing underlying social cognition and motivations. Although verbal communication has become the primary form of social interaction, eye contact still plays a crucial role in conveying hidden psychological states and enhancing verbal communication. To better understand human communication, it is necessary not only to analyze language but also to conduct systematic research on the mechanisms of human eye contact interaction. Current technologies cannot fully extract truly effective information, neglecting the potential interactive relationships between different individuals' behaviors, thus weakening the effectiveness of reasoning. Furthermore, a large amount of background noise also interferes with the results of relational reasoning. Therefore, developing a method and device for automatically detecting eye contact, which can effectively overcome the shortcomings of the aforementioned technologies, has become a pressing technical problem for the industry. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, embodiments of the present invention provide a method and device for automatically detecting eye contact.

[0004] In a first aspect, embodiments of the present invention provide a method for automatically detecting eye contact, comprising: step 1, extracting features from a node image to obtain a feature image; step 2, updating the structure of the feature image; step 3, updating the network nodes of the structure-updated feature image; and step 4, classifying and outputting eye contact behavior.

[0005] Based on the above method embodiments, the automatic detection method for eye contact provided in this embodiment of the invention includes the following steps in step 1: Step 1.1, extracting features from the node bounding box using ResNet50 for each node as initial features; Step 1.2, to reduce the number of parameters, using a fully connected layer to compress all node features into 6 dimensions, where formula (1) represents the feature x of each person's facial rectangle or the rectangle of the gaze target. v :R 6 Extracting features using ResNet50 or convolution, where R6 indicates that the image features of the node rectangle are 6-dimensional vectors.

[0006] x v =f(Nodes) (1)

[0007] Where, x vThe extracted image features of node v; Nodes is the image bounding box of 4 nodes in a consecutive frame; f represents the image features of the head region and gaze target region extracted using networks such as ResNet50 or convolution; Step 1.3, obtain the information of the 4 sides and center position p of each person's face rectangle and gaze target rectangle. v :R 06 , where R 06 The node position information is represented as a 6-dimensional feature vector; Step 1.4, generate the information representation of node v:

[0008] xp v = <x v ,p v > (2)

[0009] Among them, xp v Information representation for node v; xp v :R 12 It is a vector with dimension 12; Step 1.5, generate edge information edge_feature v,w :R 12+12 ,

[0010] edge_feature v,w = <xp v XP w > (3)

[0011] Among them, edge_feature v,w The edge information consists of two different nodes v and w in the same frame; xp w Let v represent the information of node w; v represents a node in the graph network; w represents other nodes besides node v; and V represents the set of all nodes in the graph network.

[0012] Based on the above method embodiments, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 2: using a facial-gaze target spatial trajectory extraction method, the image features of nodes have little impact on the dynamic changes in eye contact interaction compared to the node's position information. Spatial trajectory information is extracted using the node's position information. For each frame, an information aggregation weight matrix message_aggregate_weight[4,4] is generated to extract information from the facial-gaze target spatial trajectory. Since any change in the position of any node in the time domain will have a predetermined impact, the information aggregation must consider not only the current frame but also the node position in the previous frame. For the first frame, intra-frame information transmission and feature update are performed. For the following four frames, each node must consider the features of the node in the previous frame. The node feature update is equivalent to aggregating the information of the four nodes in the current frame and the node information in the previous frame. This avoids relying solely on known atmaps for inter-node information transmission and node feature state updates. Otherwise, in a frame, only target1->head1 and target2->head2 information transmission would be performed, and only the features of the two head nodes would be updated.

[0013] nodes_all = [xp h1 XP h2 XP t1 XP t2 ] T

[0014] `nodes_all` represents the total features of two human face nodes and two gaze target nodes in this frame. h1 For the information representation of node h1, xp h2 For the information representation of node h2, xp t1 For the information representation of node t1, xp t2 This represents the information of node t2.

[0015] For a node v,

[0016] Among them, edge_feature v The edge_feature is the information of all nodes and node v forming an edge in the same frame. v,h1 edge_feature is the edge information composed of two different nodes v and h1 in the same frame. v,h2 edge_feature is the edge information composed of two different nodes v and h2 in the same frame. v,t1 For the edge information composed of two different nodes v and t1 in the same frame, the weights of the edges between node v and each node are obtained, and the edge_feature is calculated. v After a fully connected layer fc :R 4×24 →R 4 A sigmoid function is used to obtain the aggregated weights of all nodes in the same frame from node v, called message_aggregate_weight. v ;

[0017] message_aggregate_weight v =σ(f c (edge_feature v )) represents the weights from the remaining nodes to node v, and is normalized to the range [0,1]. σ is the sigmoid activation function, and f c (·) represents a function expression;

[0018] For intra-frame nodes, pos_message = f c ([p h1 p h2 p t1 p t2 ] T ), where pos_message is the information transmitted from all nodes to one node in the same frame, p h1 p represents the position information of the block diagram of node h1 within the frame image. h2 For the position information of the block diagram of node h2 in the frame image, p t1 p represents the position information of the block diagram of node t1 in the frame image. t2 To determine the location of node t2 within the frame image, the message passing of the node's position is obtained by linearly transforming the nodes within the frame.

[0019] The method involves calculating the sum of the implicit edges of all non-neighbor nodes and the adjacency matrix weights of neighbor nodes to form the final weights of each of the remaining nodes in the frame image to node v (message_tmp). v ,

[0020] message_tmp v =attmap v +message_aggregate_weight v

[0021] message_tmp v After expanding the dimension from [4] to [4,6], multiplying it with pos_message yields the information transfer message_aggregate of all edges from nodes to node v and the spatial changes of the real neighbor nodes to node v. v .

[0022] Based on the above method embodiments, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 3 as follows: when frame_id = 0, the updated position information of node v is the sum of the spatial information of the other nodes transmitted in this frame. Tensors of dimension 6:

[0023]

[0024] in, For the updated node position information of node v in frame number frame_id, message_aggreate v The position information of all nodes in the same frame is passed to node v. When frame_id is not equal to 0, the information of this node in the previous frame must also be passed. Therefore, the position information of the node in the previous frame is passed through a fully connected layer to obtain last_message:

[0025]

[0026] Where last_message is the information passed from a node in the previous frame to the same node in the current frame. To obtain the updated node position information of node v in frame number frame_id-1, we sum the messages passed to each node position in the current frame and then add them to last_message to obtain the impact of the remaining node position information on node v.

[0027]

[0028] The loss function used is cross-entropy.

[0029] Based on the above method embodiments, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 4 as follows: the node image features nodes_feature of dimension [batch_size,5,4,6] and the updated node position information pos_updata of dimension [batch_size,5,4,6] are concatenated in the last dimension to obtain a tensor of dimension [batch_size,5,4,12], which is then transformed into dimension [batch_size,240]. After passing through several fully connected layers, the [batch_size,240]-dimensional tensor is transformed into a [batch_size,6]-dimensional tensor, which is the Atomic_class of all classification results in a batch; where batch_size is the number of samples taken in one training session.

[0030] Secondly, embodiments of the present invention provide an apparatus for automatically detecting eye contact, comprising: a first main module for implementing step 1, extracting features from a node image to obtain a feature image; a second main module for implementing step 2, updating the structure of the feature image; a third main module for implementing step 3, updating network nodes in the structure-updated feature image; and a fourth main module for implementing step 4, classifying and outputting eye contact behavior.

[0031] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0032] At least one processor; and

[0033] At least one memory communicatively connected to the processor, wherein:

[0034] The memory stores program instructions that can be executed by the processor. The processor can call the program instructions to execute the automatic detection of eye contact method provided by any of the various implementations of the first aspect.

[0035] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute a method for automatically detecting eye contact provided by any of the various implementations of the first aspect.

[0036] The method and device for automatically detecting eye contact provided in this invention improve the accuracy of predicting eye contact social behavior by extracting context-specific information of a person's gaze behavior, network topology information, and synchronization timing information between gaze events, and assigning different attention coefficients to the message propagation path of the graph network. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a method for automatically detecting eye contact provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of the device for automatically detecting eye contact provided in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0042] By introducing latent factors of the spatiotemporal variation features of the "face-gaze target" between non-neighboring nodes, the correlation of non-neighboring nodes is learned, which expands the effectiveness of information transmission between graph neural network nodes and improves the accuracy of the prediction model. To address the problem of feature noise caused by dynamic changes in the face and target, a "face-gaze target" spatial trajectory extraction method is designed. This method calculates the spatiotemporal trajectory differences between different gaze interaction behaviors and learns attention weights to reduce the impact of dynamic image noise. Based on this idea, this invention provides a method for automatically detecting eye contact. See [link to relevant documentation]. Figure 1 The method includes: Step 1, extracting features from the node image to obtain a feature image; Step 2, updating the structure of the feature image; Step 3, updating the network nodes of the structure-updated feature image; Step 4, classifying and outputting the gaze behavior.

[0043] Specifically, gaze communication is defined in six basic atomic categories: Single refers to an individual's gaze without any intention of social communication; Mutual refers to two people looking into each other's eyes, which is the most powerful way for humans to establish communication connections. Mutual gaze attracts attention, initiates conversation, maintains communication, expresses feelings of trust and extroversion, and signals the availability of interaction in situations such as passing items to a partner; Avert refers to avoiding gaze. To avoid mutual gaze, one subject's gaze will be moved away from the other subject. Avoidance of gaze indicates distrust, introversion, fear, modulates intimacy, communicates thoughts, or indicates cognitive effort, such as shifting gaze before answering a question; Referent gaze refers to referencing gaze, when one person attempts to draw another person's attention to a target through gaze. Referent gaze indicates the intention to notify, share, or request something. Referent gaze is used to eliminate uncertainty of reference and to react quickly; Follow refers to when one person perceives another person's gaze and shifts their gaze to the target that person is looking at. Eye following is used to understand the partner's intention; Share refers to two people looking at the same target together.

[0044] Define a social graph g = (V, E), where nodes v ∈ V take a specific value in {1, 2, ..., |V|} to represent entities in the social scene (such as scene objects, people). An edge e = (v, w) ∈ E indicates a directed edge v → w, representing all person-to-person gaze interactions or person-to-scene relationships. There is a special node s ∈ V representing the social scene. For node v, a V-dimensional vector xv ∈ RV is used to represent the node. Similarly, an E-dimensional vector xv, w ∈ RE is used to represent the edge e = (v, w). Each node v (excluding s) has an output state lv ∈ {single, mutual, vert, refer, follow, share}, and an adjacency matrix is ​​used. Let attmapv,w represent the communication structure of the entire social graph g. Each element attmapv,w represents the connection relationship from node v to w. The goal is to determine the atomic-level social gaze type of the two people based on the head features and positions of the two people in N consecutive frames of images, as well as the features and positions of their gaze targets.

[0045] Initialization: Define two people's faces and their gaze targets as nodes, for a total of 4 nodes. Based on the features and positional information of these 4 nodes, there is an input tensor.

[0046] enter:

[0047] Nodes—[batch_size,N,4,3,224,224] tensor, a block diagram of 4 nodes in N frames;

[0048] Poses is a tensor of [batch_size, N, 4, 6], which contains the position information of 4 nodes in N frames.

[0049] attmap——[batch_size,N,4,4] tensor, attention map (adjacency matrix) for N frames;

[0050] Output:

[0051] Atomic_class——[batch_size,6] tensor, representing the probability of classifying gaze-based social interaction types, with 6 categories;

[0052] The adjacency matrix attmap is set. If the gaze target of head1 is head2, then attmap[0][1] = 1; if the gaze target of head1 is not head2, then attmap[0][2] = 1; if head1 has no gaze target, then attmap[0][:] = 0. If the gaze target of head2 is head1, then attmap[1][0] = 1; if the gaze target of head2 is the same as the gaze target of head1, then attmap[1][2] = 1; if the gaze target of head2 is different from the gaze target of head1, then attmap[1][3] = 1; if head2 has no gaze target, then attmap[1][:] = 0 and attmap[2:][:] = 0. This technical solution sets N = 5, only considering the case where the total number of gaze targets of the two people in 5 consecutive frames is >= 7. The message passing steps are designed in three stages: node image feature extraction stage, graph structure update stage, and node update stage.

[0053] Based on the above method embodiments, as an optional embodiment, the automatic detection method for eye contact provided in this embodiment of the invention specifically includes the following steps in step 1: Step 1.1, extracting features from the node bounding box using ResNet50 for each node as initial features; Step 1.2, to reduce the number of parameters, using a fully connected layer to compress all node features into 6 dimensions, where formula (1) represents the feature x of each person's facial rectangle or the rectangle of the gaze target. v :R 6 Extract using ResNet50 or convolution, where R 6 The image features representing the node rectangle are 6-dimensional vectors.

[0054] x v =f(Nodes) (1)

[0055] Where, x vThe extracted image features of node v; Nodes is the image bounding box of 4 nodes in a consecutive frame; f represents the image features of the head region and gaze target region extracted using networks such as ResNet50 or convolution; Step 1.3, obtain the information of the 4 sides and center position p of each person's face rectangle and gaze target rectangle. v :R 06 , where R 06 The node position information is represented as a 6-dimensional feature vector; Step 1.4, generate the information representation of node v:

[0056] xp v = <x v ,p v > (2)

[0057] Among them, xp v Information representation for node v; xp v :R 12 It is a vector with dimension 12; Step 1.5, generate edge information edge_feature v,w :R 12+12 ,

[0058] edge_feature v,w = <xp v XP w > (3)

[0059] Among them, edge_feature v,w The edge information consists of two different nodes v and w in the same frame; xp w Let v represent the information of node w; v represents a node in the graph network; w represents other nodes besides node v; and V represents the set of all nodes in the graph network.

[0060] Based on the above method embodiments, as an optional embodiment, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 2: using a face-gaze target spatial trajectory extraction method, the image features of nodes have little impact on the dynamic changes of eye contact interaction compared to the position information of nodes, using the position information of nodes to extract spatial trajectory information, generating an information aggregation weight matrix message_aggregate_weight[4,4] for each frame, extracting information from the face-gaze target spatial trajectory, since any change in the position of any node in the time domain will have a predetermined impact, the aggregation of information must consider not only the current frame but also the node position of the previous frame, performing intra-frame information transmission and feature update for the first frame, and for the following 4 frames, each node must consider the features of the node in the previous frame, the feature update of the node is equivalent to aggregating the information of the 4 nodes in the current frame and the information of the node in the previous frame; avoiding relying solely on known atmaps for inter-node information transmission and node feature state update, otherwise only target1->head1 and target2->head2 information transmission would be performed in a frame, only updating the features of the two head nodes.

[0061] nodes_all = [xp h1 XP h2 XP t1 XP t2 ] T

[0062] `nodes_all` represents the total features of two human face nodes and two gaze target nodes in this frame. h1 For the information representation of node h1, xp h2 For the information representation of node h2, xp t1 For the information representation of node t1, xp t2 This represents the information of node t2.

[0063] For a node v,

[0064] Among them, edge_feature v The edge_feature is the information of all nodes and node v forming an edge in the same frame. v,h1 edge_feature is the edge information composed of two different nodes v and h1 in the same frame. v,h2 edge_feature is the edge information composed of two different nodes v and h2 in the same frame. v,t1 For the edge information composed of two different nodes v and t1 in the same frame, the weights of the edges between node v and each node are obtained, and the edge_feature is calculated. vAfter a fully connected layer f c :R 4×24 →R 4 A sigmoid function is used to obtain the aggregated weights of all nodes in the same frame from node v, called message_aggregate_weight. v ;

[0065] message_aggregate_weight v =σ(f c (edge_feature v )) represents the weights from the remaining nodes to node v, and is normalized to the range [0,1]. σ is the sigmoid activation function, and f c (·) represents a function expression;

[0066] For intra-frame nodes, pos_message = f c ([p h1 p h2 p t1 p t2 ] T ), where pos_message is the information transmitted from all nodes to one node in the same frame, p h1 p represents the position information of the block diagram of node h1 within the frame image. h2 For the position information of the block diagram of node h2 in the frame image, p t1 p represents the position information of the block diagram of node t1 in the frame image. t2 To determine the location of node t2 within the frame image, the message passing of the node's position is obtained by linearly transforming the nodes within the frame.

[0067] The method involves calculating the sum of the implicit edges of all non-neighbor nodes and the adjacency matrix weights of neighbor nodes to form the final weights of each of the remaining nodes in the frame image to node v (message_tmp). v ,

[0068] message_tmp v =attmap v +message_aggregate_weight v

[0069] message_tmp v After expanding the dimension from [4] to [4,6], multiplying it with pos_message yields the information transfer message_aggregate of all edges from nodes to node v and the spatial changes of the real neighbor nodes to node v. v .

[0070] Based on the above method embodiments, as an optional embodiment, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 3 as follows: when frame_id = 0, the updated position information of node v is the sum of the spatial information of the other nodes transmitted in this frame. Tensors of dimension 6:

[0071]

[0072] in, For the updated node position information of node v in frame number frame_id, message_aggreate v The position information of all nodes in the same frame is passed to node v. When frame_id is not equal to 0, the information of this node in the previous frame must also be passed. Therefore, the position information of the node in the previous frame is passed through a fully connected layer to obtain last_message:

[0073]

[0074] Where last_message is the information passed from a node in the previous frame to the same node in the current frame. To obtain the updated node position information of node v in frame number frame_id-1, we sum the messages passed to each node position in the current frame and then add them to last_message to obtain the impact of the remaining node position information on node v.

[0075]

[0076] The loss function used is cross-entropy.

[0077] Based on the above method embodiments, as an optional embodiment, the automatic detection of eye contact method provided in this embodiment of the invention specifically includes step 4 as follows: the node image features nodes_feature of dimension [batch_size, 5, 4, 6] and the updated node position information pos_updata of dimension [batch_size, 5, 4, 6] are concatenated in the last dimension to obtain a tensor of dimension [batch_size, 5, 4, 12], which is then transformed into dimension [batch_size, 240]. After passing through several fully connected layers, the [batch_size, 240]-dimensional tensor is transformed into a [batch_size, 6]-dimensional tensor, which is the Atomic_class of all classification results in a batch; where batch_size is the number of samples taken in one training session.

[0078] The method for automatically detecting eye contact provided in this invention improves the accuracy of predicting eye contact social behavior by extracting context-specific information about a person's gaze behavior, network topology information, and synchronization timing information between gaze events, and assigning different attention coefficients to the message propagation path of the graph network.

[0079] The various embodiments of the present invention are implemented through programmed processing using a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention can be encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide an apparatus for automatically detecting eye contact, which is used to execute the method for automatically detecting eye contact in the above method embodiments. See also... Figure 2 The device includes: a first main module for implementing step 1, extracting features from the node image to obtain a feature image; a second main module for implementing step 2, updating the structure of the feature image; a third main module for implementing step 3, updating the network nodes of the structure-updated feature image; and a fourth main module for implementing step 4, classifying and outputting gaze behavior.

[0080] The automatic eye contact detection device provided in this embodiment of the invention employs... Figure 2 Several modules in the model extract contextual information about a person's gaze behavior, network topology information, and synchronization timing information between gaze events, and assign different attention coefficients to the message propagation path of the graph network, thereby improving the prediction accuracy of gaze-based social behavior.

[0081] It should be noted that the apparatus in the device embodiments provided by the present invention can be used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The difference lies only in the setting of corresponding functional modules. Its principle is basically the same as that of the above device embodiments provided by the present invention. As long as those skilled in the art, based on the above device embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and improve the apparatus in the above device embodiments while ensuring the practicality of the technical solutions, they can obtain corresponding device-type embodiments for implementing the methods in other method-type embodiments. For example:

[0082] Based on the above device embodiments, as an optional embodiment, the device for automatically detecting eye contact provided in this embodiment of the invention further includes: a first submodule, used to implement step 1, specifically including: step 1.1, extracting features from the node bounding box using ResNet50 for each node as initial features; step 1.2, to reduce the number of parameters, using a fully connected layer to compress all node features into 6 dimensions, where formula (1) represents the feature x of each person's facial rectangle or the rectangle of the gaze target. v :R 6 Extract using ResNet50 or convolution, where R 6 The image features representing the node rectangle are 6-dimensional vectors.

[0083] x v =f(Nodes) (1)

[0084] Where, x v The extracted image features of node v; Nodes is the image bounding box of 4 nodes in a consecutive frame; f represents the image features of the head region and gaze target region extracted using networks such as ResNet50 or convolution; Step 1.3, obtain the information of the 4 sides and center position p of each person's face rectangle and gaze target rectangle. v :R 06 , where R 06 The node position information is represented as a 6-dimensional feature vector; Step 1.4, generate the information representation of node v:

[0085] xp v = <x v ,p v > (2)

[0086] Among them, xp v Information representation for node v; xp v :R 12 It is a vector with dimension 12; Step 1.5, generate edge information edge_feature v,w :R 12+12 ,

[0087] edge_feature v,w = <xp v XP w > (3)

[0088] Among them, edge_feature v,w The edge information consists of two different nodes v and w in the same frame; xp w Let v represent the information of node w; v represents a node in the graph network; w represents other nodes besides node v; and V represents the set of all nodes in the graph network.

[0089] Based on the above-described device embodiments, as an optional embodiment, the device for automatically detecting eye contact provided in this embodiment of the invention further includes: a second submodule, used to implement step 2, specifically including: using a facial-gaze target spatial trajectory extraction method, where the image features of a node have little impact on the dynamic changes in eye contact interaction compared to the node's position information, using the node's position information to extract spatial trajectory information, generating an information aggregation weight matrix message_aggregate_weight[4,4] for each frame, extracting information from the facial-gaze target spatial trajectory, since any change in the position of any node in the time domain will have a predetermined impact, the information aggregation must consider not only the current frame but also the node position of the previous frame, performing intra-frame information transmission and feature update for the first frame, and for the following 4 frames, each node must consider the features of the node in the previous frame, and the node feature update is equivalent to aggregating the information of the 4 nodes in the current frame and the node information of the previous frame; avoiding relying solely on known atmaps for inter-node information transmission and node feature state update, otherwise only target1->head1 and target2->head2 information transmission would be performed in a frame, only updating the features of the two head nodes.

[0090] nodes_all = [xp h1 XP h2 XP t1 XP t2 ] T

[0091] `nodes_all` represents the total features of two human face nodes and two gaze target nodes in this frame. h1 For the information representation of node h1, xp h2 For the information representation of node h2, xp t1 For the information representation of node t1, xp t2 This represents the information of node t2.

[0092] For a node v,

[0093] Among them, edge_feature v The edge_feature is the information of all nodes and node v forming an edge in the same frame. v,h1 edge_feature is the edge information composed of two different nodes v and h1 in the same frame. v,h2 edge_feature is the edge information composed of two different nodes v and h2 in the same frame. v,t1For the edge information composed of two different nodes v and t1 in the same frame, the weights of the edges between node v and each node are obtained, and the edge_feature is calculated. v After a fully connected layer f c :R 4×24 →R 4 A sigmoid function is used to obtain the aggregated weights of all nodes in the same frame from node v, called message_aggregate_weight. v ;

[0094] message_aggregate_weight v =σ(f c (edge_feature v )) represents the weights from the remaining nodes to node v, and is normalized to the range [0,1]. σ is the sigmoid activation function, and f c (·) represents a function expression;

[0095] For intra-frame nodes, pos_message = f c ([p h1 p h2 p t1 p t2 ] T ), where pos_message is the information transmitted from all nodes to one node in the same frame, p h1 p represents the position information of the block diagram of node h1 within the frame image. h2 For the position information of the block diagram of node h2 in the frame image, p t1 p represents the position information of the block diagram of node t1 in the frame image. t2 To determine the location of node t2 within the frame image, the message passing of the node's position is obtained by linearly transforming the nodes within the frame.

[0096] The method involves calculating the sum of the implicit edges of all non-neighbor nodes and the adjacency matrix weights of neighbor nodes to form the final weights of each of the remaining nodes in the frame image to node v (message_tmp). v ,

[0097] message_tmp v =attmap v +message_aggregate_weight v

[0098] message_tmp vAfter expanding the dimension from [4] to [4,6], multiplying it with pos_message yields the information transfer message_aggregate of all edges from nodes to node v and the spatial changes of the real neighbor nodes to node v. v .

[0099] Based on the above-described device embodiments, as an optional embodiment, the device for automatically detecting eye contact provided in this invention further includes: a third submodule, used to implement step 3, specifically including: when frame_id = 0, the updated position information of node v is the sum of the spatial information of the remaining nodes transmitted in that frame. Tensors of dimension 6:

[0100]

[0101] in, For the updated node position information of node v in frame number frame_id, message_aggreate v The position information of all nodes in the same frame is passed to node v. When frame_id is not equal to 0, the information of this node in the previous frame must also be passed. Therefore, the position information of the node in the previous frame is passed through a fully connected layer to obtain last_message:

[0102]

[0103] Where last_message is the information passed from a node in the previous frame to the same node in the current frame. To obtain the updated node position information of node v in frame number frame_id-1, we sum the messages passed to each node position in the current frame and then add them to last_message to obtain the impact of the remaining node position information on node v.

[0104]

[0105] The loss function used is cross-entropy.

[0106] Based on the above device embodiments, as an optional embodiment, the device for automatically detecting eye contact provided in this embodiment of the invention further includes: a fourth submodule, used to implement step 4, specifically including: concatenating the node image features nodes_feature of dimension [batch_size, 5, 4, 6] and the updated node position information pos_updata of dimension [batch_size, 5, 4, 6] in the last dimension to obtain a tensor of dimension [batch_size, 5, 4, 12], and then transforming it into dimension [batch_size, 240]; after passing through several fully connected layers, the [batch_size, 240]-dimensional tensor is transformed into a [batch_size, 6]-dimensional tensor, which is the Atomic_class of all classification results in a batch; where batch_size is the number of samples taken in one training session.

[0107] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, this embodiment of the invention provides an electronic device, such as... Figure 3 As shown, the electronic device includes at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor can invoke logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0108] Furthermore, when the logical instructions in at least one of the aforementioned memories can be implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Based on this understanding, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatically detecting eye contact, characterized in that, include: Step 1: Extract features from the node image to obtain the feature image; Step 2: Update the structure of the feature image; Step 3: Update the network nodes of the feature image after the structure update; Step 4: Classify and output the gaze behavior. Step 2 specifically includes: using a face-gaze target spatial trajectory extraction method, where the image features of nodes have a smaller impact on the dynamic changes in human gaze interaction compared to the node's position information, and using the node's position information to extract spatial trajectory information, generating an information aggregation weight matrix message_aggregate_weight[4,4] for each frame, and extracting information from the face-gaze target spatial trajectory. Since any change in the position of any node in the time domain will have a predetermined impact, the information aggregation must consider not only the current frame but also the node position in the previous frame. For the first frame, information transmission and feature updates are performed within the frame, while for the following four frames, each node must consider the features of the node in the previous frame. The node feature update is equivalent to aggregating the information of the four nodes in the current frame and the node information in the previous frame; avoiding relying solely on known atmaps for information transmission between nodes and node feature state updates, otherwise only target1->head1 and target2->head2 information transmission would be performed in a frame, and only the features of the two head nodes would be updated. This represents the total features of two human face nodes and two gaze target nodes in this frame. For the information representation of node h1, For the information representation of node h2, For the information representation of node t1, This represents the information of node t2. For a node v, , in, This refers to the information about the edges formed by all nodes and node v in the same frame. The edge information consists of two different nodes v and h1 in the same frame. The edge information consists of two different nodes v and h2 in the same frame. For the edge information composed of two different nodes v and t1 in the same frame, the weights of the edges between node v and each node are obtained. After a fully connected layer A sigmoid function is used to obtain the aggregated weights of information from all nodes to node v in the same frame. ; Let be the weights from the remaining nodes to node v, and normalized to the range [0,1]. It is the sigmoid activation function. It is a function expression; For intra-frame nodes: ,in The information transmitted from all nodes in the same frame to one node. This refers to the position information of the block diagram of node h1 within the frame image. This provides the position information of the block diagram of node h2 within the frame image. This refers to the position information of the block diagram of node t1 within the frame image. To determine the location of node t2 within the frame image, the message passing of the node's position is obtained by linearly transforming the nodes within the frame. The method involves calculating the sum of the implicit edges of all non-neighbor nodes and the adjacency matrix weights of neighbor nodes to form the final weights of all other nodes in the frame image relative to node v. , Will After the dimension is expanded from [4] to [4,6], and Multiplying these together yields information about the spatial changes of all edges from nodes to node v and the actual neighbor nodes to node v. .

2. The method for automatically detecting eye contact according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1, using ResNet50 to extract features from the node bounding box for each node as initial features; Step 1.2, to reduce the number of parameters, using a fully connected layer to compress all node features into 6 dimensions, where formula (1) represents the features of each person's facial rectangle or the rectangle of the gaze target. Extract using ResNet50 or convolution, where The image features representing the node rectangle are 6-dimensional vectors. （1） in, The extracted image features of node v; Nodes is the image bounding box of 4 nodes in a consecutive frame; This indicates that ResNet50 or a convolutional network is used to extract image features of the head region and the gaze target region; step 1.3, obtain the information of the four sides and the center position of each person's face rectangle and gaze target rectangle. ,in, The node position information is represented as a 6-dimensional feature vector; Step 1.4, generate the information representation of node v: （2） in, The information representation for node v; It is a vector with dimension 12; Step 1.5, generate edge information. , （3） in, The edge information consists of two different nodes v and w in the same frame; Let v represent the information of node w; v represents a node in the graph network; w represents other nodes besides node v; and V represents the set of all nodes in the graph network.

3. The method for automatically detecting eye contact according to claim 2, characterized in that, Step 3 specifically includes: when frame_id=0, the node The updated location information is the sum of the spatial information of the other nodes transmitted in this frame. Tensors of dimension 6: in, This refers to the updated position information of node v in frame number frame_id. To pass the position information of all nodes in the same frame to node v, when frame_id is not equal to 0, the information of that node in the previous frame must also be passed. Therefore, the position information of the node in the previous frame is obtained by passing it through a fully connected layer. : in, This refers to the information passed from a node in the previous frame to the same node in the current frame. To obtain the updated node position information of node v in frame number frame_id-1, we sum the messages passed to each node position in the current frame and then add them to last_message to obtain the impact of the remaining node position information on node v. The loss function used is cross-entropy.

4. The method for automatically detecting eye contact according to claim 3, characterized in that, Step 4 specifically includes: concatenating the node image features nodes_feature of dimension [batch_size, 5, 4, 6] and the updated node position information pos_updata of dimension [batch_size, 5, 4, 6] in the last dimension to obtain a tensor of dimension [batch_size, 5, 4, 12], which is then transformed into dimension [batch_size, 240]. After passing through several fully connected layers, the [batch_size, 240]-dimensional tensor is transformed into a [batch_size, 6]-dimensional tensor, which is the Atomic_class of all classification results in a batch. Here, batch_size is the number of samples taken in one training session.

5. A device for automatically detecting eye contact, characterized in that, To implement the method according to any one of claims 1 to 4, the apparatus comprises: a first main module for implementing step 1, extracting features from a node image to obtain a feature image; a second main module for implementing step 2, updating the structure of the feature image; a third main module for implementing step 3, updating network nodes in the structure-updated feature image; and a fourth main module for implementing step 4, classifying and outputting gaze behavior.

6. An electronic device, characterized in that, include: At least one processor, at least one memory, and a communication interface; wherein, The processor, memory, and communication interface communicate with each other; The memory stores program instructions that can be executed by the processor, which invokes the program instructions to perform the method described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method described in any one of claims 1 to 4.

Images