A family dining behavior interaction modeling and analysis method based on a graph neural network

Abstract

The application discloses a kind of based on graph neural network's family dining behavior interaction modeling and analysis method, it is related to the technical field of behavior identification.Extract the features of family members, actions and environmental states by collecting multi-modal raw behavior data, and construct a family interaction heterogeneous graph.Based on the family interaction heterogeneous graph, use a learnable attribute sampling algorithm to calculate the saliency of node attributes, and use a dual-path feature extraction model to capture local subtle action features and global long-range interaction features;By compression-excitation channel attention algorithm and multiple attention aggregation algorithm, the feature channels are reconstructed and dynamically weighted, and finally the interaction probability between members is calculated to analyze the interaction properties of family dining.The application effectively captures the high-order interaction relationship across members and behaviors, improves the analysis accuracy of family dining behavior, and solves the problems of redundant information interference and lack of directionality in feature extraction in the prior art.

Description

Technical Field

[0001] This invention relates to the technical field of behavior recognition, and in particular to a method for modeling and analyzing family dining behavior interactions based on graph neural networks. Background Technology

[0002] In recent years, shared meals have become a high-frequency and socially significant interactive activity in daily family life. Often accompanied by verbal communication, coordinated actions, and changes in the environment, they reflect the interaction patterns and relationship status among family members. However, due to the characteristics of shared meals—multiple participants, complex behavioral types, and implicit interaction relationships—related behavioral data typically exhibits multimodal, highly heterogeneous, and complex temporal correlations, posing significant challenges to behavioral modeling and analysis.

[0003] Currently, Chinese invention patent application CN112465226A discloses a user behavior prediction method based on feature interaction and graph neural networks. This method obtains a dataset containing relational network data, node feature data, and user behavior data as the original dataset; preprocesses the original data to obtain the graph's adjacency matrix, user node feature matrix, and user behavior labels; uses the preprocessed data for feature interaction, and performs graph propagation using the resulting user node feature matrix and relational network adjacency matrix; uses the user features obtained after graph propagation to predict user behavior labels, thereby training the graph neural network model; inputs test data into the trained graph neural network model, and outputs the predicted results. However, related technologies lack a unified structured modeling mechanism for multiple entities such as family members, actions, and environmental states, making it difficult to accurately depict the complex interaction relationships between multiple members in a family meal scenario. They also fail to adaptively filter and align multimodal features under graph structure constraints, making it difficult to effectively suppress the interference of redundant information on behavior analysis results and lacking the ability to dynamically adjust the importance of different interaction paths. Summary of the Invention

[0004] The technical problem solved by this invention is that existing technologies are difficult to implement heterogeneous modeling mechanisms that constrain the structural conditions of multiple entities such as family members, behaviors, and environmental states in the context of family meals. They are also difficult to achieve adaptive filtering and alignment of multimodal features in the process of interaction relationship modeling. Furthermore, they fail to simultaneously characterize local behavioral features and high-order interaction relationships across members and behaviors, and lack the ability to dynamically adjust the importance of different interaction paths, resulting in inaccurate and unstable results in the analysis of family meal behavior interactions.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for modeling and analyzing family dining behavior interactions based on graph neural networks, comprising the following steps: Step S1: Obtain multimodal raw behavioral data, define entities and extract relationships from the multimodal raw behavioral data, and construct a heterogeneous graph of family interactions; Step S2: Nonlinearly map the initial node features of each node in the family interaction heterogeneous graph to obtain each modal feature component. By calculating the correlation entropy value of each modal feature component in the time domain and spatial domain, obtain the modal feature weight distribution map and generate the initial embedding matrix. Step S3: Based on the initial embedding matrix, extract local subtle action features and global remote interaction features through local random feature enhancement and random walk mechanism, and construct a family meal behavior feature map; Step S4: Perform feature channel calibration on the family dining behavior feature map to obtain the fused interaction feature matrix; Step S5: Based on the fused interaction feature matrix, the final interaction probability between family member node pairs is calculated using a multi-attention aggregation algorithm. The final interaction probability is then compared with a preset behavior association threshold to obtain the behavioral analysis results of the family meal interaction.

[0006] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S1 specifically includes: Step S11: Select family member samples according to the preset sociological role differentiation criteria, collect them through a multimodal perception device, and obtain multimodal raw behavioral data; The multimodal sensing device includes a depth camera, an infrared thermal imager, an RGB camera, a directional voice acquisition head, a microphone array, a photosensitive sensor, a noise sensor, and a pressure sensor; The multimodal raw behavioral data includes video image sequences, audio signal streams, and environmental perception parameters; The environmental sensing parameters include ambient light intensity, sound decibels of the monitored environment, and pressure data. Step S12: The time synchronization calibration algorithm is used to perform benchmark unification processing on the multimodal raw behavior data, and the entity feature components of the multimodal raw behavior data within the sampling window are extracted to obtain the quasi-synchronous behavior feature sequence. Step S13: Extract the pose features, visual focus features, and verbal emotion features of family members from the quasi-synchronous behavior feature sequence, and define a heterogeneous node set based on the entity attributes in the entity feature components; The heterogeneous node set includes family member nodes, action nodes, and environmental attribute nodes; Step S14: Combining the heterogeneous node set, the semantic association strength of the quasi-synchronous behavior feature sequence between different nodes is calculated to establish the connection edges between each node, and the graph construction operator is called to convert the connection edges and corresponding weight values ​​into an adjacency matrix to construct the family interaction heterogeneous graph.

[0007] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S2 specifically includes: Step S21: The learnable attribute sampling operator is used to perform nonlinear spatial mapping on the initial features of nodes in the family interaction heterogeneous graph to obtain each modal feature component, and each modal feature component is mapped to the normalized node attribute space of the same dimension to obtain the normalized node attribute set. Step S22: Based on the entity attributes in the heterogeneous node set, calculate the correlation entropy values ​​of each modal feature component in the time and spatial domains to obtain the modal feature weight distribution map. Step S23: Perform a weighted selection operation on the normalized node attribute set based on the modal feature weight distribution map, identify and remove redundant feature components with weights lower than the preset significance threshold, and obtain the target node attribute subset. Step S24: Feed the target node attribute subset back to the family interaction heterogeneous graph, use the adjacency matrix to extract the neighborhood attribute information of each node, and use the feature propagation mechanism to interact and fuse the neighborhood attribute information and the target node attribute subset to obtain the initial embedding features of each node. The neighborhood attribute information includes the action semantic components of neighboring nodes, the environmental state components, and the behavioral representation components of associated members. The initial embedding features of each node are dimension-aligned using linear projection transformation to obtain the initial embedding matrix.

[0008] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, in step S22, the modal feature weight distribution map is obtained by calculating the correlation entropy values ​​of each modal feature component in the time domain and the spatial domain. The processing logic includes: Step S221: The information entropy calculation method is used to extract the numerical fluctuation components of the multimodal raw behavioral data within the sampling time window, the probability distribution uncertainty of each modal feature component is calibrated, and the time dimension entropy component is calculated. Step S222: The mutual information evaluation algorithm is used to extract the association strength between nodes in the heterogeneous node set. The association strength is calibrated according to the probability difference of the spatial coupling degree between nodes, and the spatial dimension entropy component is calculated. Step S223: The temporal entropy components and spatial entropy components are linearly weighted and superimposed. The Sigmoid function is used to map the superimposed values ​​to a probability interval to obtain the modal feature weight distribution map corresponding to each modal feature component.

[0009] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, the processing logic for weighted selection of normalized node attribute sets based on modal feature weight distribution graph in step S23 includes: Step S231: Extract the feature vectors from the normalized node attribute set and the corresponding weight coefficients from the modal feature weight distribution map. Perform a weighted operation using the multiplication operator to obtain the weighted feature vector. The calculation formula is as follows: ; in, Represents the weighted eigenvector. This represents the weight coefficients corresponding to the modal feature weight distribution plot. Represents the feature vector in the normalized node attribute set; Step S232: Construct a selection operator using a preset significance threshold. The calculation formula is as follows: ; in, Indicates the selection operator, This indicates a preset significance threshold; The weighted eigenvectors are mapped using a selection operator, and then eliminated... Corresponding redundant components; Step S233, extract the required parameters. Linear aggregation is performed on the weighted feature vectors to obtain a subset of the target node attributes.

[0010] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S3 specifically includes: Step S31: Input the initial embedding matrix into the local path of the dual-path feature extraction model, and process the individual neighborhood of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. Step S32: Extract the adjacency matrix corresponding to the heterogeneous graph of family interactions. Use the weight mapping operator to map the weight coefficients in the modal feature weight distribution graph to the corresponding elements of the adjacency matrix. Introduce a jump factor to calculate the cross-level transition probability between any two nodes in the heterogeneous node set. Use Markov chain state transition logic to calculate the transition probability between any two nodes in the heterogeneous node set. The transition probabilities form a probability distribution mapping set, the calculation formula of which is: ; in, Represents nodes in a heterogeneous node set With nodes The transition probability between them Indicates the jump factor. This represents the weight coefficients corresponding to the modal feature weight distribution plot. This represents the numerical value indicating the connection relationships between nodes in the adjacency matrix. Indicates a node All first-order neighbor nodes, The jump gain function represents the sampling across nodes. Represents a node The set of nodes corresponding to the individual's neighborhood. Indicates traversing nodes The variable index number of the set of nodes corresponding to the individual's neighborhood. This represents any node in the heterogeneous node set. Represents another arbitrary node in the heterogeneous node set; Step S33: Invoke the jump-type random walk mechanism to perform state transitions with a preset step size from family member nodes in the heterogeneous node set based on the probability distribution mapping set; During the jump process, the jump factor is used to drive the sampling points to cross the heterogeneous topology constraints to extract the interaction path between non-directly adjacent nodes, and the initial path sequence is obtained. The heterogeneous topology consists of family member nodes, action nodes, environment attribute nodes, and connecting edges between nodes. We use a feature enhancement operator to perform weighted operations on the node features in the initial path sequence, and use a spatial pooling operator to compress the dimension of the node feature vectors under the same path to obtain global remote interaction features. Step S34: Spatial dimension aggregation of global remote interaction features and local subtle action features to obtain the corresponding feature vector; By utilizing the components of each dimension of the feature vector to construct the corresponding feature channels, a feature map of family dining behavior is constructed based on the feature channels.

[0011] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, in step S31, feature aggregation and convolution processing are performed on the individual neighborhoods of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. The processing logic includes: Step S311: Extract the initial feature vector of each node in the initial embedding matrix, determine the individual neighborhood of each node according to the adjacency matrix, and use the random mask operator to perform probability discard processing on the connecting edges in the individual neighborhood to obtain the local adjacency descriptor. Step S312: Based on the local adjacency descriptor, the feature vectors of each neighboring node in the individual's neighborhood are weighted using the aggregation operator to obtain the neighborhood aggregation feature components. The calculation formula is as follows: ; in, Represents a node The corresponding neighborhood aggregation feature components, Represents a non-linear activation function. Represents nodes in a local adjacency descriptor With nodes Connection weights between them Represents the nodes in the initial embedding matrix eigenvectors, This represents the learnable transformation matrix corresponding to local random feature enhancement. This indicates the step size of the random walk; Step S313: Use a convolution operator to perform nonlinear feature mapping of the spatial dimension on the neighborhood aggregated feature components to obtain local subtle action features.

[0012] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S4 specifically includes: Step S41: Extract the feature map of family dining behavior, and use the global average pooling operator to perform a spatial dimension statistical average operation on the feature values ​​in each feature channel. The calculation formula is as follows: ; in, Indicates the first The global description component corresponding to each feature channel. The diagram showing the characteristics of family dining behavior is shown in Figure 1. Within the first feature channel The feature values ​​of each node position Indicates the total number of nodes. Indicates the node index number; Step S42: Input the global description vector into the activation network, and perform dimensionality reduction mapping, nonlinear activation, and dimensionality increase mapping in sequence to calculate the dynamic activation coefficients corresponding to each feature channel. The calculation formula is as follows: ; ; ; in, This represents the dynamic excitation coefficient corresponding to the c-th feature channel. This represents the first learning parameter matrix. This represents the second learning parameter matrix. Indicates the dimensionality reduction compression ratio. This represents the activation function. Indicates the number of feature channels; Step S43: Use the multiplication operator to multiply the dynamic excitation coefficients with the feature components in the corresponding channels of the family dining behavior feature map element-wise, and rescale the response intensity of each feature channel to obtain the reconstructed feature map. Step S44: Integrate the feature vectors corresponding to the positions of each node in the reconstructed feature map, keeping the spatial correspondence of each node unchanged, to obtain the fusion interaction feature matrix of each node feature.

[0013] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S5 specifically includes: Step S51: Obtain the fusion interaction feature matrix of each node's features. Use the fusion interaction feature matrix of each node's features as the hidden feature matrix of each node. Extract the hidden feature vectors corresponding to each family member node from the hidden feature matrix of each node. Use the multi-attention aggregation algorithm to perform vector concatenation on the hidden feature vectors and perform nonlinear mapping operation through the linear projection algorithm to calculate the original association score between any two family member nodes. The calculation formula is as follows: ; in, Represents family member nodes With family member nodes The original correlation score between them This represents the vector concatenation operator. Represents a linear transformation matrix. This represents the attention transpose vector. This represents the activation function. This represents the hidden feature vector corresponding to any family member node. This represents the hidden feature vector corresponding to another family member node; Step S52: The original association scores are normalized using the Softmax function to obtain the evolutionary attention weights; Step S53: The hidden feature vectors of each family member node are weighted and summed using evolutionary attention weights to obtain the aggregate vector. The calculation formula is as follows: ; in, Represents family member nodes The corresponding aggregate vector, This represents the preset feature mapping matrix. Represents a nonlinear function. Represents family member nodes With family member nodes Evolutionary attention weights between them This represents the key vector in the multiple attention mechanism.

[0014] As a preferred embodiment of the family dining behavior interaction modeling and analysis method based on graph neural networks described in this invention, step S5 further includes: Step S54: Construct the interaction representation vectors corresponding to each family member node pair using the aggregation vector, and input the interaction representation vectors into the preset fully connected classification layer to obtain the interaction response strength between each family member node pair. Each family member node pair consists of any two family member nodes; The interaction response strength and evolutionary attention weight are weighted and fused to obtain the final interaction probability; The final interaction probability is compared with a preset behavior association threshold; If the final interaction probability is greater than or equal to the preset behavior association threshold, it is determined that there is a substantial shared meal interaction behavior between the corresponding family member node pairs. If the final interaction probability is less than the preset behavior association threshold, it is determined that there is no effective interaction between the corresponding family member node pairs; Step S55: For node pairs determined to have substantial interaction, extract the corresponding aggregation vector and interaction response intensity, and use a preset classification operator to map the aggregation vector to obtain the behavioral analysis results of the family meal interaction. The behavioral analysis results of the family dining interaction included collaborative dining, verbal communication, negative conflict, lack of attention, and emotional interaction.

[0015] The beneficial effects of this invention are as follows: This invention innovatively injects dynamic modal weights and heterogeneous graph topological characteristics into a dual-path feature extraction model. Through the synergistic effect of local random feature enhancement and a jump-type random walk mechanism, it achieves deep coupling between subtle individual actions and global far-end interaction features. Unlike traditional random walks, which are limited to the physical connections of first-order neighborhoods, this invention uses a weight mapping operator to map modal weights reflecting sociological roles to the adjacency matrix in real time, and utilizes a jump factor to drive sampling points to cross the physical constraints of heterogeneous topology. This mechanism enables deep path extraction between indirectly adjacent nodes, effectively capturing long-range association features that cross dining space and modal constraints, solving the technical pain points of existing technologies in handling irregular family dining scenarios, such as the lack of directionality in feature extraction and the overly smoothing of traditional graph calculations. This invention also introduces a compressed-incentivized channel attention algorithm and a multi-attention aggregation algorithm. Through the reconstruction and weighting of feature channels and the precise calculation of the final interaction probability between family members, a complete modeling closed loop from low-level perception to high-level social behavior quantitative analysis is constructed. Attached Figure Description

[0016] Figure 1The flowchart illustrates the steps of a family dining behavior interaction modeling and analysis method based on graph neural networks, as provided in one embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example, refer to Figure 1 This paper presents a method for modeling and analyzing family dining behavior interactions based on graph neural networks, including the following steps: Step S1: Obtain multimodal raw behavioral data, define entities and extract relationships from the multimodal raw behavioral data, and construct a heterogeneous graph of family interactions; Step S2: Nonlinearly map the initial node features of each node in the family interaction heterogeneous graph to obtain each modal feature component. By calculating the correlation entropy value of each modal feature component in the time domain and spatial domain, obtain the modal feature weight distribution map and generate the initial embedding matrix. Step S3: Based on the initial embedding matrix, extract local subtle action features and global remote interaction features through local random feature enhancement and random walk mechanism, and construct a family meal behavior feature map; Step S4: Perform feature channel calibration on the family dining behavior feature map to obtain the fused interaction feature matrix; Step S5: Based on the fused interaction feature matrix, the final interaction probability between family member node pairs is calculated using a multi-attention aggregation algorithm. The final interaction probability is then compared with a preset behavior association threshold to obtain the behavioral analysis results of the family meal interaction.

[0019] In specific implementation, step S1 includes: Step S11: Select family member samples according to the preset sociological role differentiation criteria, collect them through a multimodal perception device, and obtain multimodal raw behavioral data; Multimodal sensing devices include depth cameras, infrared thermal imagers, RGB cameras, directional voice acquisition heads, microphone arrays, photosensors, noise sensors, and pressure sensors; Multimodal raw behavioral data includes video image sequences, audio signal streams, and environmental perception parameters; Environmental sensing parameters include ambient light intensity, sound levels in decibels of the monitored environment, and pressure data. Step S12: The time synchronization calibration algorithm is used to perform benchmark unification processing on the multimodal raw behavior data, and the entity feature components of the multimodal raw behavior data within the sampling window are extracted to obtain the quasi-synchronous behavior feature sequence. Step S13: Extract the pose features, visual focus features, and verbal emotion features of family members from the quasi-synchronous behavior feature sequence, and define a heterogeneous node set based on the entity attributes in the entity feature components; The heterogeneous node set includes family member nodes, action nodes, and environment attribute nodes; Step S14: Combining the heterogeneous node set, the semantic association strength of the quasi-synchronous behavior feature sequence between different nodes is calculated to establish the connection edges between each node, and the graph construction operator is called to convert the connection edges and corresponding weight values ​​into an adjacency matrix to construct the family interaction heterogeneous graph.

[0020] Specifically, the pre-defined sociological role differentiation criterion means that during the data collection phase, family members are not treated as homogeneous individuals, but rather sampled and labeled differently based on their social roles within the family, including parents, children, and elders. Different roles often exhibit systematic differences in behavioral patterns, interaction styles, and emotional expressions during shared meals. For example, parents may exhibit more caregiving behaviors, while children may initiate communication or requests more frequently. Role differentiation allows for a more nuanced capture of the impact of the family's internal social structure on interactive behavior, thereby enhancing the interpretive depth and accuracy of behavioral analysis.

[0021] The acquired multimodal raw behavioral data comes from various sensors such as depth cameras and microphone arrays, and these devices need to be precisely aligned in time. Therefore, a time synchronization calibration algorithm is required for processing. This is achieved through software timestamp alignment, for example, by using a unified network time protocol to timestamp all data streams and fine-tuning the sampling time to ensure that the actions in the video, the speech in the audio, and the environmental sensor readings correspond on the same time base. This eliminates noise caused by device latency or different sampling rates, enabling the accurate fusion of entity feature components such as pose, speech content, and ambient light extracted from different modalities at a certain moment, forming a quasi-synchronous behavioral feature sequence describing the scene at the same moment.

[0022] Based on these synchronized feature sequences, a heterogeneous node set is defined based on the entity attributes in the entity feature components. Specifically, the algorithm identifies different types of entities from the data: each family member is defined as a family member node, their actions such as picking up food or speaking are defined as action nodes, and environmental states such as lighting and noise are defined as environmental attribute nodes. Each node is represented by its feature vector; for example, family member nodes include role and posture features, while action nodes include action type and duration. By calculating the co-occurrence relationship of different node features in time, weighted connection edges are established, ultimately constructing a heterogeneous graph of family interactions that can simultaneously contain multiple entity types and their complex relationships. For example, the action node of the father node speaking and the action node of the child node gazing highly overlap in time.

[0023] This application addresses the challenges of insufficient modeling depth and heterogeneous data distortion in family scenarios by combining sociological criteria with multimodal alignment. It not only eliminates temporal discrepancies between modalities but also achieves precise characterization of deep interaction logic through heterogeneous node definition, capturing implicit relationship features that traditional algorithms cannot perceive. This complete chain from low-level data synchronization to high-level semantic modeling effectively improves the depth and accuracy of family interaction analysis, laying a solid technical foundation for obtaining accurate analysis results of the nature of family dining interactions.

[0024] In specific implementation, step S2 includes: Step S21: The learnable attribute sampling operator is used to perform nonlinear spatial mapping on the initial features of nodes in the family interaction heterogeneous graph to obtain each modal feature component, and each modal feature component is mapped to the normalized node attribute space of the same dimension to obtain the normalized node attribute set. Step S22: Based on the entity attributes in the heterogeneous node set, calculate the correlation entropy values ​​of each modal feature component in the time and spatial domains to obtain the modal feature weight distribution map. Step S23: Perform a weighted selection operation on the normalized node attribute set based on the modal feature weight distribution map, identify and remove redundant feature components with weights lower than the preset significance threshold, and obtain the target node attribute subset. Step S24: Feed the target node attribute subset back to the family interaction heterogeneous graph, use the adjacency matrix to extract the neighborhood attribute information of each node, and use the feature propagation mechanism to interact and fuse the neighborhood attribute information and the target node attribute subset to obtain the initial embedding features of each node. Neighborhood attribute information includes the action semantic components of neighboring nodes, the environmental state components, and the behavioral representation components of associated members; The initial embedding features of each node are dimension-aligned using linear projection transformation to obtain the initial embedding matrix.

[0025] Specifically, in the time domain, for each feature dimension, its information entropy is calculated across the entire sampling window to measure the complexity and uncertainty of the feature's changes over time; a higher entropy value often indicates that it contains more dynamic behavioral information. In the spatial domain, the mutual information between any two nodes in the graph is calculated for that feature dimension to quantify the strength of the feature in revealing the relationship between nodes. By weighting and fusing the temporal entropy of a feature dimension with its average spatial mutual information entropy in the global graph, and then normalizing it using the Softmax function, a modal feature weight distribution map in the form of a probability distribution is obtained. The weight coefficients in this map clearly indicate the relative importance of each feature dimension.

[0026] In the feature propagation mechanism, the fusion mechanism follows the logic of message passing and semantic aggregation. Using the adjacency matrix constructed in step S14, all direct or indirect neighboring nodes of each central node are determined, and a subset of target node attributes, denoised in step S23, is retrieved from these neighboring nodes. The feature propagation mechanism then converges the features of these neighboring nodes along the topological path of the graph to the central node. During this process, the interaction fusion operator is typically a nonlinear aggregation function, which concatenates and fuses the central node's own attribute vector with the converged neighboring attribute vectors. Through this propagation, the features of each node are no longer isolated data points, but rather absorb the interactive features of surrounding actions, environmental noise, and feedback from others, enabling deep coupling of node attributes within a unified latent space.

[0027] The advantage of this step compared to traditional techniques lies in its effective resolution of the feature silos and false alarm problems inherent in existing technologies when processing multimodal data. Traditional methods often directly concatenate all modalities, making it difficult to distinguish the contribution of different sensor information to specific behaviors. This solution achieves "pre-screening and then fusion" of features through learnable attribute sampling and association entropy calculation. That is, after unifying the dimensions using nonlinear mapping, redundant interference is identified and eliminated through a weight distribution map, retaining only high-value target attributes for propagation. This approach not only significantly reduces computational redundancy when the model handles complex family interactions but also endows the initial embedded features with a global topological perspective through the propagation mechanism.

[0028] This feature propagation-based fusion mechanism enables the model to keenly capture the meaning of specific actions in a particular environmental context. Through this refined and context-aware feature extraction method, this invention provides a purer and more relevant initial embedding matrix for subsequent steps to accurately determine complex properties that are highly dependent on the interaction context, such as emotional interactions or negative conflicts. This is the key technology that demonstrates higher robustness and accuracy in handling irregular family dining behaviors.

[0029] In specific implementation, in step S22, the modal feature weight distribution map is obtained by calculating the correlation entropy values ​​of each modal feature component in the time domain and the spatial domain. The processing logic includes: Step S221: The information entropy calculation method is used to extract the numerical fluctuation components of the multimodal raw behavioral data within the sampling time window, the probability distribution uncertainty of each modal feature component is calibrated, and the time dimension entropy component is calculated. Step S222: The mutual information evaluation algorithm is used to extract the association strength between nodes in the heterogeneous node set. The association strength is calibrated according to the probability difference of the spatial coupling degree between nodes, and the spatial dimension entropy component is calculated. Step S223: The temporal entropy components and spatial entropy components are linearly weighted and superimposed. The Sigmoid function is used to map the superimposed values ​​to a probability interval to obtain the modal feature weight distribution map corresponding to each modal feature component.

[0030] Specifically, in step S221, the uncertainty of the probability distribution of each modal feature component is calibrated. This is achieved by extracting each feature component from the original multimodal behavioral data, such as pose features in a video sequence or emotion features in an audio stream, and statistically analyzing the frequency distribution of the feature values ​​within a set sampling time window. This is typically done by constructing a histogram by discretizing the feature value range. Based on the obtained probability distribution, its information entropy value is calculated using the following formula: ; in, Represents the information entropy value. Represents eigenvalues. Represents eigenvalues The probability of occurrence Represents a logarithm. Represents the set of possible values ​​for the eigenvalues; The entropy value directly characterizes the degree of uncertainty of the feature during its evolution over time: the higher the entropy value, the more drastic the feature fluctuations and the lower the predictability; conversely, the lower the entropy value, the more stable the feature changes over time. Through this calculation, each modal feature component is assigned a scalar entropy value component in the form of a time dimension, thereby transforming temporal dynamics into a quantifiable measure of uncertainty.

[0031] In step S222, the spatial coupling degree between nodes is calibrated based on the probabilistic differences. Specifically, a mutual information evaluation algorithm is used to measure the statistical dependency between the feature vectors of any two nodes in the heterogeneous node set, reflecting the strength of the association in the spatial structure. For node pairs representing family members, actions, and environmental attributes in the graph, their corresponding feature vectors are extracted and treated as random variables. The mutual information between these two variables directly quantifies the degree to which the information of one variable contains the information of another. A higher mutual information value indicates tighter spatial coupling between nodes, meaning that the state change of one node has a higher predictive value for the state of another node; conversely, a lower mutual information value indicates weaker coupling. By traversing all relevant nodes, the system calculates a spatial dimension entropy component for each modal feature component. This component comprehensively reflects the interaction and dependency pattern of the feature across nodes in the graph.

[0032] Regarding step S223, which involves linearly weighting and superimposing the temporal and spatial entropy components, this process is a multi-scale information integration based on weighted fusion. The calculation formula is as follows: ; in, Represents the overall entropy value. The weighting coefficient represents the time entropy value. Represents the entropy component in the time dimension. Represents the entropy components of spatial dimensions. Indicates the weighting coefficient of spatial entropy; This weighting method flexibly balances the contribution of temporal uncertainty and spatial correlation to the importance of features. For example, it increases the weight of temporal entropy when focusing on instantaneous behavioral changes, while emphasizing spatial entropy when highlighting interaction patterns among members.

[0033] The sigmoid function is used to map the superimposed values ​​to a probability interval to obtain the weight coefficients corresponding to each modal feature component. The comprehensive entropy value is input into the sigmoid function, and the output value is compressed to the (0,1) interval, forming normalized weights. This mapping process transforms the entropy value in the real number domain into a probabilistic form, intuitively representing the relative importance of each modal feature in the overall analysis. Performing this operation on all modal feature components yields a weight set that constitutes a modal feature weight distribution map. This distribution map provides a basis for weighted selection and feature filtering in subsequent steps, enabling focus on feature dimensions that are information-rich and significantly correlated.

[0034] In specific implementation, the processing logic for weighted selection of the normalized node attribute set based on the modal feature weight distribution map in step S23 includes: Step S231: Extract the feature vectors from the normalized node attribute set and the corresponding weight coefficients from the modal feature weight distribution map. Perform a weighted operation using the multiplication operator to obtain the weighted feature vector. The calculation formula is as follows: ; in, Represents the weighted eigenvector. This represents the weight coefficients corresponding to the modal feature weight distribution plot. Represents the feature vector in the normalized node attribute set; Step S232: Construct a selection operator using a preset significance threshold. The calculation formula is as follows: ; in, Indicates the selection operator, This indicates a preset significance threshold; The weighted eigenvectors are mapped using a selection operator, and then eliminated... Corresponding redundant components; Step S233, extract the required parameters. Linear aggregation is performed on the weighted feature vectors to obtain a subset of the target node attributes.

[0035] Specifically, the weighted selection operation on the normalized node attribute set based on the modal feature weight distribution map is based on a binary selection operator constructed from a preset significance threshold. This operator filters and condenses the information of the weighted feature vectors based on their importance. The selection operator is defined as a threshold logic function, essentially a binary mask controlled by the comparison between the weight coefficients and the threshold. This operator is applied to each weighted feature vector. The mapping process does not involve complex mathematical transformations but rather performs a "keep or reset" decision: for each feature component in the weighted feature vector, if its corresponding modal feature weight coefficient is greater than or equal to the preset significance threshold, the selection operator outputs 1, and the feature value of that dimension is retained; if the weight coefficient is lower than the preset significance threshold, the selection operator outputs 0, and the feature value of that dimension is set to zero. Through this element-wise mapping, all feature components considered insignificant, redundant, or noisy are systematically filtered out, leaving only those feature elements that have proven to have high information entropy and strong correlation in both time and space dimensions.

[0036] Linear aggregation is performed on all weighted feature vectors that satisfy the condition that "the modal feature weight coefficient is greater than or equal to a preset significance threshold". This aggregation process is not a simple concatenation of all vectors, but rather the integration of all selected feature components (i.e., those feature values ​​that are not set to zero) according to their original structural relationships, usually through vector addition, to form a new, compact, and information-dense feature representation—the target node attribute subset. This linear aggregation method achieves effective compression and redundancy removal of multimodal feature information, while also ensuring that the aggregated feature subset retains the most relevant discriminative information about the interaction behavior in the original data to the greatest extent. The final target node attribute subset, as the refined feature representation of the nodes, is fed back into the structure of the family interaction heterogeneous graph.

[0037] This processing step offers significant advantages over traditional techniques, effectively addressing the issues of false alarms and computational redundancy in multimodal data processing. Traditional methods typically perform full stitching of all sensor modalities, making it difficult to distinguish the contribution of different information to specific behaviors and easily leading to noise interference in the model. This solution achieves "first screening, then fusion" of features through a dual mechanism of association entropy weighting and hard threshold screening, ensuring that only high-value target attributes participate in subsequent graph feature propagation. This coarse-and-fine processing method not only significantly reduces the computational overhead of the model when handling complex family interactions and improves the system's real-time performance, but also endows node features with stronger discriminative power.

[0038] In specific implementation, step S3 includes: Step S31: Input the initial embedding matrix into the local path of the dual-path feature extraction model, and process the individual neighborhood of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. Step S32: Extract the adjacency matrix corresponding to the heterogeneous graph of family interactions. Use the weight mapping operator to map the weight coefficients in the modal feature weight distribution graph to the corresponding elements of the adjacency matrix. Introduce a jump factor to calculate the cross-level transition probability between any two nodes in the heterogeneous node set. Use Markov chain state transition logic to calculate the transition probability between any two nodes in the heterogeneous node set. The transition probabilities form a probability distribution mapping set, the calculation formula of which is: ; in, Represents nodes in a heterogeneous node set With nodes The transition probability between them Indicates the jump factor. This represents the weight coefficients corresponding to the modal feature weight distribution plot. This represents the numerical value indicating the connection relationships between nodes in the adjacency matrix. Indicates a node All first-order neighbor nodes, The jump gain function represents the sampling across nodes. Represents a node The set of nodes corresponding to the individual's neighborhood. Indicates traversing nodes The variable index number of the set of nodes corresponding to the individual's neighborhood. This represents any node in the heterogeneous node set. Represents another arbitrary node in the heterogeneous node set; Step S33: Invoke the jump-type random walk mechanism to perform state transitions with a preset step size from family member nodes in the heterogeneous node set based on the probability distribution mapping set; During the jump process, the jump factor is used to drive the sampling points to cross the heterogeneous topology constraints to extract the interaction path between non-directly adjacent nodes, and the initial path sequence is obtained. Heterogeneous topology consists of family member nodes, action nodes, environment attribute nodes, and connecting edges between nodes; We use a feature enhancement operator to perform weighted operations on the node features in the initial path sequence, and use a spatial pooling operator to compress the dimension of the node feature vectors under the same path to obtain global remote interaction features. Step S34: Spatial dimension aggregation of global remote interaction features and local subtle action features to obtain the corresponding feature vector; By utilizing the components of each dimension of the feature vector to construct the corresponding feature channels, a feature map of family dining behavior is constructed based on the feature channels.

[0039] It is important to emphasize that this invention innovatively integrates dynamic modal weights and heterogeneous graph topological characteristics into existing dual-path feature extraction models. When extracting features using this model, this invention does not directly apply general algorithms, but instead achieves deep analysis of multi-scale behavioral features in family dining scenarios by processing local subtle actions and global remote interactions in parallel. In the local path, by randomly enhancing the features of an individual's neighborhood, the system can accurately capture subtle body movements and immediate feedback from members. In the global path, this invention establishes a path guidance mechanism based on weight mapping in step S32. This mechanism introduces a jump factor and Markov chain state transition logic, using a weight mapping operator to map modal feature weights reflecting sociological roles to the adjacency matrix in real time. This transforms the existing random walk process from blind topological jumps into directional sampling driven by semantic weights. This improvement enables the model to overcome the hierarchical limitations of heterogeneous topologies, specifically enhancing feature capture of key interaction paths among family members. It solves the problems of lack of directionality in feature extraction when dealing with irregular family dining scenarios and the excessive smoothing of traditional graph computation when handling long-distance dependencies. Building upon this foundation, a jump factor is used to drive sampling points to overcome the physical constraints of the non-homogeneous structure of "family member-action-environment." Sampling points refer to the logical node positions where state transitions occur within the heterogeneous graph topology during random walks. This invention achieves deep path extraction between non-directly adjacent nodes, effectively capturing long-range association features that transcend dining table space and modal constraints, overcoming the implicit social interaction signals easily overlooked by traditional methods. By performing feature enhancement and spatial pooling on the path sequences, a family dining behavior feature map is obtained. This improved approach of "semantic reconstruction" and "parameter alignment" of existing models achieves logical alignment from the underlying data distribution to the mid-level feature extraction path. While retaining the efficiency of the dual-path architecture, it endows the feature extraction process with sociological interpretability, representing the core technology for accurately determining the complex interactive nature of family dining.

[0040] By introducing a jump factor, this invention transcends the constraints of heterogeneous topology, thereby extracting interaction paths between indirectly adjacent nodes. In traditional random walks, transitions are typically restricted to first-order neighbors as defined by the adjacency matrix. This invention, however, combines the jump factor with modal feature weights and a jump gain function, constructing a term in the transition probability formula that allows cross-level jumps. When calculating the transition probability from node i to node j, not only are the regular transitions between node i and all its first-order neighbors considered, but a jump probability determined by the modal feature weights and jump gain function of node j is also weighted by the jump factor. This allows the random walk process to potentially ignore direct topological connections at each step, jumping directly to another semantically highly relevant node in the graph—a node with high weights and large gain function values ​​but not directly connected. For example, the walk path might jump directly from the parent node, across the intermediate "picking up food" action node, to the child's smiling expression node, thus capturing this end-to-end emotional transmission path that transcends specific actions and extracting the deep interactive semantics inherent in indirect connections. The value range of the jump factor is [0.6-0.9]. When extracting features, the value of the jump factor is adjusted in real time according to the association entropy value calculated in step S2: when the association entropy value decreases, the jump factor is increased to improve the sampling weight of local subtle action features; when the association entropy value increases, the jump factor is decreased to improve the walking depth of global far-end interaction features.

[0041] After obtaining the initial path sequence, a feature enhancement operator is used to weight the node features. This operator does not treat all nodes in the path equally, but rather enhances the features of each node based on its importance in the constructed modal feature weight distribution map. The modal feature weight coefficient corresponding to each node in the path sequence is used as a scaling factor for that node's feature vector. Through this element-wise weighting, the feature strength of nodes judged to have high information entropy and strong correlation in the temporal and spatial dimensions is significantly improved in the embedded representation of the path sequence, while the features of secondary or transitional nodes are relatively weakened. This weighting operation allows the global features extracted from the path through pooling to focus more on key interaction events, enhancing the discriminative power of the features.

[0042] Residual weighted aggregation is used to spatially aggregate global far-end interaction features and local subtle action features. This aggregation logic effectively suppresses the oversmoothing problem that occurs during multi-layer propagation in graph neural networks by superimposing the local feature components (i.e., local subtle action features) of the initial embedding matrix after linear transformation onto the global far-end interaction features. By preserving the discriminative power of local subtle action features, the model can accurately identify subtle behavioral differences between individuals, even in scenes with a consistent macroscopic atmosphere. Here, the spatial dimension corresponds to the position or order of nodes in the graph. Both global far-end interaction features and local subtle action features maintain their correspondence with nodes in the original graph during generation; therefore, aggregation is performed independently at each node position. For each node position in the graph, the system fuses its corresponding local feature vector and global feature vector. This fusion is usually not a simple concatenation but is achieved through a learnable nonlinear transformation, such as a small neural network or a fusion gating unit. This transformation can adaptively determine how much information to extract from local and global features based on the context of the current node, ultimately outputting a unified feature vector that fuses local details and global context at that node position.

[0043] Each node acquires a unified feature vector. Each dimension of this feature vector is considered an independent feature channel, carrying a specific, multi-scale fused interactive semantic. For example, one channel might specifically encode "attention based on eye contact," while another encodes "collaboration based on voice and action synchronization." Arranging these channels of all nodes according to their topological order constructs a two-dimensional feature map of family dining behavior. Structurally, this feature map resembles an image: its width and height dimensions implicitly represent the spatial or topological relationships between nodes, while the "number of channels" represents the rich interactive semantic dimensions.

[0044] In specific implementation, in step S31, feature aggregation and convolution processing are performed on the individual neighborhoods of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. The processing logic includes: Step S311: Extract the initial feature vector of each node in the initial embedding matrix, determine the individual neighborhood of each node according to the adjacency matrix, and use the random mask operator to perform probability discard processing on the connecting edges in the individual neighborhood to obtain the local adjacency descriptor. Step S312: Based on the local adjacency descriptor, the feature vectors of each neighboring node in the individual's neighborhood are weighted using the aggregation operator to obtain the neighborhood aggregation feature components. The calculation formula is as follows: ; in, Represents a node The corresponding neighborhood aggregation feature components, Represents a non-linear activation function. Represents nodes in a local adjacency descriptor With nodes Connection weights between them Represents the nodes in the initial embedding matrix eigenvectors, This represents the learnable transformation matrix corresponding to local random feature enhancement. This indicates the step size of the random walk; Step S313: Use a convolution operator to perform nonlinear feature mapping of the spatial dimension on the neighborhood aggregated feature components to obtain local subtle action features.

[0045] Specifically, this application introduces local random feature enhancement, significantly improving the accuracy and robustness of the model in capturing individual actions in family dining scenarios. By using a random masking operator to probabilistically discard neighboring edges of nodes, dynamic data augmentation is achieved for heterogeneous graph data, forcing the model to extract key action features even with missing information or environmental noise interference, such as limb occlusion and sudden changes in lighting during dining. This effectively overcomes the excessive reliance of traditional methods on the complete topology and enhances the system's robustness. Based on a neighborhood aggregation mechanism using a learnable transformation matrix, the connection weights between nodes can be automatically adjusted according to the behavioral characteristics of different family members, achieving deep refinement of subtle action components such as picking up food and serving food.

[0046] This processing method not only effectively alleviates the oversmoothing problem that easily occurs in the feature propagation process of deep graph neural networks, ensuring the discrimination differences of individual features among different family members, but also transforms scattered neighborhood information into local subtle action features with high-order semantics through the spatial nonlinear mapping of convolution operators. This improvement provides high-quality and high-purity underlying input for subsequent global path extraction of long-range social connections, and is a key technical prerequisite for achieving accurate identification of implicit interaction behaviors among family members, demonstrating the significant technical progress of this invention in processing irregular and heterogeneous family behavior data.

[0047] In specific implementation, step S4 includes: Step S41: Extract the feature map of family dining behavior, and use the global average pooling operator to perform a spatial dimension statistical average operation on the feature values ​​in each feature channel. The calculation formula is as follows: ; in, Indicates the first The global description component corresponding to each feature channel. The diagram showing the characteristics of family dining behavior is shown in Figure 1. Within the first feature channel The feature values ​​of each node position Indicates the total number of nodes. Indicates the node index number; Step S42: Input the global description vector into the activation network, and perform dimensionality reduction mapping, nonlinear activation, and dimensionality increase mapping in sequence to calculate the dynamic activation coefficients corresponding to each feature channel. The calculation formula is as follows: ; ; ; in, This represents the dynamic excitation coefficient corresponding to the c-th feature channel. This represents the first learning parameter matrix. This represents the second learning parameter matrix. Indicates the dimensionality reduction compression ratio. This represents the activation function. Indicates the number of feature channels; Step S43: Use the multiplication operator to multiply the dynamic excitation coefficients with the feature components in the corresponding channels of the family dining behavior feature map element-wise, and rescale the response intensity of each feature channel to obtain the reconstructed feature map. Step S44: Integrate the feature vectors corresponding to the positions of each node in the reconstructed feature map, keeping the spatial correspondence of each node unchanged, to obtain the fusion interaction feature matrix of each node feature.

[0048] Specifically, in step S43, the response intensity of each feature channel is rescaled, and the overall importance of each channel in the original feature map is adaptively adjusted using the calculated dynamic excitation coefficients. The dynamic excitation coefficients obtained in step S42 are then multiplied element-wise with the family dining behavior feature map. For the c-th feature channel of the feature map, the feature values ​​at all node positions within that channel are uniformly multiplied by the same scaling factor. If the scaling factor is close to 1, it indicates that the channel is judged as highly important by the excitation network, and its feature response is almost completely preserved or even relatively enhanced; if the scaling factor is close to 0, it indicates that the channel is judged as information redundancy or has low relevance to the current interaction context, and its feature response is significantly suppressed. This process is mathematically equivalent to an attention mask in one channel dimension. It is not based on manual rules but is automatically learned and generated by the excitation network based on the global context, thus achieving adaptive and dynamic recalibration of multi-channel features. After this operation, the "volume" of each channel in the reconstructed feature map is differentially adjusted, allowing the model to focus its limited representational power on the feature dimensions that best characterize the current family dining interaction state.

[0049] In step S44, the integration of feature vectors corresponding to each node position in the reconstructed feature map refers to converting the data, which has been rescaled and organized in a multi-channel "image" format, back into a regular data structure that corresponds one-to-one with the nodes in the map and facilitates subsequent node-level processing. Each "pixel" position in the reconstructed feature map (corresponding to a node in the original image) contains a multi-dimensional feature vector, the dimension of which is equal to the number of feature channels. The integration operation extracts all the feature vectors of these node positions and arranges them according to the order of the nodes in the original family interaction heterogeneous map, usually forming a two-dimensional matrix. In this matrix, each row corresponds to a specific node, and each column corresponds to a specific feature channel. This process strictly preserves the spatial correspondence of each node, that is, the identity and topological context information of the node in the heterogeneous map are preserved through its row order in the matrix. The resulting fused interaction feature matrix has row vectors that are the final feature representations of each node that incorporate global channel attention information. Each row vector in the fused interaction feature matrix is ​​used as a node-level, semantically rich interaction state descriptor and is directly fed into the multi-attention aggregation algorithm for fine-grained calculation and property determination of inter-member interaction relationships.

[0050] Compared to traditional methods that directly concatenate multimodal features or use fixed weights to calculate interactions, this application achieves adaptive dynamic calibration of feature dimensions through a compressed-excitation channel attention algorithm. This solves the problem that existing technologies struggle to distinguish the contribution of different feature channels to specific behaviors in complex family dining contexts. Traditional techniques typically process all sensor modalities in their entirety, making them highly susceptible to environmental noise and computational redundancy. In contrast, this invention utilizes dynamically generated excitation coefficients learned automatically by an excitation network, which can rescale the response intensity of feature channels, actively enhancing high-value interaction features and suppressing redundant or noisy channels, much like an "attention mask." This feature purification mechanism based on global context not only effectively reduces false alarms but also ensures that the final generated fused interaction feature matrix possesses extremely high semantic discriminative power.

[0051] In specific implementation, step S5 includes: Step S51: Obtain the fusion interaction feature matrix of each node's features. Use the fusion interaction feature matrix of each node's features as the hidden feature matrix of each node. Extract the hidden feature vectors corresponding to each family member node from the hidden feature matrix of each node. Use the multi-attention aggregation algorithm to perform vector concatenation on the hidden feature vectors and perform nonlinear mapping operation through the linear projection algorithm to calculate the original association score between any two family member nodes. The calculation formula is as follows: ; in, Represents family member nodes With family member nodes The original correlation score between them This represents the vector concatenation operator. Represents a linear transformation matrix. This represents the attention transpose vector. This represents the activation function. This represents the hidden feature vector corresponding to any family member node. This represents the hidden feature vector corresponding to another family member node; Step S52: The original association scores are normalized using the Softmax function to obtain the evolutionary attention weights; Step S53: The hidden feature vectors of each family member node are weighted and summed using evolutionary attention weights to obtain the aggregate vector. The calculation formula is as follows: ; in, Represents family member nodes The corresponding aggregate vector, This represents the preset feature mapping matrix. Represents a nonlinear function. Represents family member nodes With family member nodes Evolutionary attention weights between them This represents the key vector in the multiple attention mechanism.

[0052] Specifically, compared to traditional methods for determining member relationships based on fixed weights or a single distance metric, this application achieves fine-grained dynamic modeling of interactions among family members through a multi-attention aggregation algorithm. Traditional methods often struggle to capture asymmetric and nonlinear member interactions in shared dining scenarios, while this application utilizes vector concatenation and linear projection operations to deeply mine the potential original association scores between any two family member nodes, accurately characterizing the interaction intensity of different roles in specific contexts. Evolutionary attention weights generated through the Softmax function achieve adaptive weighting of member features, effectively extracting highly discriminative aggregation vectors and endowing the interaction analysis process with clear sociological interpretability.

[0053] In specific implementation, step S5 also includes: Step S54: Construct the interaction representation vectors corresponding to each family member node pair using the aggregation vector, and input the interaction representation vectors into the preset fully connected classification layer to obtain the interaction response strength between each family member node pair. Each family member node pair consists of any two family member nodes; The interaction response strength and evolutionary attention weight are weighted and fused to obtain the final interaction probability; The final interaction probability is compared with a preset behavior association threshold; If the final interaction probability is greater than or equal to the preset behavior association threshold, it is determined that there is a substantial shared meal interaction behavior between the corresponding family member node pairs. If the final interaction probability is less than the preset behavior association threshold, it is determined that there is no effective interaction between the corresponding family member node pairs; Step S55: For node pairs determined to have substantial interaction, extract the corresponding aggregation vector and interaction response intensity, and use a preset classification operator to map the aggregation vector to obtain the behavioral analysis results of the family meal interaction. The behavioral analysis results of family dining interactions include collaborative dining, verbal communication, negative conflict, lack of attention, and emotional interaction.

[0054] Specifically, the "no effective interaction" judgment in step S54 corresponds to the lack of attention in the behavior analysis results of step S55. "Lack of attention" behaviors in family meals include each person eating with their head down and ignoring the other, which are reflected in the model as extremely low interaction response intensity and evolutionary attention weights. Therefore, when the final interaction probability is less than the preset behavior association threshold, the system directly maps the interaction nature of that node pair to lack of attention. This design ensures that the behavior analysis results can cover the entire spectrum of family meal scenarios from "strong interaction" to "very weak interaction," achieving a logical closed loop in the judgment. Strong interaction includes conflict and communication, while weak interaction includes alienation and ignoring.

[0055] When constructing the interaction representation vectors corresponding to family member node pairs using aggregated vectors, the aggregated vectors of any two family member nodes are extracted and merged using a vector concatenation operator, thus forming a higher-dimensional feature vector that simultaneously encompasses the individual behavioral characteristics of both parties and the background of bilateral interaction. The pre-defined fully connected classification layer is essentially a nonlinear mapping network composed of learnable weight matrices. To map the abstract high-dimensional interaction representation vectors to a quantized scalar space, thereby deriving the interaction response intensity that represents the intensity of interaction between members, this layer calculates the original interaction components between node pairs by performing linear transformations and activation function operations on the input interaction representation vectors. To ensure the rigor of the judgment results, the interaction response intensity and evolutionary attention weights are weighted and fused. That is, a pre-defined adjustment coefficient is used to linearly superimpose and integrate the instantaneous action intensity feedback with the long-term spatiotemporal evolution logic to obtain the final interaction probability, which is used to measure whether there is a substantial shared meal interaction behavior between two family members. The preset classification operators are a set of classification decision functions designed for specific sociological semantics, aiming to map complex aggregation vectors to specific behavioral category probability spaces. When a substantial interaction is determined to exist, the operator performs feature purification and normalization on the aggregation vectors and response intensity, and finally outputs accurate analysis results including collaborative dining, verbal communication, negative conflict, lack of attention, and emotional interaction.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0057] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A method for modeling and analyzing family dining behavior interactions based on graph neural networks, characterized in that, Includes the following steps: Step S1: Obtain multimodal raw behavioral data, define entities and extract relationships from the multimodal raw behavioral data, and construct a heterogeneous graph of family interactions; Step S2: Nonlinearly map the initial node features of each node in the family interaction heterogeneous graph to obtain each modal feature component. By calculating the correlation entropy value of each modal feature component in the time domain and spatial domain, obtain the modal feature weight distribution map and generate the initial embedding matrix. Step S3: Based on the initial embedding matrix, extract local subtle action features and global remote interaction features through local random feature enhancement and random walk mechanism, and construct a family meal behavior feature map; Step S4: Perform feature channel calibration on the family dining behavior feature map to obtain the fused interaction feature matrix; Step S5: Based on the fused interaction feature matrix, the final interaction probability between family member node pairs is calculated using a multi-attention aggregation algorithm. The final interaction probability is then compared with a preset behavior association threshold to obtain the behavioral analysis results of the family meal interaction.

2. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 1, characterized in that, Step S1 specifically includes: Step S11: Select family member samples according to the preset sociological role differentiation criteria, collect them through a multimodal perception device, and obtain multimodal raw behavioral data; The multimodal sensing device includes a depth camera, an infrared thermal imager, an RGB camera, a directional voice acquisition head, a microphone array, a photosensitive sensor, a noise sensor, and a pressure sensor; The multimodal raw behavioral data includes video image sequences, audio signal streams, and environmental perception parameters; The environmental sensing parameters include ambient light intensity, sound decibels of the monitored environment, and pressure data. Step S12: The time synchronization calibration algorithm is used to perform benchmark unification processing on the multimodal raw behavior data, and the entity feature components of the multimodal raw behavior data within the sampling window are extracted to obtain the quasi-synchronous behavior feature sequence. Step S13: Extract the pose features, visual focus features, and verbal emotion features of family members from the quasi-synchronous behavior feature sequence, and define a heterogeneous node set based on the entity attributes in the entity feature components; The heterogeneous node set includes family member nodes, action nodes, and environmental attribute nodes; Step S14: Combining the heterogeneous node set, the semantic association strength of the quasi-synchronous behavior feature sequence between different nodes is calculated to establish the connection edges between each node, and the graph construction operator is called to convert the connection edges and corresponding weight values ​​into an adjacency matrix to construct a heterogeneous graph of family interaction.

3. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 2, characterized in that, Step S2 specifically includes: Step S21: The learnable attribute sampling operator is used to perform nonlinear spatial mapping on the initial features of nodes in the family interaction heterogeneous graph to obtain each modal feature component, and each modal feature component is mapped to the normalized node attribute space of the same dimension to obtain the normalized node attribute set. Step S22: Based on the entity attributes in the heterogeneous node set, the modal feature weight distribution map is obtained by calculating the correlation entropy values ​​of each modal feature component in the time domain and the spatial domain. Step S23: Perform a weighted selection operation on the normalized node attribute set based on the modal feature weight distribution map, identify and remove redundant feature components with weights lower than the preset significance threshold, and obtain the target node attribute subset. Step S24: Feed the target node attribute subset back to the family interaction heterogeneous graph, use the adjacency matrix to extract the neighborhood attribute information of each node, and use the feature propagation mechanism to interact and fuse the neighborhood attribute information and the target node attribute subset to obtain the initial embedding features of each node. The neighborhood attribute information includes the action semantic components of neighboring nodes, the environmental state components, and the behavioral representation components of associated members. The initial embedding features of each node are dimension-aligned using linear projection transformation to obtain the initial embedding matrix.

4. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 3, characterized in that, In step S22, the modal feature weight distribution map is obtained by calculating the correlation entropy values ​​of each modal feature component in the time and spatial domains. The processing logic includes: Step S221: The information entropy calculation method is used to extract the numerical fluctuation components of the multimodal raw behavioral data within the sampling time window, the probability distribution uncertainty of each modal feature component is calibrated, and the time dimension entropy component is calculated. Step S222: The mutual information evaluation algorithm is used to extract the association strength between nodes in the heterogeneous node set. The association strength is calibrated according to the probability difference of the spatial coupling degree between nodes, and the spatial dimension entropy component is calculated. Step S223: The temporal entropy components and spatial entropy components are linearly weighted and superimposed. The Sigmoid function is used to map the superimposed values ​​to a probability interval to obtain the modal feature weight distribution map corresponding to each modal feature component.

5. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 3, characterized in that, In step S23, the processing logic for weighted selection of the normalized node attribute set based on the modal feature weight distribution map includes: Step S231: Extract the feature vectors from the normalized node attribute set and the corresponding weight coefficients from the modal feature weight distribution map. Perform a weighted operation using the multiplication operator to obtain the weighted feature vector. The calculation formula is as follows: ; in, Represents the weighted eigenvector. This represents the weight coefficients corresponding to the modal feature weight distribution plot. Represents the feature vector in the normalized node attribute set; Step S232: Construct a selection operator using a preset significance threshold. The calculation formula is as follows: ； in, Indicates the selection operator, This indicates a preset significance threshold; The weighted eigenvectors are mapped using a selection operator, and then eliminated... Corresponding redundant components; Step S233, extract the required parameters. Linear aggregation is performed on the weighted feature vectors to obtain a subset of the target node attributes.

6. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 1, characterized in that, Step S3 specifically includes: Step S31: Input the initial embedding matrix into the local path of the dual-path feature extraction model, and process the individual neighborhood of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. Step S32: Extract the adjacency matrix corresponding to the heterogeneous graph of family interactions. Use the weight mapping operator to map the weight coefficients in the modal feature weight distribution graph to the corresponding elements of the adjacency matrix. Introduce a jump factor to calculate the cross-level transition probability between any two nodes in the heterogeneous node set. Use Markov chain state transition logic to calculate the transition probability between any two nodes in the heterogeneous node set. The transition probabilities form a probability distribution mapping set, the calculation formula of which is: ； in, Represents nodes in a heterogeneous node set With nodes The transition probability between them Indicates the jump factor. This represents the weight coefficients corresponding to the modal feature weight distribution plot. This represents the numerical value indicating the connection relationships between nodes in the adjacency matrix. Indicates a node All first-order neighbor nodes, The jump gain function represents the sampling across nodes. Represents a node The set of nodes corresponding to the individual's neighborhood. Indicates traversing nodes The variable index number of the set of nodes corresponding to the individual's neighborhood. This represents any node in the heterogeneous node set. Represents another arbitrary node in the heterogeneous node set; Step S33: Invoke the jump-type random walk mechanism to perform state transitions with a preset step size from family member nodes in the heterogeneous node set based on the probability distribution mapping set; During the jump process, the jump factor is used to drive the sampling points to cross the heterogeneous topology constraints to extract the interaction path between non-directly adjacent nodes, and the initial path sequence is obtained. The heterogeneous topology consists of family member nodes, action nodes, environment attribute nodes, and connecting edges between nodes. We use a feature enhancement operator to perform weighted operations on the node features in the initial path sequence, and use a spatial pooling operator to compress the dimension of the node feature vectors under the same path to obtain global remote interaction features. Step S34: Spatial dimension aggregation of global remote interaction features and local subtle action features to obtain the corresponding feature vector; By utilizing the components of each dimension of the feature vector to construct the corresponding feature channels, a feature map of family dining behavior is constructed based on the feature channels.

7. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 6, characterized in that, In step S31, feature aggregation and convolution processing are performed on the individual neighborhoods of each node in the heterogeneous node set through local random feature enhancement to obtain local subtle action features. The processing logic includes: Step S311: Extract the initial feature vector of each node in the initial embedding matrix, determine the individual neighborhood of each node according to the adjacency matrix, and use the random mask operator to perform probability discard processing on the connecting edges in the individual neighborhood to obtain the local adjacency descriptor. Step S312: Based on the local adjacency descriptor, the feature vectors of each neighboring node in the individual's neighborhood are weighted using the aggregation operator to obtain the neighborhood aggregation feature components. The calculation formula is as follows: ； in, Represents a node The corresponding neighborhood aggregation feature components, Represents a non-linear activation function. Represents nodes in a local adjacency descriptor With nodes Connection weights between them Represents the nodes in the initial embedding matrix eigenvectors, This represents the learnable transformation matrix corresponding to local random feature enhancement. This indicates the step size of the random walk; Step S313: Use a convolution operator to perform nonlinear feature mapping of the spatial dimension on the neighborhood aggregated feature components to obtain local subtle action features.

8. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 1, characterized in that, Step S4 specifically includes: Step S41: Extract the feature map of family dining behavior, and use the global average pooling operator to perform a spatial dimension statistical average operation on the feature values ​​in each feature channel. The calculation formula is as follows: ； in, Indicates the first The global description component corresponding to each feature channel. The diagram showing the characteristics of family dining behavior is shown in Figure 1. Within the first feature channel The feature values ​​of each node position Indicates the total number of nodes. Indicates the node index number; Step S42: Input the global description vector into the activation network, and perform dimensionality reduction mapping, nonlinear activation, and dimensionality increase mapping in sequence to calculate the dynamic activation coefficients corresponding to each feature channel. The calculation formula is as follows: ； ； ； in, This represents the dynamic excitation coefficient corresponding to the c-th feature channel. This represents the first learned parameter matrix. This represents the second learning parameter matrix. Indicates the dimensionality reduction compression ratio. This represents the activation function. Indicates the number of feature channels; Step S43: Use the multiplication operator to multiply the dynamic excitation coefficients with the feature components in the corresponding channels of the family dining behavior feature map element-wise, and rescale the response intensity of each feature channel to obtain the reconstructed feature map. Step S44: Integrate the feature vectors corresponding to the positions of each node in the reconstructed feature map, keeping the spatial correspondence of each node unchanged, to obtain the fused interactive feature matrix.

9. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 1, characterized in that, Step S5 specifically includes: Step S51: Obtain the fused interaction feature matrix. Use the fused interaction feature matrix of each node as the hidden feature matrix of each node. Extract the hidden feature vectors corresponding to each family member node from the hidden feature matrix of each node. Perform vector concatenation on the hidden feature vectors using the multi-attention aggregation algorithm and perform nonlinear mapping operation using the linear projection algorithm to calculate the original association score between any two family member nodes. The calculation formula is as follows: ； in, Represents family member nodes With family member nodes The original correlation score between them This represents the vector concatenation operator. Represents a linear transformation matrix. This represents the attention transpose vector. This represents the activation function. This represents the hidden feature vector corresponding to any family member node. This represents the hidden feature vector corresponding to another family member node; Step S52: The original association scores are normalized using the Softmax function to obtain the evolutionary attention weights; Step S53: The hidden feature vectors of each family member node are weighted and summed using evolutionary attention weights to obtain the aggregate vector. The calculation formula is as follows: ； in, Represents family member nodes The corresponding aggregate vector, This represents the preset feature mapping matrix. Represents a nonlinear function. Represents family member nodes With family member nodes Evolutionary attention weights between them This represents the key vector in the multiple attention mechanism.

10. The method for modeling and analyzing family dining behavior interactions based on graph neural networks as described in claim 9, characterized in that, Step S5 also includes: Step S54: Construct the interaction representation vectors corresponding to each family member node pair using the aggregation vector, and input the interaction representation vectors into the preset fully connected classification layer to obtain the interaction response strength between each family member node pair. Each family member node pair consists of any two family member nodes; The interaction response strength and evolutionary attention weight are weighted and fused to obtain the final interaction probability; The final interaction probability is compared with a preset behavior association threshold; If the final interaction probability is greater than or equal to the preset behavior association threshold, it is determined that there is a substantial shared meal interaction behavior between the corresponding family member node pairs. If the final interaction probability is less than the preset behavior association threshold, it is determined that there is no effective interaction between the corresponding family member node pairs; Step S55: For node pairs determined to have substantial interaction, extract the corresponding aggregation vector and interaction response intensity, and use a preset classification operator to map the aggregation vector to obtain the behavioral analysis results of the family meal interaction. The behavioral analysis results of the family dining interaction included collaborative dining, verbal communication, negative conflict, lack of attention, and emotional interaction.

Images