A method, apparatus, equipment, medium, and product for controlling smart home devices.

Abstract

This invention provides a method, apparatus, device, medium, and product for controlling smart home devices, applied in the field of smart home technology. The method includes: controlling smart home devices in a smart home environment to enter a target operating mode in response to user commands; determining the overall emotional state of all individuals in the smart home environment based on sensor data collected by multimodal sensors in the smart home environment in the target operating mode; and controlling the smart home devices based on the overall emotional state. Through the embodiments of this invention, it is possible to control smart home devices based on the overall emotional state of all individuals in the smart home environment, enabling the devices to adjust their operating state according to the emotional state in the dynamic environment, thereby achieving fine-grained dynamic adjustment and control of the devices.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and in particular to a method, apparatus, device, medium, and product for controlling smart home devices. Background Technology

[0002] Currently, smart home systems have gradually become an important part of modern family life. By integrating various sensors, actuators, and intelligent algorithms, smart home systems provide users with a more convenient, comfortable, and personalized living environment.

[0003] In related technologies, smart home systems mainly rely on preset scene modes to control smart home devices, such as cinema mode, guest mode, and away mode. Users can preset these modes as needed and trigger corresponding preset actions in specific scenarios, such as closing curtains, dimming lights, and turning on the projector.

[0004] However, when using the above method, the smart home system can only control smart home devices based on preset scene modes, and cannot perceive and respond to dynamic changes in the environment in real time, nor can it make fine-grained dynamic adjustments and controls to smart home devices. Summary of the Invention

[0005] In view of the above problems, a method, apparatus, device, medium, and product for controlling smart home devices are proposed to overcome or at least partially solve the above problems, including:

[0006] A method for controlling smart home devices, applied to a smart home system, the method comprising:

[0007] In response to user commands, control smart home devices in the smart home environment to enter the target operating mode;

[0008] In the target operation mode, the overall emotional state of all individuals in the smart home environment is determined based on sensor data collected by multimodal sensors in the smart home environment.

[0009] The smart home devices are controlled based on the overall emotional state.

[0010] Optionally, based on sensor data collected by multimodal sensors in the smart home environment, the overall emotional state of all individuals in the smart home environment is determined, including:

[0011] Acquire sensor data collected by multimodal sensors in the smart home environment;

[0012] Based on the sensor data, determine the characteristics related to the group's emotional field;

[0013] By fusing the relevant features of the group emotional field, a group emotional field vector is obtained to characterize the overall emotional state of all individuals in the smart home environment.

[0014] Optionally, the relevant features of the group emotional field are fused to obtain a group emotional field vector representing the overall emotional state of all individuals in the smart home environment, including:

[0015] Create a node graph for all individuals in the smart home environment; wherein the node graph includes multiple nodes, each node corresponding to an individual in the smart home environment;

[0016] Based on the aforementioned group emotional field-related characteristics, determine the first feature vector of each node;

[0017] The first feature vector of each node is fused with the first feature vectors of the node's neighboring nodes to obtain the second feature vector of each node;

[0018] The second feature vectors of the multiple nodes are aggregated to obtain the third feature vector of the node graph, and a group emotional field vector is generated based on the third feature vector of the node graph to represent the overall emotional state of all individuals in the smart home environment.

[0019] Optionally, the first feature vector of each node is fused with the first feature vectors of the node's neighboring nodes to obtain the second feature vector of each node, including:

[0020] During the fusion at the Kth layer, the input feature vector of each node at the Kth layer is fused with the input feature vectors of the node's neighboring nodes at the Kth layer to obtain the output feature vector of the Kth layer; wherein, the input feature vector of the 1st layer is the first feature vector;

[0021] The output feature vector of the Kth layer is used as the input feature vector of the K+1th layer for fusion at the K+1th layer, and the output feature vector of the Lth layer is used as the second feature vector of each node.

[0022] Where K is a positive integer greater than or equal to 1, L is a positive integer greater than 1, and L is greater than K.

[0023] Optionally, during the fusion at the Kth layer, the input feature vector of each node at the Kth layer is fused with the input feature vectors of its neighboring nodes at the Kth layer to obtain the output feature vector of the Kth layer, including:

[0024] During the fusion at the Kth layer, the attention coefficient between the node and each of its neighboring nodes is determined based on the input feature vector of each node at the Kth layer and the input feature vector of the node's neighboring nodes at the Kth layer.

[0025] The attention coefficients between the node and each neighboring node are normalized to obtain the attention weights between the node and each neighboring node.

[0026] Based on the attention weights, the input feature vectors of multiple neighboring nodes of the node in the Kth layer are weighted and aggregated to obtain the output feature vector of the Kth layer.

[0027] Optionally, based on the group emotional field-related features, a first feature vector for each node is determined, including:

[0028] Based on the aforementioned group emotional field characteristics, determine the individual characteristics of each node;

[0029] Based on the individual-related characteristics, determine the first feature vector of each node.

[0030] Optionally, the group emotional field-related features include multi-dimensional features, which include any one or more of the following:

[0031] Group valence characteristics, group arousal characteristics, and group participation characteristics.

[0032] Optionally, the multi-dimensional features include auditory type and visual type;

[0033] In the auditory type, the group arousal feature includes any one of the following: log energy, fundamental frequency standard deviation; the group valence feature includes any one of the following: emotional event valence score; and the group engagement feature includes any one of the following: speaker conversion rate.

[0034] In the visual type, the group arousal feature includes any one of the following: group kinetic energy; the group valence feature includes any one of the following: postural synchronicity; and the group participation feature includes any one of the following: interpersonal closeness.

[0035] A device for controlling smart home devices, applied in a smart home system, the device comprising:

[0036] The user command response module is used to respond to user commands and control smart home devices in the smart home environment to enter the target operating mode.

[0037] The overall emotional state determination module is used to determine the overall emotional state of all individuals in the smart home environment based on sensor data collected by multimodal sensors in the smart home environment under the target operation mode.

[0038] The device control module is used to control the smart home devices based on the overall emotional state.

[0039] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0040] A computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method described above.

[0041] A computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0042] The embodiments of the present invention have the following advantages:

[0043] In this embodiment of the invention, by responding to user commands, the smart home devices in the smart home environment are controlled to enter a target operating mode. In the target operating mode, the overall emotional state of all individuals in the smart home environment is determined based on sensor data collected by multimodal sensors in the smart home environment. Based on the overall emotional state, the smart home devices are controlled, thereby realizing the control of the smart home devices according to the overall emotional state of all individuals in the smart home environment. This enables the devices to adjust their own operating state according to the emotional state in the dynamic environment, thereby achieving fine-grained dynamic adjustment and control of the devices. Attached Figure Description

[0044] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a diagram of a hybrid architecture deployed on an edge computing gateway and a cloud server working together, provided by some embodiments of the present invention;

[0046] Figure 2 This is a flowchart of the steps of a method for controlling smart home devices provided in some embodiments of the present invention;

[0047] Figure 3 This is a flowchart of the steps of a second method for controlling smart home devices provided in some embodiments of the present invention;

[0048] Figure 4 This is a structural block diagram of a smart home device control device provided in some embodiments of the present invention. Detailed Implementation

[0049] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0050] In related technologies, the control of smart home devices relies on preset scene modes, resulting in rigid and non-adaptive interactions that cannot cope with the dynamism and complexity of real-life scenarios.

[0051] For example, when the scene mode is cinema mode, it cannot distinguish whether the user is watching an art film or a light comedy, and therefore cannot make subtle adjustments to differentiate the atmosphere.

[0052] Secondly, smart home systems in related technologies lack an understanding of interpersonal atmosphere. For example, in the guest reception mode, they cannot sense whether the visiting guest is conducting a serious business meeting or a casual gathering of friends. They can only perform corresponding actions according to the preset mode and cannot adjust the style, volume, or brightness of the background music or the lighting according to the social atmosphere of the scene. Sometimes, they may even seem inappropriate.

[0053] Based on this, this invention proposes to achieve adaptive adjustment of the environment and dynamic changes in interaction by modeling a group emotional field in a space in real time and combining it with a home personality driven by a large language model that can continuously learn and dynamically adjust. This solves the problems of rigid scenes, mechanical interactions, and inability to adapt to the dynamics of real family life in related technologies.

[0054] like Figure 1 This invention proposes a hybrid architecture deployed on an edge computing gateway and a cloud server working together. By receiving user commands and using sensor data collected by multimodal sensors, it determines the features related to the group's emotional field. Based on a fusion model of graph neural network (GNN), it constructs a node graph and calculates the global comprehensive feature vector of the current scene. The global comprehensive feature vector is mapped to the group's emotional field vector through a multilayer perceptron (MLP) in the fusion model. Then, it performs structured decision output based on a large language model (LLM), such as generating natural language responses and / or generating smart home device control commands.

[0055] Specifically, non-invasive sensors (wide-angle cameras, microphone arrays) can be used to collect anonymized, group social signals of all people in the space, rather than focusing on the biometrics of individual people. These social signals can include: sound energy (volume) in the space, speech rate and conversation turn-off rate, acoustic characteristics of laughter or arguments, overall movement speed of the crowd, and their aggregation / dispersion status.

[0056] Then, a fusion model based on graph neural networks is established, in which each person can be abstracted as a node. By analyzing the interaction features between nodes (such as dialogue frequency and gaze direction), a global comprehensive feature vector is calculated, and a multilayer perceptron is used to map the global comprehensive feature vector into a group emotional field vector.

[0057] The emotional field vector of this group can include at least three features: group valence, such as whether the atmosphere is positive or negative; group arousal, such as whether the atmosphere is exciting or calm; and group engagement, such as whether members interact highly with each other or are independent.

[0058] Finally, a large language model is built into the smart home system as the core. This large language model can learn the long-term interaction patterns of family members and gradually form a unique basic personality that matches the family style (for example, a family with children may be more humorous, while a family of serious scholars may be more concise and precise).

[0059] The aforementioned group emotional field vectors are used as real-time context inputs to the large language model to dynamically adjust the generation of interactive control strategy instructions in the large language model.

[0060] For example, when interaction with users or adjustment of the environment is required, the large language model can select appropriate interaction control strategies based on the current group emotional field. For instance, in a high-arousal, high-engagement party atmosphere, it can proactively generate instructions to play fast-paced music; in a low-arousal, low-engagement focused work atmosphere, it can proactively generate instructions to turn non-urgent notifications into silent push notifications. The wording and tone of the large language model's language interaction will also change accordingly.

[0061] The present invention will be further described below with reference to the accompanying drawings:

[0062] Reference Figure 2 The diagram illustrates a flowchart of steps for controlling a smart home device according to some embodiments of the present invention, which is applied to a smart home system.

[0063] As examples, smart home systems can be integrated with edge computing gateways and / or cloud servers. An edge computing gateway can be a local processing device in a user's home, capable of processing data from local smart home devices, such as sensor data from cameras and microphones, in real time to improve response speed and protect user privacy.

[0064] Specifically, it may include the following steps:

[0065] Step 201: In response to the user's instruction, control the smart home devices in the smart home environment to enter the target operating mode.

[0066] As some examples, user instructions can be voice commands or text commands entered through the device, such as "turn on the living room lights" or "turn on the family dinner mode." The target operation mode can be a scene control mode determined based on the user instructions.

[0067] For example, if a user says "turn on the family dinner mode", the smart home system can control the smart home devices to enter the family dinner mode, so that the smart home devices can work together according to the family dinner mode, such as the smart lights dimming the living room lights and the smart speakers playing soft music.

[0068] Step 202: In the target operation mode, the overall emotional state of all individuals in the smart home environment is determined based on the sensor data collected by the multimodal sensors in the smart home environment.

[0069] As examples, a smart home environment refers to an environment containing smart home devices, such as an office, bedroom, or living room; multimodal sensors can include cameras, microphones, etc.; sensor data can include information such as images, audio, and environmental parameters; an individual refers to each member (person) in the smart home environment, whose emotional state can be determined through data fusion analysis of facial expressions, voice tone, etc.; the overall emotional state can include happiness, pleasure, calmness, etc.

[0070] For example, in target operation mode, smart home devices equipped with multimodal sensors can collect sensor data and send it to the smart home system. By analyzing and calculating the sensor data, the overall emotional state of all members in the current smart home environment can be determined.

[0071] In some embodiments of the present invention, the overall emotional state of all individuals in the smart home environment is determined based on sensor data collected by multimodal sensors in the smart home environment, including:

[0072] Acquire sensor data collected by multimodal sensors in the smart home environment; determine group emotional field related features based on the sensor data; fuse the group emotional field related features to obtain a group emotional field vector that characterizes the overall emotional state of all individuals in the smart home environment.

[0073] As examples, group emotional field related features can be calculated from sensor data and can be used as feature parameters to reflect the emotional state of a group; the group emotional field vector is a vector obtained by fusing group emotional field related features and is used to quantitatively represent the overall emotional state (such as excitement, fatigue, tension, happiness, etc.) of all individuals in the smart home environment.

[0074] For example, multimodal sensors serve as input sources, continuously collecting data from the environment (such as voice segments, motion trajectories, etc.). After the smart home system preprocesses the collected sensor data (such as noise reduction and segmentation), it calculates features related to group emotions (such as emotional tone in speech), and fuses the extracted multidimensional features through weighted aggregation and other methods to generate a group emotion field vector.

[0075] In some examples, the group sentiment field vector is a time-varying vector that represents the overall emotional state of a group within a space at time point t. This group sentiment field vector contains at least three dimensions: group valence features, group arousal features, and group participation features.

[0076]

[0077] in, It refers to the collective emotional field vector of a group within a space at time point t; It refers to the group valence characteristic of a group within a space at time point t; This refers to the arousal characteristics of a group within a space at time point t; This refers to the group participation characteristics of a group within a space at time point t; It refers to a three-dimensional real number space.

[0078] The group emotional field vector can be calculated by combining a composite function F (a graph neural network fusion model based on an attention mechanism) with a feature set. This function uses the feature set extracted from sensor data collected from multimodal sensors within a time window Δt. ) is the input:

[0079]

[0080] in, It is the auditory feature within that time window ( ) and visual features ( )gather.

[0081] In some embodiments of the present invention, the group emotional field-related features include multi-dimensional features, which include any one or more of the following: group valence features, group arousal features, and group participation features.

[0082] As examples, group valence characteristics can characterize the positive or negative tendency of the overall emotional atmosphere of a group, i.e., whether the group's emotions are positive (pleasure, excitement) or negative (sadness, anger).

[0083] For example, if everyone is having a pleasant conversation and laughing during dinner, the group valence is high; but if a dispute occurs, the group valence is neutral or negative.

[0084] Group arousal characteristics can characterize whether the overall emotional atmosphere of a group is excited or calm. For example, loud conversations during a gathering of friends indicate high arousal, while family members focusing on work or reading indicate low arousal.

[0085] Group participation characteristics can characterize the closeness of interaction among members, such as high participation when playing games or discussing in groups, and low participation when engaging in independent activities.

[0086] For example, a pleasant conversation among family members with a high speaker switching rate can be considered high participation; when children are talking but parents are attentively listening with a high degree of synchronicity, it can also be considered a high level of "focused" participation.

[0087] In some embodiments of the present invention, the multidimensional features have auditory type and visual type.

[0088] After determining the characteristics related to the group emotional field based on sensor data, the multi-dimensional characteristics of the group emotional field can be divided into two types (i.e., auditory type and visual type). The acoustic characteristics (such as log energy, fundamental frequency standard deviation, emotional event valence score, and speaker switching rate) under the auditory type and the behavioral characteristics (such as group kinetic energy, posture synchronization, and interpersonal closeness) under the visual type are calculated respectively.

[0089] In some embodiments of the present invention, in the auditory type, the group arousal feature includes any one of the following: log energy, fundamental frequency standard deviation; the group valence feature includes any one of the following: emotional event valence score; and the group engagement feature includes any one of the following: speaker conversion rate.

[0090] As some examples, logarithmic energy can be calculated using the following formula:

[0091]

[0092] in, This refers to logarithmic energy; log refers to the natural logarithm function. This refers to audio sampling points, where n is any sampling point between 1 and N; N is the total number of sampling points within the window. This refers to a small constant that prevents logarithmic underflow.

[0093] As some examples, the fundamental frequency standard deviation can be calculated using the following formula:

[0094]

[0095] in, This refers to the standard deviation of the fundamental frequency; This refers to the fundamental frequency sequence of all voice frames detected within the window; It refers to the first The fundamental frequency of the frame speech is the pitch of the speech information; M refers to the total number of speech frames detected within the analysis time window. This refers to the index, which represents any frame from frame 1 to frame M. This refers to the average value of the M-frame base frequency.

[0096] As some examples, the valence score of an emotional event can be calculated using the following formula:

[0097]

[0098] in, This refers to the valence score of emotional events; It is a lightweight convolutional neural network for the Mel spectrum of the input. The output is a vector of event category probabilities. This refers to a collection of positive emotional events (such as laughter, cheers). It refers to a set of negative emotional events (such as {the sound of arguing}).

[0099] For example, a smart home system acquires raw audio signals from a microphone and then processes the audio signals into a Mel spectrogram; inputting the Mel spectrogram features into a convolutional neural network will output a probability vector of an event category.

[0100] For example, P could be equal to P(laughter) = 0.6, P(cheer) = 0.1, P(argument) = 0.05, P(speaking) = 0.25; the smart home system has two predefined sets, a positive set {laughter, cheers} and a negative set {argument}; when the probability of the positive set is equal to P(laughter) + P(cheer) = 0.7, the probability of the negative set is equal to P(argument).

[0101] As some examples, the speaker conversion rate can be calculated using the following formula:

[0102]

[0103] in, This refers to the speaker conversion rate; This refers to the number of speaker switching events detected within a time window Δt (seconds) using speaker log technology.

[0104] In some embodiments of the present invention, in the visual type, the group arousal feature includes any one of the following: group kinetic energy, the group valence feature includes any one of the following: postural synchronicity, and the group participation feature includes any one of the following: interpersonal closeness.

[0105] As examples, assuming the system detects K anonymized individuals within a time window, the group kinetic energy can be calculated using the following formula:

[0106]

[0107] in, This refers to the group's kinetic potential energy; K represents the number of individuals detected. The first was calculated using the optical flow method. The average motion vector of each individual contour (i.e., the system-calculated first individual contour) (Individual's overall average speed and direction of movement), the The individual finger system detects a specific person out of K individuals; the contour finger system sees this person through a camera. The body shape and silhouette, the motion vector can be thought of as an arrow representing motion; the system can compare two consecutive frames of images (such as images at the 1st millisecond and the 2nd millisecond).

[0108] By analyzing this The outline determines its position in the first frame and its position in the second frame; the vector (arrow) represents this movement, and the direction of the arrow represents... The direction of movement is indicated by the length of the arrow. The speed of movement; since a person's silhouette is composed of many points, the system can calculate this. The movement arrows of all points on the outline are calculated, and then an average value is taken. The average value of this value for all K people is added together, and then the group's kinetic energy is obtained, which is used to reflect the movement or stillness of the entire group.

[0109] As examples, suppose the system detects K anonymized individuals within a time window; the pose synchronization can be calculated using the following formula:

[0110]

[0111] in, This refers to attitude synchronization; K represents the number of individuals detected. This refers to the first Pose vectors extracted from the skeletal key points of each individual; This refers to the first one. The pose vectors extracted from the skeletal keypoints of an individual are used to calculate the average cosine similarity of the pose vectors between all pairs of individuals to obtain the pose synchronization.

[0112] For example, given the total number of individuals K detected by the system in the current space, we can first calculate the pose vector of each of the K individuals (e.g., ...). The pose vector can be extracted by analyzing the skeletal key points of each person (such as the position of the head, shoulders, and elbows), and mathematically represents a person's current posture.

[0113] Then, calculate any two people ( and The cosine similarity between them is expressed in the formula:

[0114]

[0115] The system can randomly select two people from K people, such as individuals. and individuals Calculate the pose vectors of these two people ( and The cosine similarity is calculated as follows: the result can be a score between -1 and 1. If the score is close to 1, it means that the two people have almost the same posture (such as both people leaning forward in sync). If the score is close to 0, it means that the two people have completely unrelated postures.

[0116] Finally, the system can repeatedly calculate the pairwise cosine similarity for all possible pairs of individuals in the space (e.g., when K=3, it will calculate the cosine similarity between 1 and 2, 1 and 3, and 2 and 3). After summing all the calculated pairwise similarity scores, the average is taken to obtain a standardized average value, which is the attitude synchronization, representing the average consistency of the entire group's attitude at the current moment.

[0117] If the posture synchronicity value is high (e.g., 0.85), it indicates that most people in the space are adopting similar postures, representing a state of high synchronization, shared focus, or shared emotions (e.g., moments of empathy). If the posture synchronicity value is low, it indicates that everyone's postures are different, and the atmosphere is relatively casual or independent.

[0118] As examples, assuming the system detects K anonymized individuals within a time window, interpersonal closeness can be calculated using the following formula:

[0119]

[0120] in, This refers to interpersonal closeness; K represents the number of individuals detected. It refers to the first The coordinates of the center position of each individual; It refers to the first The coordinates of the center position of each individual.

[0121] For example, the system detects a total of K individuals in the current space, and then obtains the center coordinates of each of the K individuals.

[0122] If K=3, the system will obtain three coordinates, i.e. Then calculate the straight-line distance between each pair of individuals, i.e. and The distance between them is expressed in the formula as Then, calculate the reciprocal of the square between each pair of individuals, i.e. .

[0123] Finally, iterate through all possible pairs of individuals (e.g., when K=3, calculate the values ​​between 1 and 2, 1 and 3, and 2 and 3), sum and average all the inverse square values, divide this sum by all the logarithms to obtain a standardized value, namely interpersonal closeness, which quantifies whether a group is clustered or dispersed.

[0124] In the above example, since the reciprocal of distance is used in the calculation, the closer the individuals are, the higher the value of interpersonal closeness will be, which means that the group is closer; conversely, if everyone stands far apart, the value of interpersonal closeness will be very low.

[0125] In some embodiments of the present invention, the group emotional field-related features are fused to obtain a group emotional field vector characterizing the overall emotional state of all individuals in the smart home environment, including:

[0126] Sub-step 11: Create a node graph for all individuals in the smart home environment; wherein the node graph includes multiple nodes, each node corresponding to an individual in the smart home environment.

[0127] As examples, a graph neural network fusion model can be used to construct a node graph (abstract data structure graph) to model the dynamic interaction relationships between all individuals in a smart home environment. Each node in the abstract data structure graph represents an individual in the smart home environment, and the abstract data structure graph can have multiple nodes, that is, multiple individuals.

[0128] For example, we can construct an abstract data structure graph G=(N, E); where N refers to the set of nodes, N={1,...,K} represents K individuals (people); and E represents the edges between nodes. In other words, each node is treated as a carrier of a person's corresponding characteristics. The state of the nodes (the state between people) is quantified and described through these characteristics, while the edges between nodes indicate the spatial or behavioral relationships between individuals.

[0129] Sub-step 12: Determine the first feature vector of each node based on the group emotional field related features.

[0130] As examples, the first feature vector can be the initial feature vector of each individual; the group emotional field-related features of each individual can be concatenated to obtain the initial feature vector of each individual, i.e., the first feature vector.

[0131] For example, for each node The characteristics of their individual group emotional field (such as...) The average motion vector, posture vector, and acoustic features (if the speaker is present) are concatenated to form the first feature vector:

[0132]

[0133] in, d refers to the first eigenvector; d refers to the dimensional vector space.

[0134] In some embodiments of the present invention, determining the first feature vector of each node based on the group emotional field-related features includes:

[0135] Sub-step 121: Determine the individual related characteristics of each node based on the group emotional field related characteristics.

[0136] Sub-step 122: Determine the first feature vector of each node based on the individual-related characteristics.

[0137] Since the features related to the group emotional field are extracted from the real-time behavior dynamics and interaction patterns of each individual in the space, these features can be split and aligned by individual to obtain the individual-related features of each node in the group emotional field.

[0138] For example, the group kinetic potential energy is obtained by summing and averaging the average motion vectors of each individual, and the average motion vector of each individual can be extracted as part of its individual-related characteristics. Similarly, attitude synchronization can be decomposed into the relative attitude angle difference between each two individuals, and then mapped to part of its individual-related characteristics.

[0139] By concatenating the individual-related features of each node, we obtain the initial feature vector of each individual, i.e., the first feature vector.

[0140] Sub-step 13: Fuse the first feature vector of each node with the first feature vectors of the node's neighboring nodes to obtain the second feature vector of each node.

[0141] As examples, attention-based graph neural network fusion models can be used to fuse the first feature vector of each node with the first feature vectors of its neighboring nodes to obtain a fused second feature vector.

[0142] For example, if the first feature vector of node A is 1, and the first feature vector of node B, a neighbor of node A, is also 1, then the fusion results in the second feature vector 2.

[0143] In some embodiments of the present invention, the first feature vector of each node is fused with the first feature vectors of the node's neighboring nodes to obtain the second feature vector of each node, including:

[0144] Sub-step 131: During the fusion at the Kth layer, the input feature vector of each node at the Kth layer is fused with the input feature vectors of the neighboring nodes at the Kth layer to obtain the output feature vector of the Kth layer; wherein, the input feature vector of the 1st layer is the first feature vector.

[0145] In the K-th layer computation of the graph neural network fusion model, the node feature update can utilize self-fusion and neighbor fusion mechanisms. Each node first obtains its own input feature vector in the K-th layer (if it is the initial layer, i.e., the 1st layer, then the input feature vector is the first feature vector); at the same time, it collects the corresponding input feature vectors of its first-order neighbor nodes in the K-th layer, combines the node's own features with the aggregated neighbor features (such as concatenation or addition), and outputs a new feature vector in the K-th layer after nonlinear transformation, which serves as the input basis for the node to enter the next layer computation.

[0146] In some embodiments of the present invention, during the fusion at the Kth layer, the input feature vector of each node at the Kth layer is fused with the input feature vectors of the neighboring nodes at the Kth layer to obtain the output feature vector of the Kth layer, including:

[0147] Sub-step 1311: During the fusion at the Kth layer, the attention coefficient between the node and each neighbor node is determined based on the input feature vector of each node at the Kth layer and the input feature vector of the node's neighbor nodes at the Kth layer.

[0148] As examples, during the fusion of the Kth layer of a graph neural network fusion model, the interaction strength between a node and its neighbors can be dynamically calculated through an attention mechanism. For each node, its own input feature vector at the Kth layer can be obtained first, while the input feature vectors of all directly connected neighboring nodes at the Kth layer can be collected. Subsequently, the attention coefficient between the node and each neighboring node can be obtained based on these feature vectors (e.g., by concatenating the features of the node itself and its neighbors).

[0149] For example, nodes can be calculated using the following formula. and his neighbors Attention coefficient between:

[0150]

[0151] in, Represents a node and his neighbors Attention coefficient between them; It is an improved activation function in graph neural networks; This represents the learnable weight matrix of this layer, and k represents the current layer number of the graph neural network fusion model; This represents a vector concatenation operation; This represents a learnable single-layer feedforward network weight vector, where T represents the transpose of the matrix. Indicates the current layer node The input feature vector, Indicates the current layer node The input feature vector.

[0152] Sub-step 1312: Normalize the attention coefficients of the node and each neighboring node to obtain the attention weights of the node and each neighboring node.

[0153] As examples, the attention coefficients between a node and its neighbors can be normalized using the softmax function (normalized activation function). This process amplifies the differences among all attention coefficients through an exponential transformation, then divides them by the sum to normalize the weights to a probability distribution (summing to 1), which can then be used as the attention weights between the node and each of its neighboring nodes. For example, the coefficients of node A with its neighbors B and C are [2, 1], and the normalized weights are [0.7, 0.3].

[0154] The attention coefficients between a node and its neighbors are normalized using the Softmax function to obtain the attention weights between the node and each neighbor. This weight can be calculated using the following formula:

[0155]

[0156] in, This refers to the attention weights between a node and each of its neighboring nodes; softmax is a mathematical function that transforms a numerical vector into a probability distribution. Represents a node and his neighbors The attention coefficient between them; exp refers to the natural exponential function; It is a node The set of all neighbors; This refers to an index variable that represents a node. One of all its neighboring nodes; It is for nodes All neighbors Perform a traversal and calculate each neighbor. corresponding And sum them.

[0157] Sub-step 1314: Based on the attention weights, the input feature vectors of multiple neighboring nodes of the node in the Kth layer are weighted and aggregated to obtain the output feature vector of the Kth layer.

[0158] As some examples, the input feature vectors of each neighbor of a node at layer K can be assigned according to attention weights, and the neighbor information can be aggregated by weighted summation to obtain the output feature vector of the node at layer K.

[0159] For example, the node can be updated by weighting and aggregating the input feature vectors of neighboring nodes according to attention weights. The feature vector representation of the node is obtained. The output feature vector of the Kth layer:

[0160]

[0161] in, Represents a node The output feature vector at the (k+1)th layer; This refers to nonlinear activation functions (such as the Rectified Linear Unit (RELU)). This refers to the attention weights between a node and each of its neighboring nodes; k represents the current layer number of the graph neural network fusion model; W is a trainable weight matrix. This represents the learnable weight matrix of this layer; Indicates the current layer node The input feature vector; It is a node The set of all neighbors.

[0162] Combining sub-steps 1311-1314, during the fusion at layer K, the input feature vector of each node at layer K is fused with the input feature vectors of its neighboring nodes at layer K to obtain the output feature vector of layer K for further interpretation. The entire process can be divided into three steps:

[0163] The first step is to calculate the attention coefficient.

[0164] Get the node of the current layer k The first feature vector and its neighboring nodes The first feature vector is used to obtain the learnable weight matrix of this layer (the k-th layer). ), to obtain the learnable attention weight vector a.

[0165] Then, using the same weight matrix ( Multiply by the nodes respectively Features ( ) and neighbors Features ( The feature vectors undergo a linear transformation, and the two transformed feature vectors are concatenated together (in the formula). (symbol); multiply the attention vector by the concatenated long vector. (transpose) Finally, input the obtained score into... In the activation function, the attention coefficient is obtained. ( ), representing nodes To the neighbors The original attention or relevance score, this process will affect the node All neighbors (including) Each one is counted once.

[0166] The second step is to normalize the attention weights.

[0167] Get the set of all neighbors of the node calculated in the first step. Attention coefficient () ); The attention coefficients are normalized using the Softmax function, and the Softmax function is applied.

[0168] Wherein, the numerator is the exponent of the attention coefficient ( ), the denominator takes the node To all neighbors The attention coefficients are all taken exponentially and then summed ( ), and divide the numerator by the denominator.

[0169] Finally, we obtain the attention weights between a node and each of its neighboring nodes. Attention weights are values ​​between 0 and 1, representing the neighbors. For nodes The importance percentage.

[0170] After normalization, the nodes Attention weights with all its neighbors The sum of them equals 1.

[0171] The third step is to aggregate the neighbor feature vectors and update the nodes. Eigenvectors.

[0172] Obtain the nodes calculated in the second step. To all his neighbors The attention weights are obtained by retrieving all neighbors used in the first step of the calculation. The transformed feature vector. For each neighbor... Multiply its own attention weight by its transformed feature vector ( Then, the weighted feature vectors of all neighbors are aggregated, and the aggregated total feature vector is input into... In nonlinear activation functions.

[0173] Get Node The output feature vector of the (k+1)th layer ( This new feature vector incorporates important information from all neighbors (with attention weights determining which is more important), thus completing the node... One feature update.

[0174] Sub-step 132: The output feature vector of the Kth layer is used as the input feature vector of the (K+1)th layer for fusion of the (K+1)th layer, and the output feature vector of the Lth layer is used as the second feature vector of each node; where K is a positive integer greater than or equal to 1, L is a positive integer greater than 1, and L is greater than K.

[0175] As examples, in the hierarchical propagation of graph neural networks, node features can be updated layer by layer. The feature vector output by the Kth layer (such as the feature vector after attention aggregation) is directly used as the input of the K+1th layer, which is used to further integrate neighbor information (such as weighted aggregation again). When the output feature vector of a specified deep layer (such as the Lth layer, L>K and L≥2) is reached, it is used as the second feature vector of the node.

[0176] Sub-step 14: Aggregate the second feature vectors of the multiple nodes to obtain the third feature vector of the node graph, and generate a group emotional field vector to characterize the overall emotional state of all individuals in the smart home environment based on the third feature vector of the node graph.

[0177] As some examples, the second feature vectors of each node can be aggregated (such as attention pooling) first, and the information of all nodes can be fused to obtain the third feature vector of the node graph (global comprehensive feature vector); then, based on this third feature vector, it can be mapped to the group sentiment field vector; where, the multilayer perceptron refers to a classic feedforward artificial neural network that achieves nonlinear feature transformation by stacking multiple fully connected layers (Dense Layer).

[0178] After obtaining the second feature vector through L layers of updates, mean pooling can be used to aggregate the second feature vectors of all nodes to obtain the third feature vector:

[0179]

[0180] in, It refers to the first The second feature vector of each node; This refers to the third eigenvector.

[0181] After obtaining the third feature vector, the third feature vector is input into a multilayer perceptron, and finally mapped to a group emotion field vector:

[0182]

[0183] in, It refers to the group emotional field vector of a group in a space at time point t.

[0184] Example 1: When the target task mode is a family dinner mode, involving a family of three, with nodes {father, mother, child}, and K=3 in the graph neural network fusion model, the smart home system preprocesses the collected sensor data and calculates the group emotional field-related features:

[0185]

[0186] in, This refers to logarithmic energy; This refers to the standard deviation of the fundamental frequency; This refers to the valence score of emotional events; This refers to the speaker conversion rate; This refers to the kinetic potential energy of the group; This refers to the closeness of interpersonal relationships; This refers to attitude synchronization.

[0187] A node graph is constructed for the three nodes ({father, mother, child}) using a graph neural network fusion model, and the first feature vector of each node is determined. .

[0188] because =12 (high), calculated attention weights ( The distribution is relatively uniform among different node pairs, with no obvious focal point.

[0189] The third eigenvector of the node graph is obtained. Then, the third feature vector is mapped to a group emotion field vector through a multilayer perceptron. ),get .

[0190] Based on the group's emotional field vector, the large language model determines that the mood is warm and lively, and generates instructions to enable the smart home system to control smart home devices to perform operations such as playing music and adjusting warm-colored lights.

[0191] For example, in conjunction with Example 1 above, it was detected that the child was describing a difficult problem and the parents were listening attentively; the calculated logarithmic energy was -28dB, the emotional event valence score was -0.1, the speaker switching rate was 2 (the child was the only speaker), and the posture synchronicity increased to 0.85 because both parents adopted a forward-leaning synchronous listening posture.

[0192] Despite the extremely low speaker switching rate, the extremely high attitude synchronization value is encoded into the first initial feature vector of the parent and mother nodes.

[0193] When calculating attention, the graph neural network fusion model can learn to assign high attention weights between child nodes and parent nodes due to the similarity of features between parent and child nodes (high pose synchronization) and their shared visual orientation towards child nodes. This indicates that it is a highly focused scenario centered on the child.

[0194] The group emotional field vector [-0.4, -0.5, 0.7] was obtained. The graph neural network fusion model did not mistakenly judge low participation due to low speaker conversion rate, but correctly identified high non-verbal participation through visual features such as gesture synchronization.

[0195] Based on the group's emotional field vector, the large language model determines that a moment of empathy requires quiet and focus, and immediately generates instructions to smoothly fade out the music.

[0196] Step 203: Control the smart home devices based on the overall emotional state.

[0197] In practical applications, the group emotional field vector can be input into a large language model. The large language model can identify the overall emotional state as anxiety based on the group emotional field vector, and can generate control commands for smart home devices, such as turning on or dimming lights or playing soft music. The smart home system can then send control commands to smart lights, smart speakers, and other devices through Internet of Things protocols (such as Message Queuing Telemetry Transport (MQTT)) based on the control commands, thereby controlling the smart home devices.

[0198] In this embodiment of the invention, by responding to user commands, the smart home devices in the smart home environment are controlled to enter a target operating mode. In the target operating mode, the overall emotional state of all individuals in the smart home environment is determined based on sensor data collected by multimodal sensors in the smart home environment. Based on the overall emotional state, the smart home devices are controlled, thereby realizing the control of the smart home devices according to the overall emotional state of all individuals in the smart home environment. This enables the devices to adjust their own operating state according to the emotional state in the dynamic environment, thereby achieving fine-grained dynamic adjustment and control of the devices.

[0199] Reference Figure 3 This diagram illustrates a flowchart of another method for controlling smart home devices according to some embodiments of the present invention, applied to a smart home system, and specifically may include the following steps:

[0200] Step 301: In response to the user's instruction, control the smart home devices in the smart home environment to enter the target operating mode.

[0201] Step 302: In the target operating mode, acquire sensor data collected by multimodal sensors in the smart home environment.

[0202] Step 303: Determine the characteristics related to the group's emotional field based on the sensor data.

[0203] Step 304: The relevant features of the group emotional field are fused to obtain a group emotional field vector that represents the overall emotional state of all individuals in the smart home environment.

[0204] Step 305: Control the smart home devices based on the overall emotional state.

[0205] In this embodiment of the invention, by responding to user commands, the smart home devices in the smart home environment are controlled to enter a target operating mode. In the target operating mode, the overall emotional state of all individuals in the smart home environment is determined based on sensor data collected by multimodal sensors in the smart home environment. Based on the overall emotional state, the smart home devices are controlled, thereby realizing the control of the smart home devices according to the overall emotional state of all individuals in the smart home environment. This enables the devices to adjust their own operating state according to the emotional state in the dynamic environment, thereby achieving fine-grained dynamic adjustment and control of the devices.

[0206] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0207] Reference Figure 4 The diagram illustrates a structural schematic of a smart home device control apparatus according to some embodiments of the present invention, which is applied to a smart home system and may specifically include the following modules:

[0208] User command response module 401 is used to respond to user commands and control smart home devices in the smart home environment to enter the target operating mode;

[0209] The overall emotional state determination module 402 is used to determine the overall emotional state of all individuals in the smart home environment based on sensor data collected by multimodal sensors in the smart home environment under the target operation mode.

[0210] The device control module 403 is used to control the smart home device according to the overall emotional state.

[0211] In some embodiments of the present invention, the overall emotional state determination module 402 includes:

[0212] The sensor data acquisition submodule is used to acquire sensor data collected by multimodal sensors in the smart home environment;

[0213] The relevant feature determination submodule is used to determine the relevant features of the group's emotional field based on the sensor data;

[0214] The relevant feature fusion submodule is used to fuse the relevant features of the group emotional field to obtain a group emotional field vector that represents the overall emotional state of all individuals in the smart home environment.

[0215] In some embodiments of the present invention, the related feature fusion submodule includes:

[0216] A node graph creation unit is used to create a node graph for all individuals in the smart home environment; wherein, the node graph includes multiple nodes, and each node corresponds to an individual in the smart home environment;

[0217] The first feature vector determination unit is used to determine the first feature vector of each node based on the group emotional field related features;

[0218] The second feature vector determination unit is used to fuse the first feature vector of each node with the first feature vectors of the neighboring nodes of the node to obtain the second feature vector of each node.

[0219] The third feature vector determination unit is used to aggregate the second feature vectors of the multiple nodes to obtain the third feature vector of the node graph, and generate a group emotional field vector to characterize the overall emotional state of all individuals in the smart home environment based on the third feature vector of the node graph.

[0220] In some embodiments of the present invention, the second feature vector determination unit includes:

[0221] The first fusion subunit is used to fuse the input feature vector of each node at layer K with the input feature vector of the neighboring nodes at layer K during fusion at layer K to obtain the output feature vector of layer K; wherein, the input feature vector of layer 1 is the first feature vector;

[0222] The second fusion subunit is used to take the output feature vector of the Kth layer as the input feature vector of the K+1th layer to perform fusion of the K+1th layer, and take the output feature vector of the Lth layer as the second feature vector of each node.

[0223] Where K is a positive integer greater than or equal to 1, L is a positive integer greater than 1, and L is greater than K.

[0224] In some embodiments of the present invention, the first fusion subunit is configured to:

[0225] During the fusion at the Kth layer, the attention coefficient between the node and each of its neighboring nodes is determined based on the input feature vector of each node at the Kth layer and the input feature vector of the node's neighboring nodes at the Kth layer.

[0226] The attention coefficients between the node and each neighboring node are normalized to obtain the attention weights between the node and each neighboring node.

[0227] Based on the attention weights, the input feature vectors of multiple neighboring nodes of the node in the Kth layer are weighted and aggregated to obtain the output feature vector of the Kth layer.

[0228] In some embodiments of the present invention, the first feature vector determination unit is configured to:

[0229] Based on the aforementioned group emotional field characteristics, determine the individual characteristics of each node;

[0230] Based on the individual-related characteristics, determine the first feature vector of each node.

[0231] In some embodiments of the present invention, the group emotional field-related features include multi-dimensional features, which include any one or more of the following:

[0232] Group valence characteristics, group arousal characteristics, and group participation characteristics.

[0233] In some embodiments of the present invention, the multidimensional features have auditory type and visual type;

[0234] In the auditory type, the group arousal feature includes any one of the following: log energy, fundamental frequency standard deviation; the group valence feature includes any one of the following: emotional event valence score; and the group engagement feature includes any one of the following: speaker conversion rate.

[0235] In the visual type, the group arousal feature includes any one of the following: group kinetic energy; the group valence feature includes any one of the following: postural synchronicity; and the group participation feature includes any one of the following: interpersonal closeness.

[0236] Some embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method described above.

[0237] Some embodiments of the present invention also provide a computer-readable storage medium on which a computer program is stored, and which, when executed by a processor, implements the method described above.

[0238] Some embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0239] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0240] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0241] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0242] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0243] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0244] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0245] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0246] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0247] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the aforementioned element.

[0248] The above provides a detailed description of the method, apparatus, device, medium, and product for controlling smart home devices. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method of smart home device control, the method comprising: The method is applied to a smart home system, and comprises: In response to a user instruction, controlling a smart home device in a smart home environment to enter a target operation mode; In the target operation mode, determining an overall emotional state of all individuals in the smart home environment according to sensor data collected by multi-modal sensors in the smart home environment; According to the overall emotional state, the smart home device is controlled; Wherein, according to the sensor data collected by the multi-modal sensors in the smart home environment, the overall emotional state of all individuals in the smart home environment is determined, comprising: Obtaining sensor data collected by multi-modal sensors in the smart home environment; According to the sensor data, determine the group emotional field related feature; Fusion is performed on the group emotional field related feature to obtain a group emotional field vector for representing the overall emotional state of all individuals in the smart home environment; Wherein, fusion is performed on the group emotional field related feature to obtain a group emotional field vector for representing the overall emotional state of all individuals in the smart home environment, comprising: Creating a node graph for all individuals in the smart home environment; wherein the node graph comprises a plurality of nodes, each node corresponding to an individual in the smart home environment; According to the group emotional field related feature, determine the first feature vector of each node; Fusion is performed on the first feature vector of each node and the first feature vector of the neighbor node of the node to obtain the second feature vector of each node; The second feature vectors of the plurality of nodes are aggregated to obtain a third feature vector of the node graph, and a group emotional field vector for representing the overall emotional state of all individuals in the smart home environment is generated according to the third feature vector of the node graph; Wherein, the group emotional field related feature comprises multi-dimensional features, and the multi-dimensional features comprise any one or more of the following: Group valence feature, group arousal feature, group participation feature; The multi-dimensional features have an auditory type and a visual type; In the auditory type, the group arousal feature includes any one of the following: logarithmic energy, fundamental frequency standard deviation, the group valence feature includes any one of the following: emotional event valence score, and the group participation feature includes any one of the following: speaker transition rate; In the visual type, the group arousal feature includes any one of the following: group kinetic energy, the group valence feature includes any one of the following: posture synchrony, and the group participation feature includes any one of the following: interpersonal closeness.

2. The method of claim 1, wherein, Fusion is performed on the first feature vector of each node and the first feature vector of the neighbor node of the node to obtain the second feature vector of each node, comprising: In the Kth layer fusion, the input feature vector of each node in the Kth layer is fused with the input feature vector of the neighbor node of the node in the Kth layer to obtain the output feature vector of the Kth layer; wherein the input feature vector of the first layer is the first feature vector; The output feature vector of the Kth layer is taken as an input feature vector of the K+1th layer to perform K+1th layer fusion, and the output feature vector of the Lth layer is taken as a second feature vector of each node. K is a positive integer greater than or equal to 1, L is a positive integer greater than 1, and L is greater than K.

3. The method of claim 2, wherein, In the Kth layer fusion, the input feature vector of each node in the Kth layer is fused with the input feature vector of the neighbor node of the node in the Kth layer to obtain the output feature vector of the Kth layer, including: In the Kth layer fusion, the attention coefficient of the node and each neighbor node is determined according to the input feature vector of each node in the Kth layer and the input feature vector of the neighbor node of the node in the Kth layer. The attention coefficients of the node and each neighbor node are normalized to obtain the attention weights of the node and each neighbor node. According to the attention weights, the input feature vectors of the multiple neighbor nodes of the node in the Kth layer are weighted and aggregated to obtain the output feature vector of the Kth layer.

4. The method of claim 1, wherein, According to the group emotional field related features, a first feature vector of each node is determined, including: According to the group emotional field related features, an individual related feature of each node is determined. According to the individual related feature, a first feature vector of each node is determined.

5. An apparatus for smart home device control, the apparatus comprising: The device is applied to an intelligent home system, and the device includes: A user instruction response module is configured to control an intelligent home device in an intelligent home environment to enter a target operation mode in response to a user instruction. An overall emotional state determination module is configured to determine an overall emotional state of all individuals in the intelligent home environment according to sensor data collected by multi-modal sensors in the intelligent home environment in the target operation mode. A device control module is configured to control the intelligent home device according to the overall emotional state. The overall emotional state determination module includes: A sensor data acquisition submodule is configured to acquire sensor data collected by multi-modal sensors in the intelligent home environment. A related feature determination submodule is configured to determine group emotional field related features according to the sensor data. A related feature fusion submodule is configured to fuse the group emotional field related features to obtain a group emotional field vector representing the overall emotional state of all individuals in the intelligent home environment. The related feature fusion submodule includes: A node graph creation unit is configured to create a node graph for all individuals in the intelligent home environment. The node graph includes multiple nodes, and each node corresponds to an individual in the intelligent home environment. A first feature vector determination unit is configured to determine a first feature vector of each node according to the group emotional field related features. A second feature vector determination unit is configured to fuse the first feature vector of each node with the first feature vector of a neighbor node of the node to obtain a second feature vector of each node. a third feature vector determination unit configured to aggregate the second feature vectors of the plurality of nodes to obtain a third feature vector of the node graph, and generate a group affective field vector representing a collective emotional state of all individuals in the smart home environment according to the third feature vector of the node graph; wherein the group affective field related features comprise multi-dimension features, and the multi-dimension features comprise any one or more of: a group valence feature, a group arousal feature, and a group engagement feature; the multi-dimension features comprise an auditory type and a visual type; in the auditory type, the group arousal feature comprises any one of: a log energy and a fundamental frequency standard deviation, the group valence feature comprises any one of: an affective event valence score, and the group engagement feature comprises any one of: a speaker turn rate; in the visual type, the group arousal feature comprises any one of: a group kinetic energy, the group valence feature comprises any one of: a posture synchrony, and the group engagement feature comprises any one of: an interpersonal closeness.

6. An electronic device, comprising: a computer program product comprising a processor, a memory, and a computer program stored on the memory and loadable on the processor, the computer program being executable by the processor to implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, a computer readable storage medium storing a computer program, the computer program being executable by a processor to implement the method of any one of claims 1 to 4.

8. A computer program product, characterised in that, a computer program product comprising a computer program, the computer program being executable by a processor to implement the method of any one of claims 1 to 4.

Images