Equipment control method and device based on graph neural network, and smart home system

Abstract

The embodiment of the invention relates to an equipment control method and device based on a graph neural network, and a smart home system. The method comprises the steps of obtaining a corresponding object attribute information set and an object relation information set under a target equipment control system; based on the object attribute information set and the object relation information set, constructing a graph structure containing nodes and edges; performing state prediction on nodes and edges in the graph structure by using a pre-trained graph neural network model to obtain prediction state information of each detected object; and performing state adjustment on a controllable object in the at least two detected objects based on the prediction state information. According to the embodiment of the invention, the method achieves the mapping of the graph structure to a specific object state task, analyzes the relation between different objects through the graph neural network, applies the relation between different objects to the state prediction of the object, improves the accuracy of the state prediction, and achieves the more accurate control of a controllable object.

Description

Technical Field

[0001] The present application relates to the field of intelligent device control technology, and in particular to a device control method, device and intelligent home system based on graph neural network. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence, intelligent device control has been applied to all aspects. For example, the smart home system realizes automatic control and personalized services of the home environment by connecting various household appliances, sensors and user devices. However, the existing intelligent device interconnection system still faces many challenges in dealing with diverse device interactions, dynamic user behaviors and complex environmental perception.

[0003] Traditional smart device interconnection systems usually use rule engines or simple machine learning models, which make it difficult to effectively capture the complex relationships between devices, sensors, and users. Especially in heterogeneous and dynamically changing environments, it is impossible to accurately predict the state of the device that needs to be adjusted next, and the accuracy and targeting of device control need to be improved. Summary of the invention

[0004] In view of this, in order to solve some or all of the above-mentioned technical problems, the embodiments of the present application provide a device control method, apparatus and smart home system based on a graph neural network.

[0005] In a first aspect, an embodiment of the present application provides a device control method based on a graph neural network, the method comprising: obtaining an object attribute information set and an object relationship information set corresponding to a target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents an attribute of a detected object, and each object relationship information in the object relationship information set represents a relationship between two detected objects; based on the object attribute information set and the object relationship information set, constructing a graph structure comprising nodes and edges; using a pre-trained graph neural network model, performing state prediction on the nodes and edges in the graph structure to obtain predicted state information of each detected object; and based on the predicted state information, adjusting the state of controllable objects among the at least two detected objects.

[0006] In a possible implementation, the construction of a graph structure including nodes and edges based on the object attribute information set and the object relationship information set includes: obtaining object attribute time information corresponding to each object attribute information in the object attribute information set, and object relationship time information corresponding to each object relationship information in the object relationship information set; fusing each object attribute information in the object attribute information set and the corresponding object attribute time information to obtain a fused attribute information set; fusing each object relationship information in the object relationship information set and the corresponding object relationship time information to obtain a fused relationship information set; taking each fused attribute information in the fused attribute information set as a node and each fused relationship information in the fused relationship information set as an edge to construct the graph structure.

[0007] In a possible implementation, the obtaining of object attribute time information corresponding to each object attribute information in the object attribute information set and object relationship time information corresponding to each object relationship information in the object relationship information set includes: obtaining an object attribute time series corresponding to each object attribute information in the object attribute information set and an object relationship time series corresponding to each object relationship information in the object relationship information set; and encoding the time series using a preset time coding function to obtain the object attribute time information and the object relationship time information.

[0008] In one possible implementation, the graph neural network model includes a feature extraction network and a prediction network; the pre-trained graph neural network model is used to perform state prediction on the nodes and edges in the graph structure to obtain predicted state information of each detected object, including: using the feature extraction network to update the node feature vector of each node in the graph structure to obtain an updated feature vector for each node; using the prediction network to perform state prediction on the updated feature vector to obtain predicted state information corresponding to the detected object represented by each node.

[0009] In one possible implementation, the feature extraction network includes at least two cascaded sub-networks; using the feature extraction network to update the node feature vector of each node in the graph structure to obtain an updated feature vector for each node, including: in each layer of the at least two layers of sub-networks, using a preset message passing function to perform message aggregation processing on the neighbor node feature vectors corresponding to each node in the graph structure and the edge feature vector of at least one edge connected to each node to obtain a message aggregation feature vector; using a preset node update function to perform node feature vector update processing on the message aggregation feature vector and the initial node feature vector input into the layer of sub-network to obtain a node feature vector output by the current sub-network; and determining the node feature vector output by the last layer of the at least two layers of sub-networks as the updated feature vector.

[0010] In one possible implementation, the object attribute information set includes at least one type of object attribute information: device attribute information, sensor attribute information, and user attribute information; the object relationship information set includes at least one type of object relationship information: control relationship information, perception relationship information, association relationship information, spatial relationship information, and time dependency relationship information.

[0011] In one possible embodiment, the graph neural network model is pre-trained according to the following steps: obtaining a set of sample object attribute information and a set of sample object relationship information collected for at least two sample detected objects, and the annotation status information corresponding to each sample detected object; constructing a sample graph structure based on the set of sample object attribute information and the set of sample object relationship information; inputting the sample graph structure into the initial graph neural network model to obtain the predicted status information of each sample detected object; based on a preset loss function, determining a loss value representing the error between the predicted status information and the annotation status information of each sample detected object; based on the loss value, adjusting the parameters of the initial graph neural network model; in response to the initial graph neural network model after adjusting the parameters meeting the training end conditions, determining the initial graph neural network model after adjusting the parameters as the trained graph neural network model.

[0012] In a second aspect, an embodiment of the present application provides a device control device based on a graph neural network, the device comprising: an acquisition module, used to acquire an object attribute information set and an object relationship information set corresponding to a target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents an attribute of a detected object, and each object relationship information in the object relationship information set represents a relationship between two detected objects; a construction module, used to construct a graph structure comprising nodes and edges based on the object attribute information set and the object relationship information set; a prediction module, used to use a pre-trained graph neural network model to perform state prediction on the nodes and edges in the graph structure to obtain predicted state information of each detected object; an adjustment module, used to adjust the state of controllable objects among the at least two detected objects based on the predicted state information.

[0013] In a third aspect, an embodiment of the present application provides a smart home system, which includes a control device and at least two controlled devices, wherein the control device is used to execute the method of any embodiment of the device control method based on graph neural network in the above-mentioned second aspect.

[0014] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program stored in the memory, and when the computer program is executed, it implements the method of any embodiment of the device control method based on a graph neural network of the above-mentioned first aspect of the present application.

[0015] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method of any embodiment of the device control method based on a graph neural network as described in the first aspect above is implemented.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program, which includes a computer-readable code. When the computer-readable code runs on a device, the processor in the device implements a method as in any embodiment of the device control method based on a graph neural network in the first aspect described above.

[0017] The device control method, apparatus and smart home system based on graph neural network provided in the embodiments of the present application construct a graph structure including nodes and edges by acquiring the corresponding object attribute information set and object relationship information set under the target device control system, and using a pre-trained graph neural network model to predict the state of the nodes and edges in the graph structure, obtain the predicted state information of each detected object, and finally adjust the state of the controllable objects among the at least two detected objects based on the predicted state information. The embodiments of the present application realize the construction of a graph structure, mapping the graph structure to a specific object state task, and using a graph neural network to analyze the relationship between different objects, and applying the relationship between different objects to the object state prediction, thereby improving the accuracy of the state prediction and controlling the controllable objects more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0021] Figure 1 A schematic diagram of a flow chart of a device control method based on a graph neural network provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a flow chart of another device control method based on a graph neural network provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a flow chart of another device control method based on a graph neural network provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a flow chart of another device control method based on a graph neural network provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of a graph neural network model provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of another graph neural network model provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of a flow chart of another device control method based on a graph neural network provided in an embodiment of the present application;

[0028] Figure 8 A flowchart of a method for training a graph neural network model provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of a device control device based on a graph neural network provided in an embodiment of the present application;

[0030] Fig.10 A schematic diagram of the structure of a smart home system provided in an embodiment of the present application;

[0031] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of the components and steps, the numerical expressions and the numerical values ​​described in these embodiments do not limit the scope of the present application.

[0033] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present application are only used to distinguish between different steps, devices, modules and other objects, and do not represent any specific technical meanings, nor do they indicate the logical order between them.

[0034] It should also be understood that in this embodiment, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present application can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0036] In addition, the term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be understood that the description of the various embodiments in this application focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0039] Technologies, circuits, and devices known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the above-mentioned technologies, circuits, and devices should be considered as part of the specification.

[0040] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. To facilitate the understanding of the embodiments of the present application, the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0042] In order to solve the technical problem that the existing technology cannot fully explore the relationship between different objects, resulting in low accuracy of equipment state prediction, the present application provides a device control method based on graph neural network, which uses graph neural network to analyze the relationship between different objects, and applies the relationship between different objects to object state prediction, thereby improving the accuracy of state prediction and more accurately controlling controllable objects.

[0043] Figure 1A flow chart of a device control method based on a graph neural network provided in an embodiment of the present application. This method can be applied to various types of intelligent device control systems, such as smart home systems, industrial equipment control systems, etc. This method can be performed by various types of electronic devices, such as central control devices, servers, smart phones, personal computers, etc. These electronic devices can be connected to various types of detected objects, execute this method for each detected object, and adjust the state of the controllable objects therein. In addition, the execution subject of this method can be hardware or software. When the above-mentioned execution subject is hardware, the execution subject can be one or more of the above-mentioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the above-mentioned execution subject is software, this method can be implemented as multiple software or software modules, or as a single software or software module. It is not specifically limited here.

[0044] like Figure 1 As shown, the method specifically includes:

[0045] Step 101: Obtain the corresponding object attribute information set and object relationship information set under the target device control system.

[0046] In some embodiments, the target device control system may be various types of systems to which the present method is applied. For example, the target device control system may be a smart home system, a smart industrial manufacturing system, etc. The target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents an attribute of a detected object, and each object relationship information in the object relationship information set represents a relationship between two detected objects.

[0047] The above-mentioned detected objects may include various types of objects, such as various devices, sensors, users, etc. The properties of the detected objects may be different according to the types of the detected objects. For example, the detected objects may be devices such as air conditioners and lamps, and the object attribute information may include information indicating the device type, device status, set value, power consumption, location, model, etc. If two detected objects have an association relationship, the corresponding object relationship information may be set. For example, the object relationship information may represent the following types of relationships: control relationship, indicating the user's operation on the device, such as "the user controls the air conditioner switch"; perception relationship, indicating the sensor's monitoring of the device or environment, such as "the temperature sensor senses the operating environment of the air conditioner"; association relationship, indicating the logical connection between devices or users, such as "the linkage setting of lights and air conditioners"; spatial relationship, indicating the connection of nodes in space, such as "the device and sensor are located in the same room"; time dependency relationship, indicating the time association between events or nodes, such as "the user controls multiple devices at different times".

[0048] Step 102: construct a graph structure including nodes and edges based on the object attribute information set and the object relationship information set.

[0049] In some embodiments, the above-mentioned object attribute information and object relationship information may respectively include data representing certain meanings, and the data included in each object data information constitutes a node feature vector, and the data included in each object relationship information constitutes an edge feature vector.

[0050] In the graph structure, each node represents a detected object and corresponds to a node feature vector. Each edge connects two nodes, represents the relationship between a pair of detected objects, and corresponds to an edge feature vector.

[0051] As an example, for a node, its node feature vector represents the type, state, set value, power consumption, location, model, etc. of the detected object corresponding to the node. For an edge connecting a user node and an air conditioning device node, its edge feature vector represents that the user and the air conditioning device are in a control relationship (type code is 1). It can also include timestamp, operation type (such as turn on), operation intensity, state change (such as from off to on) and other data. The above edge feature vectors can more comprehensively capture the complex interactions between different nodes in the smart home system.

[0052] Optionally, the graph structure in the embodiment of the present application can be a heterogeneous knowledge graph structure, that is, the type of the detected object can be of multiple types, and there are multiple different types of nodes and edges in the graph structure. The data complexity of the heterogeneous graph structure is high, which is suitable for modeling complex systems and can capture complex interactions between different types of entities and relationships.

[0053] Step 103, using the pre-trained graph neural network model, the nodes and edges in the graph structure are predicted to obtain the predicted state information of each detected object.

[0054] In some embodiments, the graph neural network model may include multiple layers of node weights and edge weights, and calculate the node features and edge features respectively to obtain updated node features and edge features. After multiple calculations, the predicted state information representing the state of each detected object in the future period of time can be obtained.

[0055] Step 104: adjusting the state of at least two controllable objects among the detected objects based on the predicted state information.

[0056] In some embodiments, the controllable object is a detected object that can be controlled among at least two detected objects, and the states of these devices can be adjusted. For example, in a smart home system, air conditioners, refrigerators, humidifiers, lights and other devices are all controlled devices.

[0057] As an example, in a smart home environment, the following entities are included: air conditioner (AC01), lighting (Light01), temperature sensor (Temp01), motion sensor (Motion01), user A (User01). The node set of the constructed graph structure is V = {AC01, Light01, Temp01, Motion01, User01}, and the edge set is E = {(User01, AC01), (User01, Light01), (Temp01, AC01), (Motion01, User01)}. In the graph neural network, the device nodes are mapped using the matrix W. D , the sensor node uses the mapping matrix W S , the user node uses the mapping matrix W U The edge type mapping includes: Control relationship usage parameter W RC Perceived relationship using parameter W RS The graph neural network uses the above node mapping matrix and edge mapping matrix to calculate the nodes and edges included in the input graph structure and output the predicted state information.

[0058] The prediction results of the graph neural network on the device status include: predicting that user A will leave the living room in the next time period, the system automatically adjusts the air conditioning temperature or turns off the lighting to save energy. Detecting the temperature rising trend, it is predicted that the air conditioner needs to increase the cooling power, and the settings are adjusted in advance to improve comfort.

[0059] The device control method based on graph neural network provided in the embodiment of the present application obtains the corresponding object attribute information set and object relationship information set under the target device control system, constructs a graph structure including nodes and edges, uses a pre-trained graph neural network model to predict the state of the nodes and edges in the graph structure, obtains the predicted state information of each detected object, and finally adjusts the state of at least two controllable objects in the detected objects based on the predicted state information. The embodiment of the present application realizes the construction of a graph structure, maps the graph structure to a specific object state task, and uses a graph neural network to analyze the relationship between different objects, and applies the relationship between different objects to the object state prediction, thereby improving the accuracy of the state prediction and controlling the controllable objects more accurately.

[0060] In some optional implementations, such as Figure 2 As shown, step 102 includes:

[0061] Step 1021: Obtain object attribute time information corresponding to each object attribute information in the object attribute information set, and object relationship time information corresponding to each object relationship information in the object relationship information set.

[0062] Among them, the object attribute time information indicates the time point when the detected object undergoes state transition, data recording and other operations. The above time can usually be recorded by a timestamp. For example, as time goes by, the state of each node (such as device state, sensor reading or user behavior) will change, and the time of these changes is marked by the timestamp of these state changes.

[0063] Object relationship time information indicates the time when the relationship or interaction between two detected objects occurs. In the graph structure, it is the time corresponding to each edge. Usually, the above time can also be recorded by timestamp. For example, the time when the user controls the device, the data collected by the sensor on the device operation, etc. Object relationship time information describes the timing information of the interaction between nodes. For example, in a smart home system, the timestamp corresponding to the edge helps to mark the time when the device control behavior or sensor perception behavior occurs at a specific time point.

[0064] Step 1022: fuse each object attribute information in the object attribute information set with the corresponding object attribute time information to obtain a fused attribute information set.

[0065] Step 1023: fuse each object relationship information in the object relationship information set with the corresponding object relationship time information to obtain a fused relationship information set.

[0066] Specifically, object attribute information and object relationship information can be represented by feature vectors, which are referred to as node feature vectors and edge feature vectors. Each data in the node feature vector and edge feature vector can correspond to a timestamp, and these timestamps can also form a time vector, or the time vector can be encoded using a time encoding function to obtain an encoded time vector. The node feature vector is connected to the corresponding time vector, and the edge feature vector is connected to the corresponding time vector, and the new feature vector obtained is used as the fused attribute information and fused relationship information.

[0067] For example, let the initial node feature vector be X v , the corresponding time vector is Then X v and The fusion attribute information formed by connection is represented as a vector Let the initial edge eigenvector be e μv , the corresponding time vector is Then e μv and The fusion attribute information formed by connection is represented as a vector Here, μ and ν represent two nodes, namely, two detected objects, and τ represents the value after the timestamp is encoded.

[0068] Step 1024, each fused attribute information in the fused attribute information set is used as a node, and each fused relationship information in the fused relationship information set is used as an edge to construct a graph structure.

[0069] This embodiment integrates the time information corresponding to the object attribute information and the object relationship information respectively, so that the graph structure can include time information, so that the graph neural network model can obtain the time series characteristics and state change rules from the graph structure, thereby further improving the accuracy of state prediction.

[0070] In some optional implementations, such as Figure 3 As shown, step 1021 includes:

[0071] Step 10211, obtaining the object attribute time series corresponding to each object attribute information in the object attribute information set, and the object relationship time series corresponding to each object relationship information in the object relationship information set.

[0072] The object attribute time series is a sequence of different time points for recording the attribute information of a detected object, and the object relationship time series is a sequence of different time points for recording the relationship information between a pair of detected objects. The above time series can all be composed of timestamps.

[0073] Step 10212, using a preset time encoding function, encode the time series to obtain object attribute time information and object relationship time information.

[0074] Optionally, the time coding function may be a sine-cosine position coding function. The coding function is shown in the following formula (1):

[0075]

[0076] Where i is the serial number of the data in the time series. i is the frequency parameter, and its calculation method is shown in the following formula (2):

[0077]

[0078] Where d is the dimension of the time series.

[0079] Using sine and cosine position encoding functions to encode time series can well capture the periodic changes of time. For many scenarios, time information is inherently periodic (such as 24 hours a day, 7 days a week). And the sine and cosine functions are continuous, which can well represent the sequential relationship between time points. Similar timestamps will correspond to similar encoding values, and this continuity helps the model capture the sequential characteristics of time. In addition, sine and cosine encoding allows time information to be fully passed to subsequent model layers through a combination of different frequencies without losing the key characteristics of time due to discretization or over-compression.

[0080] It should be noted that the above sine-cosine position encoding function is only an example, and the present embodiment does not limit the specific encoding method. For example, the time series can also be encoded using a learnable position encoding method, a relative position encoding method, etc. related to the art.

[0081] This embodiment can fully mine the period, frequency and other information hidden in the time series by setting a time encoding function to encode the time series, thereby helping to improve the accuracy of obtaining object attribute time information and object relationship time information.

[0082] In some optional implementations, the graph neural network model includes a feature extraction network and a prediction network.

[0083] like Figure 4 As shown, step 103 includes:

[0084] Step 1031, using the feature extraction network, updates the node feature vector of each node in the graph structure to obtain an updated feature vector of each node.

[0085] like Figure 5 As shown, it shows a schematic diagram of the structure of the graph neural network model. The initial node feature vector X1-X corresponding to each node in the graph structure N The initial edge feature vector e1-e corresponding to each edge M Input the feature extraction network, and the initial node feature vector and initial edge feature vector are updated multiple times through the feature extraction network, and finally the updated feature vector h′1-h′ of each node is output N .

[0086] Step 1032, using the prediction network, perform state prediction on the updated feature vector to obtain the predicted state information corresponding to the detected object represented by each node.

[0087] The prediction network may include a fully connected layer. The updated features corresponding to each node are calculated by the fully connected layer. The predicted state information obtained can be a vector. Each value in the vector represents the probability of a state category. The state category corresponding to the maximum probability can be taken to obtain the predicted state information (such as Figure 5 Y1-Y shown N ).

[0088] As an example, the calculation process of the prediction network is shown in the following formula (3):

[0089]

[0090] in, represents the predicted data, W0 and b0 are the parameters of the prediction network respectively.

[0091] This embodiment analyzes the nodes and edges in the graph structure by setting a feature extraction network and a prediction network in the graph neural network model. After updating the node feature vector, the updated feature can accurately represent the state of the node in the future, thereby improving the accuracy of state prediction.

[0092] In some optional implementations, such as Figure 6 As shown, the feature extraction network includes at least two layers of cascaded sub-networks (including sub-network 1-sub-network L).

[0093] like Figure 7 As shown, step 1031 includes:

[0094] Step 10311, in each layer of at least two layers of subnetworks, use a preset message passing function to perform message aggregation processing on the neighbor node feature vector corresponding to each node in the graph structure and the edge feature vector of at least one edge connected to each node to obtain a message aggregation feature vector.

[0095] Specifically, for a layer of sub-network, let its corresponding layer number be l, and its corresponding message transfer function is shown in the following formula (4):

[0096]

[0097] Among them, m μ→v represents the message transmitted from the neighbor node μ to the node v, σ is the activation function, φ(e) represents an edge, and Indicates the network parameters corresponding to this edge in the l-th layer sub-network, represents the neighbor node feature vector of the input sub-network, that is, the node feature vector of the node μ output by the previous sub-network, h μvrepresents the edge feature vector between node μ and node v, and ⊙ represents element-wise multiplication.

[0098] According to the following formula, all neighbor nodes of node v can be aggregated:

[0099]

[0100] in, is the set of all neighbor nodes of node ν, is the aggregated feature vector obtained by node ν in the l-th layer sub-network.

[0101] h μv The calculation formula is as follows:

[0102]

[0103] Among them, e μv To input the initial edge feature vector of the current layer sub-network, after the parameters of the current layer sub-network are calculated, the edge feature vector of the current layer can be obtained.

[0104] Optionally, if this embodiment is different from the above Figure 3 In combination with the corresponding embodiment, the object relationship time information can be fused into the edge feature vector according to the following formula:

[0105]

[0106] in, Represents the encoded object relationship time information, and ∥ represents connecting two vectors.

[0107] Step 10312, using a preset node update function, performs node feature vector update processing on the message aggregation feature vector and the initial node feature vector input to the sub-network of this layer, to obtain the node feature vector output by the current sub-network.

[0108] Among them, the node update function can be shown as follows:

[0109]

[0110] in, represents the node feature vector output by the l-th layer sub-network. φ(ν) represents node ν, and Represents the parameters in the l-th layer sub-network corresponding to the node ν. It is the node feature vector of node ν output by the previous layer of sub-network, which is the initial node feature vector input.

[0111] It should be noted that, for the first layer sub-network, the initial node feature vector of each node and the initial edge feature vector of each edge included in the received graph structure can be embedded, that is:

[0112]

[0113] Optionally, if this embodiment is different from the above Figure 3 In combination with the corresponding embodiments, the object attribute time series can be fused into the node feature vector according to the following formula:

[0114]

[0115] in, Represents the encoded object attribute time series, and ∥ represents connecting two vectors.

[0116] Step 10313: determine the node feature vector output by the last layer of the sub-network in the at least two layers of sub-networks as the updated feature vector.

[0117] like Figure 6 As shown, for node ν, the L-th layer subnetwork outputs the updated feature vector of the node

[0118] By setting the message passing function and the node update function, this embodiment can fully analyze the complex relationship between multiple detected objects in the feature extraction network, and update the node feature vector layer by layer. The output updated feature vector can more accurately represent the state of the detected object corresponding to each node, thereby further improving the accuracy of state prediction.

[0119] In some optional implementations, the object attribute information set includes at least one of the following types of object attribute information: device attribute information, sensor attribute information, and user attribute information. The object relationship information set includes at least one of the following types of object relationship information: control relationship information, perception relationship information, association relationship information, spatial relationship information, and time dependency relationship information.

[0120] Among them, device attribute information can represent various attributes of the device, such as type, state, control amount, etc. Sensor attribute information can represent attributes of the sensor, such as location, data type, etc. User attribute information can represent attributes of the user, such as location, movement speed, habitual action, etc.

[0121] The above control relationship information is used to describe the user's operation on the device, such as "the user controls the air conditioner switch". The above perception relationship information is used to describe the sensor's monitoring of the device or environment, such as "the temperature sensor senses the operating environment of the air conditioner". The above association relationship is used to describe the logical connection between devices or users, such as "the linkage setting of lights and air conditioners". The above spatial relationship information is used to describe the spatial connection of nodes, such as "the device and sensor are located in the same room". The above time dependency relationship information is used to describe the time association between events or nodes, such as "the user controls multiple devices at different times".

[0122] The various object attribute information and object relationship information provided in the embodiments of the present application enable processing of different types of detected objects and association relationships. Heterogeneous graph structures can be constructed using this information, which helps to improve more accurate state prediction and control of device control systems in complex scenarios.

[0123] In some optional implementations, such as Figure 8 As shown in the figure, the graph neural network model is pre-trained according to the following steps:

[0124] Step 801: Acquire a sample object attribute information set and a sample object relationship information set collected for at least two sample detected objects, as well as labeling status information corresponding to each sample detected object.

[0125] The labeled state information is the real state of each sample detected object. For example, the switch state of the device is labeled as 0 or 1; the set value of the device (such as temperature) is directly labeled with the real set value; for the user behavior prediction task, the user's behavior can be labeled, such as the room entered, the type of device used, etc.

[0126] Step 802: construct a sample graph structure based on the sample object attribute information set and the sample object relationship information set.

[0127] The method for constructing the sample graph structure may be the same as the above step 102, and will not be described in detail here.

[0128] Step 803: Input the sample graph structure into the initial graph neural network model to obtain the predicted state information of each sample detected object.

[0129] The initial graph neural network model can be an untrained graph neural network model with initial parameters, or a trained graph neural network model. The initial graph neural network model can analyze the node feature vectors and edge feature vectors included in the graph structure to obtain the predicted state information of each sample detected object.

[0130] Step 804 : Based on a preset loss function, determine a loss value representing the error between the predicted state information and the labeled state information of each sample detected object.

[0131] The loss function may include a loss function for classification (e.g., a cross entropy loss function) and a loss function for regression (e.g., a mean square error loss function). That is, for each node, if a classification task is performed on the node, the corresponding probability of the state of the node can be predicted, and the classification loss function is used to calculate the error between the probability and the corresponding labeled state information to obtain a loss value representing the error between the predicted result and the expected result. If a set value prediction task is performed on the node, the regression loss function can be used to calculate the error between the predicted set value and the standard set value to obtain a loss value representing the error between the predicted result and the expected result.

[0132] Step 805: Adjust the parameters of the initial graph neural network model based on the loss value.

[0133] Machine learning methods (such as the minimum gradient method and the back propagation method) can be used to adjust the parameters of the initial graph neural network model by minimizing the loss value as the optimization goal.

[0134] Step 806, in response to the initial graph neural network model after adjusting parameters meeting the training end condition, the initial graph neural network model after adjusting parameters is determined as the trained graph neural network model.

[0135] Among them, the training end conditions may include but are not limited to at least one of the following: loss value convergence, the number of training times reaches a preset number of times, and the training time reaches a preset time.

[0136] It should be understood that the steps provided in this embodiment are performed on a set of training samples. During actual training, multiple sets of training samples can be used to iteratively train the model, and the model after the last iteration can be used as the trained graph neural network model.

[0137] Optional, refer to the above Figure 3 The method described in the corresponding embodiment integrates time information into a sample object attribute information set and a sample object relationship information set, and the trained model can analyze time series data more accurately.

[0138] This embodiment uses machine learning methods to conduct training for model application scenarios, thereby obtaining a graph neural network model with higher prediction accuracy.

[0139] Fig. 9 A schematic diagram of the structure of a device control device based on a graph neural network provided in an embodiment of the present application. Specifically comprising:

[0140] The acquisition module 901 is used to acquire the object attribute information set and the object relationship information set corresponding to the target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents the attribute of one detected object, and each object relationship information in the object relationship information set represents the relationship between two detected objects;

[0141] A construction module 902, configured to construct a graph structure including nodes and edges based on an object attribute information set and an object relationship information set;

[0142] Prediction module 903, used to use the pre-trained graph neural network model to predict the state of nodes and edges in the graph structure to obtain predicted state information of each detected object;

[0143] The adjustment module 904 is used to adjust the state of the controllable objects among the at least two detected objects based on the predicted state information.

[0144] In some optional implementations, the construction module includes: an acquisition unit, used to acquire object attribute time information corresponding to each object attribute information in the object attribute information set, and object relationship time information corresponding to each object relationship information in the object relationship information set; a first fusion unit, used to fuse each object attribute information in the object attribute information set and the corresponding object attribute time information to obtain a fused attribute information set; a second fusion unit, used to fuse each object relationship information in the object relationship information set and the corresponding object relationship time information to obtain a fused relationship information set; a construction unit, used to construct a graph structure by taking each fused attribute information in the fused attribute information set as a node and each fused relationship information in the fused relationship information set as an edge.

[0145] In some optional implementations, the acquisition unit includes: an acquisition sub-unit, used to acquire the object attribute time series corresponding to each object attribute information in the object attribute information set, and the object relationship time series corresponding to each object relationship information in the object relationship information set; an encoding sub-unit, used to encode the time series using a preset time encoding function to obtain the object attribute time information and the object relationship time information.

[0146] In some optional implementations, the graph neural network model includes a feature extraction network and a prediction network; the prediction module includes: an update unit, which uses the feature extraction network to update the node feature vector of each node in the graph structure to obtain an updated feature vector of each node; a prediction unit, which uses the prediction network to perform state prediction on the updated feature vector to obtain predicted state information corresponding to the detected object represented by each node.

[0147] In some optional implementations, the feature extraction network includes at least two cascaded sub-networks; the update unit includes: a message passing sub-unit, which is used to use a preset message passing function in each layer of the at least two sub-networks to perform message aggregation processing on the neighbor node feature vectors corresponding to each node in the graph structure and the edge feature vector of at least one edge connected to each node to obtain a message aggregation feature vector; an update sub-unit, which is used to use a preset node update function to perform node feature vector update processing on the message aggregation feature vector and the initial node feature vector input into the layer of sub-network to obtain a node feature vector output by the current sub-network; and a determination sub-unit, which is used to determine the node feature vector output by the last layer of the at least two sub-networks as the updated feature vector.

[0148] In some optional implementations, the object attribute information set includes at least one of the following types of object attribute information: device attribute information, sensor attribute information, and user attribute information; the object relationship information set includes at least one of the following types of object relationship information: control relationship information, perception relationship information, association relationship information, spatial relationship information, and time dependency relationship information.

[0149] In some optional implementations, the graph neural network model is pre-trained according to the following steps: obtaining a set of sample object attribute information and a set of sample object relationship information collected for at least two sample detected objects, and the labeled status information corresponding to each sample detected object; constructing a sample graph structure based on the sample object attribute information set and the sample object relationship information set; inputting the sample graph structure into the initial graph neural network model to obtain the predicted status information of each sample detected object; based on a preset loss function, determining a loss value representing the error between the predicted status information and the labeled status information of each sample detected object; based on the loss value, adjusting the parameters of the initial graph neural network model; in response to the initial graph neural network model after adjusting the parameters meeting the training end conditions, determining the initial graph neural network model after adjusting the parameters as the trained graph neural network model.

[0150] The device control device based on graph neural network provided in this embodiment can be as follows Fig. 9 The graph neural network-based equipment control device shown in can execute all steps of the above graph neural network-based equipment control methods, and thus achieve the technical effects of the above graph neural network-based equipment control methods. Please refer to the above relevant description for details. For the sake of brevity, it will not be repeated here.

[0151] Fig.10A structural schematic diagram of a smart home system provided for an embodiment of the present application, the system includes: a control device 1001, at least two controlled devices 1002 (including control device 1-control device N), the control device 1001 is communicatively connected with the at least two controlled devices 1002, and the control device 1001 is used to execute the above-mentioned device control method based on graph neural network. When executing the above-mentioned method, the control device 1001 can use at least two controlled devices 1002 as detected objects, and can also use users as detected objects. The above-mentioned at least two controlled devices may include various types of devices, such as air conditioners, electric lights, televisions, etc., and may also include devices such as temperature sensors, humidity sensors, and motion sensors.

[0152] The smart home system provided in this embodiment, by applying the above-mentioned graph neural network-based device control method in the control device, can construct a graph structure for each controlled object, map the graph structure to a specific object state task, and use the graph neural network to analyze the relationship between different objects, and apply the relationship between different objects to object state prediction, thereby improving the accuracy of state prediction and more accurately controlling the controllable objects.

[0153] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, Fig.11 The electronic device 1100 shown includes: at least one processor 1101, a memory 1102, at least one network interface 1104 and other user interfaces 1103. The various components in the electronic device 1100 are coupled together via a bus system 1105. It is understood that the bus system 1105 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1105 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 1105 is not described in detail. Fig.11 Various buses are labeled as bus system 1105.

[0154] The user interface 1103 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0155] It can be understood that the memory 1102 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 1102 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0156] In some embodiments, the memory 1102 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 11021 and an application program 11022 .

[0157] Among them, the operating system 11021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 11022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present application can be included in the application 11022.

[0158] In this embodiment, by calling the program or instruction stored in the memory 1102, specifically, the program or instruction stored in the application 11022, the processor 1101 is used to execute the method steps provided by each method embodiment, for example, including:

[0159] Obtain the object attribute information set and the object relationship information set corresponding to the target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents the attribute of a detected object, and each object relationship information in the object relationship information set represents the relationship between two detected objects; based on the object attribute information set and the object relationship information set, construct a graph structure including nodes and edges; use a pre-trained graph neural network model to perform state prediction on the nodes and edges in the graph structure to obtain predicted state information of each detected object; based on the predicted state information, adjust the state of the controllable objects among the at least two detected objects.

[0160] The method disclosed in the above embodiment of the present application can be applied to the processor 1101, or implemented by the processor 1101. The processor 1101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1101. The above processor 1101 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1102, and the processor 1101 reads the information in the memory 1102 and completes the steps of the above method in combination with its hardware.

[0161] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the above-mentioned functions of the present application, or a combination thereof.

[0162] For software implementation, the technology described above in this article can be implemented by a unit that performs the functions described above in this article. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0163] The electronic device provided in this embodiment may be Fig.11 The electronic device shown in can execute all the steps of the above-mentioned device control methods based on graph neural networks, and thus achieve the technical effects of the above-mentioned device control methods based on graph neural networks. Please refer to the above relevant description for details. For the sake of brevity, it will not be repeated here.

[0164] The embodiment of the present application also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0165] When one or more programs in the storage medium can be executed by one or more processors, the device control method based on graph neural network executed on the electronic device side can be implemented.

[0166] The processor is used to execute the program stored in the memory to implement the following steps of the device control method based on the graph neural network performed on the electronic device side:

[0167] Obtain the object attribute information set and the object relationship information set corresponding to the target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents the attribute of a detected object, and each object relationship information in the object relationship information set represents the relationship between two detected objects; based on the object attribute information set and the object relationship information set, construct a graph structure including nodes and edges; use a pre-trained graph neural network model to perform state prediction on the nodes and edges in the graph structure to obtain predicted state information of each detected object; based on the predicted state information, adjust the state of the controllable objects among the at least two detected objects.

[0168] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different circuits to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0169] The steps of the circuits or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, software modules executed by a processor, or a combination of the two. The software modules may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0170] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments, and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The steps, processes, and operations described in the text are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0171] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A device control method based on graph neural network, characterized in that: The method comprises: Acquire an object attribute information set and an object relationship information set corresponding to a target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents an attribute of a detected object, and each object relationship information in the object relationship information set represents a relationship between two detected objects; Based on the object attribute information set and the object relationship information set, construct a graph structure including nodes and edges; Using a pre-trained graph neural network model, predicting the states of nodes and edges in the graph structure to obtain predicted state information of each detected object; Based on the predicted state information, a state of a controllable object among the at least two detected objects is adjusted.

2. The method according to claim 1, characterized in that The step of constructing a graph structure including nodes and edges based on the object attribute information set and the object relationship information set includes: Acquire object attribute time information corresponding to each object attribute information in the object attribute information set, and object relationship time information corresponding to each object relationship information in the object relationship information set; fusing each object attribute information in the object attribute information set and the corresponding object attribute time information to obtain a fused attribute information set; fusing each object relationship information in the object relationship information set and the corresponding object relationship time information to obtain a fused relationship information set; The graph structure is constructed by taking each fused attribute information in the fused attribute information set as a node and each fused relationship information in the fused relationship information set as an edge.

3. The method according to claim 2, characterized in that The acquiring of the object attribute time information corresponding to each object attribute information in the object attribute information set and the object relationship time information corresponding to each object relationship information in the object relationship information set includes: Acquire an object attribute time series corresponding to each object attribute information in the object attribute information set, and an object relationship time series corresponding to each object relationship information in the object relationship information set; The time series is encoded using a preset time encoding function to obtain the object attribute time information and the object relationship time information.

4. The method according to claim 1, characterized in that: The graph neural network model includes a feature extraction network and a prediction network; The pre-trained graph neural network model is used to predict the state of the nodes and edges in the graph structure to obtain the predicted state information of each detected object, including: Using the feature extraction network, updating the node feature vector of each node in the graph structure to obtain an updated feature vector of each node; The prediction network is used to perform state prediction on the updated feature vector to obtain predicted state information corresponding to the detected object represented by each node.

5. The method according to claim 4, characterized in that The feature extraction network includes at least two layers of cascaded sub-networks; The step of using the feature extraction network to update the node feature vector of each node in the graph structure to obtain an updated feature vector of each node includes: In each layer of the at least two layers of subnetworks, using a preset message passing function, message aggregation processing is performed on the neighbor node feature vector corresponding to each node in the graph structure and the edge feature vector of at least one edge connected to each node to obtain a message aggregation feature vector; Using a preset node update function, the message aggregation feature vector and the initial node feature vector input to the sub-network of this layer are subjected to node feature vector update processing to obtain the node feature vector output by the current sub-network; The node feature vector output by the last layer of the sub-network in the at least two layers of the sub-network is determined as the updated feature vector.

6. The method according to claim 1, characterized in that The object attribute information set includes at least one type of the following object attribute information: device attribute information, sensor attribute information, and user attribute information; The object relationship information set includes at least one type of the following object relationship information: control relationship information, perception relationship information, association relationship information, spatial relationship information, and time dependency relationship information.

7. The method according to any one of claims 1 to 6, characterized in that: The graph neural network model is pre-trained according to the following steps: Acquire a sample object attribute information set and a sample object relationship information set collected for at least two sample detected objects, as well as labeling status information corresponding to each sample detected object; Constructing a sample graph structure based on the sample object attribute information set and the sample object relationship information set; Inputting the sample graph structure into an initial graph neural network model to obtain predicted state information of each sample detected object; Based on a preset loss function, determining a loss value representing the error between the predicted state information and the labeled state information of each sample detected object; Based on the loss value, adjusting the parameters of the initial graph neural network model; In response to the initial graph neural network model after adjusting parameters meeting the training end condition, the initial graph neural network model after adjusting parameters is determined as the graph neural network model after training.

8. A device control device based on graph neural network, characterized in that: The device comprises: an acquisition module, used to acquire an object attribute information set and an object relationship information set corresponding to a target device control system, wherein the target device control system includes at least two detected objects, each object attribute information in the object attribute information set represents an attribute of a detected object, and each object relationship information in the object relationship information set represents a relationship between two detected objects; A construction module, used to construct a graph structure including nodes and edges based on the object attribute information set and the object relationship information set; A prediction module, used to use a pre-trained graph neural network model to predict the states of nodes and edges in the graph structure to obtain predicted state information of each detected object; An adjustment module is used to adjust the state of a controllable object among the at least two detected objects based on the predicted state information.

9. A smart home system, comprising: A control device and at least two controlled devices, wherein the control device is communicatively connected to the at least two controlled devices, and the control device is used to execute the device control method based on graph neural network according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to execute a computer program stored in the memory, and when the computer program is executed, the device control method based on the graph neural network described in any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the device control method based on graph neural network described in any one of claims 1 to 8 is implemented.

Images