Home control system, method, device and medium based on user behavior prediction

Abstract

The present application provides a home control system, method, device and medium based on user behavior prediction, wherein the human body detection module is used to obtain the video image frames of the target scene monitoring video at each moment, and perform human body detection on each video image frame to obtain the human body detection result; the human body tracking module is used to determine the actual motion trajectory of the human body in the monitoring video based on the human body detection result; the posture estimation module is used to determine the actual posture of the human body based on the human body detection result and the actual motion trajectory of the human body; the behavior prediction module is used to determine the predicted posture of the human body at the target moment based on the actual posture of the human body and the preset reference posture; the home control module is used to control each home appliance in the target scene at the target moment based on the predicted posture of the human body. The above system is used to reduce the condition restrictions and user workload for home control, while improving the flexibility and convenience of home control.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a home control system, method, device and medium based on user behavior prediction. Background Art

[0002] With the rapid development of the Internet of Things and Internet technology, the demand for smart home systems is becoming increasingly strong. Smart home systems combine many modern scientific technologies such as automation control systems, computer network systems, and network communication technologies to create a networked and intelligent home control system, making it more convenient to manage fragmented, disordered, and unorganized home appliances, greatly saving time, production, management, and control costs, and optimizing the user's home control experience.

[0003] However, in the process of using the existing smart home system, users are usually required to actively operate the controller in the home system or use the client to send operating instructions to each home appliance. Then each home appliance operates independently or triggers other related devices according to the operating instructions sent by the user to realize the control of smart home devices. However, it was found in the research that the above method requires users to actively initiate trigger operations to control home appliances. If the current controller or client is not with the user, or is damaged and cannot be used, then the home appliances cannot be controlled at this time. In other words, it is necessary to ensure that the remote control device is available to realize home control, which increases the conditions for home control and makes home control lack flexibility and convenience. In addition, since the above method requires users to actively initiate trigger operations, it will increase the user's workload and labor costs when performing home control. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a home control system, method, device and medium based on user behavior prediction to reduce the condition restrictions and user workload for home control, while improving the flexibility and convenience of home control.

[0005] In a first aspect, an embodiment of the present application provides a home control system based on user behavior prediction, the system comprising a human body detection module, a human body tracking module, a posture estimation module, a behavior prediction module and a home control module;

[0006] The human body detection module is used to obtain the video image frames of the target scene monitoring video at each moment, and perform human body detection on each video image frame to obtain the human body detection results of each video image frame;

[0007] The human body tracking module is used to determine the actual movement trajectory of the human body in the monitoring video based on the human body detection results of each video image frame;

[0008] The posture estimation module is used to determine the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual motion trajectory of the human body;

[0009] The behavior prediction module is used to determine the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture;

[0010] The home control module is used to control each home appliance in the target scene at the target time based on the predicted human body posture.

[0011] Optionally, determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and a preset reference posture includes:

[0012] Get multiple preset reference poses;

[0013] Calculating the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the previous moment of the target moment and the predicted posture feature vector of each preset reference posture;

[0014] Determine the preset reference posture with the highest similarity to the actual posture feature vector as the target reference posture;

[0015] The predicted posture of the human body at the target moment is determined according to the target reference posture and the probability transfer matrix.

[0016] Optionally, determining the predicted posture of a human body at the target moment according to the target reference posture and the probability transfer matrix includes:

[0017] Querying the probability transfer matrix, and determining the conditional probability distribution of the target reference posture from the probability transfer matrix;

[0018] Performing Monte Carlo sampling on the conditional probability distribution to obtain a mean of the sampling points;

[0019] The predicted posture of the human body at the target moment is determined according to the distance between the mean value of the sampling points and each preset reference posture.

[0020] Optionally, the probability transfer matrix is ​​used to describe the probability of jumping from the current state to the next state, and the behavior prediction module is further used before determining the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix:

[0021] Acquire a model training set, wherein the model training set includes a plurality of training video frames;

[0022] The probability transfer matrix is ​​constructed based on the change of the actual posture of the human body in every two consecutive training video frames.

[0023] Optionally, the controlling each household device in the target scene at the target time based on the predicted human body posture includes:

[0024] Determine the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability of the home devices in the target scene and the graph;

[0025] Determine the control strategy for each home appliance at the target time based on the device event probability of each home appliance;

[0026] Each home appliance is controlled based on the control strategy for each home appliance at the target moment.

[0027] Optionally, the determining of the control strategy for each household device at the target time based on the device event probability of each household device includes:

[0028] For each home device, determine whether the device event probability of each device event occurring in the home device exceeds a preset threshold;

[0029] For each device event, if the device event probability of the home device occurring the device event exceeds the preset threshold, controlling the home device to execute the device event is determined as the control strategy of the home device.

[0030] Optionally, the device event probability and OR graph is used to describe the probability of the relationship between the events of two things occurring one after the other. Before determining the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability and OR graph of the home devices in the target scene, the home control module is further used to:

[0031] Acquire a model training set, wherein the model training set includes a plurality of training video frames;

[0032] The device event probability and / or graph is constructed based on the actual posture of the human body in each training video frame and the number of occurrences of each device event.

[0033] In a second aspect, an embodiment of the present application provides a home control method based on user behavior prediction, the method comprising:

[0034] The human body detection module is used to obtain the video image frames of the target scene monitoring video at each moment, and human body detection is performed on each video image frame to obtain the human body detection results of each video image frame;

[0035] Determine the actual motion trajectory of the human body in the monitoring video based on the human body detection results of each video image frame through a human body tracking module;

[0036] Determine the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual motion trajectory of the human body through the posture estimation module;

[0037] The behavior prediction module determines the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture;

[0038] The home control module controls each home appliance in the target scene at the target time based on the predicted human body posture.

[0039] Optionally, determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and a preset reference posture includes:

[0040] Get multiple preset reference poses;

[0041] Calculating the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the previous moment of the target moment and the predicted posture feature vector of each preset reference posture;

[0042] Determine the preset reference posture with the highest similarity to the actual posture feature vector as the target reference posture;

[0043] The predicted posture of the human body at the target moment is determined according to the target reference posture and the probability transfer matrix.

[0044] Optionally, determining the predicted posture of a human body at the target moment according to the target reference posture and the probability transfer matrix includes:

[0045] Querying the probability transfer matrix, and determining the conditional probability distribution of the target reference posture from the probability transfer matrix;

[0046] Performing Monte Carlo sampling on the conditional probability distribution to obtain a mean of the sampling points;

[0047] The predicted posture of the human body at the target moment is determined according to the distance between the mean value of the sampling points and each preset reference posture.

[0048] Optionally, the probability transfer matrix is ​​used to describe the probability of jumping from the current state to the next state. Before determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix, the method further includes:

[0049] Acquire a model training set through the behavior prediction module, wherein the model training set includes a plurality of training video frames;

[0050] The probability transfer matrix is ​​constructed by the behavior prediction module based on the change of the actual posture of the human body in every two consecutive training video frames.

[0051] Optionally, the controlling each household device in the target scene at the target time based on the predicted human body posture includes:

[0052] Determine the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability of the home devices in the target scene and the graph;

[0053] Determine the control strategy for each home appliance at the target time based on the device event probability of each home appliance;

[0054] Each home appliance is controlled based on the control strategy for each home appliance at the target moment.

[0055] Optionally, the determining of the control strategy for each household device at the target time based on the device event probability of each household device includes:

[0056] For each home device, determine whether the device event probability of each device event occurring in the home device exceeds a preset threshold;

[0057] For each device event, if the device event probability of the home device occurring the device event exceeds the preset threshold, controlling the home device to execute the device event is determined as the control strategy of the home device.

[0058] Optionally, the device event probability and OR graph is used to describe the probability of the sequential occurrence relationship of events of two things. Before determining the device event probability of each device event occurring in each household device at the target moment according to the predicted human body posture and the device event probability and OR graph of the household devices in the target scene, the method further includes:

[0059] Acquire a model training set through the home control module, wherein the model training set includes a plurality of training video frames;

[0060] The device event probability and / or graph is constructed by the home control module based on the actual posture of the human body in each training video frame and the number of occurrences of each device event.

[0061] In a third aspect, an embodiment of the present application provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the home control method based on user behavior prediction described in any optional implementation manner of the second aspect are performed.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the home control method based on user behavior prediction described in any optional implementation manner of the second aspect above.

[0063] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0064] The present application realizes intelligent home control through the cooperation of the human body detection module, human body tracking module, posture estimation module, behavior prediction module and home control module in the home control system based on user behavior prediction. By acquiring the video image frames of the target scene monitoring video at each time, and performing human body detection on each video image frame to obtain the human body detection results of each video image frame, then determining the actual motion trajectory of the human body in the monitoring video based on the human body detection results of each video image frame, and then determining the actual posture of the human body in each video image frame based on the human body detection results of each video image frame and the actual motion trajectory of the human body, it is possible to automatically collect the user's behavior data information and provide data support for user posture prediction for subsequent automated intelligent home control. Then, based on the actual posture of the human body in each video image frame and the preset reference posture, the predicted posture of the human body at the target time is determined, and then based on the predicted posture of the human body, each home appliance in the target scene is controlled at the target time, which can achieve the situation where the user does not need to actively start the operation through the remote control device, and then determine the home control strategy according to the user's behavior and possible control requirements, and then perform home control based on the control strategy, which can reduce the condition restrictions and user workload for home control, and at the same time improve the flexibility and convenience of home control.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 A schematic diagram of the structure of a home control system based on user behavior prediction provided by the first embodiment of the present invention is shown;

[0068] Figure 2 A schematic diagram of the structure of a home environment time-space AND / OR diagram provided by the first embodiment of the present invention is shown;

[0069] Figure 3 A flowchart of a probabilistic grammar model training provided by the first embodiment of the present invention is shown;

[0070] Figure 4 A schematic diagram of a device event probability and / or graph provided by the first embodiment of the present invention is shown;

[0071] Figure 5 A schematic diagram showing key points of a human body model provided by the first embodiment of the present invention is shown;

[0072] Figure 6 A schematic diagram of a probability and / or diagram of device events in a living room scene provided by the first embodiment of the present invention is shown;

[0073] Figure 7 A flowchart of a home control method based on user behavior prediction provided by the second embodiment of the present invention is shown;

[0074] Figure 8 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention is shown. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0076] Embodiment 1

[0077] To facilitate understanding of this application, Figure 1 The flowchart of a method for detecting causes of abnormal service operation provided by the first embodiment of the present invention is shown to describe the contents of the first embodiment of the present application in detail.

[0078] See also Figure 1 As shown, Figure 1 A structural schematic diagram of a home control system based on user behavior prediction provided by Example 1 of the present invention is shown, wherein the system includes a human detection module 101, a human tracking module 102, a posture estimation module 103, a behavior prediction module 104 and a home control module 105.

[0079] Specifically, the human detection module, human tracking module and posture estimation module do not require data from the target scene for training. The behavior prediction module needs to first train a probabilistic grammar model based on the data from the target scene, while the home control module generates control suggestions based on the prior probability and causal relationship in the probabilistic grammar model.

[0080] The human body detection module is used to obtain video image frames of the target scene monitoring video at each moment, and perform human body detection on each video image frame to obtain the human body detection results of each video image frame.

[0081] Specifically, the human body detection module obtains the surveillance video of the target scene and extracts N consecutive video image frames at each moment in the surveillance video. Taking the N consecutive video image frames as input, a visual human body detector (such as MTCNN, SSD, YOLO) is used to obtain the position and size of all human bodies in each video image frame, and the human body detection results are recorded in the format of tracking frames. The human body position tracking frame in each video image frame is recorded as a group, and n groups of human body position tracking frames can be obtained.

[0082] The human body tracking module is used to determine the actual movement trajectory of the human body in the monitoring video based on the human body detection results of each video image frame.

[0083] Specifically, n sets of human position tracking frames are used as input and tracked using a visual tracker (such as DeepSort, Kalman filter, particle filter). Assuming that there are m people in the target scene, if the tracking is successful, m target IDs indicating different human bodies and the motion trajectory of each human body in N video image frames can be obtained.

[0084] The posture estimation module is used to determine the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual movement trajectory of the human body.

[0085] Specifically, for all N groups of human position tracking frames with target IDs, the image in the human position tracking frame and the motion trajectory of each human body in N video image frames are used as input, and a 3D posture estimator (such as OpenPose, AlphaPose, MetaPose) is used to obtain the actual posture of the human body in each human position tracking frame in each video image frame, including but not limited to features such as human skeleton point coordinates and rotation angles.

[0086] The behavior prediction module is used to determine the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture.

[0087] Specifically, the preset reference postures include multiple ones, and the mapping relationship between the preset reference postures and the predicted human body postures is pre-configured. According to the similarity between the posture feature vector of the actual human body posture in each video image frame and the posture feature vector of the preset reference posture, the reference posture closest to the actual human body posture in each video image frame is determined, and the reference posture is used as the mapping basis for determining the predicted human body posture, and the predicted human body posture having a mapping relationship with the reference posture is found as the predicted human body posture at the required target moment.

[0088] The home control module is used to control each home appliance in the target scene at the target time based on the predicted human body posture.

[0089] Specifically, the predicted human posture is obtained by predicting the target moment based on the human posture characteristics. Usually, when the user has different postures, he may perform different operations on home appliances. For example, when the user stands up, he may turn on the desk lamp, and the user needs to manually turn on the light. However, in this application, the home control module can control the desk lamp to turn on instead of the user, thereby improving the user's comfort and experience.

[0090] In an optional implementation, the step of determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and a preset reference posture includes:

[0091] Acquire multiple preset reference postures; calculate the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the moment before the target moment and the predicted posture feature vector of each preset reference posture; determine the preset reference posture with the highest similarity to the actual posture feature vector as the target reference posture; determine the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix.

[0092] Specifically, each video image frame is extracted at different moments, each moment corresponds to a time point. Assuming that the video image frame extracted at the current time point is the nth frame, the target moment, that is, the video image frame corresponding to the predicted time point is set as the n+Tth frame. Assuming that each actual human posture includes q human bone points, the posture of a person in the nth frame can be expressed as a q*3-dimensional posture vector w n . Calculate the posture vector w n The similarity with multiple reference postures is measured by comparing the posture vector w n The reference pose p with the highest similarity a For the attitude vector w n Update, that is, p a is with w n The closest human reference posture is recorded as the target reference posture. Then, the predicted human posture at the target moment is determined according to the target reference posture using the probability transfer matrix obtained by pre-statistics.

[0093] In an optional implementation, determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix includes:

[0094] The probability transfer matrix is ​​queried, and the conditional probability distribution of the target reference posture is determined from the probability transfer matrix; the conditional probability distribution is subjected to Monte Carlo sampling to obtain the mean of the sampling points; and the predicted posture of the human body at the target moment is determined according to the distance between the mean of the sampling points and each preset reference posture.

[0095] Specifically, the probability transfer matrix describes the probability of jumping from the current posture state to the next posture state. Assuming that the rows represent the posture at the current moment (assuming there are 10 possibilities) and the columns represent the posture at the next moment, the probability transfer matrix is ​​a 10x10 matrix, in which the element in the 2nd row and 3rd column represents the probability of jumping from the current 2nd posture state to the 3rd posture state at the next moment. Assuming that by querying the probability transfer matrix P(w n |w n-1 ) to obtain a reference pose p of the target a The conditional probability distribution P(p a ), w nis the posture vector of the actual posture of the human body in the nth video image frame (i.e., the video image frame at the nth moment), w n-1 is the posture vector of the actual posture of the human body in the n-1th frame of the video image. Monte Carlo sampling is performed on this probability distribution, assuming that p c is the category closest to the mean of the sampling points, then the most likely reference state for the next moment n+1 is p c By analogy, we can obtain the most likely state w at time n+2, n+3, ..., n+T in turn. n+1 、w n+2 ,…,w n+T =p d , and finally we will w n+T =p d As a result of behavior prediction, the predicted posture of the human body at the target moment.

[0096] In an optional implementation, the probability transfer matrix is ​​used to describe the probability of jumping from the current state to the next state, and the behavior prediction module is used to determine the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix before it is used to:

[0097] A model training set is obtained, wherein the model training set includes a plurality of training video frames; and the probability transfer matrix is ​​constructed based on changes in actual postures of human bodies in every two consecutive training video frames.

[0098] Specifically, the probability transfer matrix is ​​part of the probability grammar model, and the probability transfer matrix can be obtained when the probability grammar model is constructed and trained. The training process of the probability grammar model is as follows:

[0099] The Spatial-Temporal-Causal And-Or Graph (STC-AOG) is used as the probability grammar dictionary, where the Spatial-Temporal-Causal And-Or Graph (ST-AOG) is used to describe all possibilities of changes in human attributes and spatial relationships in indoor monitoring scenarios, and the Device Event Probability And-Or Graph (C-AOG) is built on the entity nodes of ST-AOG to describe the causal relationship between scene events.

[0100] ST-AOG is a directed acyclic graph. In the construction and training of ST-AOG, there are many ways to define ST-AOG of home environment, see Figure 2 As shown, Figure 2The schematic diagram of the structure of a spatiotemporal AND-OR graph of a home environment provided by the first embodiment of the present invention is shown, wherein each moment corresponds to a 3-layer AND-OR graph structure, and a total ST-AOG is formed by connecting different moments through the root node. "Monitoring scene t" represents the scene at the current moment and is an "AND node", "monitoring scene t+1" represents the scene at the next moment of the current moment and is an "AND node"; its child nodes "first area", "second area", etc. are all areas divided from the two-dimensional image, for example, the first area is the left sofa area in the target scene, the second area is the area near the TV in the target scene, the third area is the right sofa area in the target scene, and the fourth area is the remaining area in the target scene. The area node is an OR node, and the connection relationship with the child node is "OR". If the first area is connected to the four child nodes "person 01", "person 02", "object 001", and "object 002", it means that visually "person 01", "person 02", "object 001", and "object 002" are all in the first area. If the second region is connected to multiple child nodes such as "Person 02", "Person 03", "Object 001", etc., it means that visually "Person 02", "Person 03", "Object 001" and other connected nodes are all in the second region. The "Person" node and the "Object" node are leaf nodes. In the legend of the above ST-AOG, undirected dotted lines represent neighboring relationships, and directed dotted lines represent time sequences. Since the region is a concept of a two-dimensional image plane, and people move in three-dimensional space, a person may visually exist in multiple image regions at the same time, so a person's node will be connected to multiple regions.

[0101] By connecting the scene nodes at the current moment with the scene nodes at the previous moment, it is possible to construct an ST-AOG that can describe the spatiotemporal changes of the scene. Each node in the ST-AOG has rich attributes. The attributes related to this application include the area and shape of the region in the "region" node, and the human body position, 3D joint position, 3D joint rotation angle and other information in the "person" node. Assuming that each 3D posture includes q human bone points, the posture of a person in the nth frame can be represented as a q*3-dimensional posture vector. By clustering the posture vectors, we can obtain s reference postures (cluster centers) p1, p2,…, p k Assume that the posture vector at the nth moment is w n And w n With the target reference posture p a The closest, then record w n =p a , so the prior probability in the probability grammar dictionary can be defined as:

[0102] The "person" node is in a certain posture p in the nth frame a The probability P(wn =p a ) is defined as:

[0103] P(w n =p a ) = training set sample occurrence w n =p a The number of times / the total number of samples in the training set;

[0104] The posture transition probability of the "person" node, that is, the n-1th frame is p a The nth frame is p b The probability P(w n =p b |w n-1 =p a ) is defined as:

[0105] P(w n =p b |w n-1 =p a ) = The first frame that appears in the adjacent frames of the training set is p a And the second frame is p b The probability can be expressed as the number of times / the total number of adjacent frames in the training set. n |w n-1 ) to indicate that p b For a Different postures.

[0106] According to the prediction requirements of different scenarios, the above probability definitions can be increased or decreased (for example, the connection probability of the "person" node and the "region node", the state transition probability of the "person" node being a subordinate region, etc. can also be defined).

[0107] In general, see Figure 3 As shown, Figure 3 A flowchart of a probabilistic grammar model training provided in Embodiment 1 of the present invention is shown, wherein, when the defined probability distribution is learned and modeled by using training data to obtain a ST-AOG model including nodes, edges and probabilities, it is first necessary to perform human body detection based on N video image frames of normal business scenarios (home scenarios), perform human body tracking based on the human body detection results, and then perform 3D posture estimation based on the human body detection results and the human body tracking results; then, the probabilistic grammar model is trained based on the 3D posture estimation results to obtain a probabilistic grammar dictionary including a spatiotemporal AND-OR graph (ST-AOG) and a device event probability AND-OR graph (C-AOG).

[0108] In an optional implementation, the controlling each household device in the target scene at the target time based on the predicted human body posture includes:

[0109] According to the predicted human body posture and the device event probability of the home appliances in the target scene and the graph, the device event probability of each device event occurring in each home appliance at the target moment is determined; based on the device event probability of each home appliance, the control strategy for each home appliance at the target moment is determined; based on the control strategy for each home appliance at the target moment, each home appliance is controlled.

[0110] Specifically, C-AOG is a probability and / or graph used to describe the relationship between the occurrence of two types of node events, namely, the probability and / or graph of device events, which includes the posture events of the "person" node in ST-AOG and the device events of the "object" node corresponding to the smart home. Among them, the posture event can be "the human body is in a certain posture for T1 consecutive moments", "the human body belongs to certain areas for T2 consecutive moments", "the human body repeatedly makes two postures within T3 time", etc., recorded as E ij , where subscript i represents the i-th person, subscript j represents the j-th gesture event; device events include “a device changes from off to on”, “a device changes from on to off”, “a device turns on a certain mode”, etc., denoted by F kl , where the subscript k represents the kth object and the subscript l represents the lth device event.

[0111] In an optional implementation scheme, the control strategy for each household device at the target time is determined based on the device event probability of each household device, including:

[0112] For each home device, determine whether the device event probability of each device event occurring in the home device exceeds a preset threshold.

[0113] For each device event, if the device event probability of the home device occurring the device event exceeds the preset threshold, controlling the home device to execute the device event is determined as the control strategy of the home device.

[0114] Specifically, assuming there are m people in the target scene, the prediction result of each person is defined as E id :w n+T =p d , where w n+T is the pose vector of the person at the n+Tth frame, p d For the predicted human posture at the target moment, this module generates control suggestions by querying C-AOG. The specific steps are as follows:

[0115] First, see Figure 4 As shown, Figure 4A schematic diagram of a device event probability and OR graph provided by the first embodiment of the present invention is shown, wherein the device event probability and OR graph (C-AOG) is a probability and OR graph used to describe the relationship between the occurrence of two types of node events, including: the posture event of the "person" node in the ST-AOG and the device event of the "object" node corresponding to the smart home. Among them, the posture event can be "the human body is in a certain posture for T1 consecutive moments", "the human body belongs to certain areas for T2 consecutive moments", "the human body repeatedly makes two certain postures within T3 time", etc., denoted as E ij , where the subscript i represents the i-th person, and the subscript j represents the j-th gesture event, that is, when i=1, it represents the first person, when i=2, it represents the second person, when j=1, it represents the first gesture event, when j=2, it represents the second gesture event, when j=3, it represents the third gesture event, and the i-th person is recorded as person0i; device events include "a certain device changes from off to on", "a certain device changes from on to off", "a certain device turns on a certain mode", etc., recorded as F kl , where the subscript k represents the kth object, and the subscript l represents the lth device event, that is, when k = 1, it represents the first object, when i = 2, it represents the second object, when l = 1, it represents the first device event, when l = 2, it represents the second device event, and the kth object is recorded as object 00k. By counting the probability of the above events occurring in sequence in the training data set, we can obtain various conditional probabilities of posture events and device events, P(F kl |E ij ) represents the conditional probability that when the i-th person (denoted as person 0i) has the j-th gesture event, the k-th object (device) (denoted as object 00k) has the l-th device event. For example, P(F 11 |E 11 )=

[0116] In the training set E 11 First happens, then F 11 Number of occurrences / F in the training set 11 The number of occurrences. If a conditional probability value P(F 12 |E 21 ) is greater than the threshold, then it can be considered that event E 12 and event F 21 There is a causal relationship between “person 01” and “object 002”, so a directed edge is added in ST-AOG with “person 01” as the starting point and “object 002” as the end point as the edge of C-AOG.

[0117] Then, traverse Figure 4 The probability of the device event shown in the figure is summed with all the "person" nodes in the figure to obtain the E ij For the relevant causal chains, take j=1 as an example, for the "person 01" node, we get 2 causal chains C 11and C 12 ,in, For the "person 02" node, we get two causal chains C 21 and C 22 ,in,

[0118] Finally, the conditional probabilities of each device event are added together according to the causal chain to obtain the total probability; if a device event F kl If the total probability of is greater than the threshold, then add 1 F to the control suggestion kl , the device event F will be executed kl Determine the control strategy, that is, control the kth object to execute the lth device event. For example, for the device event F 11 , P(F 11 |E 11 ) and P(F 11 |E 21 ) add to get the device event F 11 The total probability of device event F 11 If the total probability of is greater than the threshold, then add device event F to the control suggestion. 11 , control the first object to execute the first device event.

[0119] In an optional implementation, the device event probability and OR graph is used to describe the probability of the relationship between the events of two things occurring one after the other. Before determining the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability and OR graph of the home devices in the target scene, the home control module is further used to:

[0120] A model training set is obtained, wherein the model training set includes multiple training video frames.

[0121] The device event probability and / or graph is constructed based on the actual posture of the human body in each training video frame and the number of occurrences of each device event.

[0122] Specifically, the model training set is the aforementioned training data set. By counting the probabilities of the above events occurring one after another, various conditional probabilities of gesture events and device events can be obtained, such as P(F 11 |E 11 )=

[0123] In the training set E 11 First happens, then F 11 Number of occurrences / F in the training set 11 The number of occurrences is constructed based on the conditional probabilities of various gesture events and device events. Figure 4The device event probability and / or diagram is shown. At the same time, based on the construction and training of ST-AOG and C-AOG mentioned above, we obtain the probabilistic grammar model defined by STC-AOG.

[0124] Here, the living room scene is taken as an example to illustrate the home control system provided by this application. First, the training data is the monitoring video data of people in the living room scene and the smart home log data. The data fragments from the time when people enter the living room to the time when they leave the living room (about 30 minutes) are mainly intercepted, and a total of 150 minutes of data for 5 days are taken as the training data set. For each video clip, the video is sampled at 5 frames per second to obtain nearly 45,000 training pictures, recorded as D1. From the smart home logs contained in the training data set, two types of events, "from on to off" and "from off to on", are extracted for each switchable home appliance, and 696 events about three electrical appliances, namely TV, chandelier, and table lamp, are obtained.

[0125] Then, the probabilistic grammar model is trained. Before starting the model training, the training data needs to be structured. The specific operations are as follows: the prepared training data D1 is passed through the target detection algorithm Yolox-1 to obtain the human target detection frame; the DeepSort algorithm is used to complete the human trajectory tracking and obtain the tracking frame; the human posture detection algorithm CLIFF is used to obtain the human 3D posture.

[0126] The 3D human body model used in this application contains 24 key points, see Figure 5 As shown, Figure 5 A schematic diagram of the key points of a human body model provided by Embodiment 1 of the present invention is shown, wherein the key points of the human body model include key points 0 to 23. Taking into account that the lower body of the human body may be estimated inaccurately due to occlusion, only 14 key points of the upper body are taken. Therefore, each person contains 14 3D posture data (rotation angle along the x-axis, rotation angle along the y-axis, rotation angle along the z-axis, in radians) in each frame. Flattening the key point angles of a person in N consecutive video image frames can obtain a vector of N*14*3, denoted as W=[w1,w2,…,w n ], where w c It is a 42*1-dimensional posture vector, c∈[1,N], and c is an integer.

[0127] The 45,000 posture vectors in the training set were clustered according to the Euclidean distance, and three categories were obtained, corresponding to the three postures of walking, standing, and sitting. During the model training phase, three probabilities were defined based on the probabilistic grammar model:

[0128] The first probability is that the "person" node is in a certain posture p in the nth frame a The probability P(w n =P a ): P(wn =p a ) = training set sample occurrence w n =p a The number of times / total number of samples in the training set; the posture vectors w of all human bodies in the video frames of the training set c Classify and count by category to obtain a prior probability distribution P about the value of the posture vector 01 (w c ), and use a Gaussian mixture probability model with a parameter of 5 to fit this high-dimensional probability distribution.

[0129] The second probability is the posture transfer probability of the "person" node, that is, the n-1 frame is p a , and the nth frame is p b The probability P(w n =p b |w n-1 =p a ): P(w n =p b |w n-1 =p a ) = The first frame that appears in the adjacent frames of the training set is p a And the second frame is p b The probability can be expressed as a 90×90 probability transfer matrix P 02 (w n |w n-1 ) to indicate;

[0130] The third probability is the connection probability between the “person” node and the “region node”: the positions of all human bodies in the training set video frames are counted and recorded as P 03 (A i ), that is, area A i The prior probability of a person appearing in area A i It includes 4 subdivided areas. The first area is the sofa area on the left side of the target scene. The second area is the area near the TV in the target scene. The third area is the sofa area on the right side of the target scene. The fourth area is the remaining area in the target scene. 01 (w c ), P 02 (w n |w n-1 ) and P 03 (A i ), and completed the model training of ST-AOG.

[0131] Then, by counting the probabilities of the above events occurring one after another in the training data set, we can obtain the conditional probabilities of various gesture events and device events. If a conditional probability value P(F12 |E 21 ) is greater than 0.05, then it can be considered that event E 12 and event F 21 There is a causal relationship between them, so in ST-AOG, a directed edge is added with "person 01" as the starting point and "object 002" as the end point as the edge of C-AOG. In this embodiment, there is only one "person" node and three "object" nodes, where the behavior of the "person" node has three possible postures, and each object has only two events, so its C-AOG can be represented by a 3×6 link matrix, see Figure 6 As shown, Figure 6 A schematic diagram of a device event probability and / or diagram in a living room scene provided by the first embodiment of the present invention is shown, wherein the values ​​of the matrix elements in the device event probability and / or diagram in the living room scene represent the probability of subsequently triggering the corresponding column event based on each row of posture. Specifically, the device event probability and / or diagram in the living room scene includes three types of human posture events E 11 Go, E 12 Station and E 13 Sitting; Two Equipment Events of TV F 11 TV on and F 12 TV off, two device events of desk lamp F 21 Desk lamp on and F 22 Table lamp off, two device events of chandelier F 31 Chandelier Open and F 32 The chandelier is off.

[0132] Assuming that the person is currently walking, query the probability transfer matrix shown in the following table:

[0133]

[0134] A discrete probability distribution can be obtained: {1 TV from off to on: 0.055; 2 TV from on to off: 0.116; 3 table lamp from off to on: 0.067; 4 table lamp from on to off: 0.067; 5 chandelier from off to on: 0.067; 6 chandelier from on to off: 0.221}. Normalizing it gives the normalized result: {1 TV from off to on: 0.093; 2 TV from on to off: 0.196; 3 table lamp from off to on: 0.113; 4 table lamp from on to off: 0.113; 5 chandelier from off to on: 0.113; 6 chandelier from on to off: 0.373}. The normalized probability value is regarded as a probability distribution, and Monte Carlo sampling is performed on it. The mean of the sampling result is 2.942, corresponding to "3 table lamps from off to on", so the control suggestion "turn on the table lamp" is given, and the table lamp is controlled to perform the turn-on operation.

[0135] Embodiment 2

[0136] See also Figure 7 As shown, Figure 7 A flowchart of a home control method based on user behavior prediction provided by the second embodiment of the present invention is shown, and the method includes steps S701 to S705;

[0137] S701: obtaining video image frames of the target scene surveillance video at each time through a human body detection module, and performing human body detection on each video image frame to obtain a human body detection result of each video image frame;

[0138] S702: Determine the actual movement trajectory of the human body in the monitoring video based on the human body detection results of each video image frame by a human body tracking module;

[0139] S703: determining the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual motion trajectory of the human body through a posture estimation module;

[0140] S704: Determine the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture through the behavior prediction module;

[0141] S705: Control each home appliance in the target scene at the target time based on the predicted human body posture through a home control module.

[0142] In an optional implementation, the step of determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and a preset reference posture includes:

[0143] Get multiple preset reference poses;

[0144] Calculating the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the previous moment of the target moment and the predicted posture feature vector of each preset reference posture;

[0145] Determine the preset reference posture with the highest similarity to the actual posture feature vector as the target reference posture;

[0146] The predicted posture of the human body at the target moment is determined according to the target reference posture and the probability transfer matrix.

[0147] In an optional implementation, determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix includes:

[0148] Querying the probability transfer matrix, and determining the conditional probability distribution of the target reference posture from the probability transfer matrix;

[0149] Performing Monte Carlo sampling on the conditional probability distribution to obtain a mean of the sampling points;

[0150] The predicted posture of the human body at the target moment is determined according to the distance between the mean value of the sampling points and each preset reference posture.

[0151] In an optional implementation, the probability transfer matrix is ​​used to describe the probability of jumping from the current state to the next state. Before determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix, the method further includes:

[0152] Acquire a model training set through the behavior prediction module, wherein the model training set includes a plurality of training video frames;

[0153] The probability transfer matrix is ​​constructed by the behavior prediction module based on the change of the actual posture of the human body in every two consecutive training video frames.

[0154] In an optional implementation, the controlling each household device in the target scene at the target time based on the predicted human body posture includes:

[0155] Determine the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability of the home devices in the target scene and the graph;

[0156] Determine the control strategy for each home appliance at the target time based on the device event probability of each home appliance;

[0157] Each home appliance is controlled based on the control strategy for each home appliance at the target moment.

[0158] In an optional implementation scheme, the control strategy for each household device at the target time is determined based on the device event probability of each household device, including:

[0159] For each home device, determine whether the device event probability of each device event occurring in the home device exceeds a preset threshold;

[0160] For each device event, if the device event probability of the home device occurring the device event exceeds the preset threshold, controlling the home device to execute the device event is determined as the control strategy of the home device.

[0161] In an optional implementation, the device event probability and OR graph is used to describe the probability of the sequential occurrence of events of two things. Before determining the device event probability of each device event occurring in each household device at the target moment according to the predicted human body posture and the device event probability and OR graph of the household devices in the target scene, the method further includes:

[0162] Acquire a model training set through the home control module, wherein the model training set includes a plurality of training video frames;

[0163] The device event probability and / or graph is constructed by the home control module based on the actual posture of the human body in each training video frame and the number of occurrences of each device event.

[0164] Embodiment 3

[0165] Based on the same application concept, see Figure 8 As shown, Figure 8 FIG. 4 shows a schematic diagram of the structure of a computer device provided by Embodiment 3 of the present invention, wherein: Figure 8 As shown, a computer device 800 provided in Embodiment 3 of the present application includes:

[0166] A processor 801, a memory 802 and a bus 803, wherein the memory 802 stores machine-readable instructions executable by the processor 801. When the computer device 800 is running, the processor 801 communicates with the memory 802 via the bus 803. When the processor 801 is running, the machine-readable instructions execute the steps of the home control method based on user behavior prediction shown in the above-mentioned embodiment 2.

[0167] Embodiment 4

[0168] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the home control method based on user behavior prediction described in the above-mentioned embodiment 2 are executed.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the method described above can refer to the corresponding process in the aforementioned system embodiment, and will not be repeated here.

[0170] The computer program product for performing home control based on user behavior prediction provided in an embodiment of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0171] The home control system based on user behavior prediction provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The system provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the system embodiment can refer to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by technicians in the relevant field that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0172] In the embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0173] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0175] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0176] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0177] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A home control system based on user behavior prediction, characterized in that: The system includes a human body detection module, a human body tracking module, a posture estimation module, a behavior prediction module and a home control module; The human body detection module is used to obtain the video image frames of the target scene monitoring video at each moment, and perform human body detection on each video image frame to obtain the human body detection results of each video image frame; The human body tracking module is used to determine the actual movement trajectory of the human body in the monitoring video based on the human body detection results of each video image frame; The posture estimation module is used to determine the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual motion trajectory of the human body; The behavior prediction module is used to determine the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture; The home control module is used to control each home appliance in the target scene at the target time based on the predicted human body posture; The method of determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture includes: Get multiple preset reference poses; Calculating the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the previous moment of the target moment and the predicted posture feature vector of each preset reference posture; Determine a preset reference posture having the highest similarity to the actual posture feature vector as a target reference posture; Determine the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix; Determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix includes: Querying the probability transfer matrix, and determining the conditional probability distribution of the target reference posture from the probability transfer matrix; Performing Monte Carlo sampling on the conditional probability distribution to obtain a mean of the sampling points; The predicted posture of the human body at the target moment is determined according to the distance between the mean value of the sampling points and each preset reference posture.

2. The system according to claim 1, characterized in that The probability transfer matrix is ​​used to describe the probability of jumping from the current state to the next state. Before the behavior prediction module is used to determine the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix, it is also used to: Acquire a model training set, wherein the model training set includes a plurality of training video frames; The probability transfer matrix is ​​constructed based on the change of the actual posture of the human body in every two consecutive training video frames.

3. The system according to claim 1, characterized in that The controlling each household device in the target scene at the target time based on the predicted human body posture includes: Determine the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability of the home devices in the target scene and the graph; Determine the control strategy for each home appliance at the target time based on the device event probability of each home appliance; Each home appliance is controlled based on the control strategy for each home appliance at the target moment.

4. The system according to claim 3, characterized in that The control strategy for each household device at the target time is determined based on the device event probability of each household device, including: For each home device, determine whether the device event probability of each device event occurring in the home device exceeds a preset threshold; For each device event, if the device event probability of the home device occurring the device event exceeds the preset threshold, controlling the home device to execute the device event is determined as the control strategy of the home device.

5. The system according to claim 3, characterized in that The device event probability and OR graph is used to describe the probability of the relationship between the events of two things occurring one after the other. Before the home control module is used to determine the device event probability of each device event occurring in each home device at the target moment according to the predicted human body posture and the device event probability and OR graph of the home devices in the target scene, it is also used to: Acquire a model training set, wherein the model training set includes a plurality of training video frames; The device event probability and / or graph is constructed based on the actual posture of the human body in each training video frame and the number of occurrences of each device event.

6. A home control method based on user behavior prediction, characterized in that: The method comprises: The human body detection module is used to obtain the video image frames of the target scene monitoring video at each moment, and human body detection is performed on each video image frame to obtain the human body detection results of each video image frame; Determine the actual motion trajectory of the human body in the monitoring video based on the human body detection results of each video image frame through a human body tracking module; Determine the actual posture of the human body in each video image frame based on the human body detection result of each video image frame and the actual motion trajectory of the human body through the posture estimation module; The behavior prediction module determines the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture; Controlling each home appliance in the target scene at the target time through a home control module based on the predicted human body posture; The method of determining the predicted posture of the human body at the target moment based on the actual posture of the human body in each video image frame and the preset reference posture includes: Get multiple preset reference poses; Calculating the similarity between the actual posture feature vector of the actual posture of the human body in the video image frame at the previous moment of the target moment and the predicted posture feature vector of each preset reference posture; Determine a preset reference posture having the highest similarity to the actual posture feature vector as a target reference posture; Determine the predicted posture of the human body at the target moment according to the target reference posture and the probability transfer matrix; Determining the predicted human body posture at the target moment according to the target reference posture and the probability transfer matrix includes: Querying the probability transfer matrix, and determining the conditional probability distribution of the target reference posture from the probability transfer matrix; Performing Monte Carlo sampling on the conditional probability distribution to obtain a mean of the sampling points; The predicted posture of the human body at the target moment is determined according to the distance between the mean value of the sampling points and each preset reference posture.

7. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the home control method based on user behavior prediction as described in claim 6 are performed.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the home control method based on user behavior prediction as claimed in claim 6.

Images