A smart home system

Abstract

The present invention discloses a smart home system, including sensors, a control center, home appliances, and a mobile robot; the mobile robot is used for mapping the indoor environment and obtaining an environmental grid map; the mobile robot performs self-positioning according to the environmental grid map, and sends its own position information to the control center; the control center is used for controlling the mobile robot, and performs positioning and identification of the home appliances according to the position information of the mobile robot and a home appliance information table; the sensor is used for collecting user data, and sends the user data to the control center; the control center is also used for performing directional action and control action detection according to the user data, and based on the positioning and identification results of the home appliances, determines the target home appliances according to the directional action detection results, and controls the target home appliances according to the control action detection results; thereby improving the intelligence level of controlling the smart home appliances, being able to control the home appliances more accurately, and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of smart home technology, and in particular to a smart home system. Background Art

[0002] Smart home system is a system that uses advanced computer technology, network communication technology, integrated wiring technology and other related technologies, and combines individual needs to organically combine various parts related to home life and control and manage through the network. In actual application, users can control the relevant equipment in the smart home system through terminal devices.

[0003] However, current smart home systems are often not smart enough. For example, users can view all home appliances in the room through a mobile phone APP, but the location of each home appliance needs to be manually set by the user; when performing specific control of home appliances, users are also required to manually set the control logic in advance and use human body sensors to monitor the surrounding home appliances. When the human body sensor detects a human body, certain home appliances are turned on or off.

[0004] It can be seen that the smart home system using this configuration and interactive control method only mechanically obtains user configuration and mechanically executes the control logic preset by the user, and the control logic is generally simple. When there are many controlled devices and the control logic is complex, the configuration process will be very cumbersome, resulting in a low degree of intelligence and inability to accurately control each home device, resulting in a poor user experience. Summary of the invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a smart home system that can realize complex interactive control functions and simplify the configuration process, thereby improving the intelligence level of controlling smart home devices, and can more accurately control home devices and improve user experience.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a smart home system, the system includes a sensor, a control center and home appliances, and the system also includes a mobile robot;

[0007] The mobile robot is used to map the indoor environment and obtain an environmental grid map; it is also used to perform self-positioning according to the environmental grid map and send its own position information to the control center;

[0008] The control center is used to control the mobile robot and locate and identify the household equipment according to the position information of the mobile robot and a preset household equipment information table;

[0009] The sensor is used to collect user data and send the user data to the control center;

[0010] The control center is also used to detect directional actions and control actions based on the user data, and based on the positioning and identification results of the home appliances, determine the target home appliances according to the directional action detection results, and control the target home appliances according to the control action detection results.

[0011] Furthermore, the control center locates and identifies the home appliance, specifically including:

[0012] Controlling the mobile robot to move to a first position, capturing a first image at the first position through a camera on the mobile robot, and obtaining first position information of the mobile robot at the first position;

[0013] Controlling the mobile robot to move to a second position, capturing a second image at the second position through a camera on the mobile robot, and obtaining second position information of the mobile robot at the second position;

[0014] Acquire a first device type and a first target area corresponding to a first household device in the first image, and acquire a second device type and a second target area corresponding to a second household device in the second image;

[0015] When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points;

[0016] Acquire the three-dimensional space coordinate value of the matching feature point according to the first position information and the second position information;

[0017] Query the home device information table according to the first device type; wherein the home device information table includes a number of home devices and their corresponding device information, and the device information includes at least the device type;

[0018] When there is only one household device corresponding to the same device type as the first device type in the household device information table, the device information of the first household device is determined according to the device information corresponding to the household device, and the three-dimensional spatial coordinate value of the first household device is determined according to the three-dimensional spatial coordinate value of the matching feature point.

[0019] Furthermore, the mobile robot is also used to send the environmental grid map to the control center;

[0020] Then, the control center locates and identifies the home appliance, and further includes:

[0021] When there are M home appliances corresponding to the same device type as the first device type in the home appliance information table, obtain the three-dimensional space coordinate values ​​corresponding to the M home appliances, and obtain the working status of the first home appliance; wherein M>1;

[0022] When the first household device is running, obtaining the three-dimensional space coordinate value of the user;

[0023] Finding a home appliance in the same room as the user among the M home appliances according to the environment grid map, the three-dimensional spatial coordinate values ​​corresponding to the M home appliances, and the three-dimensional spatial coordinate values ​​of the user;

[0024] The device information of the first household device is determined according to the device information corresponding to the found household device, and the three-dimensional space coordinate value of the first household device is determined according to the three-dimensional space coordinate value of the matching feature point.

[0025] Further, the system comprises at least two of the sensors;

[0026] Then, the control center obtains the three-dimensional space coordinate value of the user, specifically including:

[0027] Based on the user, construct a probability grid map corresponding to the indoor environment;

[0028] Initialize and mark the probability grid map according to the occupancy grid map and the monitoring grid map corresponding to the indoor environment;

[0029] When any of the sensors detects the user, obtaining a first target grid where the user is located in the monitoring grid map;

[0030] The marking value of the grid corresponding to the first target grid in the marked probability grid map is updated to 1, and the marking values ​​of all grids in the marked probability grid map except the grid updated to 1 are updated to 0;

[0031] Acquire the three-dimensional space coordinate value of the user according to the updated probability grid map;

[0032] Among them, the occupancy grid map is obtained by marking the environmental grid map, and the marking value of the grid with obstacles in the occupancy grid map is 1, and the marking value of the grid without obstacles is 0; the monitoring grid map is obtained by marking the environmental grid map, and the marking value of the monitored grid in the monitoring grid map is 1, and the marking value of the unmonitored grid is 0; the marking value of each grid in the marked probability grid map corresponds to the probability value indicating the existence of the user.

[0033] Furthermore, the control center obtains the three-dimensional space coordinate value of the user, further comprising:

[0034] When none of the sensors detects the user, obtaining, according to the three-dimensional spatial coordinate value of the user at the last moment, a second target grid where the user is located in the probability grid map updated at the last moment and a mark value corresponding to the second target grid;

[0035] According to a preset moving speed, the second target grid and the mark value corresponding to the second target grid, the probability grid map updated at the last moment is updated again;

[0036] The three-dimensional space coordinate value of the user is obtained according to the probability grid map that has been updated again.

[0037] Further, the control center updates the probability grid map updated at the last moment again according to the preset moving speed, the second target grid and the mark value corresponding to the second target grid, specifically including:

[0038] According to the movement rate and the second target grid, and in combination with the occupied grid map and the monitored grid map, a feasible domain corresponding to the user in the probability grid map updated at the last moment is obtained; wherein the feasible domain includes S grids, and the feasible domain represents the location area where the user may exist at the current moment, S>0;

[0039] The marking value of each of the S grids is updated to T / S, and the marking values ​​of all grids in the probability grid map updated at the last moment except the grid updated to T / S are updated to 0; wherein T represents the marking value corresponding to the second target grid, 0<T≤1.

[0040] Furthermore, the control center performs directional motion and control motion detection according to the user data, and based on the positioning and recognition results of the home appliances, determines the target home appliance according to the directional motion detection results, and controls the target home appliance according to the control motion detection results, specifically including:

[0041] Performing human skeleton key point detection on the user data to obtain three-dimensional image coordinate values ​​of the skeleton key points;

[0042] Performing coordinate transformation on the three-dimensional image coordinate values ​​of the skeleton key points according to the posture parameters of the sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points;

[0043] Performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance;

[0044] Control-type action detection is performed according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the control-type action detection result.

[0045] Furthermore, the control center performs directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determines the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance, specifically including:

[0046] Based on the positioning and identification results of the household appliances, find Q household appliances in the same room as the user according to the environment grid map and the three-dimensional spatial coordinate values ​​of the skeleton key points; wherein Q>0;

[0047] Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point;

[0048] According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of the Q home appliances, respectively obtain the angle between the position of each of the Q home appliances and the position of the user;

[0049] The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

[0050] Furthermore, the control center is specifically used to obtain the posture parameters of the sensor in the world coordinate system through the following steps:

[0051] When the mobile robot does not exist in the monitoring field of the sensor, acquiring a background image through the sensor;

[0052] Control the mobile robot to move into the monitoring field of view of the sensor, obtain N motion images corresponding to N different positions of the mobile robot through the sensor, and obtain N position information corresponding to the N different positions of the mobile robot; wherein N>1;

[0053] Calculate the frame difference between the background image and each motion image, and obtain N frame difference images accordingly;

[0054] Each frame difference image is processed according to a preset depth threshold, and N mask images are obtained accordingly;

[0055] The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0056] The pose parameters of the sensor in the world coordinate system are calculated based on the N position information, the N cluster centers and the internal parameter matrix of the sensor.

[0057] Furthermore, the control center calculates and obtains the posture parameters of the sensor in the world coordinate system according to the N position information, the N cluster centers and the internal parameter matrix of the sensor, specifically including:

[0058] According to the formula Solve and obtain the corresponding position parameter H of the sensor in the world coordinate system d ; Among them, P i represents the i-th position information corresponding to the i-th position of the mobile robot, Indicates the location information of the i-th cluster center converted to the world coordinate system, K s represents the intrinsic parameter matrix of the sensor.

[0059] Compared with the prior art, the embodiment of the present invention provides a smart home system, which is composed of a sensor, a control center, home appliances and a mobile robot; the mobile robot is used to map the indoor environment and obtain an environmental grid map; it is also used to perform self-positioning according to the environmental grid map and send its own location information to the control center; the control center is used to control the mobile robot and locate and identify the home appliances according to the location information of the mobile robot and a preset home appliance information table; the sensor is used to collect user data and send the user data to the control center; the control center is also used to perform directional action and control action detection according to the user data, and based on the positioning and identification results of the home appliances, determine the target home appliances according to the directional action detection results, and control the target home appliances according to the control action detection results; the smart home system can realize complex interactive control functions and simplify the configuration process, thereby improving the intelligence level of controlling smart home appliances, and can more accurately control home appliances and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural block diagram of a preferred embodiment of a smart home system provided by the present invention;

[0061] Figure 2 It is a schematic diagram of calculating the three-dimensional space coordinate values ​​of matching feature points when the control center locates and identifies the household equipment provided by the embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram of updating the probability grid map when the control center obtains the three-dimensional space coordinate value of the user provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0064] The embodiment of the present invention provides a smart home system. Figure 1 , which is a structural block diagram of a preferred embodiment of a smart home system provided by the present invention, wherein the system includes a sensor, a control center and home appliances, and the system also includes a mobile robot;

[0065] The mobile robot is used to map the indoor environment and obtain an environmental grid map; it is also used to perform self-positioning according to the environmental grid map and send its own position information to the control center;

[0066] The control center is used to control the mobile robot and locate and identify the household equipment according to the position information of the mobile robot and a preset household equipment information table;

[0067] The sensor is used to collect user data and send the user data to the control center;

[0068] The control center is also used to detect directional actions and control actions based on the user data, and based on the positioning and identification results of the home appliances, determine the target home appliances according to the directional action detection results, and control the target home appliances according to the control action detection results.

[0069] The smart home system in the embodiment of the present invention is mainly composed of sensors, a control center, home appliances and mobile robots. The sensors, home appliances and mobile robots communicate with the control center respectively. In the specific implementation:

[0070] The mobile robot can use the SLAM algorithm (Simultaneous Localization and Mapping) to map the indoor environment, obtain the corresponding environmental grid map, and determine the world coordinate system. Furthermore, the mobile robot can achieve self-positioning based on the constructed environmental grid map and SLAM positioning function, that is, determine its own position information in the world coordinate system (such as the posture parameters of the mobile robot relative to the world coordinate system), and send its own position information to the control center via wireless communication;

[0071] The control center can control the mobile robot to move indoors. When the mobile robot moves to different locations, it can perform self-positioning and feed back its own location information to the control center. After receiving the location information sent by the mobile robot, the control center can locate and identify the indoor home appliances by combining the location information of the mobile robot and the pre-set home appliance information table, and accordingly obtain the location information of each indoor home appliance and its corresponding device information;

[0072] The sensor can collect user data of indoor users and send the collected user data to the control center. It can be understood that the main function of the sensor is to collect user data to obtain user information. In order to better obtain user information, the user information can also be collected through a mobile robot. For example, the mobile robot is set to follow mode to follow the user at all times to collect user information.

[0073] After receiving the user data sent by the sensor, the control center performs directional motion detection and control motion detection on the user's actions according to the received user data, and based on the positioning and identification results obtained after positioning and identifying the home appliances, the target home appliances among all the home appliances in the room can be determined according to the directional motion detection results, and the determined target home appliances can be controlled accordingly according to the control motion detection results.

[0074] It should be noted that, for home appliances, a home appliance information table is pre-set in the control center, and the home appliance information table records several home appliances in the room and the device information corresponding to each home appliance, and the device information at least includes the device type (for example, air conditioner, TV, etc.) and device function, etc.; wherein, the control center can discover and collect the device information of all home appliances in the room, for example, after the home appliance is turned on, it automatically searches for the control center according to the network protocol, and reports its own device information to the control center, establishes a communication connection with the control center, and the control center obtains the home appliance information table according to the device information reported by each home appliance collected; there is a one-to-one correspondence between the home appliances and the device information in the home appliance information table; after the control center completes the positioning and identification of a certain home appliance, the positioning and identification results of the home appliance obtained can be added to the home appliance information table, and accordingly, there is a one-to-one correspondence between the home appliances, device information, and location information in the home appliance information table.

[0075] For sensors, the control center can also discover and collect all sensor information in the room. The sensor information includes at least the sensor type. For example, the sensor can be a camera, a depth sensor, a lidar, etc.; similar to home appliances, after the sensor is turned on, it can also automatically find the control center according to the network protocol, and report its own sensor information to the control center, and establish a communication connection with the control center. Therefore, the control center can know the existence of each sensor based on the collected sensor information.

[0076] For the control of home appliances, a complete user action can be divided into two types of actions, namely, directional actions and control actions. The purpose of detecting directional actions is to determine the direction of the controlled home appliance, so as to determine the controlled target home appliance according to the direction of the controlled home appliance. The purpose of detecting control actions is to determine the control actions to be performed on the target home appliance, so as to control the target home appliance accordingly according to the control actions. Control actions are generally special action control gestures. For example, setting a right fist and a "five" gesture respectively represents the two control actions of on and off. A complete user action must include both directional actions and control actions to ensure accurate control of home appliances. Among them, directional actions and control actions can be issued successively or simultaneously. For example, the user can issue a directional action first and then a control action, or issue a control action first and then a directional action, or issue a directional action and a control action simultaneously.

[0077] Understandably, Figure 1 The smart home system shown in the figure only includes one sensor, one control center, one home appliance and one mobile robot. In actual applications, it may include multiple sensors, multiple home appliances, etc. Figure 1 The structural block diagram is merely an example of the composition of the above-mentioned smart home system and does not constitute a limitation on the smart home system. The smart home system may include more or fewer components than those shown in the figure, and the embodiments of the present invention do not make specific limitations.

[0078] A smart home system provided by an embodiment of the present invention performs indoor mapping and self-positioning through a mobile robot, so that a control center can locate and identify home appliances according to the position information of the mobile robot and a home appliance information table, collects user data through sensors, and performs directional action and control action detection according to the user data based on the positioning and identification results of the home appliances through the control center, determines the target home appliance according to the directional action detection result, and controls the target home appliance according to the control action detection result. It can realize complex interactive control functions and simplify the configuration process, thereby improving the intelligence level of controlling smart home appliances, and can more accurately control home appliances and improve user experience.

[0079] In another preferred embodiment, the control center locates and identifies the household equipment, specifically including:

[0080] Controlling the mobile robot to move to a first position, capturing a first image at the first position through a camera on the mobile robot, and obtaining first position information of the mobile robot at the first position;

[0081] Controlling the mobile robot to move to a second position, capturing a second image at the second position through a camera on the mobile robot, and obtaining second position information of the mobile robot at the second position;

[0082] Acquire a first device type and a first target area corresponding to a first household device in the first image, and acquire a second device type and a second target area corresponding to a second household device in the second image;

[0083] When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points;

[0084] Acquire the three-dimensional space coordinate value of the matching feature point according to the first position information and the second position information;

[0085] Query the home device information table according to the first device type; wherein the home device information table includes a number of home devices and their corresponding device information, and the device information at least includes the device type;

[0086] When there is only one household device corresponding to the same device type as the first device type in the household device information table, the device information of the first household device is determined according to the device information corresponding to the household device, and the three-dimensional spatial coordinate value of the first household device is determined according to the three-dimensional spatial coordinate value of the matching feature point.

[0087] Specifically, in combination with the above embodiments, first, the control center controls the mobile robot to move to a first position in the room, and uses a camera on the mobile robot to capture a first image containing household appliances at the first position. The mobile robot performs self-positioning at the first position, and correspondingly obtains first position information at the first position, and feeds it back to the control center. The control center obtains the first position information of the mobile robot at the first position, and the first position information specifically includes a first pose parameter S1={R1|t1} of the mobile robot in the world coordinate system and its corresponding three-dimensional space coordinate value; similarly, the control center controls the mobile robot to move to a second position in the room, and correspondingly obtains a second image containing household appliances captured at the second position and second position information of the mobile robot at the second position, and the second position information specifically includes a second pose parameter S2={R2|t2} of the mobile robot in the world coordinate system and its corresponding three-dimensional space coordinate value; wherein the first pose parameter S1 and the second pose parameter S2 are both values ​​in the world coordinate system, and the pose parameter S specifically represents a space transformation matrix or a pose transformation matrix, including both a rotation component R and a translation component t (i.e., world coordinates);

[0088] Secondly, the control center uses the image recognition method provided by the prior art to perform image recognition on the first image, and accordingly recognizes the first device type corresponding to the first household device in the first image, and uses the moving target detection method provided by the prior art to perform moving target detection on the first image, and accordingly detects the first target area corresponding to the first household device in the first image; similarly, the second device type and the second target area corresponding to the second household device in the second image are obtained;

[0089] Next, the control center determines whether the first device type and the second device type are the same. When it is determined that the first device type is the same as the second device type (that is, the first home appliance and the second home appliance are the same home appliance), a SIFT algorithm (Scale Invariant Feature Transform) or a SURF algorithm (Speeded Up Robust Features) can be used to extract feature points from a local image in the first target area and a local image in the second target area, and feature point matching is performed based on the extracted feature points corresponding to the first target area and the feature points corresponding to the second target area, thereby obtaining matching feature points of the first target area and the second target area.

[0090] Then, the control center calculates the three-dimensional space coordinate value corresponding to the matching feature point according to the obtained first position information and the second position information; for example, Figure 2 As shown in FIG. 1 , it is a schematic diagram of calculating the three-dimensional space coordinate values ​​of the matching feature points when the control center according to an embodiment of the present invention locates and identifies the household equipment. After obtaining the matching feature points, since the matching feature points are based on the feature points in the image coordinate system, it is necessary to first convert the matching feature points into the first matching feature points in the three-dimensional camera coordinate system through the camera imaging model corresponding to the camera used when shooting the image, and then convert the first matching feature points into the world coordinate system (i.e., Figure 2 The coordinate system W in the image is formed, and the first ray (i.e. Figure 2 Similarly, the first matching feature point is converted to the world coordinate system by the obtained second pose parameter S2, and a second ray (i.e. Figure 2 The intersection point of the first ray and the second ray (i.e., Figure 2 The three-dimensional space coordinate value corresponding to the intersection point O in Figure 2 The three-dimensional space coordinate value corresponding to the intersection point (O) in;

[0091] Finally, the control center queries a preset household appliance information table according to the first device type. The household appliance information table records several household appliances in the room and the device information corresponding to each household appliance (for example, device type, device function and other related device information) to find out the household appliances in the household appliance information table whose device type is the same as the first device type. When there is only one household appliance corresponding to the same device type as the first device type found in the household appliance information table, the found household appliance is the first household appliance. The device information corresponding to the found household appliance is used as the device information of the first household appliance, and the three-dimensional spatial coordinate value of the obtained matching feature point is used as the three-dimensional spatial coordinate value of the first household appliance, thereby associating the device information of the household appliance with the three-dimensional spatial coordinate value one-to-one, and realizing the positioning of the household appliance and the identification of the household appliance at the same time.

[0092] It can be understood that only when the first household appliance identified based on the first image and the second household appliance identified based on the second image are the same household appliance, can two sets of information corresponding to the same household appliance at different shooting positions be obtained accordingly, and the positioning and identification of the same household appliance be completed based on the two sets of information; when the first household appliance identified based on the first image and the second household appliance identified based on the second image are not the same household appliance, if the first household appliance is used as the household appliance to be located and identified, the second image containing the first household appliance can be re-photographed at the second position and corresponding processing can be performed, and the image containing the first household appliance can be re-photographed at other positions and subsequent processing can be performed, which is not specifically limited in the embodiments of the present invention.

[0093] A smart home system provided by an embodiment of the present invention takes two sets of images corresponding to the same home appliance at different locations, and determines the device information and three-dimensional spatial coordinate values ​​corresponding to the home appliance based on the relevant information of the two sets of images and a pre-set home appliance information table, and associates the device information of the home appliance with the three-dimensional spatial coordinate values, thereby being able to simultaneously realize the positioning of the home appliance and the identification of the home appliance without incurring additional costs.

[0094] As an improvement of the above solution, the mobile robot is further used to send the environmental grid map to the control center;

[0095] Then, the control center locates and identifies the home appliance, and further includes:

[0096] When there are M home appliances corresponding to the same device type as the first device type in the home appliance information table, obtain the three-dimensional space coordinate values ​​corresponding to the M home appliances, and obtain the working status of the first home appliance; wherein M>1;

[0097] When the first household device is running, obtaining the three-dimensional space coordinate value of the user;

[0098] Finding a home appliance in the same room as the user among the M home appliances according to the environment grid map, the three-dimensional spatial coordinate values ​​corresponding to the M home appliances, and the three-dimensional spatial coordinate values ​​of the user;

[0099] The device information of the first household device is determined according to the device information corresponding to the found household device, and the three-dimensional space coordinate value of the first household device is determined according to the three-dimensional space coordinate value of the matching feature point.

[0100] Specifically, in combination with the above embodiment, after obtaining the environment grid map, the mobile robot also sends the environment grid map to the control center. After receiving the environment grid map, the control center can convert the environment grid map into an indoor room floor plan, thereby obtaining indoor room information; when the control center queries the preset home appliance information table according to the first device type, if there are M home appliances corresponding to the same device type as the first device type found in the home appliance information table (M>1), that is, there are multiple home appliances of the same device type, then the three-dimensional spatial coordinate values ​​corresponding to the M home appliances are further obtained, and the control center communicates with the indoor home appliances according to the preset communication protocol, and obtains the corresponding The current working status of the first home appliance is used to determine whether the first home appliance is currently running; when it is determined that the first home appliance is currently running, the three-dimensional spatial coordinate value corresponding to the user in the room is obtained, and according to the three-dimensional spatial coordinate values ​​of the M home appliances and the three-dimensional spatial coordinate values ​​of the user, combined with the indoor room information, the home appliance in the M home appliances that is in the same room as the user is determined, and the determined home appliance is used as the first target home appliance. The first target home appliance is the first home appliance, and the device information corresponding to the first target home appliance is used as the device information of the first home appliance, and the three-dimensional spatial coordinate value of the obtained matching feature point is used as the three-dimensional spatial coordinate value of the first home appliance.

[0101] In yet another preferred embodiment, the system comprises at least two of the sensors;

[0102] Then, the control center obtains the three-dimensional space coordinate value of the user, specifically including:

[0103] Based on the user, construct a probability grid map corresponding to the indoor environment;

[0104] Initialize and mark the probability grid map according to the occupancy grid map and the monitoring grid map corresponding to the indoor environment;

[0105] When any of the sensors detects the user, obtaining a first target grid where the user is located in the monitoring grid map;

[0106] The marking value of the grid corresponding to the first target grid in the marked probability grid map is updated to 1, and the marking values ​​of all grids in the marked probability grid map except the grid updated to 1 are updated to 0;

[0107] Acquire the three-dimensional space coordinate value of the user according to the updated probability grid map;

[0108] Among them, the occupancy grid map is obtained by marking the environmental grid map, and the marking value of the grid with obstacles in the occupancy grid map is 1, and the marking value of the grid without obstacles is 0; the monitoring grid map is obtained by marking the environmental grid map, and the marking value of the monitored grid in the monitoring grid map is 1, and the marking value of the unmonitored grid is 0; the marking value of each grid in the marked probability grid map corresponds to the probability value indicating the existence of the user.

[0109] It should be noted that the smart home system in the embodiment of the present invention includes at least two sensors for realizing collaborative positioning; before the specific execution of the steps of this embodiment, the control center pre-acquires the occupation grid map and the monitoring grid map corresponding to the indoor environment according to the environment grid map; for the occupation grid map, the corresponding grid of the obstacle in the environment grid map is determined based on the actual position of the obstacle in the room, and all the grids with obstacles in the environment grid map are marked as 1, and all the grids without obstacles in the environment grid map are marked as 0, so as to obtain the occupation grid map corresponding to the indoor environment according to the marked environment grid map, and accordingly, the marking value of each grid in the occupation grid map indicates whether there is an obstacle at the actual position corresponding to the grid in the room; for the monitoring grid map, each sensor is first obtained. The grid position of the sensor in the acquired occupancy grid map and the monitoring field of view (i.e., monitoring field angle) corresponding to each sensor are obtained, and then according to the grid position and monitoring field of view corresponding to each sensor in the occupancy grid map, and based on the grids with obstacles in the occupancy grid map, the monitored grids of each sensor in the occupancy grid map are obtained respectively, and the grids corresponding to all the monitored grids in the environment grid map are marked as 1, and the grids corresponding to all the grids that cannot be monitored in the environment grid map are marked as 0, so as to obtain the monitoring grid map corresponding to the indoor environment according to the marked environment grid map, and accordingly, the marking value of each grid in the monitoring grid map indicates whether the grid can be monitored at the actual position corresponding to the grid in the room (i.e., whether it can be monitored by any sensor in the room).

[0110] When the steps of this embodiment are specifically executed to obtain the three-dimensional spatial coordinate values ​​corresponding to the user indoors, in combination with the above embodiments, the control center constructs a probability grid map corresponding to the indoor environment based on the user himself, and initializes the mark value of the corresponding grid in the constructed probability grid map according to the mark value of the grid in the occupied grid map corresponding to the indoor environment obtained in advance, and initializes the mark value of the corresponding grid in the constructed probability grid map according to the mark value of the grid in the monitored grid map corresponding to the indoor environment obtained in advance. Accordingly, the mark value of each grid in the probability grid map after initialization corresponds to the probability value indicating the presence of the user, that is, the probability value indicating the user appears at the actual position corresponding to the grid in the room; at least two sensors (such as laser sensors, depth sensors, cameras, cameras, etc.) are set up indoors, and when any one of the sensors detects the user , determine that the actual position of the user corresponds to the grid in the monitoring grid map, and use the determined grid as the first target grid; according to the determined first target grid, the mark value of the corresponding grid in the probability grid map after the initialization mark is updated accordingly, and the mark value of the grid corresponding to the first target grid in the probability grid map after the initialization mark is updated to 1, and then the mark values ​​of all grids in the probability grid map after the initialization mark except the grid updated to 1 are updated to 0, so as to obtain the user's positioning result according to the mark value of the grid in the updated probability grid map. Correspondingly, the larger the mark value of a grid in the probability grid map, the greater the probability value of the user appearing at the actual position corresponding to the grid indoors, and the three-dimensional space coordinate value of the grid with the largest mark value at the actual position corresponding to the indoors can be used as the user's positioning result, that is, the user's three-dimensional space coordinate value is obtained.

[0111] Among them, when the constructed probability grid map is initialized and marked according to the occupation grid map and the monitoring grid map, those skilled in the art can understand that there will definitely be no users at the grid positions where there are obstacles in the occupation grid map. Therefore, all grids with a mark value of 1 in the occupation grid map and the corresponding grids in the constructed probability grid map are marked as 0; although the monitored grids in the monitoring grid map can be monitored by a certain sensor, it is initialized by default that all sensors have not detected the user. Therefore, all the mark values ​​in the monitoring grid map are 1. The grids corresponding to the grids in the constructed probability grid map are all marked as 0; in the constructed probability grid map, in addition to the grids that have been initialized and marked as 0 according to the occupied grid map and the monitored grid map, there are still several unmarked grids left, then all the unmarked grids except the grids marked as 0 in the constructed probability grid map are obtained, assuming that there are X unmarked grids in total, X>0, and each of the X grids is marked as 1 / X, indicating that at the time of initialization, the probability value of the user appearing at each grid position in these X unmarked grids is equal.

[0112] A smart home system provided by an embodiment of the present invention can achieve multi-sensor collaborative positioning by setting up at least two sensors, and can more intuitively display the spatial position of users in the monitoring area and the possible areas of appearance through a probability grid map, thereby quickly and conveniently tracking and positioning mobile objects.

[0113] As an improvement of the above solution, the control center obtains the three-dimensional space coordinate value of the user, further comprising:

[0114] When none of the sensors detects the user, obtaining, according to the three-dimensional spatial coordinate value of the user at the last moment, a second target grid where the user is located in the probability grid map updated at the last moment and a mark value corresponding to the second target grid;

[0115] According to a preset moving speed, the second target grid and the mark value corresponding to the second target grid, the probability grid map updated at the last moment is updated again;

[0116] The three-dimensional space coordinate value of the user is obtained according to the probability grid map that has been updated again.

[0117] Specifically, in combination with the above embodiment, when at least two sensors do not detect the user, it means that the user has entered a monitoring blind spot from the monitoring field of view of the sensor. At this time, the possible location of the user can be predicted based on the three-dimensional spatial coordinate value of the user obtained at the last moment; in specific implementation, the control center first determines the second target grid where the user is located in the probability grid map updated at the last moment and the mark value corresponding to the second target grid based on the three-dimensional spatial coordinate value of the user obtained at the last moment, and then updates the probability grid map updated at the last moment based on the preset user's movement speed, the determined second target grid and the mark value corresponding to the second target grid, so as to obtain the three-dimensional spatial coordinate value of the user according to the updated probability grid map.

[0118] As an improvement of the above solution, the control center updates the probability grid map updated at the last moment again according to the preset moving speed, the second target grid and the mark value corresponding to the second target grid, specifically including:

[0119] According to the movement rate and the second target grid, and in combination with the occupied grid map and the monitored grid map, a feasible domain corresponding to the user in the probability grid map updated at the last moment is obtained; wherein the feasible domain includes S grids, and the feasible domain represents the location area where the user may exist at the current moment, S>0;

[0120] The marking value of each of the S grids is updated to T / S, and the marking values ​​of all grids in the probability grid map updated at the last moment except the grid updated to T / S are updated to 0; wherein T represents the marking value corresponding to the second target grid, 0<T≤1.

[0121] Specifically, in combination with the above embodiment, when the probability grid map updated at the last moment is updated again, the control center can obtain the feasible domain corresponding to the user in the probability grid map updated at the last moment according to the preset movement speed and the determined second target grid, and in combination with the pre-acquired occupied grid map and monitoring grid map. The feasible domain specifically represents the location area where the user may be at the current moment. Assuming that the feasible domain includes S grids, S>0, the mark value corresponding to the determined second target grid is T, 0<T≤1, then the mark value of each of the S grids is updated to T / S, and the mark values ​​of all other grids in the probability grid map updated at the last moment except the grid updated to T / S are updated to 0.

[0122] See also Figure 3As shown in FIG. 1 , it is a schematic diagram of updating the probability grid map when the control center obtains the three-dimensional spatial coordinate value of the user provided by an embodiment of the present invention. Assume that there are two sensors indoors, namely Figure 3 C1 and C2 in the figure, the monitoring angle of sensor C1 is the first quadrant of the plane rectangular coordinate system with the center of the grid where C1 is located as the origin. Figure 3 The grids in the gray area in the figure represent grids with obstacles. Starting from the center of the grid where sensor C1 is located, several rays are emitted in the first quadrant until they hit an obstacle. The grids that the rays can pass through (or reach) indicate that the grids can be monitored by sensor C1. At time t, the position of the user monitored by sensor C1 is Figure 3 At point A in the probability grid map, the marking value of the grid at point A is P(A) = 1, and the marking values ​​of other grids in the probability grid map are all 0. At time t+1, neither C1 nor C2 can monitor the user. Assuming that the maximum moving distance of the user in a unit time is 2 grids, then except for the grids with obstacles and the grids that can be monitored, it can be determined that the feasible domain corresponding to the user in the probability grid map after the update at time t is Figure 3 The shaded area in the figure indicates the possible grid positions of the user at time t+1. Then P(A)=1 is evenly distributed to the four grids, that is, the mark value of each of the four grids is updated to 0.25.

[0123] It can be understood that if at time t+2, sensor C1 and sensor C2 still cannot monitor the user, the feasible domain corresponding to the user in the probability grid map updated at time t+1 can be further determined based on the user's position obtained at time t+1, that is, the corresponding feasible domain is determined according to each of the four grids in the shaded area, and the grid mark value 0.25 is evenly distributed to each grid in the corresponding feasible domain. If a grid is in multiple feasible domains at the same time, the probability value assigned to the grid can be accumulated.

[0124] In another preferred embodiment, the control center performs directional motion and control motion detection according to the user data, and based on the positioning and recognition results of the home appliances, determines the target home appliance according to the directional motion detection results, and controls the target home appliance according to the control motion detection results, specifically including:

[0125] Performing human skeleton key point detection on the user data to obtain three-dimensional image coordinate values ​​of the skeleton key points;

[0126] Performing coordinate transformation on the three-dimensional image coordinate values ​​of the skeleton key points according to the posture parameters of the sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points;

[0127] Performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance;

[0128] Control-type action detection is performed according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the control-type action detection result.

[0129] Specifically, in combination with the above embodiments, after receiving the user data sent by the sensor, the control center first performs human skeleton key point detection on the user data to obtain the three-dimensional image coordinate values ​​of the human skeleton key points, and converts the obtained three-dimensional image coordinate values ​​of the human skeleton key points to the world coordinate system according to the pre-obtained posture parameters of the sensor in the world coordinate system, and correspondingly obtains the three-dimensional space coordinate values ​​of the human skeleton key points; then, based on the obtained three-dimensional space coordinate values ​​of the human skeleton key points, directional motion detection is performed on the user's actions, so as to determine the target home appliance among all the indoor home appliances according to the directional motion detection results, and control-type motion detection is performed on the user's actions based on the obtained three-dimensional space coordinate values ​​of the human skeleton key points, so as to perform corresponding control on the determined target home appliance according to the control-type motion detection results.

[0130] Among them, the key points of the human skeleton may include the head, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, chest, pelvis, left hip, right hip, left knee, left ankle, right knee and right ankle, etc.; when the control center performs control-type action detection, it can pre-set several control action templates, and different control action templates respectively represent different control commands issued to home appliances. The obtained three-dimensional spatial coordinate values ​​of the key points of the human skeleton are matched with the pre-set control action templates. The control action issued by the user can be determined according to the successfully matched control action template, that is, the corresponding control command is determined accordingly. Combined with the target home appliance determined by the directional action detection result and the user's control action determined by the control-type action detection result, the target home appliance can be controlled accordingly according to the determined user's control action.

[0131] As an improvement of the above scheme, the control center performs directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determines the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance, specifically including:

[0132] Based on the positioning and identification results of the household appliances, find Q household appliances in the same room as the user according to the environment grid map and the three-dimensional spatial coordinate values ​​of the skeleton key points; wherein Q>0;

[0133] Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point;

[0134] According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of the Q home appliances, respectively obtain the angle between the position of each of the Q home appliances and the position of the user;

[0135] The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

[0136] Specifically, in combination with the above embodiments, when the control center performs directional motion detection, first, according to the positioning and identification results obtained after positioning and identifying the household appliances (i.e., the three-dimensional spatial coordinate values ​​of the household appliances in the world coordinate system and the corresponding device information), and in combination with the obtained environmental grid map and the three-dimensional spatial coordinate values ​​of the key points of the human skeleton, find out Q household appliances in all the indoor household appliances that are in the same room with the user, Q>0; then, pre-select any two skeletal key points among the key points of the human skeleton as the starting point key points and the ending point key points respectively, obtain the three-dimensional spatial coordinate values ​​of the starting point key points and the three-dimensional spatial coordinate values ​​of the ending point key points according to the obtained three-dimensional spatial coordinate values ​​of all the key points of the human skeleton, and calculate the angle between the position of each household appliance in the Q household appliances and the position of the user according to the obtained three-dimensional spatial coordinate values ​​of the starting point key points, the three-dimensional spatial coordinate values ​​of the ending point key points and the three-dimensional spatial coordinate values ​​of the Q household appliances, select the minimum angle therefrom, and use the household appliance corresponding to the minimum angle as the target household appliance, and the three-dimensional spatial coordinate value and device information of the household appliance corresponding to the minimum angle are the three-dimensional spatial coordinate value and device information of the target household appliance.

[0137] For example, let Pstart and Pend represent the starting key point and the ending key point respectively. Assume that the three-dimensional spatial coordinate values ​​of Q home appliances in the world coordinate system correspond to V1, V2, ..., VQ. For the i-th (i=1, 2, ..., Q) home appliance, calculate the angle between the vectors (Vi-Pstart) and (Pend-Pstart) (or the angle between other two vectors can also be calculated). Then the angle between these two three-dimensional vectors can be expressed as the angle between the position of the i-th home appliance and the user's position. Similarly, the Q corresponding angles between the positions of the Q home appliances and the human body can be obtained, and the home appliance with the smallest angle is found as the target home appliance.

[0138] A smart home system provided by an embodiment of the present invention converts the three-dimensional image coordinate values ​​of the key points of the human skeleton into the world coordinate system to unify the coordinate data of the key points of the human skeleton and the environmental data into the same coordinate system, and performs pointing action and control action detection based on the coordinate data of the key points of the human skeleton in the world coordinate system, so as to perform corresponding control on the determined target home appliances according to the pointing action detection results and the control action detection results, thereby improving the accuracy of home appliance control.

[0139] In another preferred embodiment, the control center is specifically used to obtain the position and posture parameters of the sensor in the world coordinate system through the following steps:

[0140] When the mobile robot does not exist in the monitoring field of the sensor, acquiring a background image through the sensor;

[0141] Control the mobile robot to move into the monitoring field of view of the sensor, obtain N motion images corresponding to N different positions of the mobile robot through the sensor, and obtain N position information corresponding to the N different positions of the mobile robot; wherein N>1;

[0142] Calculate the frame difference between the background image and each motion image, and obtain N frame difference images accordingly;

[0143] Each frame difference image is processed according to a preset depth threshold, and N mask images are obtained accordingly;

[0144] The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0145] The pose parameters of the sensor in the world coordinate system are calculated based on the N position information, the N cluster centers and the internal parameter matrix of the sensor.

[0146] Specifically, in combination with the above embodiment, the control center can control the mobile robot to move indoors. Then, first, the mobile robot is controlled to move outside the field of view of the sensor. When the mobile robot does not exist in the field of view of the sensor, the background image in the field of view is acquired through the sensor; the mobile robot is controlled to move into the field of view of the sensor and respectively move to N (N>1) different positions. When the mobile robot exists in the field of view of the sensor, N images corresponding to the mobile robot at the N different positions are acquired through the sensor, and N position information corresponding to the mobile robot at the N different positions is acquired respectively through the self-positioning of the mobile robot;

[0147] Next, the control center calculates the frame difference between the obtained background image and each motion image in the N images, and obtains N frame difference images accordingly, and processes each frame difference image accordingly according to a preset depth threshold, and obtains N mask images accordingly; wherein, the processing method for each frame difference image is the same, and here the processing method for the i-th frame difference image is taken as an example, and the depth value of each pixel on the i-th frame difference image is compared with the preset depth threshold, and when it is determined that the depth value of any pixel is greater than the preset depth threshold, the pixel is marked as 1, and when it is determined that the depth value of any pixel is not greater than the preset depth threshold, the pixel is marked as 0, and accordingly, after marking each pixel on the i-th frame difference image with 1 or 0, the i-th mask image is obtained according to the i-th frame difference image after the marking process;

[0148] Then, the control center calculates the average value of the coordinate values ​​and depth values ​​of all pixels marked as 1 on each of the N mask images, and obtains N cluster centers accordingly; among them, taking the processing method of the i-th mask image as an example, the uv coordinate values ​​and depth values ​​d of all pixels marked as 1 on the i-th mask image are averaged, and the corresponding cluster center corresponding to the i-th mask image is expressed as p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N;

[0149] Finally, the control center calculates the posture parameters of the sensor in the world coordinate system based on the N position information corresponding to the mobile robot, the N cluster centers and the intrinsic parameter matrix of the sensor. Among them, the intrinsic parameter matrix of the sensor is generally given by the manufacturer or calibrated in advance and is a known parameter.

[0150] It should be noted that the present embodiment involves depth thresholds, averaging the depth values ​​of pixels on the image, etc., all of which use depth sensors as examples to illustrate specific schemes for calibrating sensors. If it is other types of sensors, other calibration schemes provided by the prior art can be used to obtain the corresponding posture parameters of the sensor in the world coordinate system.

[0151] As an improvement of the above solution, the control center calculates and obtains the posture parameters of the sensor in the world coordinate system according to the N position information, the N cluster centers and the internal parameter matrix of the sensor, specifically including:

[0152] According to the formula Solve and obtain the corresponding position parameter H of the sensor in the world coordinate system d ; Among them, P i represents the i-th position information corresponding to the i-th position of the mobile robot, Indicates the location information of the i-th cluster center converted to the world coordinate system, K s represents the intrinsic parameter matrix of the sensor.

[0153] Specifically, in combination with the above embodiment, the control center can use the formula Solve and obtain the corresponding pose parameter H of the sensor in the world coordinate system d , P i represents the i-th position information corresponding to the mobile robot at the i-th position, Indicates that the i-th cluster center p i =(u i , v i , d i ) is converted to the position information in the world coordinate system, K s represents the intrinsic parameter matrix of the sensor, Represents the Euclidean distance between two 3D vectors.

[0154] A smart home system provided by an embodiment of the present invention obtains a background image and N motion images within the monitoring field of view of a sensor, and respectively obtains the position information of a mobile robot corresponding to each motion image in a world coordinate system, so as to calculate N mask images according to the background image and the N motion images, and calculates the posture parameters of the sensor according to the N position information corresponding to the mobile body, the N mask images and the intrinsic parameter matrix of the sensor, which can greatly reduce the error caused by manually adjusting the test position, thereby improving the accuracy of the sensor's posture calibration.

[0155] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A smart home system, comprising sensors, a control center and home appliances, characterized in that: The system also includes a mobile robot; The mobile robot is used to map the indoor environment and obtain an environmental grid map; it is also used to perform self-positioning according to the environmental grid map and send its own position information to the control center; The control center is used to control the mobile robot and locate and identify the household equipment according to the position information of the mobile robot and a preset household equipment information table; The sensor is used to collect user data and send the user data to the control center; The control center is further used to detect directional actions and control actions according to the user data, and based on the positioning and recognition results of the home appliances, determine the target home appliances according to the directional action detection results, and control the target home appliances according to the control action detection results; The control center locates and identifies the home device, specifically including: Controlling the mobile robot to move to a first position, capturing a first image at the first position through a camera on the mobile robot, and obtaining first position information of the mobile robot at the first position; Controlling the mobile robot to move to a second position, capturing a second image at the second position through a camera on the mobile robot, and obtaining second position information of the mobile robot at the second position; Acquire a first device type and a first target area corresponding to a first household device in the first image, and acquire a second device type and a second target area corresponding to a second household device in the second image; When the first device type is the same as the second device type, extracting and matching feature points of the first target area and the second target area to obtain matching feature points; Acquire the three-dimensional space coordinate value of the matching feature point according to the first position information and the second position information; Query the home device information table according to the first device type; wherein the home device information table includes a number of home devices and their corresponding device information, and the device information at least includes the device type; When there is only one household device corresponding to the same device type as the first device type in the household device information table, the device information of the first household device is determined according to the device information corresponding to the household device, and the three-dimensional spatial coordinate value of the first household device is determined according to the three-dimensional spatial coordinate value of the matching feature point.

2. The smart home system according to claim 1, characterized in that: The mobile robot is further used to send the environmental grid map to the control center; Then, the control center locates and identifies the home appliance, and further includes: When there are M home appliances corresponding to the same device type as the first device type in the home appliance information table, obtain the three-dimensional space coordinate values ​​corresponding to the M home appliances, and obtain the working status of the first home appliance; wherein M>1; When the first household device is running, obtaining the three-dimensional space coordinate value of the user; Finding a home appliance in the same room as the user among the M home appliances according to the environment grid map, the three-dimensional spatial coordinate values ​​corresponding to the M home appliances, and the three-dimensional spatial coordinate values ​​of the user; The device information of the first household device is determined according to the device information corresponding to the found household device, and the three-dimensional space coordinate value of the first household device is determined according to the three-dimensional space coordinate value of the matching feature point.

3. The smart home system according to claim 2, characterized in that: The system comprises at least two of the sensors; Then, the control center obtains the three-dimensional space coordinate value of the user, specifically including: Based on the user, construct a probability grid map corresponding to the indoor environment; Initialize and mark the probability grid map according to the occupancy grid map and the monitoring grid map corresponding to the indoor environment; When any of the sensors detects the user, obtaining a first target grid where the user is located in the monitoring grid map; The marking value of the grid corresponding to the first target grid in the marked probability grid map is updated to 1, and the marking values ​​of all grids in the marked probability grid map except the grid updated to 1 are updated to 0; Acquire the three-dimensional space coordinate value of the user according to the updated probability grid map; Among them, the occupancy grid map is obtained by marking the environmental grid map, and the marking value of the grid with obstacles in the occupancy grid map is 1, and the marking value of the grid without obstacles is 0; the monitoring grid map is obtained by marking the environmental grid map, and the marking value of the monitored grid in the monitoring grid map is 1, and the marking value of the unmonitored grid is 0; the marking value of each grid in the marked probability grid map corresponds to the probability value indicating the existence of the user.

4. The smart home system according to claim 3, characterized in that: The control center obtains the three-dimensional space coordinate value of the user, and also includes: When none of the sensors detects the user, obtaining, according to the three-dimensional spatial coordinate value of the user at the last moment, a second target grid where the user is located in the probability grid map updated at the last moment and a mark value corresponding to the second target grid; According to a preset moving speed, the second target grid and the mark value corresponding to the second target grid, the probability grid map updated at the last moment is updated again; The three-dimensional space coordinate value of the user is obtained according to the probability grid map that has been updated again.

5. The smart home system according to claim 4, characterized in that: The control center updates the probability grid map updated at the last moment again according to the preset moving speed, the second target grid and the mark value corresponding to the second target grid, specifically including: According to the movement rate and the second target grid, and in combination with the occupied grid map and the monitored grid map, a feasible domain corresponding to the user in the probability grid map updated at the last moment is obtained; wherein the feasible domain includes S grids, and the feasible domain represents the location area where the user may exist at the current moment, S>0; The marking value of each of the S grids is updated to T / S, and the marking values ​​of all grids in the probability grid map updated at the last moment except the grid updated to T / S are updated to 0; wherein T represents the marking value corresponding to the second target grid, 0<T≤1.

6. The smart home system according to claim 2, characterized in that: The control center detects directional motion and control motion according to the user data, and based on the positioning and recognition results of the home appliances, determines the target home appliances according to the directional motion detection results, and controls the target home appliances according to the control motion detection results, specifically including: Performing human skeleton key point detection on the user data to obtain three-dimensional image coordinate values ​​of the skeleton key points; Performing coordinate transformation on the three-dimensional image coordinate values ​​of the skeleton key points according to the posture parameters of the sensor in the world coordinate system to obtain the three-dimensional space coordinate values ​​of the skeleton key points; Performing directional motion detection according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and determining the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance; Control-type action detection is performed according to the three-dimensional spatial coordinate values ​​of the skeleton key points, and the target home appliance is controlled according to the control-type action detection result.

7. The smart home system according to claim 6, characterized in that: The control center performs directional motion detection according to the three-dimensional space coordinate values ​​of the skeleton key points, and determines the target home appliance according to the directional motion detection results based on the positioning and recognition results of the home appliance, specifically including: Based on the positioning and identification results of the household appliances, find Q household appliances in the same room as the user according to the environment grid map and the three-dimensional spatial coordinate values ​​of the skeleton key points; wherein Q>0; Acquire the three-dimensional space coordinate value of the preset starting point key point and the three-dimensional space coordinate value of the preset end point key point according to the three-dimensional space coordinate value of the skeleton key point; According to the three-dimensional space coordinate value of the starting point key point, the three-dimensional space coordinate value of the end point key point and the three-dimensional space coordinate value of the Q home appliances, respectively obtain the angle between the position of each of the Q home appliances and the position of the user; The target home appliance is determined according to the three-dimensional space coordinate value of the home appliance corresponding to the minimum angle and the device information.

8. The smart home system according to claim 6, characterized in that: The control center is specifically used to obtain the position and posture parameters of the sensor in the world coordinate system through the following steps: When the mobile robot does not exist in the monitoring field of the sensor, acquiring a background image through the sensor; Control the mobile robot to move into the monitoring field of view of the sensor, obtain N motion images corresponding to N different positions of the mobile robot through the sensor, and obtain N position information corresponding to the N different positions of the mobile robot; wherein N>1; Calculate the frame difference between the background image and each motion image, and obtain N frame difference images accordingly; Each frame difference image is processed according to a preset depth threshold, and N mask images are obtained accordingly; The coordinate values ​​and depth values ​​of all pixels marked as 1 on each mask image are averaged to obtain N corresponding cluster centers; the cluster center corresponding to the i-th mask image is p i =(u i , v i , d i ), (u i , v i ) represents the average coordinate value of all pixels marked as 1 on the i-th mask image, d i represents the average depth value of all pixels marked as 1 on the i-th mask image, i = 1, 2, ..., N; The pose parameters of the sensor in the world coordinate system are calculated based on the N position information, the N cluster centers and the internal parameter matrix of the sensor.

9. The smart home system according to claim 8, characterized in that: The control center calculates and obtains the posture parameters of the sensor in the world coordinate system according to the N position information, the N cluster centers and the internal parameter matrix of the sensor, specifically including: According to the formula Solve and obtain the corresponding position parameter H of the sensor in the world coordinate system d ; Among them, P i represents the i-th position information corresponding to the i-th position of the mobile robot, Indicates the location information of the i-th cluster center converted to the world coordinate system, K s represents the intrinsic parameter matrix of the sensor.

Images