System and method for continuously monitoring entities

Abstract

A system and method for continuously monitoring entities in an Internet of Things (IoT) environment are disclosed. A plurality of behaviors associated with an entity are monitored at a first location within an IoT environment via a first sensor present at the first location. Further, a second location within the IoT environment associated with the presence of the entity is identified such that the second location exceeds the detection range of the first sensor. Further, a second sensor for monitoring a subset of the plurality of behaviors of the entity present at a second location is determined from the plurality of sensors. Further, a subset of the plurality of behaviors of the entity is monitored by a second sensor based on one or more trained models selected from the plurality of trained models.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to monitoring systems, and, for example, to monitoring entities via multiple sensors in an Internet of Things (IoT) comprising multiple edge devices. BACKGROUND

[0002] In a smart home monitoring system, monitoring of a home involves installing IoT cameras and sensors at various locations such as front door, living room, kitchen, and other areas. Each of these cameras and sensors has a specific field of view (e.g., 120 degrees, 180 degrees, etc.) and range based on their respective capabilities. Currently available home monitoring solutions mainly rely on a single sensor or combination of sensors provided by a single vendor.

[0003] In case an entity moves out of the range of a particular sensor, the monitoring or prediction process stops. Thus, if there are no similar sensors or capabilities in other areas, the monitoring is interrupted when the entity crosses the range of a particular sensor, which is undesirable.

[0004] Figure 1 An exemplary scenario of interrupted monitoring of an entity moving around in a house is shown. As depicted, the house can have multiple camera sensors placed at various locations like living room 101 and kitchen 105. These cameras have limited monitoring range, represented by the designated areas 107 and 109. Thus, if an entity such as a toddler or baby moves beyond the range of the camera sensor in the living room 101, the monitoring of the baby will stop. Further, in case the baby moves to the bedroom 103 where no camera sensor is installed, there is no way to monitor the baby in that particular room.

[0005] In another scenario, consider a scenario where the bedroom of elderly parents is equipped with a fall detection sensor to enhance their safety. However, due to budget constraints, no sensor is installed in the kitchen. Due to poor visibility, especially at night, the elderly can face several potential difficulties such as the risk of falling or accidentally breaking a sharp object. Unfortunately, since there is no fall sensor in the kitchen, there is no way for a user such as a caretaker or guardian to take any action immediately or receive an alert about any issues or accidents that can occur.

[0006] In yet another scenario, consider a case where a pet is left alone in a room (e.g., bedroom) of a house and its activities are monitored by a remote video camera (RVC). The camera sends updates to the user about the pet’s behavior. However, when the pet moves out of the line of sight of the RVC and enters another room (e.g., living room), the user can not receive any updates and is unaware of its current activities.

[0007] Thus, there is a need to provide a solution to the above-mentioned limitations. Summary of the Invention

[0008] Technical solution In an embodiment, this disclosure provides a method for continuously monitoring an entity by a controller associated with an Internet of Things (IoT) environment. The method includes: monitoring multiple behaviors associated with an entity present at a first location within the IoT environment via a first sensor of a plurality of sensors. Furthermore, the method includes: identifying a second location within the IoT environment, wherein the second location is associated with the presence of the entity and is outside the detection range of the first sensor. Additionally, the method includes: determining a subset of the multiple behaviors of the entity present at the second location from the plurality of sensors based on pre-stored information associated with the IoT environment. Furthermore, the method includes: monitoring the subset of the multiple behaviors of the entity at the second location via the second sensor based on one or more trained models.

[0009] In an embodiment, this disclosure provides a system for continuously monitoring entities in an Internet of Things (IoT) environment. The system includes: a plurality of sensors; at least one processor including processing circuitry; and a memory storing instructions, wherein the instructions, when executed individually or collectively by the at least one processor, cause the system to: monitor, at a first location within the IoT environment, a plurality of behaviors associated with an entity present at the first location via a first sensor of the plurality of sensors; identify a second location within the IoT environment, wherein the second location is associated with the presence of an entity and is outside the detection range of the first sensor; select a second sensor from the plurality of sensors present at the second location for monitoring the plurality of behaviors of the entity based on pre-stored information associated with the IoT environment; and monitor, at the second location, a subset of the plurality of behaviors of the entity based on one or more trained models via the second sensor.

[0010] To further illustrate the advantages and features of this disclosure, a more specific description of the disclosure will be presented with reference to specific embodiments of the disclosure shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the disclosure and should not be considered as limiting its scope. The disclosure will be described and explained in the accompanying drawings with additional features and details. Attached Figure Description

[0011] The above and other features, aspects and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which similar reference numerals denote similar parts, and wherein: Figure 1 This is an exemplary scenario of interrupted monitoring of entities moving around a house, based on existing technology; Figure 2 A schematic block diagram of an IoT environment in a smart home according to various embodiments of the present disclosure is shown; Figure 3 This is a flowchart illustrating the operational flow of a controller according to various embodiments of the present disclosure enabling continuous monitoring within a smart home; Figure 4 This is a schematic diagram illustrating a continuous monitoring system in a smart home according to various embodiments of the present disclosure; Figure 5 This is a block diagram illustrating multiple modules of a controller within a continuous monitoring system according to various embodiments of the present disclosure; Figure 6 This is a schematic diagram illustrating exemplary execution of various modules 411 according to various embodiments of the present disclosure; Figure 7a , Figure 7b , Figure 7c and Figure 7d This is a schematic diagram illustrating exemplary inputs and outputs of modules of a continuous monitoring system according to various embodiments of the present disclosure; and Figure 8 This is a block diagram illustrating a method 800 for continuously monitoring entities in an IoT environment according to various embodiments of the present disclosure. Detailed Implementation

[0012] To aid in understanding the principles of this disclosure, reference will now be made to various exemplary embodiments, which will be described using specific language. However, it should be understood that this is not intended to limit the scope of the disclosure, and such changes and further modifications in the systems shown, as well as such further applications of the principles of the disclosure as illustrated therein, are considered to be common knowledge of the art to which this disclosure pertains.

[0013] Those skilled in the art will understand that the foregoing general description and the following detailed description are for the purpose of interpreting this disclosure and are not intended to limit it.

[0014] Throughout this specification, references to "aspect," "on the other hand," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, throughout this specification, the phrases "in an embodiment," "in another embodiment," and similar language may, but not necessarily all, refer to the same embodiment.

[0015] It should be understood that, as used herein, terms such as “comprising,” “including,” and “having” are intended to indicate that one or more listed features or elements are within the defined element, but the element is not necessarily limited to the listed features and elements, and additional features and elements may be included within the meaning of the defined element. Conversely, terms such as “consisting of” are intended to exclude features and elements not listed.

[0016] The embodiments described herein, along with their various features and advantageous details, are explained more fully with reference to the non-limiting embodiments illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques have been omitted to avoid unnecessarily obscuring the embodiments herein. Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein means non-exclusive or. The examples used herein are intended only to facilitate understanding of how the embodiments described herein can be practiced and to further enable those skilled in the art to practice the embodiments described herein. Therefore, these examples should not be construed as limiting the scope of the embodiments described herein.

[0017] As is conventional in the art, embodiments can be described and illustrated in terms of blocks that perform the described functions. These blocks (which may be referred to herein as units or modules, etc.) are physically implemented by analog or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuitry, passive electronic components, active electronic components, optical components, hardwired circuitry, etc.) and may optionally be driven by firmware and software. The circuitry may be embodied, for example, in one or more semiconductor chips or on a substrate support such as a printed circuit board. The circuitry constituting a block may be implemented by dedicated hardware, a processor (e.g., one or more programmed microprocessors and associated circuitry), or a combination of dedicated hardware (for performing some functions of the block) and a processor (for performing other functions of the block). Without departing from the scope of this disclosure, each block of an embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of this disclosure, the blocks of an embodiment may be physically combined into more complex blocks.

[0018] The accompanying drawings are provided to aid in the easy understanding of the various technical features, and it should be understood that the embodiments presented herein are not limited to the drawings. Therefore, this disclosure should be construed as extending to any changes, equivalents, and alternatives other than those specifically set forth in the drawings. Although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are generally used only to distinguish one element from another.

[0019] To address the aforementioned issues, this disclosure provides a mechanism for continuously monitoring one or more entities in an IoT environment. Continuous monitoring is performed using multiple sensors. These sensors may have different ranges and capabilities. Furthermore, the sensors may be supplied by the same vendor or different vendors.

[0020] Furthermore, according to various embodiments of this disclosure, a system and method are provided for continuously monitoring an entity by a controller associated with an IoT environment. Multiple behaviors of a target entity (e.g., a child) are monitored via one or more first sensors (e.g., cameras) present in a first location (e.g., a living room) within the IoT environment. The target entity is detected to have moved from the first location to a second location within the IoT environment, such that the second location is outside the monitoring range of the first sensors. One or more second sensors present in the second location are identified, and at least a subset of the monitored behaviors of the target entity that can be monitored by the second sensors are determined. Finally, one or more trained edge models associated with the subset of behaviors are invoked for use by the second sensors to continue monitoring the behavior of the target entity in the second location.

[0021] For example, a user who may be particularly concerned about the baby's health could monitor the baby using a camera installed in the living room of the house. However, the baby may begin to crawl and may move to different areas (such as the kitchen) outside the field of view of the camera in the living room. According to various embodiments of this disclosure, other sensors present in the kitchen can be identified. According to various embodiments of this disclosure, one or more training models related to the baby's health (such as baby crying, baby coughing, glass breaking, etc.) are identified. Furthermore, sensors placed in the kitchen capable of running such training models are identified. For example, smart speakers and smart refrigerators may already be installed in the kitchen. According to various embodiments of this disclosure, relevant training models are deployed on smart refrigerators and smart speakers or controllers to alert the user when the baby begins to cry, thereby continuing to monitor the baby. Now combined Figures 2 to 8 The technology provided in this disclosure is described in detail.

[0022] Figure 2A schematic block diagram of an IoT environment in a smart home 200 according to an embodiment of the present disclosure is shown. The smart home 200 may include a continuous monitoring system 200A. The continuous monitoring system 200A may include a controller 201 communicatively connected to multiple sensors, such as a camera sensor 203, a microphone within a smart speaker 205, a smart TV 207, and a proximity sensor 209. The multiple sensors may be installed in different locations in various rooms of the smart home 200. For example, the camera sensor 203 may be installed in the living room 211, the smart speaker 205 may be installed in the kitchen 213, and the smart TV 207 and proximity sensor 209 may be installed in the bedroom 215. It should be understood that the continuous monitoring system 200A may include additional sensors installed in the same or additional rooms of the smart home 200.

[0023] According to embodiments of this disclosure, controller 201 may be configured to monitor one or more entities within smart home 200 using monitoring data obtained from multiple sensors installed across various rooms of smart home 200. Controller 201 may be communicatively connected to user device 217, which is associated with a user who wishes to monitor one or more entities within smart home 200. In the event of an accident or unwanted activity, controller 201 may notify the user by sending an alarm on user device 217. User device 217 may be any device capable of receiving alarms from controller 201. User device 217 may include, but is not limited to, smartphones, laptops, and tablets. Controller 201 may perform various operations to achieve the purpose of this disclosure, namely, enabling continuous monitoring throughout smart home 200. In one embodiment, controller 201 may be implemented in a separate device within smart home 200. In another embodiment, controller 201 may be implemented on a cloud-based server. In yet another embodiment, controller 201 may be implemented in a distributed manner, such that one or more components of the controller are implemented on separate devices, and one or more components are implemented on a cloud-based server. The following is in conjunction with... Figure 3 Describes the operation of controller 201 to achieve the purposes of this disclosure.

[0024] Figure 3This is a flowchart 300 illustrating the operational flow of controller 201 enabling continuous monitoring within smart home 200. In operation 301, controller 201 identifies at least one target entity to be monitored, and a first sensor involved in monitoring the target entity. Hereinafter, the term "target entity" is used interchangeably with the term "monitored entity." In operation 302, controller 201 detects whether the monitored entity has crossed the range of the first sensor. In operation 305, controller 201 determines whether the entity is out of range. If the entity has not yet crossed the range of the first sensor, the controller continues monitoring the entity. When controller 201 detects that the monitored entity has crossed the range of the first sensor, in operation 307, controller 201 determines one or more possible locations within smart home 200 that the monitored entity may have visited. Furthermore, controller 201 may determine one or more behaviors associated with the monitored entity. In operation 309, controller 201 is able to determine an exact second location from the one or more possible locations the monitored entity has moved to. In operation 311, controller 201 is able to determine a second set of sensors, wherein the second set of sensors can be used to continue monitoring the monitoring behavior of the target entity at a second location.

[0025] In operation 313, controller 201 can invoke one or more edge models on a second set of sensors based on a subset of monitored behaviors. In home monitoring within an IoT environment, one or more edge models can refer to artificial intelligence (AI)-based models that are directly deployed on edge devices (such as smart speakers, smart refrigerators, smart TVs, etc.) to enable local analysis and real-time decision-making on events (such as accidents or unwanted activities).

[0026] In operation 315, using one or more edge models, controller 201 may be able to continue monitoring the target entity using the second sensor set, and in operation 317, controller 201 may determine whether to generate an alarm or modify the edge model. In operation 319, when the second sensor set detects an unexpected event or unwanted activity, controller 201 may generate an alarm and notify the user by sending an alert to the user device. No alarm is generated if no unexpected event or unwanted activity occurs. In an embodiment, the controller may also determine to modify the deployed edge model in the second sensor set. Modification of the edge model may be necessary when a different entity enters the second location, or when one of the devices associated with the second sensor set stops working. In operation 323, controller 201 may continue monitoring the target entity and different entities at the second location via the modified edge model on the second sensor set. Now combined Figure 4 Detailed description of the 200A continuous monitoring system, which includes multiple sensors and controllers.

[0027] Figure 4 This is a schematic diagram 400 illustrating a continuous monitoring system 200A in a smart home 200 according to an embodiment of the present disclosure. As shown, the continuous monitoring system 200A may include multiple sensors 401 and a controller 201 to monitor multiple behaviors of entities within the smart home 200. In embodiments, the monitored entities may include, but are not limited to, pets, infants, or elderly people living in the smart home 200. In embodiments, multiple behaviors of an entity may be determined based on at least one of monitoring history, at least one parameter associated with the movement of the monitored entity, profile information associated with the monitored entity, and learned habits of the monitored entity. For example, profile information may include, but is not limited to, user profiles, added family members, and entity-related orders (such as ordering dog food if the user has a dog).

[0028] Multiple sensors 401 may include different types of sensors installed at different locations within the smart home 200. In an embodiment, multiple sensors 401 may be present within one or more edge devices installed in the smart home 200. One or more edge devices may include, but are not limited to, smart speakers, smart refrigerators, and smart TVs. Therefore, multiple sensors 401 may include, but are not limited to, camera sensors, microphones, proximity sensors, temperature sensors, noise sensors, and motion sensors. Figure 2 The sensor is used as Figure 4 Examples are provided. Multiple sensors 401 may have different sensing capabilities and varying sensing ranges. In embodiments, different locations within the smart home 200 may include different rooms (such as, but not limited to, living room 211, kitchen 213, and bedroom 215). According to various embodiments of this disclosure, one or more of the multiple sensors 401 may be enabled to monitor at least a subset of multiple behaviors of an entity.

[0029] Controller 201 may include at least one processor 403, memory 405 (e.g., RAM), storage 407 (e.g., ROM), network interface 409, and multiple modules 411. Processor 403 is configured to execute instructions stored in memory 405 and perform various operations as described in embodiments of this disclosure. Processor 403 may include dedicated processing units such as integrated system (bus) controllers, memory management control units, floating-point units, graphics processing units, digital signal processing units, etc. In one embodiment, processor 403 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. Processor 403 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other means of analyzing and processing data now known or hereafter developed. Processor 403 may execute one or more instructions (such as manually generated (i.e., programmed) code) to perform one or more operations disclosed herein.

[0030] Memory 407 may include one or more databases to store one or more data and information that may be needed to implement the continuous monitoring system 200A. In an embodiment, memory 407 may include one or more AI-based training models, wherein the one or more AI-based training models may be deployed on one or more edge devices, enabling one or more sensors within the edge devices to monitor at least a subset of multiple behaviors of a target entity. One or more data and information may include a house floor plan associated with smart home 200, which provides details about the layout and location of various rooms within smart home 200 and information about edge devices installed at various locations in these rooms. In an embodiment, information and data may include one or more of the following: layout associated with multiple locations in the IoT environment, configuration information associated with each of a plurality of sensors, known locations previously visited by the monitored entity, and placement information associated with the presence of each of a plurality of sensors at multiple locations in the IoT environment, wherein pre-stored information is stored in storage devices associated with the IoT environment. Network interface 409 provides network connectivity and enables communication with user device 217 via a network.

[0031] Multiple modules 411 may include an instruction set, wherein the instruction set can be executed to enable the continuous monitoring system 200A to monitor entities in the IoT environment within the smart home 200, such as in combination with Figure 5 and Figure 6Described in more detail. In an embodiment, the plurality of modules 411 may be hardware units external to memory 405. At least one of the plurality of modules may be implemented via an AI model. AI-related functions may be performed via non-volatile memory, volatile memory, and processor 403.

[0032] Processor 403 may include one or more processors. In this case, the one or more processors may be general-purpose processors (such as central processing units (CPUs), application processors (APs), graphics processing units only (such as GPUs), vision processing units (VPUs), and / or AI-specific processors (such as neural processing units (NPUs)). The one or more processors control the processing of input data according to specified operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The specified operating rules or AI models are provided through training or learning.

[0033] Here, "learning provides" means creating specified operational rules or AI models with desired characteristics by applying learning techniques to multiple learning datasets. Learning can be performed within the device itself that executes the AI ​​according to the embodiment, and / or can be implemented via a separate server / system.

[0034] AI models can consist of multiple neural network layers. Each layer has multiple weight values, and layer operations are performed by computing the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0035] Learning techniques are methods for training a predetermined target device (e.g., a robot) using multiple learning datasets to enable, allow, or control the target device to make determinations or predictions. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0036] According to this disclosure, a method for continuously monitoring entities in an IoT environment can use an artificial intelligence (AI) model to recommend / run multiple instructions using sensor data. The processor can perform preprocessing operations on the data to transform it into a form suitable for use as input to the AI ​​model. The AI ​​model can be obtained through training. Here, "obtained through training" means training a basic AI model with multiple training data using training techniques to obtain a specified operating rule or AI model configured to perform a desired function (or purpose). The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers includes multiple weight values, and neural network computation is performed by calculating the results of the previous layer with the multiple weight values.

[0037] Inference and prediction are techniques for logically inferring and predicting based on determined information, and include, for example, knowledge-based inference, optimization prediction, and preference-based planning or recommendation. In embodiments, user device 217 may also include a processor, memory, and a network interface having characteristics similar to those of the corresponding components in controller 201. For the sake of brevity, these components are not described in detail here. They will now be discussed in conjunction with... Figure 5 and Figure 6 Describes multiple modules 411.

[0038] Figure 5 This is a block diagram 500 showing a plurality of modules 411 of a controller 201 within a continuous monitoring system 200A according to an embodiment of the present disclosure. Figure 6 This is a schematic diagram 600 illustrating exemplary execution of a plurality of modules 411 according to embodiments of the present disclosure. The plurality of modules 411 may include an entity behavior monitoring module 501, an out-of-range detector module 503, an entity monitoring continuity module 505, an edge pipeline interruptor module 507, and an alarm generator and edge modifier module 509.

[0039] Entity monitoring module 501 can be configured to determine a target entity selected by a user for monitoring within smart home 200, monitoring statistics corresponding to information collected during monitoring of the target entity in the current monitoring scenario to provide insights into the target entity's behavior, and a first sensor responsible for generating most of the monitoring statistics. The first sensor may be located at a first location within smart home 200 and may have a specified detection range. In embodiments, the first sensor may include one or more sensors present at the first location. If the target entity moves from the first location to another location within smart home 200, the target entity may exceed the specified detection range of the first sensor monitoring the target entity. Out-of-range detector module 503 can be configured to determine that the target entity has moved to some other location (i.e., one or more possible locations beyond the specified detection range of the first sensor), such that one or more possible locations are associated with the presence of the monitored entity. Out-of-range detector module 503 can also be configured to determine the monitoring behavior of the target entity.

[0040] Furthermore, the entity monitoring continuity module 505 can be configured to determine a second location where the target entity has moved and a subset of second sensors available for monitoring the target entity's behavior. In an embodiment, the second sensors may include one or more sensors present at the second location. The edge pipeline interruptor module 507 can be configured to invoke an edge model based on the capabilities of the second sensors and a subset of the target entity's monitoring behavior. The alarm generator and edge modifier module 509 can be configured to determine whether to generate an alarm or modify the edge model. When it is determined that an alarm should be generated, an alarm is generated and sent to the user on the user device. However, when the edge model needs to be modified due to the entry of a different entity at the second location or one of the second sensors becoming ineffective, the alarm generator and edge modifier module 509 can modify the previously invoked edge model and enable continuous monitoring at the second location via the modified edge model. Each of the modules 411 will be described in detail below.

[0041] The entity monitoring module 501 identifies the target entity to be monitored and a first sensor at a first location within the IoT environment that monitors multiple behaviors associated with the monitored entity (i.e., the target entity). The first sensor may be from multiple sensors associated with the IoT environment and may be located at the first location. The entity monitoring module 501 determines the target entity to be monitored based on the output of a monitoring service used by the user and information shared with the user, wherein the information corresponds to alarms, notifications, routines, and similar data associated with the target entity.

[0042] Furthermore, the entity monitoring module 501 determines a first sensor by identifying the sensor that provides the maximum output associated with the target entity. To determine the sensor providing the maximum output associated with the target entity, sensors providing relevant data about the target entity are aggregated. Then, the percentage of time the sensor is focused on the target entity is calculated based on past events associated with the sensor. Subsequently, one or more sensors with events exceeding a predetermined threshold are identified as the first sensor, and the capability of the first sensor is determined in relation to the target entity. In this embodiment, the predetermined threshold is calculated based on the total number of monitored events and the total number of events specific to the target entity.

[0043] After determining the capabilities of the first sensor and the first sensor associated with the target entity, the monitoring statistics generated by the first sensor are determined by scoring and filtering the highest monitoring statistics associated with the target entity that may be useful to the user. To score and filter the highest monitoring statistics, all available monitoring statistics associated with the target entity provided by the monitoring service are collected. Then, each monitoring statistics is scored based on parameters such as, but not limited to, the association between the monitoring statistics and the first sensor, the capabilities of the first sensor, and the user's interest in the monitoring statistics. Then, the monitoring statistics with the highest scores are filtered according to the definition of the monitoring service. Therefore, the entity monitoring module 501 produces the following results: a identified target entity, a first sensor monitoring the identified target entity, and monitoring statistics associated with the target entity generated by the first sensor.

[0044] For example, such as Figure 6 As depicted in box 601, the entity monitoring module 501 can identify the infant 603 as the target entity and the camera sensor 605 as the first sensor. In this example, the monitoring statistics associated with the infant 603 generated by the camera sensor 605 may include, but are not limited to, infant movement, infant detection, infant crying, infant falling, infant playing, and infant sleeping. Figure 7a The document depicts exemplary inputs and outputs of the entity monitoring module 501. In one embodiment, a first sensor can determine that the target entity has moved out of the sensor's sensing range. For example, as... Figure 6 As depicted in box 607, while monitoring the baby's movement, camera sensor 605 can determine that the baby 603 has moved out of the camera sensor's sensing range 609. At this point, the baby 603 may be unattended and unmonitored, and the user may not receive any alerts associated with the baby 603. Therefore, the user may become continuously concerned about the baby 603's health, well-being, and protection.

[0045] The out-of-range detector module 503 determines one or more possible locations where the target entity may exist. The out-of-range detector module 503 also determines the monitoring behavior of the target entity. In an embodiment, one or more possible locations may be determined by using the last tracking data and the last remaining timestamp of the monitored entity to determine the direction of movement of the monitored entity, and by associating this with knowledge of a house floor plan associated with the smart home, wherein the knowledge includes detailed information about the layout and location of the various rooms within the smart home and information about edge devices installed at various locations within these rooms.

[0046] In this embodiment, the monitoring behavior of a target entity can be determined by sending monitoring statistics generated by a first sensor to a pre-trained model to obtain a monitoring behavior association score, and selecting monitoring behavior association scores above a specified threshold to determine a general set of monitoring behaviors of the target entity. In this embodiment, family alerts, notifications, and routines related to the next activity on the first sensor are aggregated, and the next activity of all family members is collected. Then, the relevance of the next activity and the user's current status are calculated based on the monitoring. Furthermore, pause events above a specified threshold are obtained. Pause events correspond to events that would not occur when entity monitoring is stopped. For example, continuous notifications about the activities of the monitored entity are provided.

[0047] Therefore, the output of the out-of-range detector module 503 produces the following results: determining one or more possible locations of the target entity, a general set of monitoring behaviors of the target entity, and a pause event. For example, when the baby 603 moves out of the sensing range 609 or camera sensor 605, the out-of-range detector module 503 may identify the kitchen or balcony as one or more possible locations, health, well-being, and protection as the general set of monitoring behaviors of the target entity, and the absence of user alarms and routines as a pause event. Figure 7b The diagram depicts exemplary inputs and outputs of the out-of-range detector module 503.

[0048] The entity monitoring continuity module 505 determines a second location from one or more possible locations where the target entity has moved, and determines a subset of second sensors that can be used to monitor the monitoring behavior of the target entity. The entity monitoring continuity module 505 determines one or more sensors present at the one or more possible locations, and identifies the second location from the one or more possible locations based on the monitoring of the physical environment by the determined one or more sensors. In an embodiment, monitoring the physical environment includes detecting changes in the physical environment around the at least one of the one or more sensors present at the corresponding possible location among the one or more possible locations.

[0049] Changes in the physical environment can be associated with the presence of the monitored entity at a corresponding possible location, and can correspond to changes in odor, taste, sound, light, temperature, humidity, movement, or any other change around at least one of one or more sensors that can trigger automatic actions or alarms. Upon detecting a change in the physical environment, the entity monitoring continuity module 505 can be configured to identify the possible location corresponding to at least one of one or more sensors as a second location. Furthermore, the entity monitoring continuity module 505 is configured to determine a subset of second sensors that can be used to monitor the monitoring behavior of the target entity. In an embodiment, the second sensor may include one or more sensors that may be present at the second location.

[0050] To determine the second location, the entity detection continuity module 505 may be configured to identify one or more sensors that may be present at one or more possible locations obtained from the out-of-range detector module 503. The entity detection continuity module 505 then identifies data values ​​from sensor information associated with the one or more sensors that may indicate the presence of a new entity. In an embodiment, past sensor data can be used to reduce background noise, thus allowing any new signal to be inferred as the presence of an entity using an appropriate threshold. The location of the sensor indicating the presence of the new entity can be identified as the second location. Furthermore, the entity detection continuity module 505 may determine the capabilities of a second sensor and the capabilities of a second sensor that may be present at the second location. For example, the second sensor may include a microphone with sound sensing capabilities in a smart speaker edge device and an inertial unit with vibration sensing capabilities in a tablet computer.

[0051] Furthermore, the entity monitoring continuity module 505 can retrieve executable relevant edge models based on the capabilities of the second sensor. In an embodiment, a database stored in memory 407 can be used to obtain a list of accessible edge model categories and the corresponding sensor data required for edge model inference. For example, for a microphone, sound detection models with categories such as, but not limited to, dog barking, baby crying, and falling can be used as relevant edge models.

[0052] Furthermore, the entity monitoring continuity module 505 can correlate model inference with monitoring behavior to determine the most relevant model inference. In an embodiment, a list of categories related to the capabilities of sensors present at a second location is obtained and passed through a predetermined neural network-based model type to determine the score of the monitoring behavior. Then, the category with the highest score for the monitoring behavior can be selected based on a specified threshold, and mapped to a superset of monitoring behaviors to filter out relevant model categories. Finally, the most relevant model inference for the corresponding sensor is selected. Therefore, the entity monitoring continuity module 505 can produce the following results: a second location, a second sensor at the second location, and a model inference associated with the second sensor. For example, a living room can be identified as the second location, and a microphone sensor and motion sensor in edge devices (such as smart speakers and smart air conditioners) present in the living room can be identified as the second sensor. Furthermore, sound detection models and corresponding categories (such as coughing, broken glass, and crying) can be determined. Similarly, motion detection models and corresponding categories (such as running and crawling) can be determined. In an example, as... Figure 6 As depicted in box 609, the kitchen can be identified as the second location, and the microphone in the smart speaker 611 located in the kitchen can be identified as the second sensor. Figure 7c The document describes exemplary inputs and outputs of the entity monitoring continuity module 505.

[0053] Edge pipeline interruptor module 507 invokes the edge model based on the capabilities of the second sensor and a subset of the monitored behavior of the target entity. In an embodiment, the availability of the edge device at the second location is obtained. A pipeline of the edge model is created using inputs and outputs, as well as serial and parallel models. Edge devices capable of running specific models are being determined. Based on the availability of the edge devices, the relevant edge model is invoked and loaded onto the appropriate edge device. The pipeline begins by inference using sensor data as input.

[0054] In an embodiment, the edge pipeline interruptor module 507 takes as input a second location determined by the entity monitoring continuity module 505, a second sensor at the second location, and a model inference associated with the second sensor. According to embodiments of this disclosure, the edge pipeline interruptor module 507 produces the following results: determining the edge distribution corresponding to the second sensor present at the second location, and a serial or parallel model for creating the pipeline, such as... Figure 7d As depicted. In the example, such as Figure 6 As depicted in box 613, the relevant edge model can be loaded onto the microphone in the smart speaker 611 or the smart refrigerator (not shown) and can be used to continue monitoring the target entity (i.e., the baby 603).

[0055] The alarm generator and edge modifier module 509 generates an alarm when unwanted activity is detected by the second sensor. In one embodiment, the alarm is sent to the user on the user device. In another embodiment, the alarm generator and edge modifier module 509 also determines the need to modify the edge model based on the entry of a different entity at the second location or the edge device at the second location becoming inoperable. In either case, the edge model can be modified by adding a model based on the behavior of a different entity or by removing the edge model corresponding to the inoperable edge device. Monitoring can continue via the modified edge model. In an example, such as Figure 6 As depicted in box 615, it can identify events of an infant crying and provide the user with an alert associated with that event.

[0056] Figure 8 This is a block diagram illustrating a method 800 for continuously monitoring an entity in an IoT environment according to an embodiment of the present disclosure. Method 800 includes, at operation 801, monitoring multiple behaviors associated with a monitored entity at a first location within the IoT environment via a first sensor present at the first location from a plurality of sensors associated with the IoT environment. In embodiments, the multiple behaviors associated with the monitored entity are determined based on at least one of monitoring history, at least one parameter associated with movement of the monitored entity, profile information associated with the monitored entity, and learned patterns of the monitored entity.

[0057] Then, method 800 includes: in operation 803, detecting, via a first sensor, that movement of the monitored entity exceeds a specified detection range of the first sensor. Then, method 800 includes: in operation 805, identifying a second location within the IoT environment associated with the presence of the monitored entity, wherein the second location exceeds the specified detection range of the first sensor. In an embodiment, identifying the second location includes: determining one or more possible locations exceeding the specified detection range of the first sensor, wherein the one or more possible locations are associated with the presence of the monitored entity. Then, the method includes: determining one or more sensors present at the one or more possible locations, and identifying the second location from the one or more possible locations based on monitoring of the physical environment by the determined one or more sensors.

[0058] In an embodiment, the operation of identifying a second location from one or more possible locations based on monitoring of the physical environment by one or more determined sensors includes: detecting changes in the physical environment around the at least one of the one or more sensors located at a corresponding possible location among the one or more possible locations, wherein the changes in the physical environment are associated with the presence of a monitored entity at the corresponding possible location. The method then includes: upon detecting a change in the physical environment around the at least one of the one or more sensors, identifying the corresponding possible location as the second location.

[0059] In one embodiment, the identification of a second sensor from multiple sensors is based on the association between one or more sensing capabilities of the second sensor and a subset of multiple behaviors of the monitored entity. In another embodiment, one or more possible locations are determined based on at least one parameter associated with the movement of the monitored entity tracked by a first sensor and pre-stored information associated with the IoT environment.

[0060] Subsequently, method 800 includes, at operation 807, determining, from a plurality of sensors, a subset of second sensors present at a second location for monitoring a plurality of behaviors of a monitored entity, based on pre-stored information associated with an IoT environment. In embodiments, the pre-stored information associated with the IoT environment may include one or more of the following: layout associated with a plurality of locations in the IoT environment, configuration information associated with each of the plurality of sensors, known locations previously visited by the monitored entity, and placement information associated with the presence of each of the plurality of sensors at the plurality of locations in the IoT environment, wherein the pre-stored information is stored in memory associated with the IoT environment.

[0061] Subsequently, method 800 includes: at operation 809, at a second location, monitoring a subset of multiple behaviors of the monitored entity based on one or more trained models selected from a plurality of trained models via a second sensor. In an embodiment, for monitoring, the method may include: selecting one or more trained models from a plurality of trained models based on the association between the subset of multiple behaviors of the monitored entity and one or more capabilities of the second sensor.

[0062] Based on the foregoing, this topic offers at least the following advantages: The methods described in the embodiments provide continuous monitoring to the user by understanding and determining the monitoring characteristics of interest to the user, using edge-distributed AI-based models and additional sensors available in other locations, even when sensors reach their range limitations. Furthermore, the methods described in the embodiments ensure that monitoring never stops unless intentionally stopped. This reduces user stress and alleviates the burden on each device to monitor exactly what it wants to monitor. Additionally, the methods described in the embodiments ensure that all smart devices in the user's home environment are utilized to the maximum extent, working not only when the user explicitly wants them to, but also when the situation requires input / data or processing from these devices. Moreover, the methods described in the embodiments help users reduce their monitoring costs because they do not have to purchase redundant sensors for their smart home environment and do not have to worry even in disadvantageous situations where there are fewer sensors. Finally, the methods described in the embodiments enable edge computing to run models not always on the edge environment that consumes data and computation. This disclosure makes it demand-based and therefore allows models to be scaled down to those only needed in the current scenario, thus helping to avoid edge bottlenecks.

[0063] While specific language has been used to describe the subject matter, no limitation is intended as a result. It will be apparent to those skilled in the art that various working modifications can be made to the method to achieve the inventive concept as taught herein. The accompanying drawings and the foregoing description provide examples of embodiments. Those skilled in the art will understand that one or more of the described elements can be well combined into a single functional element. Optionally, a particular element may be divided into multiple functional elements. Elements from one embodiment may be added to another embodiment.

Claims

1. A method (800) for monitoring an entity by a controller associated with an Internet of Things (IoT) environment, the method comprising: monitoring a plurality of behaviors associated with the entity present at a first location within the IoT environment via a first sensor of a plurality of sensors (801); identifying a second location within the IoT environment, wherein the second location is associated with presence of the entity and the second location is beyond a detection range of the first sensor (805); determining, based on pre-stored information associated with the IoT environment, a second sensor of the plurality of sensors present at the second location for monitoring a subset of the plurality of behaviors of the entity (807); and monitoring, based on one or more trained models, the subset of the plurality of behaviors of the entity at the second location via the second sensor (809).

2. The method of claim 1, further comprising: selecting the one or more trained models from a plurality of trained models based on an association of the subset of the plurality of behaviors of the entity with one or more capabilities of the second sensor.

3. The method of claim 1, wherein, The operation of identifying the second location comprises: determining one or more locations beyond the detection range of the first sensor, wherein the one or more locations are associated with presence of the entity; determining one or more sensors at the one or more locations; and identifying the second location from the one or more locations based on monitoring of a physical environment by the determined one or more sensors.

4. The method of claim 3, wherein, The operation of identifying the second location from the one or more locations based on monitoring of a physical environment by the determined one or more sensors comprises: detecting, by at least one sensor of the one or more sensors at a respective location of the one or more locations, a change in the physical environment around the at least one sensor of the one or more sensors, wherein the change in the physical environment is associated with presence of the entity at the respective location; and identifying the respective location as the second location upon detecting the change in the physical environment around the at least one sensor of the one or more sensors.

5. The method of claim 2, wherein, The second sensor is determined based on an association of one or more sensing capabilities of the second sensor with the subset of the plurality of behaviors of the entity.

6. The method of claim 3, wherein, The one or more locations are determined based on at least one parameter associated with movement of the entity tracked by the first sensor and the pre-stored information associated with the IoT environment.

7. The method of claim 1, wherein, The pre-stored information associated with the IoT environment comprises at least one of: a layout associated with a plurality of locations in the IoT environment, configuration information associated with each of the plurality of sensors, locations visited by the entity, or placement information associated with presence of each of the plurality of sensors at the plurality of locations in IoT environment, wherein the pre-stored information is stored in a memory associated with the IoT environment.

8. The method of claim 1, wherein, Before identifying the second location, the method comprises detecting movement of the entity beyond a detection range of the first sensor via the first sensor (803).

9. The method of claim 1, wherein, The plurality of behaviors associated with the entity are determined based on at least one of a monitored history, at least one parameter associated with movement of the entity, profile information associated with the entity, or learned routine of the entity.

10. A system (200A) for monitoring an entity in an Internet of Things, IoT, environment, the system comprising: a plurality of sensors (401); at least one processor (201) comprising processing circuitry; a memory (405) storing instructions that, when executed by the at least one processor individually or collectively, cause the system to: monitor, via a first sensor of a plurality of sensors, a plurality of behaviors associated with the entity present at a first location within the IoT environment; identify a second location within the IoT environment, wherein the second location is associated with presence of the entity and the second location is beyond a detection range of the first sensor; determine, based on pre-stored information associated with the IoT environment, a second sensor from the plurality of sensors that is present at the second location for monitoring a subset of the plurality of behaviors of the entity; monitor, at the second location via the second sensor, the subset of the plurality of behaviors of the entity based on one or more trained models.

11. The system (200A) of claim 10, wherein, To monitor, the instructions, when executed by the at least one processor (404) individually or collectively, further cause the system to: select the one or more trained models from a plurality of trained models based on an association of the subset of the plurality of behaviors of the entity with one or more capabilities of the second sensor, wherein the second sensor is determined based on an association of one or more sensing capabilities of the second sensor with the subset of the plurality of behaviors of the entity.

12. The system (200A) of claim 10, wherein, To identify the second location, the instructions, when executed by the at least one processor (404) individually or collectively, further cause the system to: determine one or more possible locations beyond a specified detection range of the first sensor, wherein the one or more possible locations are associated with presence of the entity; determine one or more sensors at the one or more possible locations; and identify the second location from the one or more possible locations based on monitoring of a physical environment by the determined one or more sensors.

13. The system (200A) of claim 12, wherein, the one or more locations are determined based on at least one parameter associated with movement of the entity tracked by the first sensor and pre-stored information associated with the IoT environment, wherein the pre-stored information associated with the IoT environment includes at least one of: a layout associated with a plurality of locations in the IoT environment, configuration information associated with each of the plurality of sensors, a location accessed by the entity, or placement information associated with a presence of each of the plurality of sensors at the plurality of locations in the IoT environment, wherein the pre-stored information is stored in a memory associated with the IoT environment.

14. The system (200A) of claim 10, wherein, prior to identifying the second location, the instructions, when executed by the at least one processor (404), singly or collectively, further cause the system to: detect, via the first sensor, movement of the entity beyond a detection range of the first sensor, wherein the plurality of behaviors associated with the entity are determined based on at least one of: a monitoring history, at least one parameter associated with movement of the entity, profile information associated with the entity, or learned routine of the entity.

15. One or more non-transitory computer-readable storage media storing one or more computer programs comprising computer-executable instructions that, when executed by at least one processor, cause a system to perform operations comprising: monitoring, via a first sensor of a plurality of sensors at a first location within an Internet of Things (IoT) environment, a plurality of behaviors associated with an entity present at the first location; identifying a second location within the IoT environment, wherein the second location is associated with a presence of the entity and the second location is beyond a detection range of the first sensor; determining, based on pre-stored information associated with the IoT environment, a second sensor from the plurality of sensors present at the second location for monitoring a subset of the plurality of behaviors of the entity; and monitoring, via the second sensor at the second location, the subset of the plurality of behaviors of the entity based on one or more trained models.

Images