Predictive commissioning and management of virtual sensors

Abstract

The present invention describes an apparatus and method that can be executed on a smart device such as a smartphone or smart hub and that can commission and / or manage a virtual sensor / actor by one or more of the following: collecting requirements for a virtual sensor / actor, determining a virtual sensor / actor based on existing sensors / actors (for example, by adjusting sensors / actors in one or more networks and / or by temporary sensor / actor and opportunity events), determining the parameters of a virtual sensor / actor given the measurements of existing sensors / actors, and exposing the parameters of a virtual sensor / actor.

Description

Technical Field

[0001] The present invention relates to a commissioning and / or management system applicable to local networks (such as home networks or home automation systems), sensor networks, or sensors, actuators, or other data generation devices arranged in extended reality / virtual reality (AR / VR) or metaverse applications that can be connected by different wireless connection technologies including short-distance and long-distance. Short-distance technologies include IEEE 802.15.4-based networks, Wi-Fi (registered trademark) networks, etc., and sensors may be connected by, for example, 3GPP (registered trademark)-based networks or long-distance networks (such as LoRa).

Background Art

[0002] Sensors are increasingly being used in local networks such as home networks and home automation systems. Sensors can measure not only temperature and light conditions, but also personal data such as respiratory rate, occupancy status, identity, blood pressure (mobile sensors), and all other types of medical data, movement patterns, etc.

[0003] Some actual devices can cooperate to create virtual sensors and actuators that sense or actuate one or more parameters of interest of a type different from the type they were designed for. Often, the parameter in question may exceed the sensing or actuating capabilities of any one actual device. As an example, such virtual sensors or actuators may be used in the audio area, such as virtual microphones or speakers, but in principle any type of parameter may be relevant. Virtual sensors are also used in industrial Internet of Things (IoT) applications adapted to predict / learn which physical sensors in a set of physical sensors can be appropriately replaced by a virtual sensor composed of some subset of the remaining sensors.

[0004] To create a virtual sensor from a set of physical sensors, known types of physical sensors at a location of interest must be deployed so that their data can be compared with data from the remaining sensors. Known systems provide such temporary or learned sensors by leveraging existing, already installed, permanent physical sensors and learning to emulate them using other sensors.

[0005] Optimal use of virtual sensors often requires processing data through machine learning models. However, a machine learning model trained in one scene (e.g., a scene with physical sensors) may not function natively when the activity is moved to another scene (i.e., a scene with new virtual sensors). For example, in a typical IoT scenario, data obtained from real or virtual sensors may be subject to further processing, particularly by machine learning approaches. A machine learning model may be trained on data from a certain context or scene, and then it may be desired to use it in a new context. In the example of a human activity recognition model that works with Wi-Fi® data, transfer learning can be applied as a technique to move such a trained model to a new scene by using only a few samples collected in the new scene. [Overview of the project] [Problems that the invention aims to solve]

[0006] The potential capabilities of virtual sensors depend not only on the installation environment but also on the properties of several different physical sensors, which can make deploying virtual sensors unintuitive for users. Known systems for learning potential virtual sensors typically require the presence of physical sensors at the location of interest, at least during the learning mode. However, this is not practical in smart home scenarios, as it is not possible to extend virtual sensors into areas not covered by physical sensors.

[0007] Furthermore, when (re)deploying physical sensors to optimize the deployment of virtual sensors, deploying / repositioning (existing) physical sensors to accommodate the maximum number or best number of virtual sensors to meet the needs of (third-party) applications installed in smart environments such as smart homes may not be intuitive for users.

[0008] Existing systems (e.g., speaker array calibration routines) provide means for testing or managing groups of similar physical devices that form part of a virtual sensor or actuator, but they cannot provide such capabilities for heterogeneous devices, heterogeneous networks (e.g., Wi-Fi® and Thread (802.15.4-based) networks, or multiple Wi-Fi® networks), or devices that need to operate in non-standard modes while forming part of a virtual sensor.

[0009] The present invention aims to provide a user-friendly system for creating potential virtual sensors within a sensor environment. [Means for solving the problem]

[0010] This objective is achieved by the apparatus, hub device, mobile device, system, method, and computer program product specified in the attached claims.

[0011] According to the first embodiment (for example, relating to a controller of a hub device (hub) or a mobile device (for example, a smartphone)), a device is provided for discovering, commissioning, or managing virtual data or action generating devices (210) in a network. This device, Collect information indicating at least one virtual data or action generation device, Send a request to an available auxiliary device to provide information. Based on the collected information and information about existing data or action generation devices, at least one virtual data or action generation device is determined. Based on measurements from existing data or action generation devices and information from available auxiliary devices, determine at least one parameter of at least one virtual data or action generation device. Outputs at least one determined parameter for at least one virtual data or action generation device.

[0012] According to a second embodiment, a network hub device (hub) is provided, comprising one or more physical data or action generation devices and the apparatus of the first embodiment.

[0013] According to a third embodiment, a mobile device (e.g., a smartphone) equipped with the apparatus of the first embodiment is provided.

[0014] According to a fourth aspect, a system is provided for discovering, commissioning, or managing virtual data or action-generating devices. This system comprises a network of one or more physical data or action-generating devices and a hub device according to a second aspect.

[0015] According to a fifth aspect, a method is provided for discovering, commissioning, or managing virtual data generating devices in a network. This method is A step of collecting information indicating at least one virtual data or action generation device, A step of determining at least one virtual data or action generation device based on the collected information and information about existing data or action generation devices, A step of determining at least one parameter of at least one virtual data or action generation device, based on existing data or measurements of an action generation device, The process includes the step of outputting at least one determined parameter of at least one virtual data or action generation device.

[0016] According to the sixth aspect, a computer program product is provided which includes coding means for generating steps of the method of the fifth aspect when executed on a computer device.

[0017] Accordingly, the present invention provides a system that can be used for the discovery, orchestration, commissioning, and management of virtual sensors / actors, which can sense events simultaneously identified in a network of physical sensors using temporary auxiliary sensors or existing physical sensors, thereby enabling the "discovery" of virtual sensors (or other data generating devices) or virtual actors (or other action generating devices) within an existing network of physical sensors or actors without requiring the deployment of additional sensors or actors, by serving as a foundation for emulation. The virtual sensors / actors can be orchestrated by a central hub without requiring user input by learning their orchestration requirements as part of the discovery process described above and applying them at runtime.

[0018] This enhances the user's commissioning of virtual sensors / actors, and therefore, the performance of virtual sensors / actors or downstream models can also be improved through appropriate gesture design.

[0019] Virtual sensors / actors can be proactively managed by the hub without necessarily requiring user input, by using gestures similar to those described above or by re-executing the sensor discovery process.

[0020] According to a first option that can be combined with any one of the first to sixth aspects, at least one virtual data or action generation device is determined by adjusting existing data or action generation devices in a network and / or other networks. Therefore, it is possible to control (e.g., orchestrate) existing devices in a network or other networks to obtain the information necessary to determine a desired virtual device.

[0021] According to a second option that can be combined with the first option or any one of the first to sixth aspects, the above adjustment can be achieved by time synchronization, frequency synchronization, optical color synchronization, and / or beamforming. Therefore, it is guaranteed that the necessary information is obtained at the appropriate place and at the appropriate timing.

[0022] According to a third option that can be combined with the first or second option or any one of the first to sixth aspects, information indicating at least one virtual data or action generation device is collected by sending an information request to available auxiliary devices and / or by sending an information request to available physical data or action generation devices. Thereby, appropriate existing auxiliary and / or physical devices in the network can be triggered to provide the information necessary to determine a virtual device.

[0023] According to a fourth option that can be combined with any one of the first to third options or any one of the first to sixth aspects, ground truth information (e.g., obtained from an auxiliary device) is compared with information collected from existing data or action generation devices, and a virtual device model for determining at least one parameter of at least one virtual data or action generation device can be estimated. Therefore, reliable ground truth information is advantageously used as a reference for estimating the virtual device model.

[0024] According to the fifth option, which can be combined with any one of the first to fourth options or any one of the first to sixth aspects, at least one virtual data or action generation device is determined by one or more primary data or action generation devices and opportunity events. Therefore, the determined virtual device can be adapted to the temporary device and opportunity events to optimize the function of the virtual device.

[0025] According to the sixth option, which can be combined with any one of the first to fifth options or any one of the first to sixth aspects, opportunity events are related to the actions or actions of the user or other people, and / or the actions or actions of the device, and ground truth sensing data is available for them. Thereby, a reference action or action can be selected to optimize the function of the virtual device.

[0026] According to the seventh option, which can be combined with any one of the first to fifth options or any one of the first to sixth aspects, at least one virtual data or action generation device is emulated by a set of existing physical data or action generation devices. Thereby, the new virtual device can utilize the output signals of the existing devices to provide without the need to add network devices.

[0027] According to an eighth option, which can be one of the first to fifth options or combined with one of the first to sixth embodiments, a transient external source of ground truth sensing data is provided by correlating it with available data streams resulting from data or action generating devices that are not part of the set of data or action generating devices used to provide at least one virtual data or action generating device, or from the use of at least one data or action generating device that is part of the set of data or action generating devices used to provide at least one data or action generating device, but is in a non-standard mode (e.g., high-energy mode or high-sampling-rate mode). This provides appropriate information for optimizing the determination of at least one parameter of the virtual device.

[0028] According to the ninth option, which can be one of the first to fifth options or combined with one of the first to sixth embodiments, the functionality of the virtual data or action generation device is enhanced or tested by designing at least one operation or action performed by the user and / or device to add ground truth data. Thus, the functionality of the virtual device can continuously adapt to changes in the network environment by providing updated ground truth data.

[0029] It should be understood that the apparatus of claim 1, the hub device of claim 12, the mobile device of claim 13, the system of claim 14, the method of claim 15, and the computer program product of claim 16 have similar and / or identical embodiments, in particular embodiments as defined in the dependent claims.

[0030] It should also be understood that preferred embodiments may be dependent claims with corresponding independent claims or any combination of the above embodiments.

[0031] These and other embodiments will become apparent from the embodiments described below and will be explained with reference to those embodiments. [Brief explanation of the drawing]

[0032] [Figure 1] Figure 1 schematically shows block diagrams of systems for discovering, commissioning, and managing sensors according to various embodiments. [Figure 2] Figure 2 schematically shows a process flow diagram based on the system in Figure 1. [Figure 3] Figure 3 schematically shows a flowchart of the virtual sensor discovery process according to the first embodiment. [Figure 4] Figure 4 schematically shows a flowchart of the virtual sensor commissioning process according to the second embodiment. [Figure 5] Figure 5 schematically shows a flowchart of the virtual sensor management process according to the third embodiment. [Figure 6] Figure 6 schematically illustrates an example of the commissioning process for a virtual sensor in an exemplary sensor environment. [Modes for carrying out the invention]

[0033] Embodiments based on sensor network systems will be described. Such systems include home networking systems, metaverse applications, or sensor systems for commercial, industrial, or public spaces (e.g., hospitals). Such networks may use long-range (e.g., cellular) communication technologies or short-range communication technologies such as Wi-Fi®, UWB, or Bluetooth®.

[0034] The following explanation describes a networked configuration, but please note that the same operations can be applied within a single device.

[0035] Throughout the following disclosure, “home network” or “sensor network” is understood as a network of sensors and actuators that facilitate a specific task (e.g., lighting or healthcare-related). This includes a home network hub (e.g., a data distribution entity) responsible for managing the home network, allowing multiple devices and nodes (e.g., sensors and actors) to connect to the network. The home network hub is also an entity that orchestrates secure data distribution (e.g., data originating from the network). The home network hub may include or provide access to a router device for linking the home network to an external network (e.g., the Internet), and / or may add or remove devices from the network.

[0036] A "sensor" or "sensor device" should be understood as a data-generating device (e.g., within a home network) capable of (wireless) sensing, positioning, and sensing of things like light, presence, sound, and images.

[0037] Furthermore, a “virtual sensor” or other “virtual device” is understood as a virtual entity instantiated by a certain type of software that processes what a physical sensor or other device would otherwise process. A “virtual sensor” or other “virtual device” learns to interpret relationships between different variables and observes measurements from different instruments. Virtual sensing technology is used to provide a viable and economical alternative to expensive and impractical physical measuring instruments. A virtual sensing system uses information obtained from other measurements and process parameters to calculate an estimate of the quantity of interest. These virtual devices may use data to gather information that cannot be measured by a single device. In this way, information that cannot be directly measured can be obtained.

[0038] Furthermore, "user" or "homeowner" is understood to refer to the person who owns the home network and, in most cases, the data collected / gathered / generated by the home network's sensors / actuators.

[0039] In addition, the term "metaverse" is understood to refer to a persistent shared set of interactable spaces where users can interact with each other in parallel with mutually perceived virtual features (i.e., augmented reality (AR)), or where those spaces are entirely composed of virtual features (i.e., virtual reality (VR)). VR and AR are sometimes commonly referred to as "mixed reality" (MR).

[0040] Furthermore, the term “data” is understood to refer to a representation of known or agreed-upon information in a known or agreed-upon form that is stored, transmitted, or otherwise processed. Information includes, in particular, one or more channels of audio, video, images, haptics, motion, or other forms of information, or environmental / personal characteristics (e.g., temperature / body temperature, heart rate, etc.), which may be synchronized or derived from sensors (e.g., microphones, cameras, motion detectors, etc.).

[0041] Furthermore, the term “events of opportunity” should be understood to refer to events whose location is known and for which a source of ground truth sensing data exists. A first example is an action performed by a user (such as walking in a defined location), which can be sensed by a specific device (such as a smartphone carried in the user's pocket) or a wireless receiver such as a Wi-Fi® access point or a 3GPP® access device such as a 5G gNB. Another example is an action performed by a robot (e.g., a drone, vacuum cleaner, or assistant) instructed to move in a predetermined direction, which can be measured and is known to the system. A further example is an action performed by a user / installer / robot during the installation phase, carrying non-standard (e.g., high-quality) hardware / sensors.

[0042] The term "ground truth" should be understood as information that is real or known to be true, provided not by inference, but by direct observation and measurement (i.e., empirical evidence). Therefore, "ground truthing" refers to the process of gathering appropriate objective (provable) data. For example, a stereo vision system is tested to see how well it can estimate 3D position. In this case, the "ground truth" is the position given by a laser rangefinder, which is known to be far more accurate than a camera system.

[0043] Furthermore, the term "gesture" should be understood as a related movement or action performed by a user or device, as sensed by a sensor network. Examples include standing in a specific location, making a physical gesture in a specific location, or walking to a specific location (for a user), and / or making a sound or performing a task in a specific place and time (for a device).

[0044] Over the past few years, the smart home market has grown significantly. The World Economic Forum estimates that in 2022, more than 130 million households owned smart home devices in a market once dominated by tech-savvy early adopters. Matter is a smart home standard developed in 2019 by Project Connected Home Over IP (Project Chip). It was officially launched in November 2022 and is maintained by the Connectivity Standards Alliance (CSA), formerly known as the Zigbee Alliance. The standard encourages interoperability between devices and platforms, allowing devices to work offline without requiring continuous access to the cloud or various cloud services.

[0045] As a result, services can be provided to owners using specific sensor data, allowing them to better control which devices and / or entities inside and outside the home are accessing data generated by other devices. Such services may include health monitoring, presence detection, lighting control, heating, ventilation, and air conditioning (HVAC) control. It is also conceivable that certain devices may perform specific actions based on data input from other devices without actually accessing the data themselves. Furthermore, it is conceivable that data generated in the owner's home may be analyzed / processed in the cloud when it is desirable for the cloud not to have access to the data. It may also be conceivable that certain devices can only have access to data if their attributes or context allow it. In such situations, access control architectures may not be sufficient.

[0046] Furthermore, data (e.g., derived from measurements) may be presented to the user or application in an aggregated / processed form. For example, if there are multiple light sensors in a room, the user may not be interested in knowing the measured values ​​of light from each light sensor (intensity, color, etc.), but may be interested in knowing the aggregated light value in the room (e.g., average light intensity). For example, if there are several physical sensors / actuators, it is feasible to create a virtual sensor / actuator. For example, the virtual actuator may be formed by multiple actuators, or the virtual sensor may provide a sensing output based on measurements from one or more physical sensors.

[0047] The following embodiments relate to a system for discovering, commissioning, and / or managing potential virtual sensors that can be created within a given environment, their potential properties, and any coordination or orchestration requirements, the system comprising one or more underlying sensor networks. Learning sensors or temporary sensors may be temporarily provided using other sensors that are known to be available or may be available, without burdening the user.

[0048] Throughout this disclosure, only the blocks, components, and / or devices related to the proposed data distribution functionality are shown in the accompanying drawings. Other blocks are omitted for brevity. Furthermore, blocks designated with the same reference number are intended to have the same or at least similar functionality, and therefore their functionality is not described below.

[0049] Figure 1 schematically shows block diagrams of systems for discovering, commissioning, and managing sensors (referred to as "sensor learning systems") according to various embodiments.

[0050] The system includes one or more sensor networks 20, each containing one or more (two or more in this example) sensors installed in a given space (such as a user's home network (e.g., a smart home)). The sensors are physical (i.e., hardware-based) sensors (PS) 220, and include all kinds of hardware capable of performing one or more sensing functions. The sensing function may be the primary function of the physical sensor 220 (e.g., a microphone installed to sense sound) or a derived function obtained by using physical sensor hardware installed for one or more purposes of sensing something in the environment (e.g., using wireless sensing from a Wi-Fi® / 3GPP® communication device to sense the movements of people nearby).

[0051] The physical sensor further includes appropriate network hardware for establishing communication with the hub 10 (via Wi-Fi®, Thread, Bluetooth®, etc.) or an access device.

[0052] Optionally, additional auxiliary sensors (AS) 230 may be provided, which may or may not be available. Examples of such auxiliary sensors include a user's smartphone or similar device that is not part of the permanently installed sensor network 20, physical sensors in non-standard modes (such as high-energy sensing modes, high-sampling-rate modes, or modes requiring high computational load) that would be very expensive to operate continuously as part of the standard operation of the sensing network 20, non-standard hardware (e.g., high-quality sensors used in specialized installation tools), and / or sensors integrated into mobile devices (e.g., robots).

[0053] Furthermore, the sensor network 20 may include one or more virtual sensors (VS) 210, which can be derived, for example, by fusing the outputs of multiple physical sensors 220, and which may be created based on a logical description of additional sensing data that is not normally available when considering the output of a single physical sensor 220.

[0054] In this embodiment, the physical sensor 220 has certain modes or behaviors required when used as part of the virtual sensor 210, and therefore requires orchestration functionality. For example, the physical sensor 220 needs to stop network communication while operating as part of the virtual sensor 210.

[0055] In this embodiment, the multiple physical sensors 220 include only one physical sensor but use two or more different sensing functions (for example, a virtual sensor is configured using both audio sensing from a single smart speaker and Wi-Fi® sensing).

[0056] In other embodiments, the multiple physical sensors 220 include multiple physical sensors located in different networks (e.g., Wi-Fi® or Thread) or systems (e.g., two nearby smart home systems).

[0057] Furthermore, the system in Figure 1 also includes physical actuators, auxiliary actuators, and virtual actuators ("action generating devices") having the same or similar characteristics as described above, except that in the case of actuators, "sensing data" should be replaced with "actual output." In this context, an actuator is understood to be a device that generates an action as an actual output in response to an electrical signal it receives. The action can be mechanical (e.g., linear or rotational motion) or an emission of something (e.g., light or heat). Therefore, the corresponding actuator network does not detect but instead generates an action. Thus, the teachings of the following embodiments can be transferred to such an actuator network.

[0058] Hub 10 is the central device of an Internet of Things (IoT) or smart system for controlling devices on the sensor network 20, such as Wi-Fi® access points and 3GPP® access devices, and for receiving their data. For example, the system uses multiple networks, as seen in Matter systems. Hub 10, for example, is implemented on a controller, bridge, and / or border router, either as defined in the Matter standard or in other ways. Hub 10 may be distributed across multiple devices. At least part of the hub logic can be provided to an edge server or cloud and made available locally through one or more local units of Hub 10. Alternatively, Hub 10 or part of its functionality may reside in a sensor / action generating device or be distributed across multiple such devices.

[0059] More specifically, the hub 10 includes one or more of the following: a computing unit, a network controller (NC) 110, a user interface (UI) 120, a location module (LM) 130, an event module (EM) 140, a measurement model (MM) 150, and a virtual sensor database (VS-DB) 160.

[0060] The network controller 110 is a hardware module that manages network communication with sensors (e.g., Wi-Fi®, Thread, Bluetooth®, 3GPP® Radio Access Network, etc.). It should be understood that the network controller 110 has sub-components, some of which may be related to functions other than simple network control. The network controller 110 controls the network / sensing / action functions of the physical sensor 220 (e.g., data communication, communication time, movement of the robot device to a specific location, use of specific sensing capabilities / parameters / modalities, synchronization of sensing from multiple devices, synchronization of sensing inputs from multiple devices, use of a given frequency, etc.), and the synchronization of multiple networks (e.g., Wi-Fi®, Thread) so that they do not interfere with each other. For example, when Wi-Fi® is used for sensing, the Thread network should not be used for communication so that Wi-Fi® sensing measurements are appropriately accurate and noise-free. Furthermore, the network controller 110 can receive sensing data from the physical sensor 220 and the auxiliary sensor 230.

[0061] The user interface 120 may be provided directly to the hub 10, or it may be located remotely and connected to the hub 10 via (local / cloud) communication with an external device such as the user's smartphone.

[0062] The location module 130 is a first software module that maintains and updates an approximate map of the space where the hub 10 and sensor network 20 are installed (such as a user's smart home).

[0063] In one example, the smart home space is obtained from the internet, where multiple sites maintain information about the existing property. In another example, the smart home space is learned by a (robot) device (e.g., by scanning and / or data retrieval). In yet another example, the location (e.g., of home objects) is learned by a camera or other image processing device. In yet another example, auxiliary objects are placed to track specific (home) objects.

[0064] The event module 140 is a second software module that calculates when an event of opportunity is detected by the sensor network 20.

[0065] One or more measurement models 150 are models that obtain (e.g., receive or acquire) sensing data from the sensing network 20 and derive meaning and context. An example is a machine learning model (e.g., a neural network) that receives sensing data related to human activity and derives information about the state and well-being of the stakeholders in question. These may run locally on the hub 10 or be accessed remotely from a cloud service.

[0066] The virtual sensor database 160 stores the parameters necessary to derive the virtual sensor 210 from the data of the physical sensor 220, along with the relevant operating mode, approximate location, and adjustment and / or orchestration requirements.

[0067] In this embodiment, assuming that data export is enabled by the corresponding function of the hub 10, if the hub 10 is modified based on the data stored in the virtual sensor database 160 (for example, by a change in the hub provider), the virtual sensors are retained. The virtual sensors may be exposed to application programs; that is, such programs can access or receive the output they generate. This means that the smart home system may be operated by different providers. Each provider can use a different hub 10 to enable smart home functionality (for example, through a general-purpose smart speaker or dedicated hardware). If a user decides to switch smart home providers, the virtual sensors used by the user and stored in the virtual sensor database 160 can also be moved. As an example, this can be achieved by including a list of virtual sensors and, for each virtual sensor, information about the structure of that virtual sensor (for example, in the virtual sensor database 160).

[0068] Furthermore, the system in Figure 1 includes a gesture design system 30 for designing gestures (hereinafter, "design" also includes "exchange"), which may run on the hub 10 or remotely.

[0069] The gesture design system includes at least one of the following: a commissioning gesture module (CG) 310 for designing commissioning gestures, a management gesture module (MD) 320 for designing management gestures, and a gesture database (G-GB) 330 for storing commissioning gestures and management gestures. The commissioning gesture module 310 designs gestures that a user can use to commission a virtual sensor 210, while simultaneously improving its performance. Such commissioning gestures are used by a (robot) device to commission the virtual sensor 210, which is steered by the hub 10 or an external party. The commissioning gestures are linked to an event and / or measurement model 150 of the opportunity associated with the commissioned virtual sensor 210.

[0070] The management gesture module 320 designs gestures used to manage or test the virtual sensor 210. These may be the same as commissioning gestures, but may be specifically designed to facilitate input or may occur naturally as a continuous virtual sensor testing function.

[0071] Figure 2 schematically shows a flow chart of the sensor detection process based on the system in Figure 1.

[0072] After the procedure begins (S), the user indicates to the hub 10 that a virtual sensor setup is required. In response, the hub enters discovery mode and broadcasts, multicasts, or multicasts sensing data requests (SD-REQ) to available auxiliary sensors 230 and sensing data requests to available physical sensors 220, along with mode switching requests (MS-REQ) and orchestration control information (ORCH).

[0073] Furthermore, the hub 10 provides the event module 140 with information regarding the timing of the discovery mode (T-DM).

[0074] The physical sensor 220 obtains / receives arbitrarily selected virtual sensing data (VS-DM) from the virtual sensor 210.

[0075] The auxiliary sensor transmits / transfers location data (LD) to the location module 130. The location module 130 generates matching sensing data and location streams (M(SD,LS)) and transfers them to the event module 140. Furthermore, the event module 140 and the hub 10 receive ground truth sensing data (GT-SD) from the auxiliary sensor 230.

[0076] Furthermore, the physical sensor 220 transmits sensing data (SD) and location information (L) to the event module 140, and the hub 10 receives information from the physical sensor 220 regarding physical sensor sensing (PS-S) and orchestration (ORCH), as well as mode switching data (MS-D).

[0077] The event module 140 determines an opportunity event (E-OPP) based on ground truth sensing data, discovery mode timing, and at least one of sensing data and location provision, and transmits information about such opportunity events to the hub 10.

[0078] Hub 10 compares ground truth data with physical sensor data, estimates a virtual sensor model, and stores information about the estimated sensor model (E-VSM) in the virtual sensor database 160.

[0079] Additional modeling information (MI) of your choice is provided from among 150 available measurement models.

[0080] In one embodiment, the construction of a virtual sensor model is based on training with time-aligned sensor data (e.g., from all sensors) collected over a period of time and stored in a data repository. In one example, the virtual sensor model is trained to generate desired sensor data (as output) from physical sensor data (as input). The virtual sensor model may be a convolutional or trans-based neural network model trained with mean squared error (MSE) for the desired output, or it may be a very simple model such as a linear regression model.

[0081] Since most of the advantages of the proposed concept can be derived from simple models, even if more complex models are not as good, the complexity of the virtual sensor model implementation can be adapted to the available computational circumstances.

[0082] Edge artificial intelligence (edge ​​AI) is the implementation of artificial intelligence in an edge computing environment. In other words, AI computations are performed at the edge of a given network, typically on the device where the data was created, rather than in a central cloud computing facility or off-site data center. Given the availability of edge AI chips, training relatively small neural networks can be done relatively quickly and easily. This can be based on a continuous online training process.

[0083] Figures 3 to 5 schematically illustrate the respective flowcharts of the virtual sensor discovery, commissioning, and management processes in various embodiments, using the components of the sensor discovery, commissioning, and management system shown in Figure 1. The steps in these flowcharts are executed or initiated, at least in part, by instructions in the respective software programs / routines that control the hub controller (e.g., the network controller 110 in Figure 1) or the remote network device.

[0084] Note that not all steps in Figures 3-5 are always necessary. Furthermore, some or all steps may be performed at once or in multiple steps, and in a different order.

[0085] More specifically, the processes of discovering, commissioning, and managing virtual sensors are described based on the relevant first to third embodiments. A common feature is the use of opportunity events occurring in the scene to enhance the hub's ability to discover or manage virtual sensors, or (in the commissioning embodiment) using opportunity events specifically designed to enhance the performance of the virtual sensor at the moment of commissioning as an "output" (i.e., gesture).

[0086] Therefore, these three embodiments relate to the discovery and orchestration of virtual sensors, the commissioning of virtual sensors, and the management of virtual sensors.

[0087] Figure 3 schematically shows a flowchart of the virtual sensor discovery process according to the first embodiment.

[0088] Virtual sensor discovery includes one or more of the following steps shown in Figure 3. The order may differ from that shown, and the steps may be repeated.

[0089] In step S301, the user indicates that they want to discover potential virtual sensors in the system, which may include one or more physical sensors that are already committed to and managed by the hub (H), or this is communicated to the hub via a third-party application, or the hub runs a program that proposes the creation of virtual sensors that would be beneficial to the user or one or more third-party applications (e.g., the hub, or applications running on a smart device connected to the system (e.g., a smartphone)) in using the system, or the (user) application triggers the discovery / creation of virtual sensors (e.g., a new IoT application triggers a request to create a useful virtual sensor), or the operating system (OS, e.g., iOS or Android®) triggers the discovery / creation of virtual sensors so that they are available to third-party apps.

[0090] In response, the hub enters discovery mode (DM).

[0091] For example, a user request could be a one-time input requesting a virtual sensor at a specific location via a user interface, or a persistent desire to discover potential virtual sensors in all or specific locations within a scene, or in a given part of a scene. The user may or may not specify the desired type and location of the virtual sensors. If the user specifies a desired location, the subset of physical sensors used will be limited to those available at that location.

[0092] In another example, a discovery request is initiated by a third-party application ("app"). For instance, a user might access an "app store" or equivalent interface to download an app corresponding to a virtual sensor, or the hub might provide sensing data from physical sensors abstracted into specific types of virtual sensors for use by third-party apps. For example, a "smart home app store" might contain many third-party virtual sensor apps (these could be different types of sensors, such as "fall detection" or "voice control microphone," or sensors with different performance levels, such as "basic microphone" or "high-sensitivity microphone"). Each virtual sensor app contains a (hidden) list of hub requirements that must be met to install each app. In this case, a user installing a new app, or a request for new abstracted sensing data from an installed app, constitutes a request for the hub to discover the appropriate virtual sensor.

[0093] In step S302, the hub enters discovery mode and requests and receives sensing data from all or some of the installed physical sensors, as well as from auxiliary sensors (such as the user's smartphone) if available (e.g., via the network controller).

[0094] If necessary, the hub instructs at least some physical sensors to enter a specific mode of the discovery phase. Examples include high-sensitivity modes (e.g., using a high sampling rate, using high input energy for active sensing (such as certain types of radio frequency (RF) sensing), and / or sending raw sensing data to the hub instead of locally processed data (standard operating mode)), scan modes (e.g., performing some type of beamforming for wireless sensing to maintain / sens a particular area, or injecting audio into a particular area and / or listening to that area), or non-standard modes (e.g., using a physical sensor that is not sensing for a primary task to temporarily provide sensing input; for example, a device that would otherwise not have sensing capabilities (such as a smart light bulb) (e.g., a light bulb with a Bluetooth® interface can sense the presence of a particular person carrying a smartphone (and running Bluetooth®), or a Wi-Fi® sensing-capable light bulb can sense the presence of a person performing an action as a presence sensor, or sense an event using network hardware that temporarily produces an abnormal output (e.g., the output properties of a smart light temporarily match those required by a nearby camera to enhance sensing)).

[0095] In another example, the hub provides enhanced coordination / orchestration to physical sensors during the discovery phase (and this can be replicated in the use phase if it improves performance).

[0096] According to the first embodiment, the hub orchestrates communication tasks and sensing tasks to avoid interference between, for example, wireless communication from nearby devices and RF sensing data from physical sensors constituting a virtual sensor. Such orchestration can be performed between physical sensors themselves or between physical sensors and other devices outside the sensor network. Furthermore, such orchestration can be performed between different wireless protocols (e.g., between Wi-Fi® and IEEE 802.15.4), between different networks implementing the same protocol (e.g., between two nearby Wi-Fi® networks to which the hub is registered or accessible), and / or between adjacent smart home systems by coordinating adjacent hubs.

[0097] In further embodiments, some forms of RF sensing require input from communications of nearby devices. For example, one method of human activity recognition using Wi-Fi® involves using one device to generate traffic from an access point (AP) and another device to detect channel status information (CSI) using the signals returned from the AP. In such embodiments, a hub orchestrates one or more physical sensors (e.g., by transmitting orchestration control information) to sense the CSI, while another device (which may be inside or outside the sensor network) generates known traffic from the AP.

[0098] According to the second embodiment, the hub orchestrates between sensing tasks and energy management tasks. For example, the hub restricts physical sensors from entering low-power or sleep states while they are being used as part of a virtual sensor (for example, by transmitting orchestration control information) (and consequently, also during the discovery phase).

[0099] According to a third embodiment, the hub orchestrates sensing tasks. For example, to prevent sensing-related emissions from an active sensor from interfering with sensing from a nearby passive sensor, the hub can instruct the first sensor to stop sensing for a specific period of time (e.g., by transmitting orchestration control information) or coordinate sensing activities with the second sensor to achieve a common time base.

[0100] The first to third embodiments described above can also be optionally applied to auxiliary sensors. In the case of auxiliary sensors, an external party (e.g., the installer) can control the use of the auxiliary sensors and / or the hub. Therefore, the external party managing such auxiliary sensors can also request and receive sensing data through the hub.

[0101] The details of the non-standard modes and orchestration described above can be stored in the virtual sensor database.

[0102] In step S303, the hub is controlled to identify opportunity events (E-OPPs) that are sensed by the installed physical sensors. In this case, the location of the event is known and / or a source of ground truth sensing data exists, and a temporary physical sensor is constructed using physical sensors or auxiliary or mobile sensors (if available).

[0103] To achieve this, the hub collects data from physical sensors (e.g., cameras) and derives suggestions for the locations of potential virtual sensors or temporary sensors. For example, a user might use their mobile phone camera to record a room and record objects of interest in that room (e.g., plants, tables, doors, windows, etc.) as potential candidates for virtual sensor placement. For instance, a plant might benefit from a virtual sensor tracking whether it needs watering, a table from a virtual sensor tracking whether someone is sitting on it and whether there is sufficient lighting, and a door from a virtual sensor tracking whether it is open or closed and whether someone is indoors. The hub then derives the locations of these potential virtual and temporary physical sensors and provides suggestions for them.

[0104] The hub, given a set of third-party applications installed on the user's smartphone, provides only suggestions relevant to the user.

[0105] In step S304, the hub compares (COMP) sensing data (SD) recorded by physical sensors during the discovery period with ground truth sensing data (GT-SD) at the location of interest to identify which subset of physical sensors can best emulate ground truth data from temporary physical sensors or potential virtual sensors. More specifically, the hub's event module attempts to identify an event of opportunity (i.e., an event sensed by installed physical sensors, for which the location of the event is known and a source of ground truth sensing data exists) by comparing sensing data from physical sensors with (ground truth) sensing data from auxiliary sensors (and / or other physical sensors).

[0106] For example, the location of an opportunity event is derived by the Location module, for instance, by one or more of the following: i. While a source of the user's location exists (e.g., from an auxiliary sensor such as the user's smartphone), sense events within the entire area of ​​interest (e.g., the entire home network space), and correlate (almost) simultaneously with sensing data of similar sensing types, using sensing data of similar locations (i.e., from physical sensors) and sensing data of known locations (e.g., from a smartphone). ii. Sensing events at a specific location (for example, when a user requests a virtual sensor at a known location). In this case, the location module filters sensing data from physical sensors and any auxiliary sensors according to location estimation (for example, from an existing map). iii. Instruct the user to place an auxiliary sensor (such as the user's smartphone) in a specific location. iv. Instruct the user to place auxiliary objects in a given location, such as an RF or optical reflective surface (e.g., metamaterial) on an object (e.g., a door, window) that reflects or modifies RF or optical signals when the object moves, or a QR code® that determines the location / area / home object to be sensed by a smart home system or hub, thereby allowing, for example, beamforming to be trained / executed toward that area. For example, a QR code® can be attached to a plant that needs to be monitored by a camera so that the camera recognizes the plant. In this way, a virtual plant monitoring sensor can be created to indicate when the plant needs watering.

[0107] In another example, an event of opportunity is identified by one or more linked data from auxiliary sensors and / or objects known to correlate well with the event type (e.g., a smartphone correlates well with the event type "user is walking"), the location of a potential virtual sensor is suggested to the user, the linked data from several physical sensors appear to have similar characteristics, and the specific event is actively output to the user in the form of a gesture (described later in relation to the second embodiment in Figure 4).

[0108] In a further example, ground truth sensing data is derived using auxiliary sensors, auxiliary objects (which may be provided or placed within the target environment), and / or nearby physical sensors that are temporarily in non-standard mode (as described, for example, in relation to step S302 above) to derive the resulting sensing data that best correlates with the event of the opportunity.

[0109] In yet another embodiment, the event originates from a device or a user.

[0110] In yet another embodiment, for example, before the process proceeds to step S304, during the time of the opportunity event, the ground truth sensing data is combined with the location data.

[0111] If the emulation is enhanced (ENH) by adjustments between physical sensors or the use of non-standard operating modes, this is also calculated by the hub in step S305.

[0112] In step S306, the hub optionally further identifies an alternative subset of physical sensors (ALT PS) that can emulate the transient physical sensors with lower (but still useful) accuracy.

[0113] In one example, the hub compares data recorded by physical sensors (e.g., when operating in "standard" mode) with ground truth sensing data related to the event of the opportunity. The hub identifies which subset of physical sensors can best emulate ground truth data from auxiliary sensors or temporarily non-standard physical sensors (e.g., using the techniques in Reference 1) and / or utilizes auxiliary objects.

[0114] If the emulation does not reach a given threshold, the hub may provide instructions to (re)position existing physical sensors (for example, the hub may suggest alternative locations where sensors (e.g., speakers or presence / light sensors) can be placed) and / or place auxiliary objects.

[0115] The goal of step S306 is to identify a subset of physical sensors that can best emulate ground truth data (i.e., that of auxiliary or transient physical sensors) during the time of an opportunity event when operating in a known mode (preferably a standard operating mode, but optionally a non-standard mode). This requires the hub to physically operate the physical sensors in several modes (i.e., their standard mode and associated non-standard modes) during the discovery phase and compare the differences in emulation fidelity. As part of this, the hub attempts to provide the physical sensors with additional orchestration that may improve the fidelity of the emulation and measure whether this actually improved the emulation. Details of such orchestration are stored in a virtual sensor database.

[0116] In a further embodiment, the hub uses a machine learning approach (e.g., the approach in Reference 1) to identify a subset of physical sensors that best approximate ground truth sensing data during an opportunity event.

[0117] Optionally, a score can be applied to indicate how closely a subset of physical sensors emulates ground truth data, and the estimation performance of that subset can be ranked. The hub then identifies several different subsets and their estimation performance. Further options include assigning weights and / or other factors related to the necessary transformations that should be applied to the physical sensor sensing data when emulating opportunity events.

[0118] In step S307, the properties and locations of temporary physical sensors or potential virtual sensors are stored in a database (e.g., a virtual sensor database) and matched with a subset of physical sensors, their operating modes, and adjustment requirements necessary to emulate them. This mapping then constitutes a newly discovered virtual sensor.

[0119] In one example, one or more of the following are stored in the virtual sensor database: a subset of physical sensors, the relevant types and locations of events for emulated opportunities, estimated fidelity, and orchestration details.

[0120] This stored information can be used by the hub to operate virtual sensors during future use, for example, by applying the relevant operating modes and orchestration modes, along with their associated weights, to a subset of identified physical sensors.

[0121] In step S308, the hub presents the discovered virtual sensors and their estimated performance to the user via a user interface (UI) (e.g., user interface 120 in Figure 1), and the user selects to commission the virtual sensors located at a desired (given) location. In one example, the hub presents the user with the location, type, and estimated performance of the virtual sensors.

[0122] In one example, the hub hides the physical sensors and provides only virtual sensors.

[0123] The commissioning of virtual sensors (i.e., granting them privileges within the network related to their type and use) can be achieved by any standard means (e.g., established commissioning procedures within the applicable smart home environment) or by the commissioning process described later with reference to Figure 4.

[0124] According to implementation examples, users may want to extend their smart home with an alarm system. For this purpose, the user installs an alarm actuator. Instead of adding (actually) sensors to doors / windows, the system suggests virtual sensors to the user, along with locations to install these virtual sensors on the doors / windows. The virtual sensors determine, for example, whether a door is open or closed, by wireless sensing, for example by using RF sensing from multiple devices in the smart home, and / or by a camera or other video device. Upon detection of a virtual sensor on a door, the user is prompted to close or open the door, or to initiate another action that causes a change in RF propagation, and thus can determine which door or window is open or closed. If the change in RF propagation is insufficient, the user may be prompted to attach, for example, RF reflective material (e.g., a transparent sticker) to the door or window.

[0125] Figure 4 schematically shows a flowchart of the virtual sensor commissioning process according to the second embodiment.

[0126] The process described above in the first embodiment with reference to Figure 3 is, in particular, aimed at discovering potential virtual sensors or transient (learning) sensors and their performance.

[0127] However, if the opportunity events used in the discovery phase are not sufficiently identified, or if they do not closely match the typical events that the virtual sensor is expected to sense, the performance of the virtual sensor may not yet be optimal.

[0128] Therefore, an additional commissioning process is described to use the moment of sensor commissioning as an additional event of opportunity, which is intelligently designed by the hub and can be used to improve the performance of the virtual sensor. The commissioning process includes one or more of the following steps. These steps may be repeated or performed in different orders.

[0129] In step S401, the user indicates that they wish to commission a virtual sensor (COMM(VS)) via the user interface and agree to use, for example, gesture-based commissioning. The hub retrieves the relevant virtual sensor model from a database (e.g., a virtual sensor database).

[0130] In step S402, in response to user instructions, the hub accesses the gesture design system to design a commissioning gesture (CG) to be performed by the user. This gesture relates to a type or class of event that is not sufficiently included in the observed opportunity events or is not adequately modeled in the existing sensor data model, and is therefore expected to add value to the performance of the virtual sensor.

[0131] More specifically, a gesture design system can design one or more commissioning gestures based on several factors, including the following cases:

[0132] According to the first case, the design gesture relates to a type or class of event that was not sufficiently included in the observed opportunity events during sensor discovery, or that had noisy or incomplete ground truth sensing data. In this case, the gesture is designed to prompt the user to provide such input, and thus data from the virtual sensor during a known period of time, prompting via the user interface, can be associated with that action with high certainty (e.g., with certainty above a predefined threshold). As an example, a virtual sensor sensing walking motion at a particular location may have been discovered, but the opportunity events during sensor discovery did not include many samples of the user walking slowly. An appropriate commissioning gesture would be to instruct the user via the user interface to go to the relevant location and walk slowly.

[0133] According to the second case, the designed gesture relates to an action or event that cannot be directly sensed by a physical or virtual sensor, but instead relies on data processing using a selected measurement model. In this case, the commissioning gesture is designed to function as an additional input to the measurement model, thereby improving its ability to transfer to a new scene (e.g., as in Reference 2). Furthermore, in this case, the commissioning gesture consists of a representative example of an event of interest that the user is instructed to perform in the new “scene” (i.e., where the virtual sensor is expected to sense). An example of this case would be a virtual sensor commissioned to enable Wi-Fi® sensing-based gesture control of smart lights in a new room, using an existing gesture detection model (measurement model) trained on data acquired in another room. The commissioning gesture designed here could be the same as the intended on / off gesture used for the lights in the existing room, but performed in the new scene.

[0134] In both of the first and second cases described above, the commissioning gesture is designed to be performed by a trusted device (i.e., an already commissioned device) rather than by the user.

[0135] In step S403, the user is instructed by the hub's user interface (UI) to perform a gesture if they wish to commission a virtual sensor. During this time, the user's actions are sensed by both the virtual sensor and any auxiliary sensors (if available).

[0136] In one example, a designed commissioning gesture is communicated to the user via the user interface of a hub (or a trusted device via a network controller), and the user can choose to perform it.

[0137] Virtual sensors (and optionally auxiliary sensors) are used to sense user or device actions completing commissioning gestures. Sensing data from a short period after a prompt is given to the user via the user interface is labeled as most likely to be related to the prompted event.

[0138] Next, the obtained sensing data is used as ground truth sensing data for the opportunity event and is compared individually with the sensing data from each physical sensor. The virtual sensor emulation calculation process of the first embodiment shown in Figure 3 is re-executed for this new opportunity event. In this comparison, while the user performs a commissioning gesture, the hub must alternate between emulating the virtual sensor using the physical sensors (using previously learned weights, settings, and orchestration requirements) and using each physical sensor individually. This alternation can be done quickly in time to "sample" the same event for both sensor sets.

[0139] If the comparison yields a different optimal subset of physical sensors (or different optimal weights or orchestration requirements) than previously calculated, the hub updates the virtual sensor model in the virtual sensor database.

[0140] If commissioning gestures are used to enhance downstream models, the model management function is updated with new training data, and the models are retrained.

[0141] In step S404, if an action is sensed, the virtual sensor (VS) is commissioned (COMM) (i.e., authorized to operate permanently within a network with the appropriate authority).

[0142] In step S405, sensing data (SD) related to the user performing a commissioning gesture is stored by the hub (H) and used to improve the performance of the virtual sensor (this includes changing a subset or weighting of the physical sensors in use, and / or transferring training data related to downstream models into the virtual sensor context).

[0143] Once the user completes the gesture, the virtual sensor is commissioned with the updated parameter / orchestration details obtained above. If the user does not complete the commissioning gesture, or if the expected commissioning gesture is not sensed with sufficient fidelity, the user is prompted to complete the commissioning gesture again via the user interface, or is allowed to commission the virtual sensor in a different way (although this may result in reduced performance).

[0144] Alternatively, instead of using gestures, the user may select one of the proposed virtual sensor locations determined in the previous embodiment (for example, via the hub's user interface).

[0145] Furthermore, instead of using gestures, users may place a QR code (registered trademark) or an auxiliary object where they want to place the virtual sensor.

[0146] Furthermore, an external device may drive the generation of commissioning gestures (or alternative approaches) and exchange that data with the hub.

[0147] In the above case, steps S402 to S404 of the process in Figure 4 are applied in accordance with the acquisition of commissioning gestures.

[0148] Figure 5 schematically shows a flowchart of the virtual sensor management process according to the third embodiment.

[0149] Testing and / or management are essential to improve the performance of virtual sensors throughout their service life. This is especially true when physical sensors are added to or removed from a sensor network, or when their location / orientation changes. Therefore, a process similar to that of the second embodiment is described below to identify or create events that present opportunities for managing virtual sensors. This management process includes one or more of the following steps. These steps may be repeated or performed in different orders.

[0150] Step S501 indicates the user and / or hub's requirements for managing or testing already commissioned virtual sensors. To achieve this, the hub (H) enters management mode (MM).

[0151] For example, a hub identifies the need for virtual sensor management (e.g., a new physical sensor has been added to the network or an already installed sensor has been removed) or a change in state. In response, the hub enters management mode. For example, this identification process may include collecting information from sensors and monitoring changes, collecting cross-measurements from sensors to identify changes in the location of one of them, collecting sensor behavior to identify changes in the location of one of them, collecting system events (e.g., a new device becomes available, a device becomes unreachable), and / or exchanging functions with devices (functions that can be used to track the behavior of the device itself or other devices).

[0152] Next, the hub accesses the gesture design system to design one or more management gestures (or alternatives used in other embodiments). These management gestures are provided to the user or device and are designed to optimally test the functionality of the virtual sensor. This process may also be driven by a third device that transmits the selected management gestures to the hub.

[0153] When a management gesture is performed by a user, the management gesture consists of the most common or most reliable action or event that the virtual sensor would sense. This maximizes the likelihood of detection even if the virtual sensor's performance is degraded by the removal of a physical sensor.

[0154] When a management gesture is performed by a device, the management gesture consists of a list of outputs (subject to the feasibility of the target device that generates the outputs), where and / or when the device should perform the management gesture.

[0155] In step S502, the user or device is instructed to perform a management gesture (MG) (which may also be the same as a "commissioning gesture") at a known location and time.

[0156] In one example, management gestures are communicated to the user via the user interface or to the device via the network controller.

[0157] In step S503, the management gesture is sensed by a virtual sensor, and the sensing data is compared to what would be expected if the virtual sensor were functioning correctly.

[0158] While a user / device is performing a management gesture, the hub uses such events as opportunity events and compares the sensing data associated with them using the currently commissioned virtual sensors and one or more of the physical sensors individually (and auxiliary sensors if available). The hub then updates the virtual sensor model as in the second embodiment described above.

[0159] In one example, the hub informs the user via the user interface that the virtual sensor has been updated.

[0160] In step S504, if no (or sufficient) matches are found, the system re-enters discovery mode to attempt to enhance the virtual sensors by finding an improved subset of the physical sensors and / or an improved operating mode for an existing subset of the physical sensors.

[0161] For example, if the performance of a virtual sensor is significantly reduced due to the removal of a physical sensor, the user will be warned via the user interface and, optionally, will be instructed on which physical sensors need to be reinstalled to restore the virtual sensor performance as needed.

[0162] As an alternative to the gesture-based management described above in the third embodiment, the hub may instead use opportunity events to periodically re-execute the virtual sensor discovery process of the first embodiment.

[0163] Figure 6 schematically illustrates an example of the commissioning process for a virtual sensor in an exemplary sensor environment.

[0164] The thick black lines represent the building walls of the sensor environment (such as a home network). Shaded dots represent physical sensors, while solid dots represent identified potential virtual sensors.

[0165] In step S1, a network of physical sensors (circular areas with diagonal lines) senses data from the scene and runs a machine learning model. In step S2, the hub identifies temporary sensors (e.g., the user's smartphone) at known locations and uses these temporary sensors to predict potential virtual sensors (black circular areas), their locations, and properties, and displays options to the user (e.g., via the smartphone display). To commission the virtual sensors, in step S3, the user is instructed to perform some action (commissioning gesture) at their intended physical location, which is likely to enhance the transfer of the model to the new scene. White circular areas represent auxiliary sensors or unregistered physical sensors.

[0166] In the above embodiment, the application for detection, commissioning, and / or management specifies one or more policies that define a specific type of virtual sensor by specifying the requirements of the actual sensor. The hub then uses this information to map the sensors and (smart) home deployed within the system to determine their locations and provide the user with location-related indicators.

[0167] Policies describe how system behavior is managed. Policies express higher-level objectives that are automatically enforced. Predefined policies allow systems to dynamically adjust their behavior at runtime without requiring user intervention. Many policy languages ​​are defined. For example, the system supports the Adaptive and Programmable Policy Environment and Language (APPEL) as a policy language.

[0168] Policies can be programmed in a chosen policy language, or a policy wizard can be designed to allow non-computing users to create and edit policies. The wizard can be web-based, enabling policies to be defined anywhere using a familiar interface.

[0169] In summary, a device / method is provided that can commission and / or manage virtual sensors / actors, provided on a smart device such as a smartphone or smart hub, by one or more of the following: collecting requirements for virtual sensors / actors, determining virtual sensors / actors based on existing sensors / actors (e.g., by coordinating sensors / actors in one or more networks and / or by temporary sensor / actor and opportunity events), determining the parameters of virtual sensors / actors given the measurements of existing sensors / actors, and exposing the parameters of virtual sensors / actors.

[0170] While embodiments have been described in the context of home networking and the metaverse, their applications are not limited to such types of operation. Embodiments can be applied to devices or networks of devices where multiple sensors exist and some correlation occurs between sensor outputs, such as in healthcare, retail stores, shopping malls, and sports clubs. Therefore, embodiments can be applied not only to single multi-sensor devices such as smartphones, but also to IoT or 5G networks, etc.

[0171] Furthermore, at least some of the embodiments can be applied to various types of UEs or terminal devices such as mobile phones, vital sign monitoring / telemetry devices, smartwatches, detectors, vehicles (for vehicle-to-vehicle (V2V) communication or more commonly vehicle-to-vehicle / vehicle-to-infrastructure (V2X) communication), V2X devices, Internet of Things (IoT) hubs, IoT devices (such as low-power medical sensors for health monitoring, medical (emergency) diagnostic and treatment devices for use in hospitals or first responders), and virtual reality (VR) headsets.

[0172] Other variations of the disclosed embodiments will be understood and implementable by those skilled in the art in carrying out the claims, from the examination of the drawings, disclosures, and appended claims. In the claims, the word “equipped with” does not exclude other elements or steps, and the singular form does not exclude the plural. A single processor or other unit may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously. The above description details certain embodiments. However, it will be understood that no matter how much the above description is detailed in the text, these embodiments can be carried out in many ways and are therefore not limited to the disclosed embodiments. Notwithstanding, the use of certain terms when describing certain features or aspects should not be construed as implying that the terms are redefined herein to be limited to including any particular features of the features or aspects to which the terms relate. Furthermore, the expression “at least one of A, B, and C” shall be understood as disjunctive, i.e., “A and / or B and / or C.”

[0173] A single unit or device can perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously.

[0174] The operations described in the embodiments described above (for example, Figures 3 to 5) may be implemented as program code means of a computer program, and / or as dedicated hardware for a related network device or function (for example, the network controller 110 in Figure 1). The computer program may be stored and / or distributed on a suitable medium such as a supplied optical storage medium or solid-state medium, together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0175] References: 1.Ant Colony Inspired Machine Learning Algorithm for Identifying and Emulating Virtual Sensors“,Mani et al,https: / / arxiv.org / pdf / 2011.00836.pdf 2.Small CSI Samples-Based Activity Recognition:A Deep Learning Approach Using Multidimensional Features“, Tian et al.,2021.https: / / www.hindawi.com / journals / scn / 2021 / 5632298 /

Claims

1. A device for discovering, commissioning, or managing virtual data or action-generating devices within a network, Collect information indicating at least one virtual data or action generation device, A request is sent to an available temporary assistive device to provide information. Based on the collected information and information about existing data or action generation devices, the at least one virtual data or action generation device is determined. Based on the measurements of the existing data or action generation device and the information from the available temporary auxiliary device, determine at least one parameter of the determined at least one virtual data or action generation device. A device that outputs the determined at least one parameter of the at least one virtual data or action generation device.

2. The apparatus according to claim 1, which determines the at least one virtual data or action generating device by adjusting existing data or action generating devices in the aforementioned network and / or other networks.

3. The apparatus according to claim 2, wherein the adjustment is achieved by time synchronization and / or beamforming.

4. The apparatus according to any one of claims 1 to 3, which collects the information indicating the at least one virtual data or action generating device by transmitting an information request to an available physical data or action generating device.

5. Furthermore, the apparatus according to any one of claims 1 to 4, which compares ground truth information with information collected from existing data or an action generation device and estimates a virtual device model that determines the at least one parameter of the at least one virtual data or action generation device.

6. The apparatus according to any one of claims 1 to 5, wherein the at least one virtual data or action generating device is determined by one or more temporary data or action generating devices and event of opportunity.

7. The apparatus according to claim 6, wherein the events of the aforementioned opportunity relate to the actions or movements of a user or another person, and / or the actions or movements of a device, for which ground truth sensing data is available.

8. The apparatus according to claim 7, which emulates at least one virtual data or action generation device using an existing set of physical data or action generation devices.

9. The apparatus according to claim 7 or 8, which provides a temporary external source of ground truth sensing data by correlating with available data streams resulting from data or action generating devices that are not part of the set of data or action generating devices used to provide the at least one virtual data or action generating device, or from the use of at least one data or action generating device that is part of the set of data or action generating devices used to provide the at least one data or action generating device, which is in a non-standard mode.

10. The apparatus according to claim 9, wherein the non-standard mode is a high-energy mode or a high-sampling-rate mode.

11. The apparatus according to any one of claims 1 to 10, for enhancing or testing the functionality of a virtual data or action generating device by designing at least one operation or action performed by a user and / or device to add ground truth data.

12. A network hub device comprising one or more physical data or action generating devices, comprising the device described in any one of claims 1 to 11.

13. A mobile device comprising the apparatus described in any one of claims 1 to 11.

14. A system for discovering, commissioning, or managing virtual data or action generating devices, comprising a network of one or more physical data or action generating devices and the hub device described in claim 12.

15. A method for discovering, commissioning, or managing virtual data generating devices within a network, A step of collecting information indicating at least one virtual data or action generation device, The steps include sending a request to a temporary assistive device available to provide information, A step of determining the at least one virtual data or action generation device based on the collected information and information about existing data or action generation devices, A step of determining at least one parameter of the determined at least one virtual data or action generation device based on the measured values ​​of the existing data or action generation device and the information from the available temporary auxiliary device, The steps include outputting the determined at least one parameter of the at least one virtual data or action generation device, Methods that include...

16. A computer program that, when executed on a computer device, includes coding means for generating steps of the method according to claim 15.

Images