Predictive virtual sensor commissioning and management

Abstract

The invention discloses a device and a method which can be operated on a smart device (such as a smart phone or a smart hub). The apparatus and method can debug and / or manage virtual sensors / actuators by one or more of acquiring requirements regarding the virtual sensors / actuators, determining, based on existing sensors / actuators, the virtual sensors or actuators (e.g., sensors, actuators, etc. By coordinating sensors / actuators in one or more networks and / or by temporary sensors / actuators and opportunistic events), parameters of virtual sensor / actuator (s) are determined and virtual sensor / actuator parameter (s) are exposed given measurements by existing sensors / actuators.

Description

TECHNICAL FIELD

[0001] The present invention relates to a commissioning and / or management system that can be applied to sensors, actuators or other data producing devices arranged in a local network, e.g. a home network or home automation system, a sensor network or augmented / virtual reality (AR / VR) or metaverse applications, which can be connected through different wireless connection technologies, including short and long distances. Short distance technologies can include IEEE 802.15.4 based networks, Wi-Fi networks, etc. Sensors can also be connected by means of, e.g., 3GPP based networks or long distance networks like LoRa. BACKGROUND

[0002] Sensors are increasingly used in local networks, e.g. home networks and home automation systems. They not only measure temperature, light conditions, but also more personal data like respiration rate, occupancy, identity, blood pressure (mobile sensors) and all kinds of other medical data, movement patterns, etc.

[0003] Virtual sensors and actuators can be created, where several real devices collaborate to sense or actuate one or more parameters of interest, which can be of a different type than the type they were designed for. In many cases, the parameter of interest can be beyond the capability of any one real device to sense or actuate. As an example, such virtual sensors or actuators can be used in the audio domain, e.g. virtual microphones or loudspeakers, but in principle any type of parameter can be relevant. Virtual sensors can also be used in industrial internet of things (loT) applications, which are suitable for predicting / learning which of a set of physical sensors can be adequately replaced by a virtual sensor consisting of some subset of the remaining sensors.

[0004] To create a virtual sensor from a set of physical sensors, it can be necessary to deploy a known type of physical sensor in the location of interest, so that data from the remaining sensors can be compared to it. Known systems provide such temporary or learning sensors by exploiting existing, already installed permanent physical sensors and learning to emulate them using other sensors.

[0005] Optimal use of virtual sensors can often require processing of their data through machine learning models. However, when an activity is transferred to a different scene (i.e. the scene in which the new virtual sensor is located), a machine learning model trained in one scene (e.g. the scene containing physical sensors) can not work properly. As an example, in a typical IoT scenario, data obtained from real or virtual sensors can need further processing, in particular through machine learning methods. Machine learning models can have been trained on data from one context or scene and can then be desired to be used in a new context. In an example (e.g. for a human activity recognition model running on Wi-Fi data), transfer learning can be applied as a technique to transfer such learned models to a new scene by using only a few samples collected in the new scene.

[0006] It can not be intuitive for a user to deploy virtual sensors as their potential capabilities depend on the properties of several different physical sensors and on the installation environment. Known systems for learning potential virtual sensors typically require the presence of physical sensors at least during the learning mode at the location of interest. However, this is not practical for a smart home scenario and does not allow extending virtual sensors into areas not yet covered by physical sensors.

[0007] Furthermore, in case of (re)deploying physical sensors to optimize the deployment of virtual sensors, it can not be intuitive for a user to deploy / reposition (existing) physical sensors such that the most or best virtual sensors can be deployed to meet the needs of (third party) applications installed in a smart environment (e.g. a smart home).

[0008] While existing systems (e.g. calibration routines for speaker arrays) can provide ways to test or manage a group of similar physical devices forming part of a virtual sensor or actuator, they can not provide such capabilities for heterogeneous devices, heterogeneous networks (e.g. Wi-Fi and Thread (based on 802.15.4) or multiple Wi-Fi networks) or devices that need to run in non-standard modes while forming part of a virtual sensor. SUMMARY

[0009] It is an object of the present invention to provide a user-friendly system for creating potential virtual sensors within a sensor environment.

[0010] This object is achieved by the apparatus, hub device, mobile device, system, method and computer program product defined in the claims.

[0011] According to a first aspect (e.g. for a controller of a hub device (hub) or a mobile device (e.g. a smartphone)), there is provided an apparatus for discovering, commissioning or managing virtual data or action producing devices (210) in a network, configured to: collect information indicative of at least one virtual data or action producing device; send a request to an auxiliary device capable of providing information; determine said at least one virtual data or action producing device based on the collected information and information about existing data or action producing devices; determine at least one parameter of the determined at least one virtual data or action producing device based on measurements of said existing data or action producing devices and said information from said auxiliary device capable; and output the determined at least one parameter of said at least one virtual data or action producing device.

[0012] According to a second aspect, there is provided a hub device (hub) of a network comprising one or more physical data or action producing devices and the apparatus of the first aspect.

[0013] According to a third aspect, there is provided a mobile device (e.g. a smartphone) comprising the apparatus of the first aspect.

[0014] According to a fourth aspect, there is provided a system for discovering, commissioning or managing virtual data or action producing devices, wherein the system comprises one or more physical data or action producing devices and a network of hub devices of the second aspect.

[0015] According to a fifth aspect, there is provided a method for discovering, commissioning or managing virtual data producing devices in a network, wherein the method comprises the steps of: collecting information indicative of at least one virtual data or action producing device; determining said at least one virtual data or action producing device based on the collected information and information about existing data or action producing devices; determining at least one parameter of the determined at least one virtual data or action producing device based on measurements of said existing data or action producing devices; and outputting the determined at least one parameter of said at least one virtual data or action producing device.

[0016] According to a sixth aspect, there is provided a computer program product comprising code means for producing the steps of the method of the fifth aspect when run on a computer device.

[0017] Accordingly, the invention provides a system that can be used for virtual sensor / actuator discovery, orchestration, commissioning and management, wherein a virtual sensor (or other data producing device) or virtual actuator (or other action producing device) can be 'discovered' in a pre-existing network of physical sensors or actuators without the need to deploy any additional sensors or actuators by simultaneously sensing a commonly identified event of the physical sensor network using a temporary auxiliary sensor or an existing physical sensor, thereby serving as a basis for the emulation. The virtual sensor / actuator can be orchestrated by the central hub without user input by learning its orchestration requirements as part of the above discovery process and applying them at runtime.

[0018] Thereby, a user can enhance commissioning of the behavior of a virtual sensor / actuator, thereby improving the performance of the virtual sensor / actuator or a downstream model by designing appropriate demonstration actions.

[0019] By using demonstration actions similar to the above, or by re-running the sensor discovery process, the virtual sensor / actuator can be actively managed by the hub without user input.

[0020] According to a first option, which can be combined with any of the first to sixth aspects, the at least one virtual data or action producing device can be determined by coordinating existing data or action producing devices in the network and / or other networks. Thereby, existing devices of the network or other networks can be controlled (e.g., orchestrated) to obtain the required information for determining the desired virtual device.

[0021] According to a second option, which can be combined with the first option or any of the first to sixth aspects, the coordination can be achieved by time synchronization and / or frequency synchronization and / or light color synchronization and / or beamforming. Thereby, it can be ensured that the required information is obtained at the right place and at the right time.

[0022] According to a third option, which can be combined with the first or second option or any of the first to sixth aspects, the information indicative of the at least one virtual data or action producing device can be gathered by sending an information request to the available auxiliary devices and / or to the available physical data or action producing devices. Thereby, suitable existing auxiliary and / or physical devices in the network can be triggered to provide the required information for determining the virtual device(s).

[0023] According to a fourth option, which can be combined with any of the first to third options or any of the first to sixth aspects, reference ground truth information (obtained from e.g. the auxiliary device(s)) can be compared with information gathered from existing data or action producing devices and a virtual device model can be estimated, which determines at least one parameter of at least one virtual data or action producing device. Thus, reliable reference ground truth information is advantageously used as a reference for estimating the virtual device model.

[0024] According to a fifth option, which can be combined with any of the first to fourth options or any of the first to sixth aspects, the at least one virtual data or action producing device can be determined by means of the opportunity event and one or more temporary data or action producing devices. Thus, the determined virtual device can be adapted to the temporary devices and the opportunity event to optimize the functionality of the virtual device.

[0025] According to a sixth option, which can be combined with any of the first to fifth options or any of the first to sixth aspects, the opportunity event can involve movements or actions of a user or other person, and / or movements or actions of devices, for which reference ground truth sensing data is available. Thereby, the reference movements or actions can be selected to optimize the functionality of the virtual device.

[0026] According to a seventh option, which can be combined with any of the first to fifth options or any of the first to sixth aspects, the at least one virtual data or action producing device can be emulated by a set of existing physical data or action producing devices. Thus, a new virtual device can be provided by using output signals of already existing devices without the need for any additional network devices.

[0027] According to an eighth option, which can be combined with any of the first to fifth options or any of the first to sixth aspects, a temporary external source of reference ground truth sensing data can be provided by using and associating available data streams under a non-standard mode (e.g. a high energy or high sampling rate mode), which originate from data and action producing devices that are not part of a set of data or action producing devices used to provide the at least one virtual data or action producing device or from the use of at least one data or action producing device that is part of the set of data or action producing devices used to provide the at least one virtual data or action producing device. Thereby, appropriate information for optimizing the determination of at least one parameter of the virtual device can be provided.

[0028] According to a ninth option, which can be combined with any of the first to fifth options or any of the first to sixth aspects, the functionality of the virtual data or action generating device can be enhanced or tested by designing at least one movement or action to be performed by the user and / or the device to add the reference ground truth data. Thus, by providing updated reference ground truth data, the functionality of the virtual device can be constantly adapted to changes in the network environment.

[0029] It will be appreciated that the apparatus of claim 1, the hub device of claim 12, the mobile device of claim 13, the system of claim 14, the method of claim 15, and the computer program product of claim 16 can have similar and / or identical embodiments, in particular as defined in the dependent claims.

[0030] It will be appreciated that preferred embodiments can also be dependent claims or any combination of the above embodiments with the corresponding independent claims.

[0031] These and other aspects will become apparent from and will be elucidated with respect to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0032] In the following drawings: Figure 1 a block diagram of a sensor discovery, commissioning and management system according to various embodiments is schematically illustrated; Figure 2 a process flow diagram of a system based on Figure 1 is schematically illustrated; Figure 3 a flow diagram of a virtual sensor discovery process according to a first embodiment is schematically illustrated; Figure 4 a flow diagram of a virtual sensor commissioning process according to a second embodiment is schematically illustrated; Figure 5 a flow diagram of a virtual sensor management process according to a third embodiment is schematically illustrated; and Figure 6 an example of a virtual sensor commissioning process in an exemplary sensor environment is schematically illustrated. DETAILED DESCRIPTION

[0033] Embodiments are now described based on a sensor network system. Such a system can be a home networking system, a sensor system for a metaverse application or business, industrial or public space (e.g. a hospital). Such a network can use long range communication technology (e.g. cellular) or shorter range communication technology, e.g. Wi-Fi, UWB, Bluetooth TM etc.

[0034] The following description is described in terms of a networked arrangement, but it should be noted that the same operations can be applied within a single device.

[0035] In the following disclosure, a “home network” or “sensor network” is understood to comprise a network of sensors and actuators that facilitate certain tasks (e.g., lighting or healthcare related). It can include a home network hub (e.g., a data distribution entity) that is responsible for managing the home network and allowing multiple devices or nodes (e.g., sensors and actuators) to connect to the device to participate in the network. The home network hub can also be an entity that is responsible for orchestrating secure data distribution (e.g., data originating from the network). The home network hub can include or provide access to a router device to link the home network to an external network (e.g., the Internet) and / or can allow devices to be added or removed from the network.

[0036] A “sensor” or “sensor device” is understood to be a data producing device (e.g., in a home network) that is capable of (wirelessly) sensing, locating, sensing (e.g., light, presence, sound, image, etc.).

[0037] Further, a “virtual sensor” or other “virtual device” is understood to be a virtual entity that can be instantiated by processing the software type that a physical sensor or other device would otherwise process. It learns to interpret relationships between different variables and observes readings from different instruments. Virtual sensing technology is used to provide a viable and economical alternative to expensive or impractical physical measuring instruments. Virtual sensing systems use information available from other measurements and process parameters to compute an estimate of the quantity of interest. These virtual devices can use data to gather information that a single device cannot measure. In this way, they can obtain information that cannot be measured directly.

[0038] Further, a “user” or “homeowner” is understood to mean the person who owns the home network and, in most cases, the data collected / acquired / generated by the sensors / actuators of the home network.

[0039] Further, the term “metaverse” is understood to refer to a persistent shared collection of interactive spaces within which users can interact with each other and perceive virtual features (i.e., augmented reality (AR)) or where those spaces are entirely composed of virtual features (i.e., virtual reality (VR)). VR and AR can generally be referred to as “mixed reality” (MR).

[0040] Furthermore, the term “data” is to be understood as referring to a representation in a known or agreed format of information to be stored, transmitted or otherwise processed. Said information can in particular comprise one or more channels of audio, video, image, haptics, motion or other forms of information or environmental / personal characteristics (e.g. temperature, heart rate, etc.), which can be synchronized and can be derived from sensors (e.g. microphones, cameras, motion detectors, etc.).

[0041] Furthermore, the term “opportunity event” is to be understood as referring to an event whose location is known and for which there exists a source of ground truth sensing data. A first example comprises a movement performed by a user (e.g. walking in a defined location), which can be sensed by a specific device (e.g. a smartphone carried in the user’s pocket) or a wireless receiver (e.g. a Wi-Fi access point) or a 3GPP access device (e.g. a 5G gNB). Another example comprises a movement performed by a robot (e.g. a drone, a vacuum cleaner, an assistant...) instructed to move in a predefined direction, where the movement can be measured and is known to the system. Another example comprises a movement performed by a user / installer / robot operating non-standard (e.g. high-quality) hardware / sensors during an installation phase.

[0042] The term “ground truth” is to be understood as information known to be true or real provided by direct observation and measurement (i.e. verified evidence), as opposed to information provided by inference. Thus, “ground truth” refers to the process of gathering proper objective (provable) data. For example, a stereo vision system is tested to verify the extent to which it can estimate 3D positions. In this case, the “ground truth” can be the positions given by a laser rangefinder - known to be much more accurate than the camera system.

[0043] Furthermore, the term “gestural action” is to be understood as a relevant motion or action performed by a user or a device that can be sensed by a sensor network. Examples include (for a user) standing in a specific location, performing a physical gestural action in that specific location or walking towards that specific location, and / or (for a device) emitting a sound or performing a task in a specific location and time.

[0044] The smart home market has grown significantly over the past few years. In a market dominated by tech-savvy early adopters, the World Economic Forum estimates that there were more than 130 million households with smart home devices in 2022. Matter is a smart home standard created by the IP-based Project Connected Home (Project CHIP) in 2019. It was officially released in September 2022 and is maintained by the Connectivity Standards Alliance (CSA), formerly known as the Zigbee Alliance. The standard encourages interoperability between devices and platforms. It allows devices to work offline without continuous access to the cloud and various cloud services.

[0045] Thus, certain sensor data can be used to provide services to the owner, in a way that the owner can better control which devices and / or entities in and / or outside the home are accessing data generated by other device(s). Such services can include health monitoring, presence detection, lighting control, heating, ventilation, air conditioning (HVAC) control, etc. It is also contemplated that certain devices can perform certain actions based on data input from other devices without actually accessing this data. It is also contemplated that data generated in the owner's home can be analyzed / processed in the cloud, but ideally the cloud has no access to this data. It is also contemplated that certain devices can only access data if their properties or context allow it. In these cases, access control architecture can not be enough.

[0046] Furthermore, data (e.g., derived from measurements) can be presented to users or applications in a summarized / processed way. For example, if there are multiple light sensors in a room, a user can not be interested in the light readings (intensity, color, etc.) of each light sensor, but rather in a summarized light value in the room (e.g., average light intensity). For example, given some physical sensors / actuators, it can be possible to create virtual sensors / actuators, e.g., a virtual actuator formed by multiple actuators, or a virtual sensor that provides sensing output based on measurements of one or more physical sensors.

[0047] The following embodiments relate to a system for discovering, commissioning, and / or managing potential virtual sensors that can be created within an environment, their potential properties, and any coordination or orchestration requirements, where the system can include one or more underlying sensor networks. Learning or temporal sensors can be provided in an ad-hoc manner using other sensors that are known or can be made available, without burdening the user.

[0048] It should be noted that in this disclosure, only those blocks, components, and / or devices that are relevant to the proposed data distribution functionality are shown in the drawings. Other blocks are omitted for the sake of brevity. Furthermore, blocks designated by the same reference numerals are intended to have the same or at least similar functionality, and thus their functionality is not repeated in later descriptions.

[0049] Figure 1 A block diagram of a sensor discovery, commissioning, and management system (which can be referred to as a "sensor learning system") according to various embodiments is schematically shown.

[0050] The system comprises one or more sensor networks 20 comprising one or more (in examples, two or more) sensors installed in a given space, such as a user's home network (e.g., a smart home). These sensors can be physical sensors (PS) 220, i.e., hardware-based sensors, comprising any type of hardware that can perform one or more sensing functions. The sensing functions can be the primary function of the physical sensor 220 (e.g., a microphone installed to sense sound), or a derived function obtained by using the physical sensor hardware installed for one or more purposes to sense things in its environment (e.g., using wireless sensing from a Wi-Fi / 3GPP communication device to sense the movement of a nearby person).

[0051] The physical sensors can also comprise suitable network hardware for establishing communication with the hub 10 (e.g., via Wi-Fi, Thread, Bluetooth, etc.) or access devices.

[0052] Optionally, additional auxiliary sensors (AS) 230 can be provided, which can be available at times and not available at other times. Examples of such auxiliary sensors include a user's smartphone or similar device (which is not a permanently installed part of the sensor network 20), and / or physical sensors in non-standard modes (e.g., high-energy sensing mode, high-sampling rate mode, or modes requiring high computational load) that are too costly to operate continuously as part of the standard operation of the sensing network 20, and / or non-standard hardware (e.g., high-quality sensors used in professional installation tools), and / or sensors integrated in mobile devices (e.g., robots).

[0053] In addition, the sensor network 20 can comprise one or more virtual sensors (VS) 210, which can be created, for example, based on a logical description of additional sensing data that can be derived by fusing the outputs of multiple physical sensors 220 but cannot be obtained by the output of any single physical sensor 220.

[0054] In embodiments, the physical sensors 220 can need specific modes or behaviors when used as part of a virtual sensor 210, and thus require orchestration functionality. For example, a physical sensor 220 can need to stop network communication when operating as part of a virtual sensor 210.

[0055] In embodiments, multiple physical sensors 220 can comprise only one physical sensor, but can use two or more different sensing functions (e.g., using both audio and Wi-Fi sensing from a single smart speaker to constitute a virtual sensor).

[0056] In other embodiments, the plurality of physical sensors 220 can comprise a plurality of physical sensors in different networks, like Wi-Fi or Thread, or different systems, like two adjacent smart home systems.

[0057] It should be noted that, Figure 1 The system of the '000' can also comprise physical, auxiliary and virtual actuators ("action producing devices") as previously described with the same or similar characteristics as defined above for the sensors, with the replacement of "sensing data" by "actuation output" for the actuators. In this case, the actuators are understood as devices that produce actions as actuation output in response to received electrical signals. These actions can be mechanical movements (e.g. linear or rotational movements) or emission of something (like light or heat). Thus, the corresponding network of actuators is not used for detection, but for generating actions. Thus, the teachings of the following embodiments can be transferred to such a network of actuators.

[0058] The hub 10 can be a central device of an Internet of Things (IoT) or smart system for controlling and receiving data from devices on the sensor network 20, e.g. a Wi-Fi access point or a 3GPP access device. For example, the system can use multiple networks, e.g. in a Matter system. The hub 10 can be implemented, e.g. on a controller and / or a bridge and / or a border router as defined in the Matter standard, or in other ways. The hub 10 can be distributed over multiple devices. At least part of the hub logic can be provided in an edge server or in the cloud and can be locally available through one or more local units of the hub 10. Furthermore, the hub 10 or part of its functionality can reside in a sensor / action producing device, or can be distributed over multiple such devices.

[0059] More specifically, the hub 10 can contain one or more of the following: a computing unit, a network controller (NC) 110, a user interface (UI) 120, a location module (LM) 130, an event module (EM) 140, a measurement model (MM) 150 and a virtual sensor database (VS-DB) 160.

[0060] The network controller 110 can be a hardware module configured to manage the network communication with the sensors (e.g. Wi-Fi, Thread, Bluetooth, 3GPP radio access network, etc.). It should be appreciated that the network controller 110 can have subcomponents, some of which can involve functions other than simple network control. It can control the network / sensing / action functions of the physical sensors 220 (e.g. control at least one of the following operations: transmission of data, communication time, movement of the robotic device to a specific location, use of specific sensing capabilities / parameters / modality, synchronization of sensing of multiple devices, synchronization of sensing inputs of multiple devices, use of specific frequencies, etc.), synchronization of multiple networks (e.g. Wi-Fi, Thread), avoiding mutual interference between networks, e.g. when Wi-Fi is used for sensing, Thread network should not be used for communication, to guarantee as much as possible the accuracy and noise-free of the Wi-Fi sensing measurements. Furthermore, the network controller 110 can be configured to receive sensing data from the physical sensors 220 and the auxiliary sensors 230.

[0061] The user interface 120 can be directly provided at the hub 10 or can be located at a remote location and connected with the hub 10 through (local / cloud) communication with an external device such as a user smartphone.

[0062] The location module 130 can be a first software module configured to maintain and update an approximate map of the space (e.g. a smart home of a user) in which the hub 10 and the sensor network 20 are installed.

[0063] In an example, the space of the smart home can be retrieved from the internet in which multiple sites maintain information about existing properties. In another example, the space of the smart home can be derived by means of (robotic) devices (e.g. by scanning and / or data retrieval). In yet another example, the location of (e.g. home objects) can be derived by means of a camera or other image processing devices. In yet another example, auxiliary objects can be placed to track specific (home) objects.

[0064] The event module 140 can be a second software module configured to compute when the sensor network 20 can have detected an opportunity event.

[0065] One or more measurement models 150 can be configured to obtain (e.g. receive or retrieve) sensing data from the sensing network 20 and derive a meaning or context. Examples will include machine learning models (e.g. neural networks) that receive sensing data related to human activities and derive information about the state or health of the relevant person in question. These can be run locally on the hub 10 or accessed remotely from a cloud service.

[0066] The virtual sensor database 160 can be configured to store parameters required for deriving virtual sensors 210 from data of physical sensors 220, as well as their related operational modes, approximate locations, and coordination and / or orchestration requirements.

[0067] In embodiments, if the hub 10 changes (e.g. due to a change in hub provider), the virtual sensors can be preserved based on the data stored in the virtual sensor database 160, as long as the respective functionality of the hub 10 supports the data derivation. The virtual sensors can be exposed to applications, i.e. such programs can access / receive the output they generate. Thus, the smart home system can be operated by different providers. Each provider can use different hubs 10 (e.g. through a universal smart speaker or a dedicated hardware) to implement the smart home functionality. When a user decides to change the smart home provider, the virtual sensors used by the user, stored in the virtual sensor database 160, can also be migrated. This can be achieved, as an example, by including a list of virtual sensors and, for each virtual sensor, information on how to construct the virtual sensor (e.g. in the virtual sensor database 160).

[0068] Furthermore, Figure 1 The system of the application comprises a choreography design system 30 for designing (in the following, designing also includes exchanging) choreographies, which can be configured to run on the hub 10 or remotely.

[0069] The choreography design system can comprise at least one of the following modules: a commissioning choreography module (CG) 310 for designing commissioning choreographies, a management choreography module (MD) 320 for designing management choreographies, and a choreography database (G-GB) 330 for storing commissioning and management choreographies. The commissioning choreography module 310 can be configured to design choreographies that can be used by a user to commission a virtual sensor 210, while improving the performance of the virtual sensor. Such commissioning choreographies can be used by a (robotic) device to commission a virtual sensor 210, wherein the (robotic) device can be manipulated by the hub 10 or an external party. The commissioning choreography can be linked to an opportunity event and / or a measurement model 150 associated with the virtual sensor 210 being commissioned.

[0070] The management choreography module 320 is configured to design choreographies that can be used to manage or test a virtual sensor 210. These can be the same as the commissioning choreographies, or can be specifically designed for ease of input or more natural motion characteristics, thus acting as a continuous virtual sensor testing function.

[0071] Figure 2 A flowchart of a sensor discovery procedure of the system of the application is schematically shown. Figure 1 A flowchart of a sensor discovery procedure of the system of the application is schematically shown.

[0072] After the program starts (S), the user indicates to the hub 10 a requirement for a virtual sensor setup. In response, the hub initiates a discovery mode and sends or broadcasts or multicasts a sensing data request (SD-REQ) to the available auxiliary sensor(s) 230 and sends or broadcasts or multicasts a sensing data request and a mode switch request (MS-REQ) and orchestration control information (ORCH) to the available physical sensor(s) 220.

[0073] In addition, the hub 10 provides information about the timing of the discovery mode (T-DM) to the event module 140.

[0074] The physical sensor(s) 220 can obtain / receive optional virtual sensing data (VS-D) from the virtual sensor(s) 210.

[0075] The auxiliary sensor(s) can send / forward location data (LD) to the location module 130, which generates and forwards a matched sensing data and location stream (M(SD, LS)) to the event module 140. In addition, the event module 140 and the hub 10 receive ground truth sensing data (GT-SD) from the auxiliary sensor(s) 230.

[0076] In addition, the physical sensor(s) 220 can transmit sensing data (SD) and location provision (L) to the event module 140, and the hub 10 receives information about physical sensor sensing (PS-S) and orchestration (ORCH) and mode switch data (MS-D) from the physical sensor(s) 220.

[0077] Based on at least one of the ground truth sensing data, the timing of the discovery mode, and the sensing data and location provision, the event module 140 determines an opportunity event (E-OPP) and forwards information about such opportunity event to the hub 10.

[0078] The hub 10 compares the ground truth data with the physical sensor data, estimates a virtual sensor model, and stores information about the estimated sensor model (E-VSM) in the virtual sensor database 160.

[0079] Optional additional modeling information (MI) can be provided from the available measurement model 150.

[0080] In embodiments, constructing the virtual sensor model can be based on training using stored time-aligned sensor data (e.g., from all sensors), which can be collected and stored in a data repository over a certain time period. In an example, the virtual sensor model can be trained to try to produce the desired sensor data (as output) from the physical sensor data (as input). The virtual sensor model can be a convolution or transformer-based neural network model (e.g., trained with mean squared error (MSE) on the desired output), or can be a very simple model (e.g., a linear regression model).

[0081] The implementation complexity of the virtual sensor model can be adjusted according to the available computing conditions, as the main advantage of the proposed concept can be obtained by a simple model even if its performance is not as good as that of a complex model.

[0082] Edge AI is the implementation of artificial intelligence technology in edge computing environments. This means that AI computation can be done at the edge of a given network (typically on the device that created the data), rather than in a centralized cloud computing facility or off-site data center. With the proliferation of Edge AI chips, training relatively small neural networks becomes relatively fast and easy. This can be achieved based on a continuous online training process.

[0083] Figures 3 to 5 The respective flowcharts of the virtual sensor discovery, commissioning and management processes are schematically illustrated according to various embodiments of components of a sensor discovery, commissioning and management system using Figure 1 The steps of these flowcharts can be at least partially executed or initiated by respective instructions of a software program / routine of a controller at the control hub (e.g., the network controller 110 of Figure 1 or a remote network device.

[0084] It should be noted that Figures 3 to 5 not all steps can be always required. Also, some or all steps can be executed one or more times, and the order of execution can be different.

[0085] More specifically, the respective processes of virtual sensor discovery, commissioning and management are described based on the related first to third embodiments, where the common feature is the use of opportunistic events occurring within the scene to enhance the hub’s ability to discover or manage virtual sensors, or (in the commissioning embodiment) to use specially designed opportunistic events as output (i.e., a demonstration action) to enhance the performance of the virtual sensor at commissioning.

[0086] Thus, the three embodiments correspond to the discovery and orchestration of virtual sensors, the commissioning of virtual sensors and the management of virtual sensors, respectively.

[0087] Figure 3 A flowchart illustrating a virtual sensor discovery process according to a first embodiment is shown schematically.

[0088] The virtual sensor discovery can comprise Figure 3 One or more of the following steps are shown. The order of the steps can differ from the illustrated order, and steps can be repeated.

[0089] In step S301, a user indicates a desire to discover potential virtual sensors in a system comprising one or more physical sensors, which can have been commissioned and are managed by a hub (H), or this desire is communicated to the hub via a third party application, or a program run by the hub suggests creating virtual sensors that can be beneficial to the user or to one or more third party applications in use in the system (e.g. applications running on the hub or on a smart device connected to the system, such as a smartphone), or an (user) application can trigger the discovery / creation of virtual sensors (e.g. a new IoT application can trigger a request to create virtual sensors that can be beneficial), or an operating system (OS, e.g. iOS or Android) can trigger the discovery / creation of virtual sensors to make them available to third party applications.

[0090] In response, the hub enters a discovery mode (DM).

[0091] In an example, the user request can be, for example, a one-off input via a user interface requesting a virtual sensor in a specific location, or a continuous desire to discover potential virtual sensors in all or a predetermined portion of the specific locations or scenarios in a scenario. The user can or can not specify the desired type and location of the virtual sensor. In the case that the user does specify a desired location, the subset of physical sensors to be used can be limited to those available at that location.

[0092] In an alternative example, the discovery request can be initiated by a third party application (“app”). For example, a user can access an “app store” or similar interface in which applications corresponding to virtual sensors can be downloaded, and / or the hub can be configured to abstract the sensing data of physical sensors into specific types of virtual sensors for use by such third party applications. For example, a “smart home app store” can contain many third party virtual sensor applications (these can be different types of sensors, like “fall detection” and “voice control microphone”, or different performance levels of sensors, like “base microphone” and “sensitive microphone”). The virtual sensor applications can contain a (hidden) list of requirements that the hub must meet in order to install each application. In this case, either a user installing a new application or an already installed application requesting new abstracted sensing data can constitute a request for the hub to discover an appropriate virtual sensor.

[0093] In step S302, the hub has entered the discovery mode and requests and receives sensing data from all or some of the installed physical sensors and available secondary sensors (like a user’s smartphone), e.g. via the network controller.

[0094] If needed, the hub can be configured to instruct at least some of the physical sensors to enter a specific mode for the discovery phase. Examples can include a high sensitivity mode (e.g. using a high sampling rate, using a higher input energy for active sensing (e.g. some types of radio frequency (RF) sensing), and / or transmitting raw sensing data rather than locally processed data to the hub (the latter can be the standard operating mode)), or a scanning mode (e.g. performing a type of beamforming, e.g. for wireless sensing to track / sense certain areas or inject audio in certain areas and / or listen in certain areas), or a non-standard mode (e.g. using a physical sensor whose primary task is not sensing to temporarily provide sensing input, e.g. using the network hardware of a device (e.g. a smart light bulb) for event sensing, while that device would not have a sensing function otherwise (e.g. a light bulb including a Bluetooth interface can be able to sense the presence of a specific person carrying a phone (and running Bluetooth), or a light bulb that can do Wi-Fi sensing can be enabled to act as a presence sensor to sense the presence of a person), or causing a temporary exceptional output (e.g. temporarily matching the output characteristics of a smart light to the output characteristics required by a nearby camera to enhance its sensing)).

[0095] In another example, the hub can provide enhanced coordination / orchestration to the physical sensors during the discovery phase (if this improves performance, this can be replicated in the usage phase).

[0096] According to a first example, the hub can coordinate communication tasks with sensing tasks. For example, to avoid interference between wireless communication of nearby devices and RF sensing data of physical sensors that make up a virtual sensor. This orchestration can occur between physical sensors themselves and / or between physical sensors and other devices outside the sensor network. Moreover, such orchestration can occur across different wireless protocols (e.g., between Wi-Fi and IEEE 802.15.4), and / or between different networks implementing the same protocol (e.g., between two nearby Wi-Fi networks that the hub is registered with or otherwise has access to), and / or between adjacent smart home systems by coordinating adjacent hubs.

[0097] In another example, some forms of RF sensing can require input of nearby device communication. For example, a method of human activity recognition using Wi-Fi can use one device to generate traffic from an access point (AP) and another device to sense channel state information (CSI) through return signals from the AP. In such an example, the hub can orchestrate (e.g., by transmitting orchestration control information) for one or more physical sensors to sense the CSI while another device (which can be within or outside the sensor network) generates known traffic from the AP.

[0098] According to a second example, the hub can coordinate sensing tasks with energy management tasks. For example, it can be configured to limit (e.g., by sending orchestration control information) physical sensors that are being used as part of a virtual sensor from entering a low-power or sleep state (and thus also do this during the discovery phase).

[0099] According to a third example, the hub can coordinate sensing tasks with sensing tasks. For example, to avoid interference of sensing-related emissions of one active sensor with sensing of nearby passive sensors, the hub can be configured to command (e.g., by transmitting orchestration control information) the first sensor to stop sensing during a particular time period, or to have it adopt a common time base with the second sensor to coordinate sensing activities of both.

[0100] The first through third examples described above can also optionally apply to auxiliary sensors. For auxiliary sensors, an external principal (e.g., an installer) can control the use of the auxiliary sensors and / or the hub. Thus, the external principal that manages such auxiliary sensors can also request and receive sensing data through the hub.

[0101] Any non-standard modes and orchestration details explained above can be stored in the virtual sensor database.

[0102] In step S303, the hub can be controlled to use the physical sensors or, where available, the auxiliary or mobile sensors to construct temporary physical sensors to identify opportunity events (E-OPP) that can be sensed by the installed physical sensors for which the location of the event is known and / or for which there is a source of ground truth sensing data.

[0103] To achieve this, the hub can gather data from the physical sensors (e.g. cameras) and derive suggestions about potential virtual sensor locations or temporary sensor locations. For example, a user can record a room with the camera of a smartphone and objects of interest in the room (e.g. a plant, a table, a door, a window...) can be recorded as potential candidates for placing virtual sensors. For example, a plant can be equipped with a virtual sensor to track whether it needs watering, a table can be equipped with a virtual sensor to track whether someone is seated and whether the light is sufficient, a door can be equipped with a virtual sensor to track the open / close state of the door and whether someone is inside the house. The hub can then derive and provide suggestions about the location of these potential virtual sensors or temporary physical sensors.

[0104] The hub can provide suggestions only relevant to the user, for example, based on a set of third party applications installed in the user's smartphone to provide suggestions.

[0105] In step S304, the hub compares (COMP) the sensing data (SD) recorded by the physical sensors during the discovery period with the ground truth sensing data (GT-SD) at the location of interest, thereby identifying which subset of the physical sensors can best emulate the ground truth data from the temporary physical sensors or potential virtual sensors. More specifically, the event module of the hub can compare the sensing data from the physical sensors with the (ground truth) sensing data of the auxiliary sensors (and / or other physical sensors) to try to identify opportunity events (i.e. those events that can be sensed by the installed physical sensors for which the location is known and for which there is a source of ground truth sensing data).

[0106] In examples, the location of the opportunity event can be derived, for example, by the location module by one or more of: i. Sensing events within the entire area of interest (e.g. the entire space of the home network) while there is a source of user location (e.g. auxiliary sensors from the user's smartphone) and correlating the sensing data of unknown location (i.e. data from the physical sensors) with the sensing data of known location (e.g. data from the smartphone) - the correlation being on the basis of similar sensing type signals at (close to) the same time.

[0107] ii. Sensing events at a specific location (e.g. the user requests a virtual sensor at a known location). In this case, the location module can filter the sensed data from the physical sensors and any auxiliary sensors according to the location estimate (e.g. from a pre-existing map).

[0108] iii. Guiding the user to place an auxiliary sensor (e.g. their smartphone) at a specific location.

[0109] iv. Guiding the user to place an auxiliary object at a given location, e.g. an RF or optical reflective surface (like a metamaterial) on an object (e.g. a door, window) that triggers a reflection of RF or optical signals or changes the RF or optical propagation when the object moves, or a QR code that can be used by the smart home system and / or hub to determine the location / area / home object to be sensed, so that e.g. beamforming can be trained / performed to this area. For example, a QR code can be attached to a plant that needs to be monitored by a camera, letting the camera identify this plant. In this way, a virtual plant monitoring sensor can be created to give an indication when the plant needs watering.

[0110] In another example, the opportunity event can be identified by one or more of: associated data from auxiliary sensors and / or objects that are known to be highly correlated with the event type (like a smartphone being highly correlated with the event type of a user walking); suggested potential virtual sensor locations to the user; associated data from several physical sensors that are similar in characteristics; and specific events that can be actively output to the user in the form of a gesture (as described later in connection with the second embodiment). Figure 4

[0111] In another example, the ground truth sensed data can be derived by using auxiliary sensors, auxiliary objects (which can be provided or placed in the target environment) and / or nearby physical sensors temporarily in non-standard mode (as described above in connection with step S302), and the resulting sensed data that is most strongly correlated with the opportunity event can be derived.

[0112] In yet another example, the event can originate from a device or a user.

[0113] In yet another example, the ground truth sensed data can be matched with the location data at the time of the opportunity event (e.g. before the process proceeds to step S304).

[0114] In the case that the simulation can be enhanced (ENH) by coordination between physical sensors and / or using non-standard operating modes, this can also be calculated by the hub in step S305.

[0115] ​In step S306, the hub can optionally further identify alternative subsets of physical sensors (ALT PS) that can emulate the temporary physical sensors with a lower (but still usable) accuracy.

[0116] In an example, the hub can compare data recorded by the physical sensors (e.g. in the "standard" mode) with the ground truth sensing data related to the opportunistic event. The hub can identify which subset of physical sensors can best emulate the ground truth data from the auxiliary sensor or the temporary non-standard physical sensor(s) and / or make use of the auxiliary object (e.g. using the techniques of reference 1).

[0117] Note that if the emulation does not reach a given threshold, the hub can also provide an indication to (replace) place an existing physical sensor (e.g. the hub can give a suggestion about an alternative location where a sensor (e.g. a loudspeaker or presence / light sensor) can be placed) and / or provide an indication to place an auxiliary object.

[0118] The goal of step S306 is to identify a subset of physical sensors that can best emulate the ground truth data (i.e. of the auxiliary sensor or the temporary physical sensor) when the opportunistic event occurs, in a known mode (preferably in its standard mode of operation, but optionally in a non-standard mode). This can require the hub to let the physical sensors physically run several modes (i.e. its standard mode and any relevant non-standard modes) during the discovery phase and compare the differences in the fidelity of the emulation. In this process, the hub can try to provide additional choreography to the physical sensors if it can increase the fidelity of the emulation, and measure whether this actually improves the emulation. Any such choreography details can be stored in the virtual sensor database.

[0119] In another example, the hub can use a machine learning approach (e.g. the machine learning approach of reference 1) to identify the subset(s) of physical sensors that best approximate the ground truth sensing data during the opportunistic event.

[0120] Optionally, a score indicating how close a subset of physical sensors emulates the ground truth data can be applied to rank the estimated performance of this subset. The hub can then identify several different subsets and their estimated performance. As another option, the subsets can be assigned a weight and / or other factors related to necessary transformations that should be applied to the sensing data of their physical sensors when emulating the opportunistic event.

[0121] In step S307, the properties and location of the temporary physical sensor or potential virtual sensor are stored in a database (e.g. a virtual sensor database) and matched with the subset(s) of physical sensors needed to emulate it, the operating mode(s) of these subsets and any coordination requirements. This mapping can then constitute the newly discovered virtual sensor.

[0122] In examples, one or more of the subset(s) of physical sensors, the related type and location of opportunity events they can emulate, the estimated fidelity and any orchestration details can be stored in the virtual sensor database.

[0123] The hub can then use this stored information to operate the virtual sensor during future uses, e.g. by applying the relevant operating and orchestration modes and related weights to the identified subset of physical sensors.

[0124] In step S308, the hub can present to the user via a user interface (UI) (e.g. the user interface 120 of the smart home environment 100) its discovered virtual sensors and their estimated performance, from which the user can select to commission a virtual sensor in a desired (given) location. In examples, the hub can present to the user the virtual sensor location, type and estimated performance. Figure 1

[0125] In examples, the hub can hide any physical sensors and only provide virtual sensors.

[0126] The commissioning of a virtual sensor (i.e. the granting of privileges related to its type and use in the network) can be achieved through any standard means (e.g. a commissioning process established within the applicable smart home environment) or through the commissioning process described below with reference to Figure 4

[0127] According to an implementation example, a user can wish to extend the alarm system for their smart home, for which the user can install an alarm actuator. Without adding (mobile) sensors to the doors / windows, the system suggests to the user virtual sensors and the locations at which these virtual sensors will be attached to the doors / windows. The virtual sensors can be configured to determine, e.g. whether a door is opened / closed, through wireless sensing (e.g. by using RF sensing from multiple devices in the smart home) and / or through a camera or other video device. Upon discovery of the virtual sensor at the door, the user can be requested to close / open the door and / or initiate another action that causes a change in RF propagation, which in turn determines whether and which door or window is opened / closed. If the change in RF propagation is insufficient, the user can be requested to attach, e.g. RF-reflective material (e.g. a transparent sticker) to the door or window.

[0128] Figure 4 ​​A flowchart schematically illustrating a virtual sensor commissioning procedure according to a second embodiment.

[0129] With reference to Figure 3 The above procedure of the first embodiment described is designed to, among other purposes, discover potential virtual sensors or temporary (learning) sensors and their performance.

[0130] However, if the opportunistic events used during the discovery phase are not well identified and / or do not match well with the typical events the virtual sensor is expected to sense, the virtual sensor performance can not be optimal yet.

[0131] Hence, an additional commissioning procedure is described that uses the moment of sensor commissioning as an additional opportunistic event that can be intelligently designed by the hub and used to try to improve the performance of the virtual sensor. The commissioning procedure can involve one or several of the following steps that can be repeated and / or performed in different order.

[0132] In step S401, the user indicates that he wishes to commission (COMM(VS)) the virtual sensor via the user interface and agrees to use, for example, gesture-based commissioning. The hub retrieves its relevant virtual sensor model from a database (e.g. virtual sensor database).

[0133] In step S402, in response to the user’s indication, the hub accesses a gesture design system to design a commissioning gesture (CG) to be performed by the user, where the gesture can involve a type or class of events that were not sufficiently covered in the observed opportunistic events and / or for which the existing sensor data model failed to model well, and are thus expected to boost the performance of the virtual sensor.

[0134] More specifically, the gesture design system can design one or several commissioning gestures based on several factors, which can include the following cases: According to a first case, designing the gesture can involve a type or class of events that were not sufficiently covered in the opportunistic events observed during the sensor discovery, or for which the ground truth sensing data is noisy or incomplete. In this case, the gesture can be designed to have the user provide an input such that the data from the virtual sensor in a known time period before and after the user interface provides a prompt can be associated with that gesture with a high certainty (e.g. higher than a predefined threshold). As an example, it can have been discovered that the virtual sensor senses walking motion in a particular location, but the opportunistic events during the sensor discovery only contained a few samples of the user walking slowly. A suitable commissioning gesture would be to direct the user via the user interface to go to the relevant location and walk slowly.

[0135] According to the second case, designing a demonstrative action can involve an action or event that cannot be directly sensed by a physical or virtual sensor but relies on data processing using a selected measurement model. In this case, the commissioning demonstrative action can be designed to act as an additional input to the measurement model, which enhances the model’s ability to migrate to new scenarios (e.g. reference 2). Furthermore, in this case, the commissioning demonstrative action can consist of a representative example of the event of interest, which the user is guided to perform in the new ‘scenario’ (i.e. the location where the virtual sensor is expected to sense). An example of this case can include a virtual sensor that is commissioned to equip a smart light in a new room with Wi-Fi sensing-based demonstrative action control using a pre-existing demonstrative action detection model (measurement model) that has been trained on data already captured in other rooms. The commissioning demonstrative action designed here can be the same as the expected on / off demonstrative action for the light in the existing rooms, but performed in the new scenario.

[0136] In either of the above first or second cases, the commissioning demonstrative action can also be designed to be performed by a trusted device (i.e. a device that has already been commissioned) rather than the user.

[0137] In step S403, if the user wishes to commission the virtual sensor, the user can be guided to perform a demonstrative action through the hub’s user interface (UI). During this time, the user’s actions are sensed by both the virtual sensor and, if available, any auxiliary sensor(s).

[0138] In an example, the designed commissioning demonstrative action is communicated to the user via the hub’s user interface (or via a trusted device of the network controller), and the user can choose to perform it.

[0139] The virtual sensor (and optionally, the auxiliary sensor(s)) are used to sense the user’s or device’s actions in completing the commissioning demonstrative action. The sensing data in the short time period after the user interface gives the user a prompt can be labelled as the most likely to be relevant to the prompted event.

[0140] The obtained sensing data can then be used as ground truth sensing data for the opportunistic event, and compared to the sensing data of each physical sensor individually. The virtual sensor simulation computation process of the above first embodiment can then be re-run on this new opportunistic event. Figure 3 The comparison can require the hub to alternate between using the physical sensor to simulate the virtual sensor (with its previously learned weights, settings and orchestration requirements) and using each physical sensor individually during the time period when the user performs the commissioning demonstrative action. This alternating can be done quickly in time to ‘sample’ the same event with both sets of sensors.

[0141] If the resulting physical sensor best subset is different from the one previously computed (or the best weights or orchestration requirements are different), the hub can update its virtual sensor model in the virtual sensor database.

[0142] In case the commissioning gesture has been used for downstream model enhancement, the model management function can be updated with new training data, and the model can be retrained as well.

[0143] In step S404, in case the action is sensed, the virtual sensor (VS) can be commissioned (COMM) (i.e. allowed to operate within the network with the appropriate rights persistently).

[0144] In step S405, the sensing data (SD) related to the execution of the commissioning gesture by the user can be stored and used by the hub (H) to enhance the performance of the virtual sensor (this can include adjusting the subset or weighting of physical sensors in use, and / or transferring training data related to the downstream model to the context of the virtual sensor).

[0145] Upon completion of the gesture by the user, the virtual sensor is commissioned using the updated parameters / orchestration details obtained above. If the user does not complete the commissioning gesture, or does not sense the expected commissioning gesture with sufficient fidelity, the user can be prompted via the user interface to complete it again, or enabled to commission the virtual sensor in another way (although the performance can be reduced).

[0146] It should be noted that the user can also not use the gesture, but select one of the suggested virtual sensor locations determined in the previous embodiments (e.g. via the user interface of the hub).

[0147] It should also be noted that the user can also not use the gesture, but place a QR code or a helper object where he wishes to place the virtual sensor.

[0148] It should also be noted that the external device can also drive the generation of the commissioning gesture (or alternative method) and exchange this data with the hub.

[0149] In the above case, Figure 5 The steps S402 to S404 of the process of the above case

[0150] Figure 5 A flowchart of a virtual sensor management process according to a third embodiment is schematically illustrated.

[0151] During the lifetime of a virtual sensor, testing and / or management can be required to enhance its performance. This is particularly important when physical sensors are for example added to the sensor network, or removed from the sensor network or change position / orientation. Therefore, the following describes a procedure similar to the second embodiment to identify or create opportunity events for virtual sensor management. The management procedure can involve one or several of the following steps which can be repeated and / or performed in a different order.

[0152] In step S501, the user and / or the hub can indicate a requirement to manage or test a virtual sensor that has been commissioned. To achieve this, the hub (H) can enter a management mode (MM).

[0153] In an example, the hub can identify a need for virtual sensor management (e.g. due to the addition of a new physical sensor to the network, or the removal of an already installed physical sensor) or a modification of the status. In response to this, the hub can enter the management mode. In an example, the identification procedure can involve collecting information from the sensors and monitoring for changes, and / or collecting mutual measurements from the sensors to identify for example a change in position of one of the sensors, and / or collecting movements of the sensors to identify a change in position of one of the sensors, and / or collecting system events (new device available, device not available, etc.), and / or collecting an exchange of capabilities with the devices (such as the capability to track movements of the device itself or other devices).

[0154] The hub can then access the choreography design system to design one or more management choreographies (or any alternative used in other embodiments). These management choreographies can be provided to the user or the devices and can be designed to best test the functionality of the virtual sensor. Note that this procedure can also be driven by a third device sending selected management choreographies to the hub.

[0155] If the management choreographies are to be performed by the user, the management choreographies can consist of actions or events that are most commonly or most reliably sensed by the virtual sensor. This enables the highest detection accuracy even if the removal of a physical sensor has caused a decrease in virtual sensor performance.

[0156] If the management choreographies are to be performed by the devices, the management choreographies can consist of a list of outputs (conditional on the feasibility of the output being produced by the target device), positions and / or times at which the devices should perform the management choreographies.

[0157] In step S502, the user or the devices can be instructed to perform the management choreographies (MG) at known positions at known times (which can be the same as the commissioning choreographies).

[0158] In one example, the management gesture can be communicated to the user via a user interface or to the device via a network controller.

[0159] In step S503, the virtual sensor senses the management gesture and compares the sensed data to the sensed data expected of the virtual sensor in normal operation.

[0160] During the time period in which the user / device performs the management gesture, the hub uses such events as opportunity events and uses the currently commissioned virtual sensor and each physical sensor (and available auxiliary sensors) individually or in combination to compare the sensed data associated with these events. The hub can then update its virtual sensor model in the manner of the second embodiment described above.

[0161] In an example, the hub can communicate to the user via a user interface that the virtual sensor has been updated.

[0162] In step S504, if no match is found (or the match is poor), the system can re-enter the discovery mode to attempt to discover an improved subset of physical sensors and / or an improved operating mode of the existing subset of physical sensors to enhance the virtual sensor.

[0163] In one example, if removing a physical sensor causes a substantial degradation in the performance of the virtual sensor, the user can be alerted via a user interface and, if necessary, can optionally be guided to reinstall which physical sensor to restore the virtual sensor.

[0164] As an alternative to the above-described management based on gestures of the third embodiment, the hub can instead use opportunity events to periodically re-run the virtual sensor discovery process of the first embodiment.

[0165] Figure 6 A top view schematically illustrating an example of virtual sensor commissioning in an example sensor environment.

[0166] The thick black lines indicate the walls of the building of the sensor environment (e.g., a home network). The shaded dots indicate physical sensors, and the solid dots indicate potential virtual sensors that have been identified.

[0167] In step S1, a network of physical sensors (shaded circular areas) sense data from the scene and can run a machine learning model. In step S2, the hub identifies a temporary sensor at a known location (e.g., a user’s smartphone) and uses this temporary sensor to predict potential virtual sensors (black circular areas), their locations and properties, and displays options to the user (e.g., via the smartphone’s display). To commission the virtual sensors, the user is guided in step S3 to perform some actions (commissioning gestures) at their intended physical locations, which likely enhance the model’s migration to new scenes. The white circular areas can be auxiliary sensors or unregistered physical sensors.

[0168] It should be noted that in the above embodiments, the application for discovery, commissioning and / or management can specify one or more policies that specify requirements for real sensors to define a particular type of virtual sensor. The hub can take this information into account next to the map of sensors and (smart) home deployed in the system to determine the location and give the user an indication about the location.

[0169] Policies express how the behavior of the system is managed. They express higher-level goals that are implemented automatically. Many policy languages have been defined. As an example, the system can support the Adaptable and Programmable Policy Environment and Language (APPEL) as a policy language.

[0170] Policies can be programmed in a chosen policy language, or a policy wizard can be designed to allow non-computing users to create and edit policies. This wizard can be web-based, which allows defining policies anywhere using a familiar interface.

[0171] In summary, apparatuses / methods have been described that can be provided on a smart device such as a smartphone or a smart hub, which can be able to commission and / or manage virtual sensors / actuators by one or more of the following ways: gathering requirements about virtual sensors / actuators, determining virtual sensors / actuators based on existing sensors / actuators (e.g., by coordinating sensors / actuators in one or more networks and / or by temporary sensors / actuators and opportunistic events), determining parameter(s) of virtual sensors / actuators given measurements of existing sensors / actuators, and exposing virtual sensor / actuator parameter(s).

[0172] Although the embodiments are described in the context of home networks and virtual reality, their application is not limited to this type of operation. They can be applied to any device or network of devices in which there are multiple sensors and there can be some correlation between the sensor outputs, for example, applications for healthcare, shops, shopping malls, sports clubs, etc. Thus, they can be applied to single multi-sensor devices such as smartphones, as well as loT or 5G networks, etc.

[0173] Furthermore, at least some embodiments can be applied to various types of UE or terminal devices, such as mobile phones, vital signs monitoring / telemetry devices, smart watches, detectors, vehicles (for vehicle-to-vehicle (V2V) or more generally vehicle-to-everything (V2X) communications), V2X devices, loT hubs, loT devices (including low-power medical sensors for health monitoring), medical (emergency) diagnostic and treatment devices for hospital use or first responder use, virtual reality (VR) headsets, etc.

[0174] Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art upon the study of the drawings, the disclosure, and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit can implement the functionality of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The foregoing description details certain embodiments. It will be appreciated, however, that no matter how detailed the foregoing appears in text, the embodiments can be practiced in many ways, and are therefore not limited to the embodiments disclosed. It should be noted that the use of particular terminology when describing certain features or aspects is not intended to limit the

[0175] A single unit or device can implement the functionality of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0176] Operations similar to those shown in the above-described embodiments (e.g., Figures 3 to 5 ) can be implemented as program code modules of a computer program and / or as dedicated hardware of the associated network device or function, respectively (e.g., Figure 1The computer program can be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid state medium supplied together with, or as part of other hardware, but can also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.

[0177] References: 1. Ant Colony Inspired Machine Learning Algorithm for Identifying and Emulating Virtual Sensors, Mani et al. https: / / arxiv.org / pdf / 2011.00836.pdf 2. Small CSI Samples-Based Activity Recognition: A Deep Learning Approach Using Multidimensional Features, Tian et al., 2021.

Claims

1. An apparatus (32) for discovering, debugging, or managing virtual data or action generating devices (210) in a network (20), wherein, The device (110) is configured to: Information is collected indicating at least one virtual data or action generating device (210); Send a request to the available temporary auxiliary equipment to provide information; The at least one virtual data or action generating device (210) is determined based on the collected information and information about existing data or action generating devices (220, 230); Based on the measurement results of the existing data or motion generating devices (220, 230) and the information from the available temporary auxiliary devices, at least one parameter of the at least one virtual data or motion generating device (210) is determined; and Output at least one parameter of the at least one virtual data or action generating device (210) determined by the output.

2. The apparatus according to claim 1, wherein, The device (110) is configured to determine the at least one virtual data or action generating device (210) by coordinating existing data or action generating devices (220) in the network (20) and / or other networks.

3. The apparatus according to claim 2, wherein, The coordination is achieved through time synchronization and / or beamforming.

4. The apparatus according to any one of the preceding claims, wherein, The device (110) is configured to acquire the information indicating the at least one virtual data or motion generating device (210) by sending an information request to an available physical data or motion generating device (220).

5. The apparatus according to any one of the preceding claims, wherein, The device (110) is also configured to compare the baseline truth information with information collected from the existing data or motion generating devices (220, 230) and estimate a virtual device model, the virtual device model determining the at least one parameter of the at least one virtual data or motion generating device (210).

6. The apparatus according to any one of the preceding claims, wherein, The device (110) is configured to determine the at least one virtual data or action generating device (210) by means of an opportunistic event and one or more temporary data or action generating devices.

7. The apparatus according to claim 6, wherein, The opportunity event involves the movement or action of a user or other person, and / or the movement or action of a device, for which reference true sensing data is available.

8. The apparatus according to claim 7, wherein, The device (110) is configured to simulate the at least one virtual data or motion generating device (210) using a set of existing physical data or motion generating devices (220).

9. The apparatus according to claim 7 or 8, wherein, The device (110) is configured to provide the reference truth sensing data in a non-standard mode by using and associating a usable data stream from a temporary external source that is not part of a set of data or motion generating devices (220) for providing the at least one virtual data or motion generating device (210), or from the use of at least one data or motion generating device (220) that is part of the set of data or motion generating devices (220) for providing the at least one virtual data or motion generating device (210).

10. The apparatus according to claim 9, wherein, The non-standard mode is the high-energy or high-sampling-rate mode.

11. The apparatus according to any one of the preceding claims, wherein, The device (110) is configured to enhance or test the functionality of the virtual data or action generating device (210) by designing at least one movement or action to be performed by a user and / or device to add baseline truth data.

12. A hub device (10) for a network (20) comprising one or more physical data or action generating devices (220), the hub device comprising means according to any one of claims 1 to 11.

13. A mobile device comprising the means according to any one of claims 1 to 11.

14. A system for discovering, debugging, or managing virtual data or motion generating devices (210), the system comprising a network (20) having one or more physical data or motion generating devices (220) and a hub device (10) according to claim 12.

15. A method for discovering, debugging, or managing virtual data generating devices (210) in a network (20), wherein, The method includes: Information is collected indicating at least one virtual data or action generating device (210); Send a request to the available temporary auxiliary equipment to provide information; The at least one virtual data or action generating device (210) is determined based on the collected information and information about existing data or action generating devices (220, 230); Based on the measurement results of the existing data or motion generating devices (220, 230) and the information from the available temporary auxiliary devices, at least one parameter of the at least one virtual data or motion generating device (210) is determined; and Output at least one parameter of the at least one virtual data or action generating device (210) determined by the output.

16. A computer program product comprising a code module for generating the steps of claim 15 when run on a computer device.

Images