Performing a sensing task with vibratory sensing surfaces

Abstract

A system (1) for performing a sensing task is configured to, in a set-up phase, transmit radar signals via at least one network device (11) and receive, via the at least one network device, a plurality of return signals in response. Each of the radar signals is transmitted with a different radar setting. The system is further configured to classify each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal, classify one or more return signals which are or correspond to vibratory return signals into one or more vibration classes, and store identifiers of vibration sensing surfaces (13, 14). Each of the vibration sensing surfaces corresponds to at least one of the one or more return signals and is associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device.

Description

[0001] PERFORMING A SENSING TASK WITH VIBRATORY SENSING SURFACES

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a system for performing a sensing task and to a sensing device.

[0004] The invention further relates to a method of performing a sensing task and to a method of receiving an information request.

[0005] The invention also relates to computer program products enabling a computer system to perform such methods.

[0006] BACKGROUND OF THE INVENTION

[0007] Classification of vibration, e.g. audio, signals has long been used to determine useful information about the environment, such as machine condition monitoring, traffic monitoring, home activity monitoring, and infrastructure monitoring. In recent years these methods primarily rely on machine learning methods to perform classifications of vibration, e.g. audio, events and derive information from the vibration, e.g. audio, signals. Such methods commonly rely on a neural network, where the neural network is initially trained on labelled examples of vibration data, e.g. audio data, and which then may be deployed on new, unlabeled data to output predicted classifications. Classifiers of this kind may be generic, in that they run on an unknown vibration, e.g. audio, snippet to assign it a set of broad classes (e.g. “human voice” vs “footfall” vs “traffic noise” vs “dog bark” etc.), or they may be specific, in that they are provided a vibration, e.g. audio, snippet that is known to contain a certain class of data (e.g. “human voice”), and they classify specific aspects of that data (e.g. “speaker gender”, “voice frequency”, “language being spoken”, etc.).

[0008] Due to the great popularity, utility and accessibility of audio classifiers, recent work has been done to utilize these classifiers in different domains, i.e. running on RF data from a radar device rather than audio data from a microphone device. Much recent work has been focused on radar extraction and classification of audio, particularly speech audio. More recently, it has been demonstrated that vibration, e.g. audio, information can be extracted via doppler radar from vibration sensing surfaces in an environment, i.e. surfaces, e.g. of objects, which are vibrating due to another vibration source, rather than themselves being vibration sources, e.g. sound sources. For example, high quality speech of people in a room may be extracted from doppler radar signals monitoring nearby vibrating picture frames.

[0009] The paper “Vital signs monitoring using pseudo-random noise coded Doppler radar with Delta-Sigma modulation” by Abouzaid, S.H., Ahmad, W.A., Eibert, T.F., & Ng, H.J., published in IET Radar, Sonar & Navigation (2020), describes the use of doppler radar for vital signs monitoring. A drawback of the existing methods of extracting vibration information via doppler radar from vibration sensing surfaces is that they require a priori knowledge of which vibration sensing surfaces exist and which sensing functions they enable.

[0010] SUMMARY OF THE INVENTION

[0011] It is advantageous to provide a system, which can be used to perform a sensing task with a vibration sensing surface without a priori knowledge of which vibration sensing surfaces exist and which sensing functions they enable.

[0012] It is advantageous to provide a method, which can be used to perform a sensing task with a vibration sensing surface without a priori knowledge of which vibration sensing surfaces exist and which sensing functions they enable.

[0013] In a first aspect, a system for performing a sensing task comprises at least one processor configured to transmit a plurality of radar signals via at least one network device, each of the plurality of radar signals being transmitted with a different radar setting of a plurality of radar settings, receive, via the at least one network device, a plurality of return signals in response to the transmitted plurality of radar signals, classify each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal, classify one or more return signals which are or correspond to a vibratory return signal into one or more vibration classes of a plurality of vibration classes, and store identifiers of a plurality of vibration sensing surfaces, each of the plurality of vibration sensing surfaces corresponding to at least one of the one or more return signals, each of the plurality of vibration sensing surfaces being associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device.

[0014] The at least one processor is further configured to receive a sensing task request, the sensing task request requesting the performance of the sensing task, the sensing task request indicating a vibration class of the plurality of vibration classes, retrieve, based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces, the one or more vibration sensing surfaces being associated with one or more network devices of the at least one network device, transmit, via the one or more network devices, one or more further radar signals towards the one or more vibration sensing surfaces, receive, via the one or more network devices, one or more further return signals in response to the one or more further radar signals, classify the one or more further return signals into one or more further vibration classes of the plurality of vibration classes, and output sensing results based on the one or more further vibration classes.

[0015] The plurality of return signals comprise both vibratory and non-vibratory return signals. On the other hand, the one or more return signals comprise return signals which are or correspond to a vibratory return signal. The one or more return signals may be a subset of the plurality of return signals or might not be included in the plurality of return signals but correspond to a subset of the plurality of return signals. In the latter case, the one or more return signals and the subset of the plurality of return signals may be received in response to radar signals transmitted with the same sets of radar settings. The vibratory return signals may be acoustic signals, the vibration sensing surfaces may be acoustic sensing surfaces, and the vibration classes may be audio classes, for example.

[0016] The system identifies useful surfaces in the environment where vibratory measurements may be taken by radar and identifies which sensing functions they enable. This may be done periodically, e.g. once per week, or upon some known change the network or environment around a network device. This may be done periodically at different times of the day. A network device is a device which can communicate with another device and has an RF antenna.

[0017] Since classifying return signals into vibration classes is a relatively complex process, it is beneficial to first classify each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal and then classify (only) the return signals which are or correspond to the vibratory return signals into vibration classes.

[0018] With this system, it may be possible to detect many kinds of vibration, e.g. audio, events in an environment with network devices capable of (e.g. doppler) radar sensing, as long as the radar can be directed toward those surfaces, e.g. of objects, within the environment that are useful for vibration, e.g. audio, data extraction. The availability of mmWave radar as part of Integrated Sensing and Communications (ISAC) systems in 5G+ networks means network devices in these networks can leverage the vibratory environment around them for performing a wide range of sensing tasks in the RF domain.

[0019] The system may comprise one or more base stations and / or one or more UEs (User Equipment) or may be a standalone system. As an example of the latter, the standalone system may comprise a dedicated radar device with internal memory. The performance of the sensing task may be centralized, e.g. performed by an appointed base station, by an appointed UE, or by the standalone system, or may be distributed.

[0020] If the sensing task is performed by a central component, either the central component or each network device may classify the one or more return signals into the one or more vibration classes and store the identifiers of the plurality of vibration sensing surfaces in the central component, for example in a memory or in an external database.

[0021] The at least one processor may be configured to classify each return signal of the one or more return signals into the one or more vibration classes of the plurality of vibration classes by, for each respective return signal of the one or more return signals, denoising the respective return signal, extracting, from the denoised return signal, doppler information on doppler shifts in the frequency domain, determining dominant vibrational frequencies based on the extracted doppler information, reconstructing the dominant vibrational frequencies in the time domain to produce a vibratory waveform, and classifying the vibratory waveform into the one or more vibration classes. This is a beneficial implementation that enables the one or more return signals to be classified.

[0022] The at least one processor may be configured to transmit an information request to a sensing device, the information request comprising data representing the vibratory waveform and requesting the sensing device to compare the vibratory waveform with a further vibratory waveform sensed by the sensing device in a time period, the information request indicating the time period, receive information from the sensing device in response to the information request, the information indicating a degree of similarity between the vibratory waveform and the further vibratory waveform, ascertain a function and / or device identifier of the sensing device, and associate the vibration sensing surface with the function and / or device identifier of the sensing device in dependence on the degree of similarity. For example, if (e.g. dedicated) sensing devices exist in the environment which have already been identified and localized (e.g., a vibration sensor attached to a machine in a factory use case), the sensing function of vibratory sensing surfaces may be learned with help of the sensing devices.

[0023] The at least one processor may be configured to transmit a data request for a further vibratory waveform to a sensing device, the further vibratory waveform being sensed by the sensing device in a time period, the data request indicating the time period, receive the further vibratory waveform from the sensing device, ascertain a function and / or device identifier of the sensing device, determine a measure of similarity between the vibratory waveform and the further vibratory waveform, compare the measure of similarity with a threshold, and associate the vibration sensing surface with the function and / or device identifier of the sensing device if the measure of similarity is determined to exceed the threshold. For example, if (e.g. dedicated) sensing devices exist in the environment which have already been identified and localized (e.g., a vibration sensor attached to a machine in a factory use case), the sensing function of vibratory sensing surfaces may be learned with help of the sensing devices. If the sensing device is willing to share sensing data, the system could determine the measure of similarity between the vibratory waveform and the further vibratory waveform itself.

[0024] The radar signals may comprise continuous wave radar signals, e.g. specifically phase-modulated continuous wave radar signals, e.g. even more specifically pseudo-random noise coded phase-modulated continuous wave radar signals. It is beneficial to use a radar scheme capable of making low-frequency vibration measurements despite noise, such as a radar scheme which uses pseudo-random noise coded phase-modulated continuous wave radar signals.

[0025] Transmitting each of the plurality of radar signals with a different radar setting and receiving the return signals may comprise stepwise adjusting a phase modulation of the transmitted radar signals and the received return signals and / or stepwise adjusting a relative phase delay of a radar antenna array, the radar signals being transmitted over the radar antenna array. For example, the different radar settings may correspond to different combinations of heading / direction and depth / range.

[0026] A common problem with using continuous wave (CW) radar to extract information from small vibrations (such as audio signals) is noise. Vibrations in audio and similar applications are usually of a very low amplitude, so noise suppression in practical scenarios (real-world and not lab-controlled) is critical. In phase modulated CW (PMCW) radars, such as pseudo-random noise (PSN) radars, the phase of the transmitted and received signals is modulated with a pseudo-random binary sequence. PSN may be generated and modulated onto the transmit path, and a delayed version of the same PSN signal may be modulated onto the receive path, producing interference which suppresses all resulting signals in the receive path except for those at a desired depth away from the antenna.

[0027] By adjusting the delay in the PSN modulations between the transmit and receive paths, the depth at which the radar is most sensitive may be “tuned”, allowing range selectivity. By using this approach, the authors of the above-mentioned paper “Vital signs monitoring using pseudo-random noise coded Doppler radar with Delta-Sigma modulation” were able to detect heart-rate amplitude signals in noisy environments, by precisely “tuning” the radar to the desired distance.

[0028] Each of the plurality of vibration sensing surfaces may be associated with a spatial location or area, the sensing task request may indicate a target spatial location or area, and the at least one processor may be configured to retrieve the one or more identifiers of the one or more vibration sensing surfaces further based on the spatial locations or areas of the plurality vibration sensing surfaces and the target spatial location or area. This may allow a more specific set of one or more network devices to be used for performing the sensing task by not involving network devices which cannot be used for sensing in the target spatial location or area, thereby saving resources.

[0029] Each respective vibration sensing surface of the plurality of vibration sensing surfaces may be associated with a spatial location or area of a network device which received a return signal corresponding to the respective vibration sensing surface. Since the spatial location of a network device is often available, it is trivial to associate the spatial location of the network device, or a spatial area around this location, with the vibration sensing surface.

[0030] The at least one processor may be configured to, for each respective vibration sensing surface of the plurality of vibration sensing surfaces, corresponding to a respective return signal of the one or more return signals, estimate a spatial location or area of the respective vibration sensing surface based on the respective return signal and / or based on a radar setting with which a respective radar signal of the plurality of radar signals was transmitted, the respective radar signal causing the respective return signal to be returned, and associate the spatial location or area of the respective vibration sensing surface with the respective vibration sensing surface. This may allow an even more specific set of one or more network devices to be used for performing the sensing task, as the spatial location or area of the vibration sensing surface is normally even more relevant than the location of the network device.

[0031] The spatial location or area may be estimated by inferring the angle of the vibration sensing surface with respect to the network device by calculating the resulting beam direction of the transmitted radar signal from the relative phase delays of the antennas of the radar hardware and by determining the range of the vibratory sensing surface with respect to the network device from the magnitude of the PSN modulation delay, for example. Since it may not be possible determine a spatial location of the vibration sensing surface precisely, it may be beneficial to estimate and use a spatial area.

[0032] The plurality of vibration classes may comprise a plurality of generic vibration classes and a plurality of specific vibration classes and the at least one processor may be configured to classify the one or more return signals into the one or more vibration classes by classifying the one or more return signals which are or correspond to vibratory return signals into a generic vibration class of the plurality of generic vibration classes, and classifying, based on the generic vibration class, the one or more return signals into a specific vibration class of the plurality of specific vibration classes. Thus, a generic vibration classification may be produced as a first step, to identify the broader class of audio event detected, e.g. “traffic noise”, “human voices”, etc. For some sensing tasks, the generic vibration classification may be enough information, e.g. a sensing task with a goal of determining if traffic noise is present or not. A specific vibration classification enables more refined sensing tasks for a given generic class of vibration information, once it has been established that that generic class of audio information is present. For example, for traffic noise, specific audio classifications may enable sensing tasks of vehicle counts, size of vehicle estimations, vehicle type estimations, etc.

[0033] In a second aspect of the invention, a sensing device comprises at least one processor, the at least one processor being configured to receive an information request from a system, the information request comprising data representing a vibratory waveform and indicating a time period, retrieve a further vibratory waveform, the further vibratory waveform relating to the time period, determine a measure of similarity between the vibratory waveform and the further vibratory waveform, and transmit information to the system in response to the information request, the information being determined based on the measure of similarity and indicating a degree of similarity between the vibratory waveform and the further vibratory waveform. The information may comprise the measure of similarity of may indicate whether the measure of similarity exceeds a threshold, for example. Thus, the degree of similarity may be indicated in different ways. The further vibratory waveform may be retrieved from a memory included in the sensing device or from another system, e.g. an external database, for example.

[0034] In a third aspect, a method of performing a sensing task comprises transmitting a plurality of radar signals via at least one network device, each of the plurality of radar signals being transmitted with a different radar setting of a plurality of radar settings, receiving, via the at least one network device, a plurality of return signals in response to the transmitted plurality of radar signals, classifying each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal, classifying one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes, and storing identifiers of a plurality of vibration sensing surfaces, each of the plurality of vibration sensing surfaces corresponding to at least one of the one or more return signals, each of the plurality of vibration sensing surfaces being associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device, The method further comprises receiving a sensing task request, the sensing task request requesting the performance of the sensing task, the sensing task request indicating a vibration class of the plurality of vibration classes, retrieving, based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces, the one or more vibration sensing surfaces being associated with one or more network devices of the at least one network device, transmitting, via the one or more network devices, one or more further radar signals towards the one or more vibration sensing surfaces, receiving, via the one or more network devices, one or more further return signals in response to the one or more further radar signals, classifying the one or more further return signals into one or more further vibration classes of the plurality of vibration classes, and outputting sensing results based on the one or more further vibration classes.

[0035] In a fourth aspect, a method of receiving an information request comprises receiving an information request, the information request comprising data representing a vibratory waveform and indicating a time period, retrieving a further vibratory waveform, the further vibratory waveform relating to the time period, determining a measure of similarity between the vibratory waveform and the further vibratory waveform, and transmitting information in response to the information request, the information being determined based on the measure of similarity and indicating a degree of similarity between the vibratory waveform and the further vibratory waveform.

[0036] Moreover, a computer program for carrying out the methods described herein, as well as a non-transitory computer readable storage-medium storing the computer program are provided. A computer program may, for example, be downloaded by or uploaded to an existing device or be stored upon manufacturing of these systems.

[0037] A non-transitory computer-readable storage medium stores at least a first software code portion, the first software code portion, when executed or processed by a computer, being configured to perform executable operations for performing a sensing task.

[0038] The executable operations comprise transmitting a plurality of radar signals via at least one network device, each of the plurality of radar signals being transmitted with a different radar setting of a plurality of radar settings, receiving, via the at least one network device, a plurality of return signals in response to the transmitted plurality of radar signals, classifying each of the plurality of return signals as a vibratory return signal or a non- vibratory return signal, classifying one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes, and storing identifiers of a plurality of vibration sensing surfaces, each of the plurality of vibration sensing surfaces corresponding to at least one of the one or more return signals, each of the plurality of vibration sensing surfaces being associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device.

[0039] The executable operations further comprise receiving a sensing task request, the sensing task request requesting the performance of the sensing task, the sensing task request indicating a vibration class of the plurality of vibration classes, retrieving, based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces, the one or more vibration sensing surfaces being associated with one or more network devices of the at least one network device, transmitting, via the one or more network devices, one or more further radar signals towards the one or more vibration sensing surfaces, receiving, via the one or more network devices, one or more further return signals in response to the one or more further radar signals, classifying the one or more further return signals into one or more further vibration classes of the plurality of vibration classes, and outputting sensing results based on the one or more further vibration classes.

[0040] A non-transitory computer-readable storage medium stores at least a second software code portion, the second software code portion, when executed or processed by a computer, being configured to perform executable operations for receiving an information request.

[0041] The executable operations comprise receiving an information request, the information request comprising data representing a vibratory waveform and indicating a time period, retrieving a further vibratory waveform, the further vibratory waveform relating to the time period, determining a measure of similarity between the vibratory waveform and the further vibratory waveform, and transmitting information in response to the information request, the information being determined based on the measure of similarity and indicating a degree of similarity between the vibratory waveform and the further vibratory waveform.

[0042] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a device, a method or a computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system." Functions described in this disclosure may be implemented as an algorithm executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon. Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium may include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer readable storage medium may be any tangible medium that can contain, or store, a program for use by or in connection with an instruction execution system, apparatus, or device.

[0043] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0044] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java(TM), Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or a central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0045] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0046] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0047] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s).

[0048] It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0049] BRIEF DESCRIPTION OF THE DRAWINGS

[0050] These and other aspects of the invention are apparent from and will be further elucidated, by way of example, with reference to the drawings, in which:

[0051] Fig. 1 is a flow chart of a first embodiment of the method of performing a sensing task;

[0052] Fig. 2 is a flow chart of a second embodiment of the method of performing a sensing task;

[0053] Fig. 3 is a flow chart of part of a third embodiment of the method of performing a sensing task;

[0054] Fig. 4 is a flow chart of part of a fourth embodiment of the method of performing a sensing task;

[0055] Fig. 5 is a flow chart of a fifth embodiment of the method of performing a sensing task;

[0056] Fig. 6 is a flow chart of a part of a sixth embodiment of the method of performing a sensing task;

[0057] Fig. 7 is a flow chart of a part of a seventh embodiment of the method of performing a sensing task and of an embodiment of the method of receiving an information request;

[0058] Fig. 8 is a flow chart of a part of an eighth embodiment of the method of performing a sensing task;

[0059] Fig. 9 is a block diagram of a first embodiment of the system; and

[0060] Fig. 10 is a block diagram of a second embodiment of the system and of an embodiment of the sensing device; and

[0061] Fig. 11 is a block diagram of an exemplary data processing system for performing the method of the invention.

[0062] Corresponding elements in the drawings are denoted by the same reference numeral.

[0063] DETAILED DESCRIPTION OF THE DRAWINGS

[0064] A first embodiment of the method of performing a sensing task is shown in Fig. 1. The method may be performed by system 1 of Fig. 9, for example. The method comprises a set-up phase and a use phase. In the set-up phase, vibration sensing surfaces in the vicinity of one or more given network devices are identified and registered, and sensing tasks which may be enabled by these vibration sensing surfaces are learned.

[0065] A Vibration Sensing Surface (VSS) is a surface, e.g. of an object, in the environment of a radar-capable network device which, when radar sensed with an appropriate radar technique, encodes the return radar signal with vibration, e.g. acoustic, information. A vibration sensing surface may be, for example, a particular passive object (e.g. a picture frame, a wall region, a window), which detectably vibrates due to vibratory, e.g. acoustic, emissions in its vicinity, or an active object (e.g. a speaker, a machine, a vehicle), which actively creates vibratory, e.g. acoustic, emissions.

[0066] Network devices may not have a global overview of their environment and from the perspective of a network device, vibration sensing surfaces may be defined by the set of radar settings the network device deploys such that it can gain vibratory information from the environment. In other words, a network device does not necessarily “know” that it is monitoring the vibration of a picture frame specifically to gain speech information in a room; it just “knows” that controlling its radar settings such that it detects signals at a particular heading and depth results in discernable speech information.

[0067] The set-up phase comprises steps 101, 103, 104, 105, 107, and 108. The use phase comprises steps 109, 111, 113, 115, 117, and 119. Step 101 comprises transmitting a plurality of radar signals via at least one network device. Each of the plurality of radar signals is transmitted with a different radar setting of a plurality of radar settings. The radar signals may comprise continuous wave radar signals, for example. The continuous wave radar signals may be phase-modulated, for example. The phase-modulated continuous wave radar signals may be pseudo-random noise coded, for example. Step 103 comprises receiving, via the at least one network device, a plurality of return signals in response to the plurality of radar signals transmitted in step 101.

[0068] Step 101 may comprise, for example, generating a radar settings table at a system and passing it from the system to a network device. Each network device may be provided an own radar settings table or multiple network devices may receive the same radar settings table. If it is the first time the set-up phase is being conducted, a pre-made default radar settings table may be provided for the specific radar hardware of a network device. Alternatively, a radar settings table may be generated such that a particular unknown region of the environment around the network device may be investigated to identify potential vibration sensing surfaces.

[0069] The radar settings table consists of a table, where each row defines a specific set of different radar settings, such that when the radar controller of the network device applies the radar settings row by row, different transmit radar signals result. The radar settings table may be ordered such that the values of radar settings in each row are an incremental difference on the values of the previous row. For example, relative phase delays of the antennas may be incremented row-by-row to result in an incrementally changing transmit radar signal beam angle and / or the magnitude of the delay between PSN phase modulations at the transmit and receive radar signals may be incremented row-by-row to result in an incremental change in depth selection.

[0070] Incremental row-by-row changes to antenna phase delay values between a maximum and minimum value would result in “sweeping” the transmit radar signal across a particular angle. Incremental row-by-row changes to PSN phase modulation delay between transmit and receive antennas would result in increased sensitivity to specific ranges from the network device. Thus, in this example, steps 101 and 103 comprise stepwise adjusting a phase modulation of the transmitted radar signals and the received return signals and / or stepwise adjusting a relative phase delay of a radar antenna array over which the radar signals are transmitted.

[0071] If a radar settings table is provided to a network device, the network device then adjusts its radar settings in sequence in accordance with the radar settings table. For each step in the sequence, i.e. for each radar setting, the radar hardware transmits a transmit radar signal and receives a return radar signal. The return radar signals are then provided to the system which passed the radar settings table.

[0072] A network device may, for example, be a base station connected to a 5G+ network, performing both communications and sensing functions. The network device comprises radar hardware capable of performing the required radar measurements. The radar hardware is capable of performing radar functions for gathering vibration data from vibration sensing surfaces, which may involve creating specific waveforms and coding schemes on the transmitted and returned radar signals, such as the above-mentioned pseudo-random noise (PSN) coded phase-modulated continuous wave (PMCW) radar signals. PSN-coded PMCW doppler radar are beneficial due to their enablement of distance-selection during use, enabling recovery of very low-amplitude vibratory signals at specific ranges.

[0073] Step 104 comprises classifying each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal. The vibratory return signals may be acoustic signals, for example. If a return signal is classified as a vibratory return signal, it is likely a vibratory return signal, but the classifier does not need to be 100% accurate. A pretrained classifier may be used on each of the return signals to perform the binary vibration / non-vibration classification. The vibration signal classifier may have been trained on examples of radar data measurements performed on known vibrating, e.g. audioproducing, surfaces. Identifying features of a signal that identify that the signal is likely a vibration signal may include amplitudes of vibrations present in the radar signal and / or the frequency range in which the detected vibrations fall, for example.

[0074] Step 105 comprises classifying one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes. Step 105 may comprise passing each of the one or more return signals to a vibration signal classifier and receiving a classification per return signal. The vibration classes may comprise generic vibration classes and / or specific vibration classes. The vibration classes may be audio classes, for example. Generic audio classification may consist of a classification of the broad class of an event, e.g. traffic noise, machine noise, footsteps, or human voices. Optionally, step 105 is implemented by a step 130. Step 130 comprises denoising the one or more return signals, processing the one or more denoised return signals and classifying the one or more processed, denoised return signals into the one or more vibration classes.

[0075] Step 107 comprises storing identifiers of a plurality of vibration sensing surfaces. Each of the plurality of vibration sensing surfaces corresponds to at least one of the one or more return signals. For example, a Vibration Sensing Surface identifier (VSS identifier) may be generated for each respective return signal of the one or more return signals and this VSS identifier may be associated with the set of radar settings with which the radar signal was transmitted which resulted in the receipt of the respective return signal. The vibration sensing surfaces may be acoustic sensing surfaces, for example.

[0076] Step 107 further comprises associating each of the plurality of vibration sensing surfaces with a network device of the at least one network device, i.e. with an identifier of this network device.

[0077] Step 108 comprises associating each of the plurality of vibration sensing surfaces with the one or more vibration classes into which the corresponding at least one return signal was classified, i.e. with one or more identifiers of these one or more vibration classes. These identifiers of the one or more vibration classes identify which sensing tasks are enabled by the vibration sensing surfaces and may be referred to as sensing task identifiers.

[0078] The VSS identifiers may be stored in a database. Other information could be associated with the VSS identifiers in multiple ways. In one of these multiple ways, the database stores one Network Device (ND) identifier and zero or more VSS identifiers per network device and further stores one set of radar settings and zero or more Sensing Task (ST) identifiers per VSS identifier. VSS identifiers are only generated and stored for return signals which are classified as vibratory return signals. At the start of the use phase, step 109 comprises receiving a sensing task request. The sensing task request requests the performance of the sensing task. The sensing task request indicates a vibration class of the plurality of vibration classes e.g. by specifying a sensing task identifier. For a given vibration sensing surface, monitored vibration, e.g. acoustic, signals, may be utilized for multiple sensing functions. For example, a vibrating region of a surface of a bridge support may be useful for both traffic monitoring and infrastructure health monitoring. Therefore, multiple sensing tasks may be possible for any given vibration sensing surface. Sensing tasks may consist of any sensing tasks which utilize vibration data to perform a classification.

[0079] A sensing task request may indicate a particular sensing task to be performed, identified by a sensing task identifier, and other specific information related to the task, such as any time constraints, like the time in which the sensing task must be performed (e.g. a particular time window defined by start and end timestamps) and / or such as a physical location at which the sensing task should be completed.

[0080] This physical location may be indicated by specifying an identifier of a network device (e.g. “measure vehicle traffic volume in the vicinity of Base Station X”), by specifying an identifier of a vibration sensing surface (e.g. “measure machine X on / off state via vibration sensing surface Y”), or by specifying a spatial location given in some shared coordinate reference system (e.g. “measure vibration at coordinates XYZ”).

[0081] Step 111 comprises retrieving, based on the vibration class indicated in the sensing task request received in step 109, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces. The one or more vibration sensing surfaces are associated with one or more network devices of the at least one network device. Step 111 may comprise searching for the specified sensing task identifier in the above-mentioned database and retrieving the associated network device identifier(s) and the associated set(s) of radar settings.

[0082] Step 113 comprises transmitting, via the one or more network devices identified in step 111, one or more further radar signals towards the one or more vibration sensing surfaces. Step 115 comprises receiving, via the one or more network devices, one or more further return signals in response to the one or more further radar signals. For example, the system may request the network device(s) identified in step 111 to participate in the sensing task, and provide the sets(s) of radar settings retrieved in step 111 plus any other requirements of the sensing task request (e.g. timing) to these one or more network devices. The participating network device(s) adjust their radar settings to match the provided radar settings, transmit a transmit radar signal, and receive a return radar signal, which is then provided to the system. Step 117 comprises classifying the one or more further return signals into one or more further vibration classes of the plurality of vibration classes. Optionally, step 117 is implemented by step 130. Step 130 comprises denoising the one or more further return signals, processing the one or more denoised further return signals and classifying the one or more processed, denoised further return signals into the one or more further vibration classes.

[0083] Step 119 comprises outputting sensing results based on the one or more further vibration classes obtained in step 117. For example, the outputted sensing results may identify the one or more further vibration classes. The vibration classification(s) may be stored in a standard format as the sensing task result and returned to the originator of the sensing task request. The sensing task result may comprise a set of generic and / or specific vibration classifications requested by the sensing task request.

[0084] Step 101 or step 109 may be repeated after step 119 has been performed. The set-up phase may be repeated periodically, e.g. once per week, or upon some known change in the network or environment around the network device. The set-up phase may be repeated at different times of the day, e.g. may be performed one or more times between 12 am and 6 am, one or more times between 6 am and 12 pm, one or more times between 12 pm and 6 pm, and one or more times between 6 pm and 12 pm, in case of vibrations / events that only / mainly occur certain times of the day.

[0085] As mentioned earlier, step 105 comprises classifying one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of the plurality of vibration classes. Fig. 2 shows a second embodiment of the method of performing a sensing task in which the one or more return signals which are classified are not return signals received in step 103 but correspond to these return signals. In an alternative embodiment, the classification is performed in step 105 on the return signals received in step 103 which have been classified as vibratory return signals in step 104.

[0086] The embodiment of Fig. 2 is an extension of the embodiment of Fig. 1. In the embodiment of Fig. 2, step 105 of Fig. 1 is implemented by a step 125 and steps 107, 121, and 123 are performed between steps 104 and 125.

[0087] Step 107 comprises storing identifiers of a plurality of vibration sensing surfaces. Each of the plurality of vibration sensing surfaces corresponds to at least one of the return signals received in step 103 which have been classified as vibratory return signals in step 104. In the embodiment of Fig. 2, step 107 comprises generating a VSS identifier for each respective return signal received in step 103 which has been classified as vibratory return signal in step 104 and storing this VSS identifier in a database along with the set of radar settings with which the radar signal was transmitted which resulted in the receipt of the respective return signal.

[0088] Step 107 further comprises associating each of the plurality of vibration sensing surfaces with a network device of the at least one network device, i.e. with an identifier of this network device.

[0089] Step 121 comprises retrieving, for each VSS identifier in the database, the set of radar settings associated with this VSS identifier and passing it to the network device(s) associated with this VSS identifier. The network device then adjusts its radar settings in accordance with the provided set of radar settings to transmit new radar signals. Step 123 comprises receiving from the network device(s) the one or more return radar signals received by the network device(s) in response to the new radar signals.

[0090] Step 125 comprises classifying the one or more return signals received in step 123 into one or more vibration classes of the plurality of vibration classes. Step 108 comprises associating the one or more vibration classes obtained in step 125 with the corresponding vibration sensing surface(s). A vibration class obtained for a return signal which is received in response to a new radar signal being transmitted with a certain set of radar settings is associated with the vibration sensing surface associated with this certain set of radar settings.

[0091] In the embodiment of Fig. 2, steps 121, 123, 125, and 108 may be repeated more often than steps 101, 103, 104, and 107. This may be done to save some resources, although steps 121, 123, 125, and 108 are likely more resource intensive than steps 101, 103, 104, and 107. Steps 121, 123, 125, and 108 may be repeated one or more times before the use phase is entered, for example.

[0092] It may be that as more classifiers become available, vibration sensing surfaces that had not previously been associated with vibration classes may later be associated with vibration classes or vibration sensing surfaces may later be associated with new vibration classes. Furthermore, certain classes may be more difficult to detect than others. For example, a vibration sensing surface may be identified which consistently vibrates due to traffic, but only later, when winds are strong, can it be determined that it can also measure wind speed.

[0093] Since step 125 is typically relative resource intensive, the performance of step 125 may be temporarily paused unless sufficient resources are available, e.g. when the battery level is sufficient, when energy is cheaper, or when there is little other communication traffic (if the computation is offloaded).

[0094] A part of a third embodiment of the method of performing a sensing task is shown in Fig. 3. The embodiment of Fig. 3 is an extension of the embodiment of Fig. 1. In the embodiment of Fig. 3, steps 105 and 117 of Fig. 1 are implemented by optional step 130 and step 130 comprises sub steps 131, 133, 135, 137, and 139. The embodiment of Fig. 3 may be combined with the embodiment of Fig. 2.

[0095] Step 131 comprises denoising each of the one or more return signals which are or correspond to vibratory return signals. This steps removes unwanted noise from the one or more return signals. The denoising may be frequency selective and the denoising algorithm may be capable of adjusting the frequency cut-offs in cases where a signal of known frequency range is being sought in the return signal. Many denoising algorithms are known in the art.

[0096] Step 133 comprises extracting, from each denoised return signal obtained in step 131, doppler information on doppler shifts in the frequency domain. These doppler shifts in the frequency domain correspond to the vibrational frequency of the vibration sensing surface, which may change over the time period covered by the denoised return signal. Step 135 comprises determining, for each denoised return signal, dominant vibrational frequencies (at any given moment) based on the doppler information extracted in step 133.

[0097] Step 137 comprises reconstructing, for each denoised return signal, the dominant vibrational frequencies in the time domain to produce a vibratory waveform, e.g. an audio waveform. A method for extracting audio signals from RF radar data is described in more detail in the afore-mentioned paper “Vital signs monitoring using pseudo-random noise coded Doppler radar with Delta-Sigma modulation”, for example. Step 139 comprises classifying each vibratory waveform produced in step 137 into the one or more vibration classes.

[0098] Optionally, step 139 is implemented by a step 140. Step 140 comprises classifying each vibratory waveform reconstructed in step 137 first into one or more generic vibration classes and then into one or more specific vibration classes. In this case, the plurality of vibration classes comprises a plurality of generic vibration classes and a plurality of specific vibration classes.

[0099] A part of a fourth embodiment of the method of performing a sensing task is shown in Fig. 4. The embodiment of Fig. 4 is an extension of the embodiment of Fig. 3. In the embodiment of Fig. 4, step 139 of Fig. 3 is implemented by optional step 140, the plurality of vibration classes comprises a plurality of generic vibration classes and a plurality of specific vibration classes, and step 140 of Fig. 3 comprises sub steps 141, 143, and 145.

[0100] Step 141 comprises classifying the one or more return signals which are or correspond to vibratory return signals into a generic vibration class of the plurality of generic vibration classes. For example, the vibratory waveform produced in step 137 may be passed to a generic vibration classifier, which produces a generic vibration classification. The generic vibration classifier may be a pre-trained classifier.

[0101] The generic vibration classification may be generic audio classification. Generic audio classification may consist of a classification of the broad class of an event described in the audio waveform produced in step 137, e.g., traffic noise, machine noise, footsteps, or human voices. Many classifiers exist in the art for classifying events occurring in vibratory, e.g. audio, waveforms, including neural network-based approaches.

[0102] The generic audio classification is always produced as a first step, to identify the broader class of vibration event detected, e.g. “traffic noise”, “human voices”, etc.

[0103] For some sensing tasks, the generic vibration classification may be enough information, e.g. a sensing task with a goal of determining if traffic noise is present or not, and therefore no specific vibration classification is required. If generic vibration classification is not enough information, the generic vibration classification is performed as initial step before performing the specific vibration classification.

[0104] Step 143 comprises using the generic vibration classification obtained in step 141 to select one or more specific vibration classifiers from a specific vibration classifier database. The specific vibration classifiers may be pre-trained classifiers. The specific vibration classifier database may contain multiple sets of pre-trained classifiers, where each set is associated with a given generic vibration classification. For example, sets within the specific vibration classifier database may include a set of audio classifiers related to traffic noise, a set of audio classifiers related to machine noise, a set of audio classifiers related to footsteps, and a set of audio classifiers related to human voices, amongst others.

[0105] Step 145 comprises classifying, with the one or more specific vibration classifiers selected in step 143, the one or more return signals into a specific vibration class of the plurality of specific vibration classes. For example, the specific vibration classifiers within each selected set may be run on the vibration waveform produced in step 137 to return a specific vibration classification. For instance, the ‘traffic noise’ set may contain specific audio classifiers which run on the vibration / audio waveform produced in step 137 to return classifications of the size of the vehicle creating noise, the timing or average frequency of vehicles passing, and / or the type of vehicle creating noise, for example.

[0106] A specific vibration classification enables more refined sensing tasks for a given class of vibration information, once it has been established that that class of vibration information is present. For example, for traffic noise, specific audio classifications may enable sensing tasks of vehicle counts, size of vehicle estimations, and / or vehicle type estimations. In an alternative embodiment, steps 105 and 117 are implemented by step 140 of Fig. 4, but not by step 130 of Fig. 3. A fifth embodiment of the method of performing a sensing task is shown in Fig. 5. The embodiment of Fig. 5 is an extension of the embodiment of Fig. 1. In the embodiment of Fig. 5, steps, 107, 109, and 111 ofFig. 1 are implemented by steps 201, 203, and 205, respectively. Step 201 differs from step 107 ofFig. 1 in that in step 201 each of the plurality of vibration sensing surfaces is associated with a spatial location or area. Each respective vibration sensing surface of the plurality of vibration sensing surfaces may be associated with a spatial location or area of the network device which received the return signal based on which the respective vibration sensing surface was detected, for example.

[0107] Step 203 differs from step 109 ofFig. 1 in that in step 203 the sensing task request indicates a target spatial location or area. Step 205 differs from step 111 ofFig. 1 in that in step 205, the one or more identifiers of the one or more vibration sensing surfaces are further determined based on the spatial locations or areas of the plurality vibration sensing surfaces, as associated in step 201, and the target spatial location or area, as obtained in step 203.

[0108] While the vibration sensing surfaces may be associated with spatial locations or areas of the network devices, e.g. in the absence of external information, it may be beneficial to learn more specific location information such as:

[0109] • Learning that a vibration sensing surface is associated with a specific object in the environment (e.g. a particular road sign, or a particular machine in a factory).

[0110] • Learning that a vibration sensing surface is associated with a particular region in space (e.g., given by a coordinate system that may be standard in the network).

[0111] Additionally, one or more steps of one or more of the embodiments of Figs. 2-4, 6-8 may be added to the embodiment of Fig. 5.

[0112] If environmental knowledge of a scene around a network device is known, this may be utilized to associate vibration sensing surfaces with particular objects, surfaces, or regions of the scene. A sixth embodiment of the method of performing a sensing task, a part of which is shown in Fig. 6, may be used to estimate a spatial location or area of a vibration sensing surface. The embodiment of Fig. 6 is an extension of the embodiment of Fig. 5.

[0113] In the embodiment of Fig. 6, step 201 of Fig. 5 is implemented by a step 213 and a step 211 is performed between steps 105 and 213. Step 211 comprises, for each respective vibration sensing surface of the plurality of vibration sensing surfaces, corresponding to a respective return signal of the one or more return signals, estimating a spatial location or area of the respective vibration sensing surface based on the respective return signal and / or based on a radar setting with which a respective radar signal (which caused the respective return signal to be returned) was transmitted. Step 211 may comprise inferring an angle of the vibration sensing surface with respect to the corresponding network device by calculating the resulting beam direction of the transmit radar signal from the relative phase delays of the antennas of the radar hardware. Step 211 may alternatively or additionally comprise determining the range of the vibration sensing surface with respect to the corresponding network device from the magnitude of the PSN modulation delay.

[0114] Step 213 differs from step 201 of Fig. 5 in that, for each respective vibration sensing surface of the plurality of vibration sensing surfaces, the spatial location or area of the respective vibration sensing surface estimated in step 211 is associated with the respective vibration sensing surface in step 213.

[0115] Step 213 may comprise identifying, in an environment map, an object of interest which corresponds to a range and angle determined in step 211. For any identified object of interest, an identifier of the object may be stored alongside the corresponding VSS identifier. The environment map may be obtained in step 213 or in a separate step (not shown in Fig. 6). An environment map may cover only the environment of one network device or the environment of multiple network devices.

[0116] The environment map is a data structure providing information on surfaces or objects in the vicinity of one or more network devices, mapped with a coordinate system that is common with the network device(s). Objects of interest may be identified within the environment map and associated with particular identifiers. Multiple types of data structure may fit this definition, with varying degrees of complexity, including:

[0117] • A full 3D representation of the scene around the network device(s), given as (for example) point cloud or 3D geometry data. In a private space use case, this may be, for example, a 3D scan of an individual’s living room.

[0118] • The spatial coordinates of known objects of interest around the network device(s). In a factory use case, these may be, for example, the locations of particular machines, which serve as objects of interest. In a public space use case, these may be, for example, the locations of surfaces such as road signs, bridge pillars, etc., which serve as objects of interest.

[0119] • 2D image or video data covering part or all of the scene around the network device(s), where associated depth data has also been learnt or inferred, and overlaid on the 2D data. This may be particularly relevant for the scenario where 5G+ base stations are instrumented with video or static image cameras.

[0120] In some cases, the learnt angle and range of the vibration sensing surface may not correspond to an already known object of interest. If the environment map is a simple environment map (i.e. a map that just consist of coordinates of objects of interest), the vibration sensing surface remains unassociated with any particular object or surface. If the environment map is a more complex environment map, such as a map consisting of full 3D scene representations, or 2D+depth video or image data, a new object of interest may be marked on the map at the determined range and angle with respect to the network device and a new corresponding object identifier may be generated and stored alongside the corresponding VSS identifier.

[0121] Additionally, one or more steps of one or more of the embodiments of Figs. 2-4 and 7 or 8 may be added to the embodiment of Fig. 6.

[0122] In scenarios where other sensing devices exist in the environment, which are already identified and localized (e.g., a vibration sensor attached to a machine in a factory use case), data from these sensing devices may be used to associate them with a particular vibration sensing surface and sensing task. This is done in a seventh embodiment of the method of performing a sensing task, of which a part is shown in Fig. 7. This embodiment is an extension of the embodiment of Fig. 3.

[0123] In this embodiment, steps 161, 163, 165, and 167 are performed in the set-up phase after step 107 of Fig. 3. Fig. 7 also shows an embodiment of the method of receiving an information request. Step 161 comprises a system 31 transmitting an information request to a sensing device 41 . The sensing device 41 is one of a set of additional vibration and / or audio sensing devices, which exist in the environment around the network device(s), and have known, specific sensing tasks (e.g. vibration sensing of a machine). The sensing devices produce sensing data and have device identifiers.

[0124] The information request comprises data representing the vibratory waveform produced in step 137 and requests the sensing device 41 to compare the vibratory waveform with a further vibratory waveform sensed by the sensing device 41 in a time period. The information request indicates the time period.

[0125] A step 171 comprises the sensing device 41 receiving the information request. A step 173 comprises the sensing device 41 retrieving the further vibratory waveform which relates to the time period. A step 175 comprises the sensing device 41 determining a measure of similarity between the vibratory waveform and the further vibratory waveform. A step 177 comprises the sensing device 41 transmitting information in response to the information request. The information is determined based on the measure of similarity and indicates a degree of similarity between the vibratory waveform and the further vibratory waveform. For example, the information may specify the measure of similarity or may specify whether the measure of similarity exceeds a threshold T. Step 163 comprises the system 31 receiving the information from the sensing device 41. Step 165 comprises the system 31 ascertaining a function and / or device identifier of the sensing device 41. Step 167 comprises associating the vibration sensing surface with the function and / or device identifier of the sensing device 41 in dependence on the degree of similarity.

[0126] Step 167 may comprise generating a new sensing task identifier of a new sensing task and associating the new sensing task identifier and the device identifier of sensing device 41 with the VSS identifier of the vibration sensing surface. This indicates that the new sensing task is capable of taking on the equivalent function as the sensing device 41. Additionally, one or more steps of one or more of the embodiments of Figs. 2,4-6 may be added to the embodiment of Fig. 7.

[0127] In the embodiment of Fig. 7, the vibratory waveform is compared with a further vibratory waveform by the sensing device. In the eighth embodiment of the method of performing a sensing task, of which a part is shown in Fig. 8, the vibratory waveform is compared with a further vibratory waveform by the system, e.g. by a network device. The embodiment of Fig. 7 may be beneficial if sensing devices are not able / willing to share sensing data.

[0128] The embodiment of Fig. 8 is an extension of the embodiment of Fig. 3. In the embodiment of Fig. 8, steps 181, 183, 165, 185, 187, and 189 are performed in the set-up phase after step 107. Step 181 comprises a system 51 transmitting a data request to a sensing device 61. The data request requests a further vibratory waveform sensed by the sensing device 61 in a time period. The data request indicates the time period.

[0129] A step 191 comprises the sensing device 61 receiving the data request. A step 193 comprises the sensing device 61 retrieving the further vibratory waveform which relates to the time period. A step 195 comprises the sensing device 61 transmitting the further vibratory waveform to the system 51.

[0130] Step 183 comprises the system 51 receiving the further vibratory waveform from the sensing device 61. Step 165 comprises the system 51 ascertaining a function and / or device identifier of the sensing device 61. Step 185 comprises the system 51 determining a measure of similarity between the vibratory waveform produced in step 137 and the further vibratory waveform received in step 183.

[0131] Step 187 comprises the system 51 comparing the measure of similarity determined in step 185 with a threshold T. Step 189 is performed if the measure of similarity is determined in step 187 to exceed the threshold T. Step 189 comprises the system 51 associating the vibration sensing surface with the function and / or device identifier of the sensing device 61, as ascertained in step 165. Additionally, one or more steps of one or more of the embodiments of Figs. 2,4-6 may be added to the embodiment of Fig. 8.

[0132] Fig. 9 is a block diagram of a first embodiment of the system for performing a sensing task. The system 1 of Fig. 9 may be configured to perform one or more of the methods of Figs. 1 to 8. In the embodiment of Fig. 9, the system 1 is separate from any base stations and UEs and may be located in the radio access network, for example. In an alternative embodiment, the system 1 may be a base station, for example.

[0133] The system 1 comprises a receiver 3, a transmitter 4, a processor 5, and a memory 7. The processor 5 is configured to transmit a plurality of radar signals via at least one network device, e.g. base station 11. Each of the plurality of radar signals is transmitted with a different radar setting of a plurality of radar settings. The processor 5 is further configured to receive, via the at least one network device, e.g. base station 11, a plurality of return signals in response to the transmitted plurality of radar signals and classify each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal.

[0134] The processor 5 is further configured to classify one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes and store identifiers of a plurality of vibration sensing surfaces, e.g. vibration sensing surfaces 13 and 14. Each of the plurality of vibration sensing surfaces corresponds to at least one of the one or more return signals and is associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device. In the example of Fig. 9, each of the vibration sensing surfaces 13 and 14 is associated with base station 11.

[0135] The processor 5 is further configured to receive a sensing task request which requests the performance of a sensing task and indicates a vibration class of the plurality of vibration classes and retrieve, based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces, e.g. vibration sensing surfaces 13 and 14. The one or more vibration sensing surfaces are associated with one or more network devices of the at least one network device. For example, based on a certain sensing task request, an identifier of vibration sensing surface 13 may be retrieved. Vibration sensing surface 13 is associated with base station 11 .

[0136] The processor 5 is further configured to transmit, via the one or more network devices, e.g. base station 11, one or more further radar signals towards the one or more vibration sensing surfaces, e.g. vibration sensing surface 13, and receive, via the one or more network devices, e.g. base station 11, one or more further return signals in response to the one or more further radar signals. The processor 5 is further configured to classify the one or more further return signals into one or more further vibration classes of the plurality of vibration classes and output sensing results based on the one or more further vibration classes.

[0137] The network devices, e.g. base station 11, comprise radar hardware. The radar hardware may operate within the mmWave RF frequency band (approx. 30 to 300 GHz). The radar hardware may, for example, comprise:

[0138] • an array of transmit and receive antennas capable of producing the radar signals and receiving their reflections off the environment (the return signals), respectively. In some cases, this may be the same antenna array used by the network device for communications.

[0139] • Gain control circuitry, to control transmit radar signal power;

[0140] • A Delta-Sigma (DS) signal generator, to provide an adjustable frequency offset to the return radar signal.

[0141] • A waveform generator capable of generating pseudo-random noise (PSN) waveforms. The waveform generator may generate a pseudo-random binary sequence waveform, for example.

[0142] • A phase modulator on the transmit radar signal antenna path for phase-modulating the transmit radar signal with the PSN waveform.

[0143] • A phase modulator on the receive radar signal antenna path. This modulator phase- modulates the return radar signal with a delayed PSN waveform. It is the magnitude of this delay which ultimately enables the distance-selectivity of the PSN PMCW radar system.

[0144] • A controller which may control the radar settings, including transmit antennas gain, relative antenna phase delays (e.g. for controlling radar pulse angle), DS signal offset, the PSN phase modulation of the transmit and return radar signals and the magnitude of the delay between them, and (if available) radar signal frequency.

[0145] Fig. 10 is a block diagram of a second embodiment of the system for performing a sensing task and of an embodiment of the sensing device. The system 31 comprises receiver 3, transmitter 4, a processor 35, and memory 7. The processor 35 is configured as described in relation to processor 5 of system 1 of Fig. 9, but is further configured to transmit an information request to a sensing device 41. The information request comprises data representing the vibratory waveform and requests the sensing device 41 to compare the vibratory waveform with a further vibratory waveform sensed by the sensing device 41 in a time period. The information request indicates the time period.

[0146] The processor 35 is further configured to receive information from the sensing device 41 in response to the information request. The information indicates a degree of similarity between the vibratory waveform and the further vibratory waveform. The processor 35 is further configured to ascertain a function and / or device identifier of the sensing device 41 and associate the vibration sensing surface 13 with the function and / or device identifier of the sensing device 41 in dependence on the degree of similarity.

[0147] The sensing device 41 comprises a receiver 43, a transmitter 44, a processor 45, and a memory 47. The processor 45 is configured to receive an information request from system 31. The information request comprises data representing a vibratory waveform and indicates a time period. The processor 45 is further configured to retrieve a further vibratory waveform which relates to the time period, determine a measure of similarity between the vibratory waveform and the further vibratory waveform, and transmit information to the system 31 in response to the information request. This information is determined based on the measure of similarity and indicates a degree of similarity between the vibratory waveform and the further vibratory waveform.

[0148] In the embodiment shown in Figs. 9 and 10, the systems 1 and 31 comprise one processor. In an alternative embodiment, the systems 1 and 31 comprises multiple processors. The processors 5 and 35 may be general-purpose processors, e.g., Intel or an AMD processors, or application-specific processors, for example. The processor 35 may comprise multiple cores, for example. The processors 5 and 35 may run a Unix-based or Windows operating system, for example. The memory 7 may comprise solid state memory, e.g., one or more Solid State Disks (SSDs) made out of Flash memory, or one or more hard disks, for example.

[0149] The receiver 3 and the transmitter 4 may use one or more communication technologies (wired or wireless) to communicate with other systems. The receiver 3 and the transmitter 4 may be combined in a transceiver. The systems 1 and 3 Imay comprise other components typical for a network system, e.g., a power supply.

[0150] In the embodiment shown in Fig. 10, the sensing device 41 comprises one processor 45. In an alternative embodiment, the sensing device 41 comprises multiple processors. The processor 45 may be a general-purpose processor, e.g., an ARM or Qualcomm processor, or an application-specific processor. The processor 45 may run Google Android or Apple iOS as operating system, for example.

[0151] The receiver 43 and the transmitter 44 of the sensing device 41 may use one or more wireless communication technologies such as Wi-Fi, LTE, and / or 5G New Radio to communicate with base stations, for example. The receiver 43 and the transmitter 44 may be combined in a transceiver. The sensing device 41 may comprise other components typical for user equipment, e.g., a battery and / or a power connector.

[0152] Example uses cases

[0153] In a factory use case, a base station may be used as a network device and the base station may be connected to a 5G+ network and serving a factory or other enclosed industrial area containing various machines, vehicles, moving objects, etc. In this use case, vibration sensing surfaces may comprise surfaces of machines, surfaces within the factory which vibrate with given events, or surfaces of particular objects such as doors or windows.

[0154] If many of the machines have sensing devices attached, the system of performing a sensing task may replace these sensing devices or augment them. This provides the potential benefits of:

[0155] • reducing resource use of sensing devices (both communications and energy resources);

[0156] • identifying potential measurement errors in the sensing device data;

[0157] • providing additional data beyond the sensing device data.

[0158] In addition, the system of performing a sensing task may identify and use vibration sensing surfaces in the enclosed factory space to obtain information on events which are not measured by any sensing devices, which may be useful for predicting safety hazards.

[0159] A road side (public spaces) use case may be primarily concerned with traffic monitoring and infrastructure monitoring, alongside crowd and pedestrian monitoring. In this use case, the vibration sensing surfaces may comprise surfaces near roads or pedestrian ways (such as road signs, traffic lights, or even trees) and surfaces on the infrastructure to be monitored (pillars of bridges, walls of buildings).

[0160] As it is broadly expected that public spaces, particularly in cities, will be increasingly well mapped in 3D, and that these maps will be made available to 5G+ networks, it may be possible to use the method of Fig. 6 in this use case. With the help of an environment map, vibration sensing surfaces may be located with roads and other infrastructure in its environment and these vibration sensing surfaces may then be associated with specific regions of a road network, particular buildings, bridges, etc. allowing the communications network to use its resources to monitor these infrastructures without the use of additional sensors. With the help of an environment map, vibration sensing surfaces may be located in known streets or pedestrian walkways. The associated base station may then be used to acoustically monitor pedestrian flows, crowd dynamics, etc. in these streets or pedestrian walkways. This would improve privacy (compared to video cameras) and not require installation of additional sensing devices like video cameras.

[0161] If an environment has not been (well) mapped, a base station with a known location may also be used to achieve the above sensing tasks, but likely with a lower targetability. For example, it would be known at installation time that a base station has been installed at a particular location, next to particular streets or roadways. Therefore, a sensing task doing traffic monitoring may be treated as a sensing task applying to that given street based purely on the known location of the base station.

[0162] A sensing task may also be performed in private spaces. Private spaces (such as homes, offices, elderly care homes, etc.) are becoming increasingly “smart” with sensors placed in the home implemented on a home or public network. Speech monitoring and vital signs monitoring are two potential applications of sensing tasks enabled by vibration sensing surfaces. Sensing tasks may also be related to the states of appliances (e.g. washing machine on / off) and sinks, showers, baths, etc.

[0163] Mixed reality devices also make it more likely that 3D scans of the home environment will be available, which may make it possible to use the method of Fig. 6. Additionally, homes that already include sensing devices on the network may be utilized, which may make it possible to use the methods of Figs. 7 and 8. Privacy is a key concern in private spaces, so being able to do event and activity monitoring based on radar-derived audio may be preferable to similar outcomes using video cameras or lidar.

[0164] Fig. 11 depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to Figs. 1-8.

[0165] As shown in Fig. 11, the data processing system 300 may include at least one processor 302 coupled to memory elements 304 through a system bus 306. As such, the data processing system may store program code within memory elements 304. Further, the processor 302 may execute the program code accessed from the memory elements 304 via a system bus 306. In one aspect, the data processing system may be implemented as a computer that is suitable for storing and / or executing program code. It should be appreciated, however, that the data processing system 300 may be implemented in the form of any system including a processor and a memory that is capable of performing the functions described within this specification.

[0166] The memory elements 304 may include one or more physical memory devices such as, for example, local memory 308 and one or more bulk storage devices 310. The local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from the bulk storage device 310 during execution.

[0167] Input / output (I / O) devices depicted as an input device 312 and an output device 314 optionally can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, or the like. Examples of output devices may include, but are not limited to, a monitor or a display, speakers, or the like. Input and / or output devices may be coupled to the data processing system either directly or through intervening I / O controllers.

[0168] In an embodiment, the input and the output devices may be implemented as a combined input / output device (illustrated in Fig. 11 with a dashed line surrounding the input device 312 and the output device 314). An example of such a combined device is a touch sensitive display, also sometimes referred to as a “touch screen display” or simply “touch screen”. In such an embodiment, input to the device may be provided by a movement of a physical object, such as e.g. a stylus or a finger of a user, on or near the touch screen display.

[0169] A network adapter 316 may also be coupled to the data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and / or networks to the data processing system 300, and a data transmitter for transmitting data from the data processing system 300 to said systems, devices and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with the data processing system 300.

[0170] As pictured in Fig. 11, the memory elements 304 may store an application 318. In various embodiments, the application 318 may be stored in the local memory 308, he one or more bulk storage devices 310, or separate from the local memory and the bulk storage devices. It should be appreciated that the data processing system 300 may further execute an operating system (not shown in Fig. 11) that can facilitate execution of the application 318. The application 318, being implemented in the form of executable program code, can be executed by the data processing system 300, e.g., by the processor 302. Responsive to executing the application, the data processing system 300 may be configured to perform one or more operations or method steps described herein. Various embodiments of the invention may be implemented as a program product for use with a computer system, where the program(s) of the program product define functions of the embodiments (including the methods described herein). In one embodiment, the program(s) can be contained on a variety of non-transitory computer-readable storage media, where, as used herein, the expression “non-transitory computer readable storage media” comprises all computer-readable media, with the sole exception being a transitory, propagating signal. In another embodiment, the program(s) can be contained on a variety of transitory computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., flash memory, floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. The computer program may be run on the processor 302 described herein.

[0171] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0172] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the implementations in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the present invention. The embodiments were chosen and described in order to best explain the principles and some practical applications of the present invention, and to enable others of ordinary skill in the art to understand the present invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CLAIMS:

1. A system (1,31,51) for performing a sensing task, the system (1,31,51) comprising at least one processor (5,35) configured to:- transmit a plurality of radar signals via at least one network device (11), each of the plurality of radar signals being transmitted with a different radar setting of a plurality of radar settings,- receive, via the at least one network device (11), a plurality of return signals in response to the transmitted plurality of radar signals,- classify each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal,- classify one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes,- store identifiers of a plurality of vibration sensing surfaces (13,14), each of the plurality of vibration sensing surfaces (13,14) corresponding to at least one of the one or more return signals, each of the plurality of vibration sensing surfaces (13,14) being associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device (H),- receive a sensing task request, the sensing task request requesting the performance of the sensing task, the sensing task request indicating a vibration class of the plurality of vibration classes,- retrieve, based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces (13,14), the one or more vibration sensing surfaces (13,14) being associated with one or more network devices of the at least one network device (11),- transmit, via the one or more network devices (11), one or more further radar signals towards the one or more vibration sensing surfaces (13,14),- receive, via the one or more network devices (11), one or more further return signals in response to the one or more further radar signals,- classify the one or more further return signals into one or more further vibration classes of the plurality of vibration classes, and- output sensing results based on the one or more further vibration classes.

2. A system (1,31,51) as claimed in claim 1, wherein the vibratory return signals are acoustic signals, the vibration sensing surfaces (13,14) are acoustic sensing surfaces, and the vibration classes are audio classes.

3. A system (1,31,51) as claimed in claim 1 or 2, wherein the at least one processor (5,35) is configured to classify each return signal of the one or more return signals into the one or more vibration classes of the plurality of vibration classes by, for each respective return signal of the one or more return signals:- denoising the respective return signal,- extracting, from the denoised return signal, doppler information on doppler shifts in the frequency domain,- determining dominant vibrational frequencies based on the extracted doppler information,- reconstructing the dominant vibrational frequencies in the time domain to produce a vibratory waveform, and- classifying the vibratory waveform into the one or more vibration classes.

4. A system (31) as claimed in claim 3, wherein the at least one processor (35) is configured to:- transmit an information request to a sensing device (41), the information request comprising data representing the vibratory waveform and requesting the sensing device (41) to compare the vibratory waveform with a further vibratory waveform sensed by the sensing device (41) in a time period, the information request indicating the time period,- receive information from the sensing device (41) in response to the information request, the information indicating a degree of similarity between the vibratory waveform and the further vibratory waveform,- ascertain a function and / or device identifier of the sensing device (41), and- associate the vibration sensing surface (13,14) with the function and / or device identifier of the sensing device (41) in dependence on the degree of similarity.

5. A system (51) as claimed in claim 3, wherein the at least one processor is configured to:- transmit a data request for a further vibratory waveform to a sensing device (61), the further vibratory waveform being sensed by the sensing device (61) in a time period, the data request indicating the time period,- receive the further vibratory waveform from the sensing device (61),- ascertain a function and / or device identifier of the sensing device (61),- determine a measure of similarity between the vibratory waveform and the further vibratory waveform,- compare the measure of similarity with a threshold, and- associate the vibration sensing surface (13,14) with the function and / or device identifier of the sensing device (61) if the measure of similarity is determined to exceed the threshold.

6. A system (1,31,51) as claimed in any one of the preceding claims, wherein the radar signals comprise continuous wave radar signals.

7. A system (1,31,51) as claimed in claim 6, wherein the radar signals comprise phase-modulated continuous wave radar signals.

8. A system (1,31,51) as claimed in claim 7, wherein the radar signals comprise pseudo-random noise coded phase-modulated continuous wave radar signals.

9. A system (1,31,51) as claimed in claim 7 or 8, wherein transmitting each of the plurality of radar signals with a different radar setting and receiving the return signals comprises stepwise adjusting a phase modulation of the transmitted radar signals and the received return signals and / or stepwise adjusting a relative phase delay of a radar antenna array, the radar signals being transmitted over the radar antenna array.

10. A system (1,31,51) as claimed in any one of the preceding claims, wherein each of the plurality of vibration sensing surfaces (13,14) is associated with a spatial location or area, the sensing task request indicates a target spatial location or area, and the at least one processor (5,35) is configured to retrieve the one or more identifiers of the one or more vibration sensing surfaces (13,14) further based on the spatial locations or areas of the plurality vibration sensing surfaces (13,14) and the target spatial location or area.

11. A system (1,31,51) as claimed in claim 10, wherein each respective vibration sensing surface of the plurality of vibration sensing surfaces (13,14) is associated with a spatial location or area of a network device (11) which received a return signal corresponding to the respective vibration sensing surface (13,14).

12. A system (1,31,51) as claimed in claim 10 or 11, wherein the at least one processor (5,35) is configured to, for each respective vibration sensing surface of the plurality of vibration sensing surfaces (13,14), corresponding to a respective return signal of the one or more return signals:- estimate a spatial location or area of the respective vibration sensing surface (13,14) based on the respective return signal and / or based on a radar setting with which a respective radar signal of the plurality of radar signals was transmitted, the respective radar signal causing the respective return signal to be returned, and- associate the spatial location or area of the respective vibration sensing surface (13,14) with the respective vibration sensing surface (13,14).

13. A system (1,31,51) as claimed in any one of the preceding claims, wherein the plurality of vibration classes comprises a plurality of generic vibration classes and a plurality of specific vibration classes and the at least one processor (5,35) is configured to classify the one or more return signals into the one or more vibration classes by:- classifying the one or more return signals which are or correspond to vibratory return signals into a generic vibration class of the plurality of generic vibration classes, and- classifying, based on the generic vibration class, the one or more return signals into a specific vibration class of the plurality of specific vibration classes.

14. A sensing device (41) comprising at least one processor (45), the at least one processor (45) being configured to:- receive an information request from a system (31), the information request comprising data representing a vibratory waveform and indicating a time period,- retrieve a further vibratory waveform, the further vibratory waveform relating to the time period,- determine a measure of similarity between the vibratory waveform and the further vibratory waveform, and- transmit information to the system (31) in response to the information request, the information being determined based on the measure of similarity and indicating a degree of similarity between the vibratory waveform and the further vibratory waveform.

15. A method of performing a sensing task, the method comprising:- transmitting (101) a plurality of radar signals via at least one network device, each of the plurality of radar signals being transmitted with a different radar setting of a plurality of radar settings;- receiving (103), via the at least one network device, a plurality of return signals in response to the transmitted plurality of radar signals;- classifying (104) each of the plurality of return signals as a vibratory return signal or a non-vibratory return signal;- classifying (105) one or more return signals which are or correspond to vibratory return signals into one or more vibration classes of a plurality of vibration classes;- storing (107) identifiers of a plurality of vibration sensing surfaces, each of the plurality of vibration sensing surfaces corresponding to at least one of the one or more return signals, each of the plurality of vibration sensing surfaces being associated with the one or more vibration classes into which the corresponding at least one return signal was classified and with a network device of the at least one network device;- receiving (109) a sensing task request, the sensing task request requesting the performance of the sensing task, the sensing task request indicating a vibration class of the plurality of vibration classes;- retrieving (111), based on the vibration class indicated in the sensing task request, one or more identifiers of one or more vibration sensing surfaces of the plurality of vibration sensing surfaces, the one or more vibration sensing surfaces being associated with one or more network devices of the at least one network device;- transmitting (113), via the one or more network devices, one or more further radar signals towards the one or more vibration sensing surfaces;- receiving (115), via the one or more network devices, one or more further return signals in response to the one or more further radar signals;- classifying (117) the one or more further return signals into one or more further vibration classes of the plurality of vibration classes; and- outputting (119) sensing results based on the one or more further vibration classes.

16. A method of receiving an information request, the method comprising:- receiving (171) an information request, the information request comprising data representing a vibratory waveform and indicating a time period;- retrieving (173) a further vibratory waveform, the further vibratory waveform relating to the time period;- determining (175) a measure of similarity between the vibratory waveform and the further vibratory waveform; and- transmitting (177) information in response to the information request, the information being determined based on the measure of similarity and indicating a degree of similarity between the vibratory waveform and the further vibratory waveform.

17. A computer program or suite of computer programs comprising at least one software code portion or a computer program product storing at least one software code portion, the software code portion, when run on a computer system, being configured for performing the method of claim 15 or 16.

Images