Self-calibrating dynamic spatiotemporal beamforming system

Abstract

This invention relates to a self-calibrating dynamic spatiotemporal beamforming system. A mobile platform for data acquisition for calibration includes: a transceiver within the mobile platform; a displacement motion unit configured to move the mobile platform within a region; a sensor coupled to the displacement motion unit and configured to output a signal; and a controller configured to: request measurement results of parameters from the sensor; remove background noise associated with the mobile platform from the measurement results, thereby focusing the measurement results on foreground noise; request a second measurement result of parameters from the sensor in response to the mobile platform reaching a new position and orientation; remove background noise associated with the mobile platform at the new position from the second measurement result; aggregate the signals from the sensor along with the associated position and orientation to create an energy map via spatial dynamic beamforming; and analyze the energy map to identify the state of devices in the region.

Description

Technical Field

[0001] This disclosure generally relates to a system and method for sensing using sound. Background Technology

[0002] Beamforming involves using sensor arrays and signal processing techniques such as phased array processing to enhance transmitted or received signals in a specific direction in space. Such techniques can also be used to map the energy distribution of signal sources within the sensor's field of view. Undesirable background noise can cause interference with the source signal. Summary of the Invention

[0003] A method for acquiring a calibrated image of a room via a controller includes: requesting a signal indicating the measurement results of parameters from sensors associated with the position and orientation of a mobile platform in the room; removing background noise associated with the mobile platform from the signal, thereby focusing the measurement results on a foreground signal, wherein the background noise is removed from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; storing the measurement results, the position and orientation of the mobile platform in the room; requesting the mobile platform to move to a second position in the room facing a second direction; in response to the mobile platform reaching the second position and the second orientation, requesting a second signal indicating a second measurement result of the parameters from the sensors associated with the second position and the second orientation of the mobile platform; and retrieving the signal from the sensor associated with the position and orientation of the mobile platform. The second measurement result is used to remove background noise associated with the mobile platform at the second position and the second orientation, wherein the background noise is removed from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; the foreground signals of interest associated with the position and orientation of the mobile platform and the second position and the second orientation of the mobile platform in the room are aggregated to create an energy map via spatial dynamic beamforming, wherein the aggregation of the foreground signals is performed by extracting background noise to provide a calibrated signal that is the generated foreground signal of interest, wherein the background noise is removed from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; the energy map is analyzed to identify the state of the device in the room; and the energy map is output.

[0004] A system for calibrating images of a room includes: a mobile platform configured to move within the room; a sensor coupled to the mobile platform and configured to measure parameters within a region relative to the sensor and output a signal associated with the parameters within the region; and a controller. The controller can be configured to: request measurement results of the parameter from the sensors associated with the position and orientation of the mobile platform; remove background noise associated with the mobile platform from the measurement results, thereby focusing the measurement results on foreground noise; store the measurement results and the position and orientation of the mobile platform within the room; move the mobile platform within the room to a new position; in response to the mobile platform reaching the new position, request a second measurement result of the parameter from the sensors associated with the new position and orientation of the mobile platform; remove background noise associated with the mobile platform at the new position from the second measurement result, thereby focusing the measurement results on foreground noise at the new position; aggregate the signals from the sensors and the associated position and orientation of the mobile platform within the room to create an energy map via spatial dynamic beamforming; analyze the energy map to identify the state of the device in the room; and output an image of the foreground beamforming.

[0005] A mobile robot platform for data acquisition for calibration includes: a transceiver within the mobile robot platform; a displacement motion unit configured to move the mobile robot platform within a region; a sensor coupled to the displacement motion unit and configured to output a signal; and a controller. The controller can be configured to: request measurement results of parameters from the sensors associated with the position and orientation of the mobile robot platform; remove background noise associated with the mobile robot platform from the measurement results, thereby focusing the measurement results on foreground noise; store the measurement results and the position and orientation of the mobile robot platform in the region; request the displacement motion unit to move the mobile robot platform in the region to a new position and orientation; in response to the mobile robot platform reaching the new position and orientation, request a second measurement result of the parameters from the sensors associated with the new position and orientation of the mobile robot platform; remove background noise associated with the mobile robot platform at the new position from the second measurement result, thereby focusing the measurement results on foreground noise at the new position; aggregate the signals from the sensors and the associated position and orientation of the mobile robot platform in the region to create an energy map via spatial dynamic beamforming; and analyze the energy map to identify the state of the device in the region. Attached Figure Description

[0006] Figure 1 This is a flowchart of a machine monitoring system that uses dynamic spatiotemporal beamforming.

[0007] Figure 2 This is a flowchart used to obtain the energy map of space dynamic beamforming.

[0008] Figure 3 This is a block diagram of a system used to acquire energy maps of space dynamic beamforming.

[0009] Figure 4 It is a graphical representation of the motion of a robotic platform configured to acquire an energy map of spatial dynamic beamforming across the entire region.

[0010] Figure 5A It is a graphical representation of an acoustic map of a region obtained by spatial dynamic beamforming of data acquired from a sensor system on a mobile robot platform.

[0011] Figure 5B It is a graphical representation of an acoustic map of a region obtained by spatial dynamic beamforming of data acquired from a sensor system on a mobile robot platform.

[0012] Figure 6 This is a flowchart of a self-calibration and self-diagnostic system that uses spatial dynamic beamforming to create environmental maps.

[0013] Figure 7 This is a block diagram of a system used for virtual sensing systems.

[0014] Figure 8 is a graphical illustration of the mathematical basis of a virtual sensing system related to a physical system.

[0015] Figure 9 It is a graphical representation of signal-to-signal transformation by mapping target sensor data to source sensor data.

[0016] Figure 10 It is another graphical representation of signal-to-signal transformation by mapping target sensor data to source sensor data.

[0017] Figure 11 This is a schematic diagram of a control system configured to control a vehicle.

[0018] Figure 12 This is a schematic diagram of a control system configured to control manufacturing machines.

[0019] Figure 13 This is a schematic diagram of a control system configured to control power tools.

[0020] Figure 14This is a schematic diagram of a control system configured to control an automated personal assistant.

[0021] Figure 15 This is a schematic diagram of a control system configured as a control and monitoring system.

[0022] Figure 16 This is a schematic diagram of a control system configured to control a medical imaging system. Detailed Implementation

[0023] Detailed embodiments of the invention are disclosed herein as needed; however, it should be understood that the disclosed embodiments are merely examples of the invention, which may be embodied in various and alternative forms. The figures are not necessarily drawn to scale; some features may be enlarged or minimized to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather serve only as a representative basis for teaching those skilled in the art to employ the invention in various ways.

[0024] The term "substantially" may be used herein to describe disclosed or claimed embodiments. The term "substantially" may modify values ​​or relative characteristics disclosed or claimed in this disclosure. In such cases, "substantially" may mean that the modified value or relative characteristic is within ±0%, 0.1%, 0.5%, 1%, 2%, 3%, 4%, 5%, or 10% of said value or relative characteristic.

[0025] The term sensor refers to a device that detects or measures a physical property and records, indicates, or otherwise responds to it. Sensors include optical, light, imaging, or photonic sensors (e.g., charge-coupled devices (CCD), CMOS active pixel sensors (APS), infrared sensors (IR), CMOS sensors), acoustic, sound, or vibration sensors (e.g., microphones, seismic detectors, hydrophones), automotive sensors (e.g., wheel speed, parking, radar, oxygen, blind spot, torque), chemical sensors (e.g., ion-sensitive field-effect transistors (ISFETs), oxygen, carbon dioxide, chemiluminescent resistors, holographic sensors), current, potential, magnetic, or radio frequency sensors (e.g., Hall effect, magnetometer, magnetoresistive, Faraday cup, galvanometer), environmental, weather, moisture, or humidity sensors (e.g., weather radar, chemiluminometer), flow rate, or fluid velocity sensors. Sensors (e.g., mass airflow sensor, anemometer), ionizing radiation or subatomic particle sensors (e.g., ionization chamber, Geiger counter, neutron detector), navigation sensors (e.g., Global Positioning System (GPS) sensor, magnetohydrodynamic (MHD) sensor), position, angle, displacement, distance, velocity or acceleration sensors (e.g., lidar, accelerometer, ultra-wideband radar, piezoelectric sensor), force, density or level sensors (e.g., strain gauge, nuclear density meter), heat, thermal or temperature sensors (e.g., infrared thermometer, pyrometer, thermocouple, thermistor, microwave radiometer), or other devices, modules, machines, or subsystems intended to detect or measure physical properties and record, indicate or otherwise respond to them.

[0026] The term "image" refers to a representation or artifact that depicts a perception (e.g., visual perception from a viewpoint), such as a similar experimental object (e.g., a physical object, scene, or property) and thus provides a depiction of it, such as a photograph or other two-dimensional picture. Images can be multidimensional because they can include components of time, space, intensity, density, or other characteristics. For example, images can include time-series images.

[0027] Systems and methods for generating high-resolution energy maps of a given space using a beamformer mounted on a mobile platform are disclosed. The energy map may indicate signal strength, such as power, according to spatial dimensions. These methods can be used to acquire information from various energy sources (e.g., acoustic, electromagnetic), but are primarily discussed in the acoustic domain. The system may include an array of receivers coupled to various beamforming algorithms mounted on a mobile platform that records spatiotemporal information as it moves through space. The methods disclosed herein use spatiotemporal information coupled to beamforming information acquired at each location to generate a high-resolution map of space. This map may be overlaid with visual information or may be used to monitor changes in acoustic or electromagnetic energy over time. Systems and methods for performing self-calibration tests during spatial dynamic beamforming operations to separate self-generated background noise are also disclosed. These methods can be used to identify self-noise or other background noise associated with the recording platform (e.g., sensor hardware, motion-capable hardware (e.g., robotic platform)). Using these methods, the output from the spatial dynamic beamforming algorithm can be improved by separating the background noise from the foreground signal, which is a signal associated with an object or region of interest. Then, background self-noise can be used to perform self-health diagnosis by monitoring changes in self-noise over time.

[0028] With increasing concerns about human health and safety in complex industrial environments, the ability to accurately monitor various energy sources in a given environment continues to become increasingly important. In particular, monitoring acoustics is crucial from both human health and industrial perspectives. For humans, monitoring acoustics is important to provide a safe working environment and prevent hearing loss. From an industrial perspective, acoustics can provide valuable insights into the health of machines or facilities in ways that visual monitoring cannot. However, acoustic monitoring of environments, spaces, and processes can be challenging for several reasons. First, sound emitted from a particular source tends to reflect off surfaces in its surrounding environment, which can result in echo versions of the signal reaching receivers such as microphones, antennas, etc. Second, when multiple sources are operating, signals from these sources overlap, making it difficult for the receiver to determine from which specific noise source it originated.

[0029] To combat these problems, numerous beamforming algorithms have been developed. Beamforming allows receiver arrays to improve signal quality by estimating the direction of arrival (DOA) of an energy source by comparing the signals recorded by each receiver. Broadly speaking, beamforming algorithms are typically categorized into three classes. The most basic category consists of maximizing the controllable response power (SRP) of the received signal, and examples of these beamformers include delay-plus-sum, filtering-plus-sum, and maximum likelihood estimation beamformers. The second category includes methods using time difference of arrival (TDOA) estimators, which take into account the arrival time of the signal to each receiver. Essentially, if a signal arrives at receiver "A" before it arrives at receiver "B," it can be estimated that the signal is closer to the origin of receiver "A," unless there is echo or other form of interference. Given a number of receivers, the TDOA estimator can be pointed in a specific direction believed to be the origin of the signal and also filters out background noise to reconstruct the original signal of interest. The third category includes spectral estimation-based locators, which incorporate multi-signal classification (MUSIC) algorithms used in many state-of-the-art devices. However, regardless of the beamforming algorithm chosen, the performance of the selected beamformer is highly dependent on the geometry of the receiver array.

[0030] Given an arbitrary receiver array, the limitation of its beamforming capability is determined by the spacing between the receivers. This depends on the Nyquist-Shannon sampling theorem, which specifies the beamforming capability for wavelengths... The minimum distance between the receivers of the signal Depend on Given the wave speed in the propagation medium. (For example, the speed of sound), can be determined by having a receiver spacing. The maximum frequency of the receiver array beamforming is therefore determined by Above this frequency, aliasing occurs, and it becomes impossible to determine the direction from which the signal was emitted. Therefore, for beamforming high-frequency signals, an array with small spacing between receivers must be used. However, if low-frequency signals are also important, a closely spaced array designed for high-frequency applications will appear small, and the phase difference between receivers will be minimal, resulting in a large beam. In essence, this prevents beamforming from improving signal quality and accurately identifying DOAs. In most beamforming applications, it is not possible to assume that the signal will be narrowband. Due to the fact that signals are typically broadband, uniformly spaced receivers cause a spectral shift in the beamformed signal. These facts have led to extensive academic and industrial research efforts on both optimal receiver array geometry and beamforming algorithms to capture information equally across a wide frequency range. The main limitation of these methods and the primary reason for making such efforts is that the receiver array hardware is often spatially static, e.g., an antenna array fixed to the top of a tower. For sources far from the array, this can make two different sources appear to originate from the same location. For sources close to the array, the far-field assumption does not hold (the incident wave cannot be modeled as a plane wave), and source localization becomes a challenging task.

[0031] The resulting aperture beamforming is a method that typically uses a 1D receiver array to generate a high-resolution 2D map by stacking numerous high-resolution 1D measurements acquired at different locations. In practice, this is often implemented by deploying 1D arrays, for example, on aircraft or ships traveling along known paths, and using sonar or radar to reconstruct images of, for example, islands or oceans. These methods are impractical for use cases such as monitoring factories or other enclosed environments, frequently monitoring changes in the environment over time, or providing location information in three dimensions.

[0032] Spatial-dynamic beamforming (SB), also known as spatial-dynamic beamforming, described in this paper, involves observing an energy source from multiple perspectives and using beamforming algorithms to stitch this information together to obtain a complete picture of the energy in a given space. SB methods can be used to obtain information about various types of energy (e.g., electromagnetic), but will be discussed from an acoustic perspective in this disclosure. However, it should not be concluded that these methods are only applicable to acoustic SB.

[0033] The field of foreground and background separation, often simply referred to as "background subtraction," involves separating the foreground and background from a signal and is highly relevant to SB (Background Subtraction). Much of the work in this field originated from computer vision and video supervision tasks. The foreground includes objects of interest, such as people or vehicles moving around the frame. The background includes static or pseudo-static objects, such as trees, buildings, and roads. By estimating the signals that constitute the background, the foreground can be extracted from the overall signal, enabling subsequent tasks to be performed with higher fidelity, such as following a person's movement throughout the video. At a high level, the basic assumption of background subtraction involves the idea that the background is relatively static, such that the background objects are identical in every frame of a given time-frame sequence from a scene. By observing many frames and identifying objects that do not change or change very little, the background can be identified and thus subtracted from the overall signal to reveal the foreground objects. Several approaches have been devised to address this problem, including kernel density estimation (KDE), Gaussian mixture models (GMM), hidden Markov models (HMM), various subspace approximation and learning techniques, and various other machine learning techniques in supervised, semi-supervised, and unsupervised learning, such as support vector machines (SVM) and deep learning, e.g., convolutional neural networks (CNN). Background subtraction techniques have also been used for acoustic background subtraction. However, this technique is applied purely for static acoustics where the recording platform does not change its spatial position. Furthermore, work in this area primarily focuses on identifying various background noises associated with the surrounding environment, such as vehicle traffic, wind, HVAC systems, or other signals that may affect technologies such as speech recognition devices. Little work has been done on isolating noise associated with the recording platform, especially in the emerging field of acoustic devices mounted on mobile robotic platforms.

[0034] Because space dynamic beamforming typically involves a platform with some kind of motion capability (e.g., unmanned aerial vehicles, robotic platforms, robotic arms, etc.), some kind of self-noise is usually associated with the signal acquired via SB. Various embodiments of these self-noise sources can include motors, servo systems, wheels, mechanical belts, fans, propellers, vents, jets, joints, gears, electronics, and other mechanical devices. Typically, these processes generate structured noise or noise in which some known knowledge about the signal exists. This knowledge can include factors such as when the noise begins and ends, but can also include more complex factors such as frequency content, statistics, or other mathematical quantities. In many cases, this structured noise is consistent across frames, making it acceptable as background noise. This background noise is undesirable when implementing space dynamic beamforming because it can alter measurements of the overall energy describing various parts of a given space. Given methods for separating this background from the rest of the signal, space dynamic beamforming methods can be improved by including only noise from the region of interest or object in the calculations. Furthermore, by isolating the background signal purely associated with the moving platform, which provides insight into the operation of the platform itself. By monitoring how this background signal changes over time and comparing it with various activities of the platform (e.g., different types of motion, load, runtime, etc.), the overall "health" of the platform can be understood.

[0035] This document describes systems and methods for generating high-resolution spatiotemporal energy maps of regions of interest or spaces via a novel combination of spatially sensed mobile beamforming receivers (referred to herein as spatiotemporal beamforming, also known as spatial-temporal beamforming). Receivers or receiver arrays, combined with beamforming algorithms, are deployed on a mobile platform that records information at various locations and uses this spatially distributed information to reconstruct a coherent model of the measured space. It should be noted that these methods can be used to acquire information about various energy sources, including acoustic emitters and electromagnetic emitters. However, for simplicity, this disclosure can be discussed primarily from an acoustic perspective. Systems and methods for isolating noise associated with the robot or mobile platform and recording equipment used in spatial dynamic beamforming to improve the output of the spatial dynamic beamforming algorithm and also to provide insight into the operation of the platform itself are also disclosed. It should be noted that these methods can be used to acquire information about various energy sources, including acoustic emitters and electromagnetic emitters. However, for simplicity, this disclosure can be discussed primarily from an acoustic perspective.

[0036] In one general aspect, a spatially conscious mobile receiver or receiver array is disclosed. The system includes some form of displacement motion unit, including but not limited to a wheeled robotic platform, track, jet, propeller, aerial drone, or robotic arm of a mobile receiver array. The mobile portion of this system includes some form of measurement and recording system telemetry technology. The telemetry portion of the system may include devices attached to the mobile platform that provide this information (e.g., optical imaging, radar, lidar, etc.), or it may also include systems not attached to the mobile platform, such as motion capture or simultaneous localization and mapping (SLAM) systems. The receiver array may consist of a single receiver or a group of many receivers organized in various 1D, 2D, or 3D geometric configurations. The system may also include onboard computing hardware and software operating independently or in combination with separate computing hardware and software. Furthermore, the entire system may consist of multiple separate mobile platforms, each with its own set of recording devices and positioning capabilities, all of which communicate with each other and / or with the main system.

[0037] In another general aspect, a method is disclosed for obtaining an acoustic image of a space by generating a coherence map from a group of beamformed information from multiple different viewpoints. This method involves applying a beamforming algorithm to data acquired by a receiver or receiver array located at a specific location and facing a specific direction to determine the acoustic output (AO) of a region of interest (ROI) or object of interest (OOI) within the receiver's field of view (FOV). The AO can consist of sound pressure level (SPL), spectrum, time-series signal, or other recording methods. The AO for each ROI / OOI is recorded, and then the acoustic array is positioned in a new location. This new location can be achieved via displacement of the receiver array itself (e.g., yaw, pitch, roll) or by completely repositioning the array in three-dimensional space. At the new location, the same AO record is recorded for all ROIs / OOIs within the FOV, including all previously recorded ROIs / OOIs within the FOV or new ROIs / OOIs not in any previous FOV. For previously measured ROIs / OOIs, the new AO is stored in a database along with previous measurements acquired from other locations. This recording and repositioning process is repeated endlessly until a complete map of the imaged space is obtained using the algorithm described in this paper.

[0038] In another general aspect, systems and methods for implementing self-calibration are disclosed, wherein noise associated with the recording instrument or platform is isolated from the noise of interest.

[0039] In another general aspect, self-health monitoring is implemented using noise associated with the recording instrument or platform.

[0040] Certain aspects will now be described in detail to provide a general understanding of the categories of devices, principles of use, designs, manufactures, and associated methods, algorithms, and outputs disclosed herein. One or more example illustrations of these aspects are shown in the accompanying drawings in various non-exhaustive embodiments. It will be understood by those skilled in the art that the methods and devices described in this disclosure and the accompanying drawings are non-limiting examples, and that the scope of this disclosure is defined only by the claims.

[0041] In many industrial, commercial, or consumer applications, monitoring the energy of the surrounding environment is crucial for human health, machine performance, infrastructure health and integrity, and the maintenance of many other assets. This energy can include, but is not limited to, acoustic energy (including audible, ultrasonic, and infrasonic), visible light and the entire electromagnetic spectrum, nuclear energy, chemical energy, thermal energy, mechanical energy, and even gravitational energy. Numerous sensors and methods have been devised to monitor these diverse forms of energy. Most commonly, these sensors are used in a static manner, i.e., they are placed at a given location and monitor their surroundings from said viewpoint, such as security cameras, microphones, or thermal imagers. However, from these locations, there are noise sources that interfere with the recording of various ROI / OOIs within the sensor's FOV. Various algorithms have been developed to account for this interference (e.g., beamforming), but this does not address two problems: (1) not all noise sources can be isolated and removed from the ground-of-interest signal; and (2) not all signals of interest can be acquired from any given location. Methods and systems designed to address these problems via a novel combination of mobile position sensing platforms and sensor technologies are described herein. These methods and systems are described primarily from an acoustic perspective. However, operation within the acoustic field should not be considered a limiting embodiment of this disclosure.

[0042] Figures 11-16This is a set of example embodiments of a space-aware robotic platform that can be used to modulate sensor hardware for data acquisition in space. In one embodiment, the robotic platform is a wheeled platform with a sensor array system directly mounted to the chassis. The platform can also be used to load other cargo. In another embodiment, the same wheeled robotic platform is used, but a specialized mounting system is used to statically or dynamically position the sensor array in a specific location or location cluster. This robotic platform can also induce displacement motion via other means, such as tracks, ball bearings, rollers, or other methods. In another embodiment, the robotic platform takes the form of an unmanned aerial vehicle (UAV) that uses an air propeller to induce displacement motion. The sensor array system can be placed directly on the UAV chassis or on a dedicated arm for arbitrary sensor positioning. In another embodiment, the robotic platform takes the form of an UAV or aerial object that uses some other form of displacement motion, such as jet propulsion or antigravity. Again, the sensor array system can be placed arbitrarily. In another embodiment, the robotic platform takes the form of a robotic arm that provides multiple degrees of freedom for sensor positioning. This robotic arm may or may not be integrated with another robotic platform to increase displacement motion capabilities. In another embodiment, the robotic platform may take the form of a simulator-like robot capable of moving in space in a manner similar to that of a human. The sensor array system may be mounted onto the body of the simulator robot or otherwise. Figures 11-16 In another embodiment not shown in the figure, the robot platform may not be a robot in essence, but a positioning device that is manually positioned and repositioned by a person.

[0043] In various ways, a robotic platform can be configured to locate itself in space relative to objects around it, or to locate to an initial position or set of positions to provide telemetry information. This telemetry information may include the distance to the ROI or OOI, precise coordinates in space given a set of reference coordinates, distance traveled, tilt, pitch, and roll angles, or other positioning means. The telemetry portion of the system may include devices attached to the mobile platform that provide this information (e.g., optical imaging, radar, lidar, RF, etc.), or it may include systems not attached to the mobile platform, such as motion capture or simultaneous localization and mapping (SLAM) systems.

[0044] Figure 1This is a flowchart of a machine monitoring system using dynamic spatiotemporal beamforming 100. In block 102, the controller performs dynamic spatiotemporal beamforming, also known as spatiotemporal beamforming. Next, the controller performs self-calibration in step 104. This self-calibration can be performed at regular or irregular intervals, such as each time the system is powered on, at a predetermined time interval, when a change in physical measurement results (e.g., temperature, pressure, detection of airborne particles such as oil droplets or smoke) is detected, or when an anomaly is detected in step 102. In step 106, the controller performs self-health monitoring as in step 104. Similar to step 104, self-calibration can be performed at regular or irregular intervals, such as each time the system is powered on, at a predetermined time interval, when a change in physical measurement results (e.g., temperature, pressure, detection of airborne particles such as oil droplets or smoke) is detected, or when an anomaly is detected in step 102. In step 108, the controller performs the region of interest (ROI) as in step 104. This self-calibration can be performed at regular or irregular intervals, such as each time the system is powered on, at a predetermined time interval, when a change in physical measurements (e.g., temperature, pressure, or airborne particles such as oil droplets or fumes) is detected, or when an anomaly is detected in step 102. When the system monitors an area, special attention can be used to focus on the ROI, and steps 102, 104, 108, and 106 can be used together or in pairs to perform machine health monitoring. This includes virtual sensing 110, assembly line monitoring 112, vehicle monitoring 114, or another machine health monitoring 116. This information can be used in a database where controllers (alone or in step 102) perform knowledge-based decision analysis.

[0045] Figure 3This is a block diagram illustrating hardware 300 in a system according to at least one aspect of this disclosure. Processor 302 is powered by power supply 304, which may take the form of various power sources, such as, but not limited to, a battery, wired power (e.g., to an AC outlet), or a solar panel. Processor 302 may be a single central processing unit (CPU) processor comprising one or more cores, a set of CPUs, a graphics processing unit (GPU), a set of GPUs, a combination of CPUs and GPUs, or multiple other computer hardware platforms and devices. Processor 302 interacts with communication hardware 306, which communicates with external devices and system 318. Some examples of communication protocols include Bluetooth, Wi-Fi, RFID, and so on. External devices 318 may be other robotic platforms, infrastructure, a central control computer, and so on. Processor 302 also interacts with sensors 308, which may include sensors (such as microphones or electromagnetic sensors) for recording environmental data for spatial dynamic beamforming, and may also include other types of sensors (such as cameras, ultrasonic detectors, lidar, radar, and other sensor modalities) for positioning, telemetry, and motion. Sensor 308 and processor 302 interact with telemetry system 310 and motion system 312, which provide various position-aware displacement motions to the robot platform. Processor 302 and motion system 312 store information in memory 314, which may be in the form of, for example, random access memory (RAM), hard disk drive storage, solid-state storage, or other types of storage. Various types of data can be stored in memory 314. Embodiments of the hardware described by at least one aspect of this disclosure may also include visualization hardware 316 and accompanying software to allow the robot platform to provide visual information to a user or other robot that indicates various information, such as robot status, status of other hardware, recording patterns, position information, reconstructed spatial dynamics, and other information. Example embodiments of visualization hardware 316 include, but are not limited to, LCD / LED screens, LEDs, digital displays, and analog devices.

[0046] Figure 2This is a block diagram of a high-level method 200 for acquiring an energy map of spatial dynamic beamforming according to at least one aspect of this disclosure. During initialization step 202, certain parameters can be selected for a specific environment of interest. In particular, several aspects (such as ROI, OOI, spatial resolution, recording parameters (e.g., sampling frequency and duration), recording location, etc.) are selected, which can vary according to the environment of interest. An example embodiment of several of these aspects that can be selected for a specific type of environment (such as a production line including various robots, machines, conveyors, and other mechanical equipment) may include ROI / OOI specific to each piece of equipment, certain parts of each piece of equipment, certain or all parts of an assembly process, human-inhabited areas, products of such a production line, and other regions of interest. In this embodiment, the sampling frequency and duration selected for sensor recording can be chosen to monitor a specific signal, i.e., a narrowband signal or a wide range of signals with varying spatial frequencies (e.g., continuous versus transient) (broadband). Other embodiments of these parameters can vary considerably according to the environment of interest. Importantly, the recording location can also be selected during or before this initialization period. These locations can be selected to represent a specific pattern or can be randomly selected. Furthermore, in one embodiment, these locations can be selected indefinitely throughout the recording duration (this can be virtually infinite), and the system can select these locations randomly or according to a certain principle or set of principles. After initialization, the robot platform positions itself (or is positioned) at the initial location in step 204. In the middle. At this position, in step 206 from position Record data of the various ROIs / OOIs selected during the initialization phase. In another embodiment, these various ROIs / OOIs are not selected during initialization, but rather as new ROIs / OOIs are discovered throughout the recording process. In step 208, the controller checks whether all recordings in step 206 are complete. If not all regions of interest (ROIs) are covered, the controller branches to step 210 and continues until data from all ROIs / OOIs is recorded. Upon completion, in step 214, the robot platform is repositioned to the new location. This pattern continues in steps 204-214 until all ROIs / OOIs have been imaged from all locations. At this point, a mathematical space dynamic beamforming method is applied in step 216, and the environment map is returned in step 218. In another embodiment briefly described above, the pattern of moving through the various locations in step 204 and recording in step 206 is repeated continuously, and Figure 2 The algorithm continues without much benefit. In this embodiment, the environment map from step 218 is returned by the spatial dynamic beamforming in step 218, which can be returned at any time and can be continuously updated.

[0047] The following are mathematical formulas for a non-limiting embodiment of a spatial dynamic beamforming algorithm used in conjunction with other methods and systems in this disclosure to reconstruct an acoustic map of an environment of interest. Given a microphone array with 𝑛∈[0,…,𝑁] microphones, the output of the frequency domain beamforming of the array at the spatial recording position 𝒎∈𝑴 for a spatial location of interest (i.e., ROI) 𝒒∈𝑸 is defined as follows:

[0048]

[0049] in and These are the frequency domain signal of each microphone at each location and the associated filter for each microphone. M and Q Let these represent the sets of spatial locations in three-dimensional space for the recorded location and the location of interest, respectively. Here, we first assume that the filter can change with position, but it is possible that... For example, filtering and summing beamforming. Element-by-element phasing via... Application and is any given Unique. Therefore, the definition The equation represents any beamforming operation for any microphone array in space. Then...

[0050]

[0051] Defined as from M The location is a matrix representation of the acoustic maps of all spatial locations recorded in the location map. At any given spatial recording location, even with optimal beamforming operations, the signal acquired for each spatial ROI cannot perfectly represent the signal emitted purely from each spatial ROI and acoustic map; that is, there may be some noise and distortion associated with each signal. This noise and distortion This could be due to a combination of signals from other sources, reflections, distortion, sensor noise, etc. Therefore, the acoustic diagram is given below.

[0052]

[0053] in This represents the ground-based acoustic map of all ROIs, regardless of recorded location. Our goal is to find... This is because it contains the real signal emitted by each source / ROI, unaltered by other sources, reflection, absorption due to attenuation and scattering, etc. Importantly, it is assumed that...

[0054]

[0055] as well as

[0056] .

[0057] Also assume that by The distortion is primarily due to sensor issues (e.g., lens scratches, damaged microphones, etc.) and does not irreversibly distort the overall signal. Under this assumption, , where 𝑰 is the identity matrix.

[0058] therefore,

[0059]

[0060]

[0061] .

[0062] Due to acoustic noise As the location changes, therefore, if acoustic maps are recorded from sufficient locations... The average acoustic map from these locations will then be close to .

[0063] Figure 4 This is an illustration of an example embodiment of the movement of a robotic platform throughout an area, room, or environment, based on at least one aspect of this disclosure. Figure 4 The robotic system 400 is most closely similar to an aerial unmanned aerial vehicle-type robotic platform, but it should be understood that any embodiment of the robotic platform (including...) can be used. Figures 11-16 Those in the above (or other embodiments) include those exhibiting motion 416 along any and all arbitrary directions or a subset of all possible directions (e.g., capable of moving with three degrees of freedom on the base plate of such an environment or enclosure). Robot 402 moves around a room or environment 404 and collects data 408. The area, room, or environment 404 can be an enclosed room of arbitrary shape and size, or it can be an open or partially open environment also of arbitrary shape and size. Data 408 can be acquired via a beamforming algorithm 406 that acquires data across an elongated column 414, which may include specific segments or slices of room 404 (e.g., ...). Figure 4The data 408 can be processed via spatial dynamic beamforming to form an energy map of room 404. Furthermore, various algorithms (such as classical signal processing techniques or modern data-driven methods from machine and / or deep learning or other fields) can be applied to the data 408 to compute various features 410 and output data 412 indicating certain states of environment 404. This data 412 can indicate things such as machine health, room health, pedestrian traffic, and other characteristics. Therefore, room 404 can contain many different items or processes of interest, which can be analyzed by features 410 and the output data 412.

[0064] Figure 5A This is an acoustic mapping system 500 for an environment acquired via spatial dynamic beamforming of data obtained from a sensor system on a mobile robotic platform, according to at least one aspect of this disclosure. In this embodiment, a 3D version of a region 502, such as a room, is shown; however, room 502 can be represented in any number of other dimensions (e.g., 1D, 2D, 3D, 4D, or more). Data 504 and 506 represent energy levels defined by a scale. In this embodiment, the scale is scaled from 0 to 1 and represents intensity. However, this scale can also indicate other characteristics, such as frequency content or object type. Data 506 and 508 are co-registered with 502 such that their data represent what has happened spatiotemporally at said particular location. In this example, data 506 indicates a region with low energy, and data 504 indicates a region with high energy. In this embodiment, room 502 is an empty room. However, room 502 in this figure can also include other objects of interest.

[0065] Figure 5B This is an acoustic mapping system 550 for an environment acquired via spatial dynamic beamforming of data acquired from a sensor system on a mobile robotic platform, according to at least one aspect of this disclosure. In this embodiment, a 3D version such as room 552 is shown, but room 552 can be represented in any number of other dimensions (e.g., 1D, 2D, 3D, 4D, or more). Data 554 and 556 represent energy levels defined by a scale. In this embodiment, the scale is scaled from 0 to 1 and represents intensity. However, this scale can also indicate other characteristics, such as frequency content or object type. Data 556 and 558 are co-registered with 552 such that their data represent what has happened spatiotemporally at the specific location. In this example, data 556 indicates a region with low energy, and data 554 indicates a region with high energy. In this embodiment, room 552 is an empty room. However, room 552 in this figure can also include other objects of interest.

[0066] However, an important aspect of the process typically associated with SB is a mobile platform that moves recording equipment around a region of interest. The motion associated with these platforms often involves noise that can be recorded by the SB acquisition equipment, thus altering the recorded measurements. This disclosure discloses methods for removing this noise from SB measurement results. Once this noise has been properly isolated and removed, it purely represents the noise associated with the mobile platform. Methods for monitoring the status and health of the mobile platform, given this noise isolation, are also disclosed herein.

[0067] Figure 6 High-level block diagrams are shown illustrating various embodiments of implementing self-calibration and self-diagnostic algorithms into a space dynamic beamforming algorithm 600 implementation according to at least one aspect of this disclosure. In the absence of self-calibration and self-diagnostic algorithms, the SB system initializes its environment in step 602 and moves to a position in step 604. (This could be its starting position), data is recorded at step 606, a check of all regions of interest (ROIs) is recorded at step 608, and a counter is incremented at step 610. Then, at step 612, it is checked whether all positions have been recorded, and the process moves to the next position 614 until all positions have been recorded. Upon completion of recording, SB is typically applied at step 616 to return the environment map at step 618. In a potential embodiment of the method in this disclosure, a self-calibration and self-diagnostic algorithm (“calibration”) can be applied at algorithm injection point 1 (i.e., step 620) to ensure that each region of interest (ROI) or object of interest (OOI) is acquired... The data is then analyzed. In another potential embodiment, calibration can be applied at algorithm injection point 2 in step 622 after all ROIs have been acquired for a given recording location. In yet another possible embodiment, calibration can be applied at algorithm injection point 3 in step 624 after all ROIs at all locations have been acquired. These three algorithm injection points offer various benefits. At algorithm injection point 1, calibration is applied only to a small amount of data. While this may result in some information loss or overfitting, it is likely the fastest algorithm injection point. In a potential embodiment of SB, various beamforming techniques (e.g., delay and summation beamforming) can be applied at step 606 to collect aggregated signals from a single ROI / OOI. In the embodiments described, it should be noted that data can be acquired at any given ROI. Calibration is performed before or after beamforming techniques. At injection point 2, calibration is applied to all ROIs from a specific recording location. This potential injection point offers the advantage of having more information about the noise associated with that specific location than injection point 1, but it is also likely to be computationally more complex than injection point 2. Injection point 3 has the advantage that calibration can be applied to all data across the entire recording domain, i.e., all ROIs / OOIs from all recording locations. This injection point is likely to be the most computationally complex due to data overhead, but it can also provide the most accurate calibration. In another possible embodiment, beamforming (SB) can be applied over time over hours, days, weeks, years, etc. In this embodiment, calibration can be applied periodically over time to understand various state changes in the system implementing SB and its surrounding environment.

[0068] Below are mathematical formulas for several non-limiting embodiments of a self-calibration algorithm used in conjunction with spatial dynamic beamforming to obtain accurate acoustic maps of various environments and to implement self-health diagnostics of the hardware used to acquire this information. Given a record Let , so that represents the number of samples in the record, and define the function. , making

[0069]

[0070] From vector form The mapping to matrix 𝑃, where .right The choice depends on the recording sampling frequency and the feature sizes of the foreground and background noise of interest. It can be selected as any value, but in its preferred embodiment it is selected such that Then, singular value decomposition (SVD) is applied to , such that...

[0071]

[0072] in ,and The basic assumption of this algorithm is that the self-noise, or "background" noise, associated with the recording platform is relatively uniform at any given time. Then, a mapping is applied. Samples of this background noise are placed in each column of , such that most of the information stored in is low-rank or even rank 1. The low-rank version of the signal includes the background noise, while the high-rank version includes the foreground data, foreground signal, or signal of interest. When used for self-diagnosis, the high-rank version of the signal includes the foreground noise, while the low-rank version includes the background data. The background noise is then reconstructed from the original signal, and the foreground portion of the signal is then approximately isolated via the rank r of ...

[0073]

[0074]

[0075] in and These represent the background and foreground portions of the signal, respectively. r The value of is typically chosen to be very small, and in its preferred embodiments is usually between 1 and 5. However, r The optimal value can be calculated using the following formula.

[0076]

[0077] in It is a ground-based foreground signal, and It is the Frebinus norm. It is certainly impossible to calculate prior values ​​without knowing the foreground signal, but it can be used, for example, for calibration and experimental testing in various environments.

[0078] In cases where the background noise is unstructured, it may be desirable to convert the data to the spectral domain before implementing SVD separation.

[0079]

[0080] An alternative implementation of spectral domain-based separation is to convert the data into spectra and use a series of spectral "images" as various frames. Then, via... A similar mapping function vectorizes these frames, and then SVD separation is applied.

[0081] Another embodiment that works particularly well when the background noise is highly structured is to use a cross-correlation function to align the vectors in 𝑃, then truncate the ends of 𝑃 to remove zero-padding, and then perform SVD background separation. Finally, using this known background, cross-correlation is used again to align the estimated background to extract the foreground.

[0082] Another embodiment involves estimating a matrix called the "shift matrix," which essentially defines the phase changes between vectorized signals in the columns of 𝑃. In essence, this shift matrix is ​​estimated, a de-shift operation is applied, and then our SVD separation algorithm is applied. Depending on how the shift matrix is ​​computed, this may ultimately be mathematically equivalent to the cross-correlation algorithm.

[0083] Therefore, in the next section, we will describe how we inject knowledge of the system's state into this computation. We want to ensure that we only compare similar states; for example, a robot will likely vocalize differently when it moves in different ways or performs different tasks. We might also want to consider the case where we are only concerned with isolating noise when the robot is simply placed in one place and not moving, which would likely be the simplest approach in any case.

[0084] Finally, given the pair A good estimate of this allows us to use it as an indicator for understanding the state of a mobile system used with an SB. We define... , which represents the total number of measurements obtained, that is, the number of times the self-noise of the mobile robot platform is calculated. The value of can vary for a variety of reasons, such as the frequency of the acoustic map calculated for a given space, the performance of the robotic platform, or how its self-noise compares to the self-noise of other robotic platforms operating in the same general area. At a very high level, we can essentially use a distance metric between what we have previously measured and what we are now measuring, along with some predefined or potential dynamic threshold, to determine if a problem exists. The distance metric we use could be something like the Frebinus norm or KL divergence, but is highly dependent on how we mathematically represent this information. The predefined or dynamic distance threshold will likely have to depend on the robotic system and its surrounding environment. We might also want to discuss adding various other information, such as environmental conditions or loads on the platform.

[0085] This is about how Figure 1 The details of defining the Region of Interest (ROI) are illustrated in box 108. The ROI algorithm, including examples of what this ROI map might look like, will be presented below. Essentially, it looks like a set of square regions of different sizes representing different levels of signal output. There are several ways it can be displayed: for example, at the dBA level, the overall SPL, which can represent some metadata about the data structure, including all records of the patch or some other information.

[0086] The algorithm set (e.g., algorithms 1-4 shown below) defines a method for dynamically assigning resolution levels in a given space for spatiotemporal beamforming. The aim of these algorithms is to use higher resolution imaging for regions of interest and lower resolution imaging for quieter regions. Algorithm 1 illustrates a high-level algorithm for dynamically generating spatial resolution mappings across multiple recording locations via spatiotemporal beamforming. In this algorithm, the input is the recording location. Locations The list is output, along with the information generated from those locations. data .

[0087] Algorithm 1: High-level mapping algorithm

[0088]

[0089] In Algorithm 1, the initial spatial resolution is used to define the size of individual regions of interest (ROIs). S It is set to an arbitrary unit value. In this embodiment, .However, S The size constraints of the given space, the acoustic distribution of the object of interest, and the physical capabilities of the recording device should be set to appropriate values. Location Index L It is set to 0 and is used to assign data structures data By adding an index, the data structure is initialized as an empty, arbitrary data structure that can vary according to hardware and software requirements.

[0090] The MOVE algorithm cited above represents the commands and subsequent actions taken by any robotic platform to move to the next position. The MOVE algorithm will not be described in detail here. For a given recorded position... L Algorithm 2 illustrates the basic investigation algorithm, which uses a recursive approach to investigate increasingly smaller regions until a sufficient resolution level is reached. This is related to the well-known binary search algorithm.

[0091] Algorithm 2: Region of Interest Survey Algorithm

[0092]

[0093] Sufficient resolution is determined by Algorithm 3 and can be based on many factors. For example, these factors can be based on both the acoustic content of a given region and the physical limitations of embodiments of these algorithms to resolve smaller regions. All of these factors are included in the "if interesting then" statement. Algorithm 4 is used to define the spatial information for each ROI. The input to this algorithm is the current region... r and current splitting factor s The simplest implementation of Algorithm 4 is by factoring only... s District r It is divided in two. However, other, more complex methods may divide it in ways that provide more useful information. Furthermore, this entire method should by no means be limited to the Cartesian system. The various regions can have different shapes and sizes, and do not need to be convex.

[0094] Algorithm 3: Interest Level Determination Algorithm

[0095]

[0096] Algorithm 4: Region of Interest Segmentation Algorithm

[0097]

[0098] Next, as follows Figure 1 The diagram in box 110 considers machine health monitoring. Virtual sensing / signal-to-signal conversion systems are disclosed here. Systems and methods are disclosed for acquiring estimates of data from one or more modes (e.g., source modes) by analyzing data from different modes or sets of modes (e.g., target modes). These methods can be used to “virtually” sense or acquire data from multiple systems or processes via a fundamental, linear, or nonlinear mapping between virtually sensed target data and acquired source data learned by observing paired data across all modes. This mapping is typically associated with some basic physical process, but it may also be derived from a more abstract process. Virtual sensing can enable the replacement of expensive and / or difficult-to-deploy sensors with cheaper and / or easier-to-deploy sensors. Furthermore, virtual sensing can achieve predictive machine or process health monitoring and diagnostics by acquiring estimates of data that provide useful information.

[0099] Sensors allow us to observe and record the world around us and estimate predictions of its future state. For example, by using sensors to observe local and global environmental indicators such as temperature, pressure, humidity, and the tracking of weather fronts, future weather conditions can be predicted with high accuracy over relatively short timeframes. Other sensors enable the observation of more microscopic processes, such as the condition of an engine, the stability of infrastructure, or human health. Often, these processes and the sensors used to observe them are related to some basic physical processes. These can be a combination of mechanical, electrical, or chemical processes, such as the current consumption of a computer processor or changes in hormonal signals within a living organism. Other times, these processes and their associated signals and sensors are more abstract, such as using stock prices as a sensor to estimate and predict market health. However, regardless of the system, all sensors used to elucidate the state of the system are related to the basic processes associated with said system, and therefore can be related via some typically nonlinear mapping or transfer function.

[0100] A common linear example of this transfer function is the ideal gas law φ = φ, where φ, φ, and φ represent the pressure, volume, and temperature of the gas, respectively. nLet represent the quantity of the gas in question, and be the invariant ideal gas constant. If the quantity of the gas in question is known, a temperature sensor can be used to measure the gas pressure without needing a pressure sensor. Since a temperature sensor can also output data from a pressure sensor given a suitable mapping equation, it can be said that a temperature sensor can "virtually" sense pressure. However, many other processes of interest exhibit sensor transfer functions with low linearity (such as degradation in various mechanical devices, such as automotive engines). In a typical engine, there are many moving parts, such as pistons, belts, fuel pumps, and so on. For example, if the goal is to analyze fuel pump degradation, several possible sensors that can be used include the temperature inside the pump, the torque or speed of rotating parts, structurally propagated vibrations, sound propagated by air released by such vibrations, flow rates induced by the pump itself, and so on. All of these sensors are essentially related to the state of the fuel pump in some way. For example, if the rotating parts that induce pump pressure begin to degrade and release particles, the friction inside the pump may increase, thereby increasing torque and temperature, thus altering the noise distribution in the air-propagated and structurally propagated sound, and likely reducing the flow rate caused by reduced pressure inside the pump or pump failure. Pump failure is defined as pump malfunction, and impending pump failure is defined as predicting pump failure within 24 hours, even though the pump may actually fail sooner. While all these sensors can observe the fuel pump in different ways and report different patterns of information, they are all related to the same fundamental physical processes associated with the pump itself. Therefore, there should exist some transfer function, likely nonlinear, that maps the values ​​of each sensor to each other in the same way that the ideal gas law maps pressure to temperature.

[0101] In many real-world scenarios, sensing physical processes can be difficult or expensive (e.g., measuring combustion pressure inside an engine or torque or flow rate inside a fuel pump while a vehicle is operating on the road). However, the ability to do so is crucial, especially for predictive diagnostics, machine health monitoring, and process control. Simultaneously, equipping such systems and processes in a laboratory setting to acquire sensory data that might be impractical or prohibitively expensive for large-scale real-world deployments may be less difficult or expensive. Given these challenges, a controlled environment can be used to “virtually” sense phenomena and / or physical processes that are difficult to sense in a field setting, combining laboratory (training) data from expensive sensors that are difficult to deploy in the field with cheaper sensors that can be deployed on a large scale. This enables signal-to-signal transformation from signals acquired via cheaper sensing solutions to signals from other sensors that would otherwise be prohibited from large-scale field deployment for various reasons.

[0102] More specifically, this paper describes systems and methods for estimating mappings (i.e., transfer functions) between sensors to achieve virtual sensing (i.e., virtual perception) between sensor modes. Multiple data-driven algorithms are trained using observations from all sensors of interest to predict outputs from one or more sensors. These algorithms are then deployed in various embodiments to enhance the capabilities of existing sensors by allowing them to virtually acquire data in a manner that replaces the sensors.

[0103] In one general aspect, various embodiments of a system for acquiring data and implementing virtual sensing in multiple physical embodiments are disclosed. The system includes some form of data processing hardware and software for implementing training and deployment portions of the virtual sensing. A high-level system evaluates the output of the data processing system to inform the operator or the observed system itself of the state of the observed system. It also includes training observable processes, which are virtually implemented at deployment time for real-world measurements of the sensors.

[0104] In another general aspect, a general method for virtual sensing is disclosed, wherein measurements of a process from one or more other modalities are estimated from measurements of one or more modalities observing the same process via a learned mapping function. The method includes methods for preprocessing data using classical and modern methods to prepare the data for virtual sensing. The method also includes a method for acquiring this physical-to-virtual sensor mapping function using a sensor training algorithm that can only be used in a training setting (e.g., in a laboratory setting, rather than for large-scale use or in a real-world environment). The method also includes technical specifications for estimating real-world sensors from a virtual domain, and methods for jointly and non-jointly learning these graphs.

[0105] In another general aspect, specific methods for calculating physical-to-virtual sensor mapping functions via data-driven models are disclosed. These methods include various non-limiting potential and preferred embodiments, including generation methods in which various types of algorithms directly generate virtual sensor data from physical sensor data.

[0106] In another general aspect, a method is disclosed for monitoring processes via virtual sensing and enabling predictive maintenance or diagnostics of various systems. The method involves observing the output of a virtual sensing system and using virtually sensed data (with or without physically sensed data) to indicate various states of the process of interest. These states may include operating states, operating conditions, failure modes, detection of specific events, and so on. Methods for implementing this monitoring include classical signal processing and statistics, machine learning, deep learning, and methods involving human-computer interaction.

[0107] In many industrial, commercial, consumer, and healthcare applications, monitoring the status of various processes is crucial. To do this, a wide variety of sensing modalities are frequently used. These sensors can include cameras, lasers, lidar, radar, SLAM systems, microphones, hydrophones, ultrasonic sensors, sonar, vibration sensors, accelerometers, torque sensors, pressure sensors, temperature sensors, fluid volume and flow rate sensors, altimeters, velocity sensors, gravity sensors, gas sensors, humidity sensors, heart rate monitors, blood pressure sensors, pulse oximeters, EEG systems, EKG systems, medical imaging equipment, and many more. Sometimes, these sensors are inexpensive and easy to deploy on a large scale, such as low-cost microphones. Other sensors are more expensive and less easy to deploy, such as high-precision lasers. In some cases, the cost of the sensor or the method required for the sensor implementation prevents the sensor from being deployed in any practical environment. For example, while relatively inexpensive torque sensors exist and can be used to evaluate various machines in controlled laboratory environments, they may be too bulky or too difficult to deploy inside the engine of every vehicle on the road. However, sometimes non-deployable sensors are most important when understanding the status or health of a machine or process. This paper describes methods and systems designed to solve this problem by learning a function that maps data from one sensor or sensor set (e.g., a “source” or “physical” sensor) to data from another sensor or sensor set (e.g., a “target” or “virtual” sensor).

[0108] Figure 7This is an overview of a general physical embodiment of the virtual sensing system. Sensor kit 702 consists of all sensors available to the virtual sensing system. Example embodiments of the sensors in sensor kit 702 include optical, acoustic, vibration, environmental, electromagnetic, and other sensors. The sensors in sensor kit 702 are those that can be observed in training state 708 (e.g., in a laboratory). Therefore, sensor kit 702 also includes sensors that cannot be easily deployed on a large scale, which are the sensors to be virtually sensed. A data processor or controller 704 interacts with and controls sensor kit 702. Data processor 704 typically consists of some computing technology. Example embodiments of the data processing system include desktop computers, mobile computers, edge computers, mobile communication devices, mobile platforms such as vehicles or aircraft, security systems, and so on. In addition to interacting with the sensors, data processor 704 also analyzes the data and implements the core virtual sensing algorithm. A subset of sensor kit 702 is a set of deployable sensors 706. Deployable sensors 706 are those sensors that can be observed in a scaled deployment 710 and used as source sensors to achieve virtual sensing of targets or virtual sensors. In addition to the data processor 704, the evaluation and instruction system 712 interacts with both the data processor 704 and the deployed observable process 710 (and therefore also accesses data from sensors 702 and 706). The evaluation system 712 is primarily used to observe the output of virtual sensing occurring on the data processor 704 and provide indications about the state of the observed process. Exemplary, non-limiting embodiments of the evaluation system 712 can be computation-based systems (which use data processing methods that may be based on prior knowledge or previously observed understanding of the virtually sensed data), machine learning algorithms (such as support vector machines, decision trees, random forests, k-nearest neighbors, k-means, neural networks, or other machine learning algorithms) designed to classify the state of the process. The output from this classification can alert artificial intelligence within the evaluation system 712, the data processor 704, or the process 710 itself to inform and justify subsequent decisions, such as changing routes, shutting down equipment, modifying operating conditions, etc. The evaluation system 712 can also incorporate human participation, where humans observe the output from the data processor 704 or work with an automated evaluation system to interpret the output from 704.

[0109] exist Figure 7 The examples shown herein include many real-world embodiments of the systems, and several examples will be discussed herein. However, those skilled in the art will recognize that, in general, the virtual sensing systems and methods disclosed herein can be virtually applied to any situation where the objective is to acquire data from difficult-to-deploy sensors via an easily deployable sensor agent, and the following examples demonstrate situations such as… Figures 11-16 Several non-limiting examples of such cases are shown in the figure.

[0110] Figure 8 The diagram illustrates the mathematical foundations of the virtual sensing problem 800 and how it relates to the physical embodiments described herein. System 802 includes process 804, which is the process of interest to be observed via a physical and / or virtual sensing domain. Sensor 806 represents all available sensors. The output from sensor 806 includes all data 808, including target sensor data. Heyuan sensor data , A subset of. An example embodiment of the virtual sensing problem is as follows. For example, let's assume that the sensor data... This indicates the source sensors that can be deployed, and the sensors... It indicates that deployment is not possible, but testing in the test settings allows you to obtain several pairs of data for time t. The target sensor. The purpose of virtual sensing is to find the target sensor. Mapped to mapping function This allows for estimation even when sensors typically used to acquire this type of data cannot be deployed. In a more general case, the expectation is to find a mapping function. 810, which will Mapped to That is, for any , , where 𝐾 represents the number of sensors. Based on this formula, the task could be from one source sensor to one target sensor, from one source sensor to multiple target sensors, from multiple source sensors to multiple target sensors, or from multiple source sensors to one target sensor.

[0111] Figure 9 The illustration illustrates the core concept behind virtual sensing via signal-to-signal transformation. The sensing setup is designed to estimate the hidden physical state of a linear or nonlinear dynamic system S 902, wherein... 904. Sensor observations y1 910 and y2 912 can be viewed as the outputs of two observation models (transfer functions) H1(s) 906 and H2(s) 908, respectively. Both y1 and y2 encode information about the hidden state S in different ways according to the properties of the observation models (i.e., H1(s) and H2(s)). The key assumption of virtual sensing is that when y1 and y2 are obtained from the underlying physical state S / phenomenon of the system, there must be some mathematical relationship (linear or nonlinear) between the two sensing modes y1 and y2. This mathematical relationship represents the mutual information between the two sensing modes y1 and y2 conditioned on S. This can be used to represent / approximate the neural network model T. 12The mathematical relationship between (s) is then established. Next, the neural network is trained by collecting a large number of y1-y2 pairs across various states S through the physical system. In a real-world deployment scenario, with y1 as input, we now estimate y2 as the output of the trained neural network transfer function model (i.e., virtually sensing y2 in the absence of a corresponding physical sensor, rather than “virtually” sensing / reconstructing y2 from physically available sensor / data y1). The choice between y1 and y2 (i.e., which sensor data to virtually sense and which to physically deploy) depends on several factors / trade-offs. For example, y2 might be expensive and difficult to deploy in a scalable manner, while y1 is cheap and easy to deploy, and estimating the physical state S from y2 (the ultimate goal of sensing for downstream tasks) might be easier, i.e., H2(s) is in a less complex, better posture. In this case, the system will deploy y1 and virtually sense y2, and then use the y2 data to reach / estimate S to implement downstream tasks, such as machine health monitoring / predictive diagnostics.

[0112] Besides mapping source sensor data to target sensor data, it may also be important to be able to map target sensor data back to source sensor data. This mapping function should be tested in a test setup where target sensor data can be obtained. When considering the effectiveness of a mapping function, its accuracy can be directly measured. However, this is less effective when deployed in environments where target sensors cannot be deployed. Figure 10 This is a method 1000 for validating virtually sensed data to map target virtual data back to the observable source domain. The observable sensor 1002 is mapped to the virtual sensor 1004 via a forward learning model 1006. The virtual sensor 1004 is then mapped via a function... The inverse learning model 1008 is represented as a mapping back to the observable sensor 1002. Importantly, the inverse learning model can be developed simultaneously with the forward learning model on real data from both the observable and virtual domains. This is significant and similar to the forward mapping function. Inverse mapping function It is possible to map all virtual sensor data to all observable sensor data or any combination of both virtual and observable data.

[0113] Below are mathematical formulas for non-limiting embodiments of virtual sensing algorithms and training processes used in conjunction with other methods and systems in this disclosure to estimate data from virtual sensors of interest. Let us first consider a physical process monitored by a set of K sensors, and let... This represents the data acquired by the nth sensor. (Data) This can correspond to unprocessed data from sensors, processed data (e.g., filtering, normalization, calculations on spectra or spectral graphs, etc.), a composition of different data processing methods, or a combination of these data. Let's next consider the problem of estimating the measurement results of the target data from the measurement results of the source data. To solve the problem of... estimate The problem can be solved by learning the mapping function parameterized by parameter φ. This allows the minimization of the reconstruction loss λ to be achieved via the following formula.

[0114] .

[0115] Next Represented as virtual sensor measurement results. Alternative sensors. Measuring phenomena to obtain non-virtual measurement results By using sensors Measure the phenomenon and map the function Applied to Such measurement results are used to obtain virtual measurement results. An example, non-limiting embodiment of the reconstruction loss is... Norm, which is used to calculate the actual sensor measurement results Comparison with virtual sensor measurement results The difference between them. The value of 𝑝 can be defined based on problem requirements and / or technical specifications or by using expert knowledge. Furthermore, in cases where the reconstruction of one particular sensor or a particular set of sensors is more important than all target sensors, the reconstruction loss 𝐿 can be adjusted to account for it and prioritize certain sensors. For example, a weighted norm of the difference can be used, where the relative importance of sensor reconstruction is conveyed via the norm's weights.

[0116] To verify the mapping function Improving the accuracy of virtual sensing, and also being able to map back from the virtual sensing domain to the non-virtual domain, may be useful. During the process of learning the parameters *f* of the function *t* that maps source sensor data to target sensor data, it is also possible to learn the parameters of the inverse mapping function *w* that maps target sensor data to source sensor data. . 𝜃 and This joint learning process can be expressed by the formula:

[0117] .

[0118] The composition of the two mapping functions, α and β, allows for virtual sensing. The reconstruction estimate, without requiring the target sensor Clear knowledge of the measurement results. Next, let... ,in This represents the reconstruction loss. The quality of the estimate of the reconstruction can also be and As part of the joint learning process, in addition to learning the mapping, parameters φ and φ are also optimized. This makes 𝛾 an accurate indicator of the reconstruction error given by the following expression.

[0119] .

[0120] This additional consideration of the process of learning the parameters used for mappings (from source to target and from target to source) is an important concept regarding the robustness of mappings. This is useful in situations where the virtual sensor of interest can never be deployed as a non-virtual sensor or cannot be deployed on a large scale. A particular mapping can be considered robust if an imperceptible change φ applied to the input of the mapping function does not result in a perceptible change in the output of the mapping function. More formally, this corresponds to the boundary given by the following representation.

[0121]

[0122] in This represents the space where disturbances are permissible (e.g., radius 1). of (Sphere). Besides approximating the virtual sensor measurements to the real sensor measurements (minimizing the loss function 𝐿), the parameters of the mapping functions 𝑓 and 𝑔 can also be considered for minimizing these boundaries. Let

[0123]

[0124] This represents a measure of robustness to the mapping function. Including these robustness measures as regularization terms in the optimization problem at hand allows for training a model that is robust to at least perturbations of the training data. Therefore, learning 𝜃 and The problem can be represented as

[0125] .

[0126] Numerous embodiments of mapping functions 𝑓 and 𝑔 exist, and they can be parameterized by various algorithms and methods. Example embodiments of methods for obtaining these functions and their parameters include regression, principal component analysis, singular value decomposition, canonical correlation analysis, sequence-to-sequence modeling methods and multimodal representation methods, artificial neural networks, deep neural networks, convolutional neural networks, recurrent neural networks, U-networks, combinations and compositions of these methods, and so on. A set of exemplary preferred embodiments of mapping functions 𝑓 and 𝑔 is a generative model, and more specifically, as a variational autoencoder. In this case, each mapping function consists of an encoder and a decoder. The encoder component of 𝑓 is received. As input and output by The parameters of a parameterized Gaussian distribution. Based on this distribution, the latent vector... The sampled data is provided as input to the decoder section of 𝑓, and its output... Conversely, the encoder component of 𝑔 receives As input and output by The parameters of a parameterized Gaussian distribution. Based on this distribution, the latent vector... The sampled components can then be provided as input to the decoder of the 𝑔, whose output... .

[0127] The parameters of the learning generative model (i.e., the encoder and decoder) can be implemented by minimizing a loss function 𝐿, which measures the difference between the real target sensor measurement and the virtual target sensor measurement, as well as the divergence measure between the distribution obtained by the encoder and the prior distribution (e.g., a Gaussian distribution with zero mean and identical covariance).

[0128] Learning maps 𝑓 and 𝑔 using a variational autoencoder can be a separate or joint process. In the case where the maps are learned separately, the parameters of the encoder-decoder pairs for 𝑓 and 𝑔 are learned independently. In the case where the maps are learned jointly, the parameters of the encoder-decoder pairs for 𝑓 and 𝑔 are learned jointly, where the encoder for 𝑓 is the inverse of the decoder for 𝑔, and the decoder for 𝑓 is the inverse of the encoder for 𝑔. One way to implement this process is by imposing distributions. and Similarity (e.g., by minimizing the divergence between the two distributions).

[0129] Figure 11This is a schematic diagram of a control system 1102 configured to control a vehicle, which may be at least partially autonomous or at least partially autonomous robot. The vehicle includes sensors 1104 and actuators 1106. Sensors 1104 may include one or more visible light-based sensors (e.g., charge-coupled device CCD or video), radar, lidar, microphone arrays, ultrasonic, infrared, thermal imaging, acoustic imaging, or other technologies (e.g., positioning sensors, such as GPS). One or more of these specific sensors may be integrated into the vehicle. Alternatively, or in addition to the one or more specific sensors identified above, control module 1102 may also include a software module configured to determine the state of actuator 1104 during execution. A non-limiting example of the software module includes a weather information software module configured to determine the current or future weather conditions near the vehicle or other location.

[0130] In embodiments where the vehicle is at least partially autonomous, actuator 1106 may be embodied in the vehicle's braking system, propulsion system, engine, powertrain, or steering system. Actuator control commands can be determined such that actuator 1106 is controlled to prevent the vehicle from colliding with detected objects. The detected objects may also be classified according to what a classifier deems most likely, such as pedestrians or trees. Actuator control commands can be determined based on said classification. For example, control system 1102 may segment images (e.g., optical, acoustic, thermal) or other inputs from sensor 1104 into one or more background categories and one or more object categories (e.g., pedestrians, bicycles, vehicles, trees, traffic signs, traffic lights, road debris, or construction cones / pipes, etc.) and send control commands to actuator 1106 (in this case, embodied in the braking or propulsion system) to avoid collisions with objects. In another example, the control system 1102 can segment an image into one or more background categories and one or more marker categories (e.g., lane markings, guardrails, roadway edges, vehicle tracks, etc.) and send control commands to the actuator 1106 (here embodied in the steering system) to cause the vehicle to avoid crossing the markings and remain in the lane. In situations where adversarial attacks may occur, the system can be further trained to better detect objects or recognize changes in lighting conditions or angles of sensors or cameras on the vehicle.

[0131] In other embodiments where vehicle 1100 is at least partially autonomous, vehicle 1100 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and stepping. The mobile robot may be at least partially autonomous lawnmower or at least partially autonomous cleaning robot. In such embodiments, actuator control commands 1106 may be determined such that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled, enabling the mobile robot to avoid collisions with identified objects.

[0132] In another embodiment, vehicle 1100 is at least partially autonomous in the form of a gardening robot. In this embodiment, vehicle 1100 can use optical sensors as sensor 1104 to determine the state of plants in the environment near vehicle 1100. Actuator 1106 may be a nozzle configured to spray chemicals. Based on the identified species and / or identified state of the plant, actuator control command 1102 can be determined to cause actuator 1106 to spray an appropriate amount of appropriate chemical onto the plant.

[0133] Vehicle 1100 may be at least partially autonomous robots in the form of household appliances. Non-limiting examples of household appliances include washing machines, stoves, ovens, microwave ovens, or dishwashers. In such a vehicle 1100, sensor 1104 may be an optical or acoustic sensor configured to detect the state of an object that will undergo processing by the household appliance. For example, in the case of a washing machine, sensor 1104 may detect the state of the clothes inside the washing machine. Actuator control commands may be determined based on the detected state of the clothes.

[0134] In this embodiment, the control system 1102 receives image (optical or acoustic) and annotation information from the sensor 1104. This information, along with a predetermined number of category and similarity metrics stored in the system, is used. The control system 1102 can be used in Figure 10 The method described herein classifies each pixel of the image received from sensor 1104. Based on this classification, signals can be sent to actuator 1106, for example, to brake or turn to avoid a collision with a pedestrian or tree, to steer to stay between detected lane markings, or any action performed by actuator 1106 as described above. Signals can also be sent to sensor 1104 based on this classification, for example, to focus or move the camera lens.

[0135] Figure 12A schematic diagram illustrates a control system 1202 of a system 1200 (e.g., a manufacturing machine), such as a punching machine, a cutting machine, or a barrel drill, configured to control a manufacturing system 102 (e.g., a part of a production line). The control system 1202 may be configured to control an actuator 14, which is configured to control the control system 100 (e.g., the manufacturing machine).

[0136] The sensor 1204 of system 1200 (e.g., a manufacturing machine) may be an optical or acoustic sensor or sensor array configured to capture one or more properties of the manufactured product. Control system 1202 may be configured to determine the state of the manufactured product based on one or more of the captured properties. Actuator 1206 may be configured to control system 1202 (e.g., the manufacturing machine) for subsequent manufacturing steps of the manufactured product 104 based on the determined state of the manufactured product. Actuator 1206 may be configured to control subsequent manufacturing processes of the system (e.g., the manufacturing machine) based on the determined state of previously manufactured products. Figure 11 (For example, the function of manufacturing machines).

[0137] In this embodiment, the control system 1202 receives image (e.g., optical or acoustic) and annotation information from the sensor 1204. This information, along with a predetermined number of category and similarity metrics stored in the system, is used. The control system 1202 can be used in Figure 10 The method described herein classifies each pixel of the image received from sensor 1204, for example, to segment an image of a manufactured object into two or more categories, detect anomalies in the manufactured product, and ensure the presence of objects such as barcodes on the manufactured product. Based on this classification, a signal can be sent to actuator 1206. For example, if control system 1202 detects an anomaly in the product, actuator 1206 can mark or remove the abnormal or defective product from the production line. In another example, if control system 1202 detects the presence of barcodes or other objects that will be placed on the product, actuator 1106 can apply or remove these objects. Signals can also be sent to sensor 1204 based on this classification, for example, to focus or move a camera lens.

[0138] Figure 13 A schematic diagram is shown of a control system 1302 configured to control a power tool 1300 (such as a drill or screwdriver) having at least a partially autonomous mode. The control system 1302 may be configured to control an actuator 1306, which is configured to control the power tool 1300.

[0139] The sensor 1304 of the power tool 1300 may be an optical or acoustic sensor configured to capture one or more properties of the work surface and / or fasteners driven into the work surface. The control system 1302 may be configured to determine the state of the work surface and / or fasteners relative to the work surface based on one or more of the captured properties.

[0140] In this embodiment, the control system 1302 receives image (e.g., optical or acoustic) and annotation information from the sensor 1304. This information, along with a predetermined number of category and similarity metrics stored in the system, is used. The control system 1302 can be used in Figure 10 The method described herein classifies each pixel of the image received from sensor 1304 to segment the image of the work surface or fastener into two or more categories or to detect anomalies in the work surface or fastener. Based on this classification, signals can be sent to actuator 1306, for example, to control the pressure or speed of the tool, or any action performed by actuator 1306 as described in the preceding sections. Signals can also be sent to sensor 1304 based on this classification, for example, to focus or move a camera lens. In another example, the image may be a time-series image of signals from power tool 1300, such as pressure, torque, revolutions per minute, temperature, current, etc., wherein the power tool is a hammer drill, electric drill, hammer (electric hammer or electric pick), impact screwdriver, reciprocating saw, oscillating multi-tool, and the power tool is either cordless or corded.

[0141] Figure 14 A schematic diagram of a control system 1402 configured to control an automated personal assistant 1401 is shown. The control system 1402 can be configured to control an actuator 1406, which in turn is configured to control the automated personal assistant 1401. The automated personal assistant 1401 can be configured to control household appliances such as a washing machine, stove, oven, microwave oven, or dishwasher.

[0142] In this embodiment, the control system 1402 receives image (e.g., optical or acoustic) and annotation information from the sensor 1404. This information, along with a predetermined number of category and similarity metrics stored in the system, is used. The control system 1402 can be used in Figure 10The method described herein classifies each pixel of an image received from sensor 1404, for example, to segment an image of an appliance or other object for manipulation or operation. Based on this classification, signals can be sent to actuator 1406, for example, to control moving parts of automated personal assistant 1401 to interact with household appliances, or any actions performed by actuator 1406 as described in the preceding sections. Signals can also be sent to sensor 1404 based on this classification, for example, to focus or move a camera lens.

[0143] Figure 15 A schematic diagram of a control system 1502 configured to control a monitoring system 1500 is shown. The monitoring system 1500 can be configured to physically control entry through a door 252. A sensor 1504 can be configured to detect and determine whether entry is permitted in a relevant scene. The sensor 1504 can be an optical or acoustic sensor or sensor array configured to generate and transmit image and / or video data. This data can be used by the control system 1502 to detect a person's face.

[0144] The monitoring system 1500 can also be a supervisory system. In such an embodiment, the sensor 1504 may be an optical sensor configured to detect the supervised scene, and the control system 1502 is configured to control the display 1508. The control system 1502 is configured to determine the classification of the scene, such as whether the scene detected by the sensor 1504 is suspicious. Disturbance objects can be used to detect certain types of objects to allow the system to identify such objects under suboptimal conditions (e.g., night, fog, rain, background noise, etc.). The control system 1502 is configured to transmit actuator control commands to the display 1508 in response to the classification. The display 1508 can be configured to adjust the displayed content in response to the actuator control commands. For example, the display 1508 may highlight objects deemed suspicious by the controller 1502.

[0145] In this embodiment, the control system 1502 receives image (optical or acoustic) and annotation information from the sensor 1504. This information, along with a predetermined number of category and similarity metrics stored in the system, is used. The control system 1502 can be used in Figure 10 The method described herein classifies each pixel of an image received from sensor 1504 to, for example, detect the presence of suspicious or unwanted objects in a scene, detect the type of lighting or viewing conditions, or detect movement. Based on this classification, signals can be sent to actuator 1506, for example, to lock or unlock a door or other entrance passage, activate an alarm or other signal, or any action performed by actuator 1506 as described in the preceding sections. Signals can also be sent to sensor 1504 based on this classification, for example, to focus or move a camera lens.

[0146] Figure 16 A schematic diagram of a control system 1602 configured to control an imaging system 1600 (e.g., an MRI apparatus, an X-ray imaging apparatus, or an ultrasound apparatus) is shown. The sensor 1604 may be, for example, an imaging sensor or an array of acoustic sensors. The control system 1602 may be configured to determine the classification of all or a portion of the sensed image. The control system 1602 may be configured to determine or select actuator control commands in response to a classification obtained by a trained neural network. For example, the control system 1602 may interpret areas of the sensed image (optical or acoustic) as potential anomalies. In this case, actuator control commands may be determined or selected such that the display 1606 displays the image and highlights the potential anomalous area.

[0147] In this embodiment, the control system 1602 receives image and annotation information from the sensor 1604. It uses these information, along with a predetermined number of category and similarity metrics stored in the system. The control system 1602 can be used in Figure 10 The method described herein classifies each pixel of the image received from sensor 1604. Based on this classification, signals can be sent to actuator 1606, for example, to detect abnormal areas in the image or any action performed by actuator 1606 as described in the preceding sections.

[0148] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform one or more aspects of one or more embodiments. Inherently non-transitory computer-readable storage media can include tangible media that implements volatile and non-volatile, as well as removable and non-removable, properties for storing information such as computer-readable instructions, data structures, program modules, or other data, using any method or technique. Computer-readable storage media can further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable compressed optical disc read-only memory (CD-ROM), or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0149] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of art including instructions that implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, the functions, actions, and / or operations specified in a flowchart or diagram may be reordered, processed sequentially, and / or processed simultaneously, consistent with one or more embodiments. Furthermore, any of the flowcharts and / or diagrams may include more or fewer nodes or boxes than those illustrated consistent with one or more embodiments.

[0150] While the entire disclosure has been illustrated by description of various embodiments, and while these embodiments have been described in considerable detail, the applicant's intention is not to limit the scope of the appended claims or in any way to such detail. Additional advantages and modifications will readily apparent to those skilled in the art. Therefore, the disclosure is not, in its broader sense, limited to the specific details, representative apparatuses and methods, and the illustrative examples shown and described. Consequently, deviations from such details may be made without departing from the spirit or scope of the overall inventive concept.

Claims

1. A method for acquiring a calibrated image of a room, comprising: Through the controller: Request signals from sensors associated with the position and orientation of the mobile platform in the room, indicating the measurement results of parameters; Background noise associated with the mobile platform is removed from the signal, thereby focusing the measurement results on the foreground signal, wherein the background noise is removed from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; Store the measurement results, and the position and orientation of the mobile platform within the room; The mobile platform is requested to move to a second position within the room facing the second direction; In response to the mobile platform reaching the second position and the second direction, a second signal is requested from the sensor associated with the second position and the second direction of the mobile platform, indicating a second measurement result of the parameter; Background noise associated with the mobile platform at the second position and in the second direction is removed from the second measurement result, wherein the background noise is removed from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; Aggregating the foreground signals of interest associated with the position and orientation of the mobile platform and the second position and second orientation of the mobile platform within the room to create an energy map via spatial dynamic beamforming, wherein the aggregation of the foreground signals is performed by extracting background noise to provide a calibrated signal that is the foreground signal of interest, and removing the background noise from the foreground signal of interest via a subspace approximation using singular value decomposition to obtain a low-rank version of the signal; Analyze the energy map to identify the status of the devices in the room; as well as Output the energy diagram.

2. The method of claim 1, further comprising superimposing a foreground beamforming image with a temporally earlier foreground beamforming image to indicate a change in the foreground beamforming image with respect to time, wherein the foreground signal includes the foreground beamforming image.

3. The method of claim 2, further comprising outputting an alarm when the magnitude of the change exceeds a threshold.

4. The method of claim 1, wherein the parameter is sound and the sensor is a microphone.

5. The method of claim 1, wherein the parameter is electromagnetic energy, and the sensor is an RF receiver, a CCD, a photodiode, or an IR receiver.

6. The method according to claim 1, wherein the sensor is a sensor array.

7. A system for calibrating an image of a room, comprising: A mobile platform configured to move within the room; A sensor coupled to the mobile platform and configured to measure parameters within a region relative to the sensor and output a signal associated with the parameters within the region; as well as The controller is configured to: The measurement results of the parameters are requested from the sensors associated with the position and orientation of the mobile platform. Background noise associated with the mobile platform is removed from the measurement results, thereby focusing the measurement results on foreground noise. Store the measurement results, as well as the position and orientation of the mobile platform within the room. Move the mobile platform in the room to a new location. In response to the mobile platform reaching the new position, a second measurement result of the parameter is requested from the sensor associated with the new position and orientation of the mobile platform. Background noise associated with the mobile platform at the new location is removed from the second measurement result, thereby focusing the measurement result on the foreground noise at the new location. The signals from the sensors are aggregated along with the associated position and orientation of the mobile platform within the room to create an energy map via spatial dynamic beamforming. Analyze the energy map to identify the status of devices in the room, and Output the image formed by the foreground beam.

8. The system of claim 7, wherein the background noise is associated with the displacement motion component of the mobile platform transmitting noise detected by the sensor.

9. The system of claim 8, wherein the background noise is associated with a component of the noise detected by the sensor during the transmission of the mobile platform.

10. The system of claim 9, wherein the sensor is a sensor array.

11. The system of claim 10, wherein the controller is further configured to superimpose the foreground beamforming image with a temporally earlier foreground beamforming image to indicate a change in the foreground beamforming image with respect to time.

12. The system of claim 11, wherein the controller is further configured to output an alarm when the magnitude of the change exceeds a threshold.

13. The system of claim 12, wherein the controller aggregates the signal from the sensor by extracting the background noise via statistical properties of the aggregated signal, thereby providing the generated signal that is the foreground signal.

14. The system of claim 13, wherein the background noise is removed from the foreground noise via a subspace approximation using singular value decomposition to obtain a low-rank version of the energy map.

15. A mobile robot platform for acquiring calibration data, comprising: The transceiver within the mobile robot platform; A displacement motion unit, the displacement motion unit being configured to move the mobile robot platform within a region; A sensor, coupled to the displacement motion unit and configured to output a signal; and The controller is configured to: Measurement results of parameters requested from the sensors associated with the position and orientation of the mobile robot platform. Background noise associated with the mobile robot platform is removed from the measurement results, thereby focusing the measurement results on foreground noise. Store the measurement results, as well as the position and orientation of the mobile robot platform within the area. The request is made to the displacement motion unit to move the mobile robot platform within the area to a new position and orientation. In response to the mobile robot platform reaching the new position and orientation, a second measurement result of the parameter is requested from the sensor associated with the new position and orientation of the mobile robot platform. The background noise associated with the mobile robot platform at the new location is removed from the second measurement result, thereby focusing the measurement result on the foreground noise at the new location. The signals from the sensors are aggregated along with the associated position and orientation of the mobile robot platform within the region to create an energy map via spatial dynamic beamforming. The energy map is analyzed to identify the state of devices in the region.

16. The mobile robot platform of claim 15, wherein the controller provides a generated signal that is a foreground signal by extracting the background noise and aggregating the signal from the sensor via the statistical properties of the aggregated signal.

17. The mobile robot platform of claim 16, wherein the background noise is removed from the foreground noise via a subspace approximation using singular value decomposition to obtain a low-rank version of the energy map.

18. The mobile robot platform of claim 17, wherein the parameter is sound and the sensor is a microphone.

19. The mobile robot platform of claim 17, wherein the parameter is electromagnetic energy, and the sensor is an RF receiver, CCD, photodiode, IR receiver, or other EM sensor.

20. The mobile robot platform of claim 17, wherein the foreground signal is generated by... It represents that the matrix is ​​directed to... P Applying singular value decomposition, we can achieve the following: and , in, ,and , r is a matrix P rank, and These represent the background and foreground portions of the signal, respectively.

Images