Method and apparatus for generating a vector representation of time series sensor data streams
By employing positional and channel-specific embeddings with transformer-based encoding, the method addresses the challenge of encoding diverse IMU sensor data, achieving improved representation and classification accuracy.
Patent Information
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2026-03-26
AI Technical Summary
Existing multimodal AI foundation models face challenges in encoding sensor data from inertial measurement units (IMUs) due to variability in sensor types, locations, and signal frequencies, which hinders consistent representation generation.
A method involving positional and channel-specific embeddings, combined with transformer-based encoding, is applied to process time-series sensor data streams, standardizing them into a canonical format and using machine learning algorithms to generate consistent vector representations.
This approach effectively handles sensor data variability, enabling improved activity classification and consistent representation generation across diverse sensor configurations.
Smart Images
Figure 00000007_0000 
Figure 00000007_0001 
Figure 00000008_0000
Abstract
Description
[0001] The invention relates to a computer-implemented method for generating a vector representation of time-series sensor data streams. The invention further relates to a method for preprocessing time-series sensor data streams for a method for generating a vector representation of time-series sensor data streams. The invention further relates to a device for generating a vector representation of time-series sensor data streams. State of the art
[0002] Recent advances in multimodal AI Foundation models are promising. One such model is “ImageBind: One Embedding Space To Bind Them All” (Girdhar et al.), which combines five data modalities in a single embedding space and enables new capabilities, such as creating images from an audio clip.
[0003] Another multimodal AI foundation model is “Meta-Transformer: A Unified Framework for Multimodal Learning, Zhang et al.”, which proposes various model architectures for modularized multimodal foundation models.
[0004] Sensor modalities such as motion sensors (often referred to as inertial measurement units or IMUs), audio and heart rate sensors, etc., are also among the additional modalities that can be incorporated into such multimodal base models. Generally, incorporating additional modalities requires specific types of encoders to encode the data of that specific modality, as described in "ImageBind: One Embedding Space To Bind Them All," Girdhar et al., with the architecture potentially being modality-specific or not.
[0005] The encoding of the IMU sensor modality has traditionally presented a challenge, mainly due to several reasons: 1) There may be different types of sensors, such as an accelerometer, a gyroscope, a magnetometer, etc. 2) The sensors can be located in various places. IMU sensors can be attached to different parts of the body, such as the head, hand, arm, leg, back, etc. If the sensors are attached to physical objects, such as a motor vehicle, they can also be located, for example, at the front, rear, or side of the vehicle. 3) Sensor signals can have different frequencies. For example, accelerometers can generate signals with different frequencies than magnetometers.
[0006] The object of the invention is therefore to develop a sensor encoder which can process sensor inputs with such variability and generate consistent representations.
[0007] The problem is solved by a method according to the features of claim 1 and a method according to the features of claim 6. The problem is further solved by a device according to the features of claim 10. Disclosure of the invention
[0008] According to a first aspect, a computer-implemented method for generating a vector representation of time series sensor data streams is proposed, with the following steps: Providing at least one time-series sensor data stream having a plurality of channels, dividing the at least one time-series sensor data stream into a plurality of channels, and dividing the time-series sensor data stream of each channel into a plurality of time segments; Assigning at least one positional embedding to each of the plurality of time intervals, applying a first machine learning algorithm to the plurality of channels of the at least one time-series sensor data stream to generate a vector representation of the at least one time-series sensor data stream comprising the positional embedding to each of the plurality of time intervals; Assigning at least one channel- and sensor-specific embedding to the plurality of channels of the at least one time-series sensor data stream; and applying a second machine learning algorithm to the plurality of channels of the at least one time-series sensor data stream to generate a vector representation of a combined embedding of the at least one channel- and sensor-specific embedding of the plurality of channels of the at least one time-series sensor data stream.
[0009] It is understood that the steps according to the invention, as well as further optional steps, do not necessarily have to be carried out in the sequence shown, but can also be carried out in a different sequence. Furthermore, additional intermediate steps may be provided. The individual steps may also comprise one or more sub-steps without thereby departing from the scope of the method according to the invention.
[0010] To cope with the variability of the sensor data, such as differences in dimensions, sampling rates, etc., the inventive method is used to generate a vector representation of time series sensor data streams and is employed as a sensor encoder in a sensor foundation model.
[0011] This transformer-based model processes sensor data by splitting it into 1D signals that overlap in time. This process is called patching. Positional embeddings are assigned to each of these 1D patches, or time segments, and fed into a common transformer.
[0012] Before encoding, all data is preprocessed. The datasets are standardized into a canonical format, where each sample is characterized by a consistent sampling rate and temporal overlap between samples. Activity classification is a practical application of the Sensor Foundation model. Using data from wearable devices, the model performs activity classification.
[0013] According to a second aspect, a method for preprocessing time series sensor data streams is proposed for the inventive method for generating a vector representation of time series sensor data streams according to the first aspect, comprising the steps: Providing a plurality of time-series sensor data streams, wherein each sensor providing a time-series sensor data stream has a different frequency; and Resample the majority of time series sensor data streams, in particular by interpolation, to a uniform frequency or divide the majority of time series sensor data streams into time segments such that the majority of time series sensor data streams have a uniform frequency.
[0014] According to a third aspect, a device for generating a vector representation of time-series sensor data streams is proposed, comprising: at least one sensor which is configured to provide at least one time series sensor data stream having multiple channels; Means for splitting the at least one time series sensor data stream into the plurality of channels and means for splitting the time series sensor data stream of each channel into a plurality of time segments; Means of assigning at least one positional embedding to each of the plurality of temporal periods; Means for applying a first machine learning algorithm to the plurality of channels of the at least one time-series sensor data stream to generate a vector representation of the time-series sensor data streams comprising the positional embedding to each of the plurality of time segments; means for assigning at least one channel- and sensor-specific embedding to the plurality of channels of the time-series sensor data streams; and Means of applying a second machine learning algorithm to the plurality of channels of the time series sensor data streams to generate a vector representation of a combined embedding of the plurality of channels of the time series sensor data streams, which includes at least one channel- and sensor-specific embedding.
[0015] The statements made regarding the procedure apply accordingly to the device. It is understood that linguistic modifications of procedurally formulated features can be reformulated for the device according to common linguistic practice, without such formulations needing to be explicitly listed here.
[0016] Compared to previous work, the present method and apparatus offer the following advantages. In particular, the present method is able to overcome the challenges arising from the encoding of time-series data with varying channel numbers and frequencies.
[0017] In a further aspect, it is proposed that at least one positional embedding for each of the plurality of time segments specifies a position in a sequence of the plurality of time segments. This advantageously enables an improved representation of the input data.
[0018] In a further aspect, it is proposed that at least one channel- and sensor-specific embedding for the majority of channels of the time-series sensor data streams includes a channel label, the position of a sensor providing the time-series sensor data stream on an object or person, and / or a sensor type. This further enables an improved representation of the input data.
[0019] In another aspect, it is proposed that the time-series sensor data stream be provided by an inertial measurement unit, with each channel comprising a univariate time-series sensor data stream, in particular an acceleration in an x, y, or z direction or an angular velocity, and with the time-series sensor data streams of the majority of channels overlapping in time. The univariate sensor data can thus each be processed separately by the subsequent machine learning algorithm.
[0020] In another aspect, it is proposed that the first and second machine learning algorithms are each formed by a transformer-encoder model. This proves to be particularly advantageous for encoding sensor data.
[0021] In another aspect, a computer program is claimed to contain program code capable of executing at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, a computer program is claimed to contain instructions that, when executed by a computer, cause it to execute the method(s) in one of its aspects.
[0022] In a further aspect, a computer-readable data carrier containing the program code of a computer program is proposed to execute at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, the invention relates to a computer-readable medium comprising instructions which, when executed by a computer, cause it to execute the method / steps of the method in one of its aspects.
[0023] The described configurations and training programs can be combined in any way desired.
[0024] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or subsequently with regard to the exemplary embodiments that are not explicitly mentioned. Brief description of the drawings
[0025] The accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain the principles and concepts of the invention.
[0026] Other embodiments and many of the aforementioned advantages become apparent with reference to the drawings. The elements depicted in the drawings are not necessarily shown to scale. Fig. Figure 1 shows a schematic flowchart of an embodiment of the present method for generating a vector representation of time series sensor data streams. Fig. Figure 2 shows a schematic flowchart of an embodiment of the present method for preprocessing time series sensor data streams for a method for generating a vector representation of time series sensor data streams. Fig. Figure 3 shows a schematic block diagram of an embodiment of a device for generating a vector representation of time series sensor data streams.
[0027] In the figures of the drawings, identical reference symbols denote identical or functionally equivalent elements, parts or components, unless otherwise stated.
[0028] The present procedure will also be explained with reference to the other figures.
[0029] Fig. Figure 1 shows a schematic flowchart of an embodiment of the present method for generating a vector representation of time series sensor data streams.
[0030] The method comprises providing S1 at least one time-series sensor data stream 200 having a plurality of channels 202, splitting S2a the at least one time-series sensor data stream 200 into the plurality of channels 202, and splitting S2b the time-series sensor data stream 200 of each channel 202 into a plurality of time segments 204, as well as assigning S3 at least one position embedding 205 to each of the plurality of time segments 204.
[0031] Furthermore, the method comprises applying S4 a first machine learning algorithm 206 to the plurality of channels 202 of the at least one time series sensor data stream 200 to generate a vector representation of the time series sensor data streams comprising the position embedding 205 to each of the plurality of time segments 204, assigning S5 at least one channel- and sensor-specific embedding 207 to the plurality of channels 202 of the time series sensor data streams, and applying S6 a second machine learning algorithm 208 to the plurality of channels 202 of the time series sensor data streams to generate a vector representation of a combined embedding 216 of the at least one channel- and sensor-specific embedding 207 of the plurality of channels 202 of the time series sensor data streams.
[0032] The at least one positional embedding 205 specifies for each of the plurality of temporal sections 204 a position in a sequence of the plurality of temporal sections 204.
[0033] The at least one channel- and sensor-specific embedding 207 to the majority of channels 202 of the time series sensor data streams includes a channel designation, a position of a sensor providing the time series sensor data stream 200 on an object or person and / or a sensor type.
[0034] The time series sensor data stream 200 is provided by an inertial measurement unit 210, wherein each channel comprises a univariate time series sensor data stream 200, in particular an acceleration in an x, y or z direction or an angular velocity, and wherein the time series sensor data streams of the plurality of channels 202 overlap in time.
[0035] The first machine learning algorithm 206 and the second machine learning algorithm 208 are each formed by a transformer-encoder model.
[0036] Fig. Figure 2 shows a schematic flowchart of an embodiment of the present method for preprocessing time series sensor data streams for a method for generating a vector representation of time series sensor data streams.
[0037] The method comprises providing S1' a plurality of time series sensor data streams, wherein each sensor providing a time series sensor data stream 200 has a different frequency, and resample S2a' the plurality of time series sensor data streams, in particular by interpolation, to a uniform frequency or divide S2b' the plurality of time series sensor data streams into time segments, such that the plurality of time series sensor data streams have a uniform frequency.
[0038] The division S2b' of the plurality of time series sensor data streams into time segments using a defined time window and embedding each time segment in such a way as to generate the same dimension, in particular using a layer of a neural convolutional network followed by a global pooling layer in the time dimension.
[0039] Reference numeral 212 designates the respective divided time periods. Reference numeral 214 designates the [missing information]. Fig. 1 described procedure and reference numeral 216 designates the output of the procedure according to Fig. 1, i.e., the vector representation of the time series sensor data streams.
[0040] Fig. Figure 3 shows a schematic block diagram of an embodiment of the present device 500 for generating a vector representation of time-series sensor data streams. The device comprises at least one sensor 502, which is configured to provide at least one time-series sensor data stream 200 having a plurality of channels 202.
[0041] Furthermore, the device comprises means 504 for splitting S2a of the at least one time-series sensor data stream 200 into the plurality of channels 202 and means 506 for splitting S2b of the time-series sensor data stream 200 of each channel 202 into a plurality of time segments 204, means 508 for assigning S3 of at least one position embedding 205 to each of the plurality of time segments 204, and means 510 for applying S4 of a first machine learning algorithm 206 to the plurality of channels 202 of the at least one time-series sensor data stream 200 to generate a vector representation of the time-series sensor data streams comprising the position embedding 205 to each of the plurality of time segments 204.
[0042] Furthermore, the device comprises means 512 for assigning S5 to at least one channel- and sensor-specific embedding 207 to the plurality of channels 202 of the time-series sensor data streams, and means 514 for applying a second machine learning algorithm 208 to the plurality of channels 202 of the time-series sensor data streams to generate a vector representation 216 of a combined embedding of the at least one channel- and sensor-specific embedding 207 of the plurality of channels 202 of the time-series sensor data streams.
Claims
[1] Computer-implemented method for generating a vector representation of time-series sensor data streams, comprising the steps: Providing (S1) at least one time series sensor data stream (200) having a plurality of channels (202); Splitting (S2a) the at least one time series sensor data stream (200) into the plurality of channels (202) and splitting (S2b) the time series sensor data stream (200) of each channel (202) into a plurality of time segments (204); Assign (S3) at least one positional embedding (205) to each of the plurality of temporal periods (204); Applying (S4) a first machine learning algorithm (206) to the plurality of channels (202) of the at least one time series sensor data stream (200) to generate a vector representation of the at least one time series sensor data stream (200) comprising the position embedding (205) to each of the plurality of time intervals (204); Assigning (S5) at least one channel- and sensor-specific embedding (207) to the plurality of channels (202) of the at least one time-series sensor data stream (200); and Applying (S6) a second machine learning algorithm (208) to the plurality of channels (202) of the at least one time series sensor data stream (200) to generate a vector representation (216) of a combined embedding of the at least one channel- and sensor-specific embedding (207) of the plurality of channels (202) of the at least one time series sensor data stream (200). [2] Computer-implemented method according to claim 1, wherein the at least one position embedding (205) to each of the plurality of time intervals (204) specifies a position in a sequence of the plurality of time intervals (204). [3] Computer-implemented method according to claim 1 or 2, wherein the at least one channel- and sensor-specific embedding (207) to the plurality of channels (202) of the at least one time-series sensor data stream (200) comprises a channel designation, a position of a sensor providing the time-series sensor data stream (200) on an object or person and / or a sensor type. [4] Computer-implemented method according to one of the preceding claims, wherein the at least one time-series sensor data stream (200) is provided by an inertial measuring unit (210), wherein each channel comprises a univariate time-series sensor data stream (200), in particular an acceleration in an x, y or z direction or an angular velocity, and wherein the time-series sensor data streams of the plurality of channels (202) overlap in time. [5] Computer-implemented method according to any of the preceding claims, wherein the first machine learning algorithm (206) and the second machine learning algorithm (208) are each implemented by a transformer-encoder model. [6] Method for preprocessing time series sensor data streams for a method for generating a vector representation of time series sensor data streams according to any one of claims 1 to 5, comprising the steps: Providing (S1') a plurality of time-series sensor data streams (200), wherein each sensor providing a time-series sensor data stream (200) has a different frequency; and Resample (S2a') the majority of time series sensor data streams, in particular by interpolation, to a uniform frequency or divide (S2b') the majority of time series sensor data streams into time segments such that the majority of time series sensor data streams have a uniform frequency. [7] Computer-implemented method according to claim 6, wherein the splitting (S2b') of the plurality of time series sensor data streams into time segments is carried out using a defined time window and embedding each time segment in such a way as to generate the same dimension, in particular using a layer of a neural convolutional network followed by a global pooling layer in the time dimension. [8] Computer program with program code to execute at least parts of a method according to any one of claims 1 to 5 when the computer program is executed on a computer. [9] Computer-readable data carrier containing program code of a computer program for executing at least parts of a method according to any one of claims 1 to 5 when the computer program is executed on a computer. [10] Device (500) for generating a vector representation of time series sensor data streams, comprising: at least one sensor (502) which is configured to provide at least one time series sensor data stream (200) having a plurality of channels (202); Means (504) for splitting (S2a) the at least one time series sensor data stream (200) into the plurality of channels (202) and means (506) for splitting (S2b) the at least one time series sensor data stream (200) of each channel (202) into a plurality of time segments (204); Means (508) for assigning (S3) at least one positional embedding (205) to each of the plurality of temporal periods (204); Means (510) for applying (S4) a first machine learning algorithm (206) to the plurality of channels (202) of the at least one time series sensor data stream (200) to generate a vector representation of the time series sensor data streams comprising the position embedding (205) to each of the plurality of time intervals (204); Means (512) for assigning (S5) at least one channel- and sensor-specific embedding (207) to the majority of channels (202) of the time-series sensor data streams; and Means (514) for applying a second machine learning algorithm (208) to the plurality of channels (202) of the time series sensor data streams to generate a vector representation (216) of a combined embedding of the at least one channel and sensor-specific embedding (207) of the majority of channels (202) of the time series sensor data streams.