Method and apparatus for generating a vector representation of a time-series sensor data stream.

Abstract

This invention provides a computer implementation method for generating a vector representation of a time-series sensor data stream. [Solution] The method involves applying a first machine learning algorithm 206 to multiple channels 202 of at least one time-series sensor data stream 200 to generate a vector representation of the time-series sensor data stream 200 including position embeddings 205 for each of multiple time sections 204 S4; associating at least one channel-specific and sensor-specific embedding 207 with multiple channels 202 of the time-series sensor data stream S5; and applying a second machine learning algorithm 208 to multiple channels 202 of the time-series sensor data stream to generate a vector representation 216 of combined embeddings of at least one channel-specific and sensor-specific embedding 207 for multiple channels 202 of the time-series sensor data stream S6.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for generating a vector representation of a time-series sensor data stream. The present invention further relates to a method for preprocessing a time-series sensor data stream for a method of generating a vector representation of the time-series sensor data stream. The present invention further relates to an apparatus for generating a vector representation of the time-series sensor data stream.

Background Art

[0002] Prior Art Recent progress in multimodal AI foundation models is highly anticipated. Such models include, for example, "ImageBind: One Embedding Space To Bind Them All" by Girdhar et al., which combines five data modalities in a single embedding space and enables new capabilities such as image creation from audio clips.

[0003] Another multimodal AI foundation model is "Meta-Transformer: A Unified Framework for Multimodal Learning" by Zhang et al., which proposes various model architectures for a modularized multimodal basic model.

[0004] Similarly, multiple sensor modalities, such as motion sensors (often also called Inertial Measurement Units or IMUs), audio sensors, and cardiac frequency sensors, also belong to the category of additional modalities that can be recorded in this multimodal basic model. Generally, introducing additional modalities requires a specific type of coding to encode the data of the specific modality here, as described in "ImageBind: One Embedding Space To Bind Them All" by Girdhar et al., where the architecture may or may not be modality-specific.

[0005] Coding IMU sensor modalities has traditionally been a challenging task, primarily due to several reasons. 1) Various types of sensors may exist, such as accelerometers, gyroscopes, magnetometers, etc. 2) The sensor can be mounted in various locations. The IMU sensor can be mounted on various parts of the body such as the head, hands, arms, legs, and back. If the sensor is mounted on a physical object such as a car, the sensor may also be located on the front, rear, or sides of the car, for example. 3) Sensor signals can have various frequencies. For example, an accelerometer can generate signals with different frequencies as a magnetometer. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Girdhar et al., “ImageBind: One Embedding Space To Bind Them All” [Non-Patent Document 2] Zhang et al., “Meta-Transformer: A Unified Framework for Multimodal Learning” [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Therefore, the object of the present invention is to develop a sensor encoder that can process such variable sensor inputs and generate a consistent display. [Means for solving the problem]

[0008] The above problems are solved by the method having the features of claim 1 and the method having the features of claim 6. The above problems are also solved by the apparatus having the features of claim 10.

[0009] Disclosure of the invention According to a first embodiment, a computer-implemented method for generating a vector representation of a time-series sensor data stream is proposed, comprising the steps of: supplying at least one time-series sensor data stream having a plurality of channels; dividing at least one time-series sensor data stream into a plurality of channels; dividing the time-series sensor data stream of each channel into a plurality of time sections; associating at least one positional embedding with each of the plurality of time sections; applying a first machine learning algorithm to the plurality of channels of at least one time-series sensor data stream to generate a vector representation of the at least one time-series sensor data stream including the positional embedding for each of the plurality of time sections; associating at least one channel-specific and sensor-specific embedding with the plurality of channels of at least one time-series sensor data stream; and applying a second machine learning algorithm to the plurality of channels of at least one time-series sensor data stream to generate a vector representation of a combined embedding of at least one channel-specific and sensor-specific embedding for each of the plurality of channels of the at least one time-series sensor data stream.

[0010] It is clear that the steps and further optional steps according to the present invention do not necessarily have to be performed in the order presented and may be performed in other orders. Furthermore, additional intermediate steps may be provided. Each individual step may further include one or more substeps, thereby without departing from the scope of the methods according to the present invention.

[0011] To handle variability in sensor data, such as differences in dimensions or sampling rates, the method for generating a vector representation of a time-series sensor data stream according to the present invention is used as a sensor encoder in a sensor foundation model.

[0012] This converter-based model processes sensor data by splitting it into time-overlapping 1D signals. This process is called patching. Each of these 1D patches or time sections is associated with a positional embedding and input to a common converter.

[0013] Before coding, all data is preprocessed. The dataset is standardized into a canonical format in which each sample is characterized by a consistent sampling rate and temporal overlap between samples. Activity classification represents a practical application of the sensor foundation model. The model performs activity classification using data from portable devices.

[0014] According to a second aspect, a method is proposed for preprocessing a time-series sensor data stream for a method for generating a vector representation of a time-series sensor data stream according to a first aspect of the present invention, the method comprising: supplying a plurality of time-series sensor data streams, wherein each sensor supplying the time-series sensor data streams has a different frequency; and resampling the plurality of time-series sensor data streams to a uniform frequency, particularly by interpolation, or dividing the plurality of time-series sensor data streams into a plurality of time sections such that one of the time-series sensor data streams has a uniform frequency.

[0015] According to a third embodiment, a device for generating a vector representation of a time-series sensor data stream is proposed, comprising: at least one sensor configured to supply at least one time-series sensor data stream having a plurality of channels; means for dividing the at least one time-series sensor data stream into a plurality of channels; means for dividing the at least one time-series sensor data stream of each channel into a plurality of time sections; means for associating at least one position embedding with each of the plurality of time sections; means for applying a first machine learning algorithm to the plurality of channels of the at least one time-series sensor data stream in order to generate a vector representation of the time-series sensor data stream including position embeddings for each of the plurality of time sections; means for associating at least one channel-specific and sensor-specific embedding with the plurality of channels of the time-series sensor data stream; and means for applying a second machine learning algorithm to the plurality of channels of the time-series sensor data stream in order to generate a vector representation of a combined embedding of at least one channel-specific and sensor-specific embedding of the plurality of channels of the time-series sensor data stream.

[0016] The descriptions given for the method also apply to the apparatus to a corresponding extent. It should be understood that linguistic changes to the features expressed in the method can be transformed into actual linguistic expressions for the apparatus without the need to explicitly include such expressions in this specification.

[0017] Compared to previous studies, this method and apparatus offer the following advantages. In particular, this method can address the challenges arising from coding time-series data using a changing number of channels and changing channel frequencies.

[0018] In a further embodiment, it is proposed that at least one positional embedding for each of the multiple time sections indicates the position in a sequence of multiple time sections. This advantageously allows for improved representation of the input data.

[0019] In yet another aspect, it is proposed that at least one channel-specific and sensor-specific embedding for a plurality of channels of a time series sensor data stream includes a channel name, a position of a sensor supplying the time series sensor data stream in an object or a person, and / or a sensor type. Thereby, further improvement of the representation of the input data becomes possible.

[0020] In yet another aspect, it is proposed that the time series sensor data stream is supplied by an inertial measurement unit, where each channel includes a univariate time series sensor data stream, particularly acceleration or angular velocity in the x-direction, y-direction or z-direction, and the time series sensor data streams of the plurality of channels overlap in time. Thus, these univariate sensor data can be processed separately by subsequent algorithms of machine learning.

[0021] In yet another aspect, it is proposed that a first algorithm of machine learning and a second algorithm of machine learning are each constituted by a transducer encoder model. This has been found to be particularly advantageous for coding sensor data.

[0022] In yet another aspect, a computer program is proposed that includes program code for implementing at least a part of the method in one of its aspects when executed on a computer. In other words, a computer program is proposed that includes instructions for causing the computer to implement the method or steps of the method in one of its aspects when executed by the computer.

[0023] In a further aspect, there is proposed a computer-readable data carrier including program code of a computer program for implementing at least a part of the method in one of those aspects when executed on a computer. In other words, the present invention relates to a computer-readable medium including instructions for causing the computer to implement the method or steps of the method in one of those aspects when executed by the computer.

[0024] The described configurations and developments can be arbitrarily combined with each other.

[0025] Further possible configurations, developments and implementations of the present invention also include implicit combinations of features of the present invention described in accordance with the examples above or below.

[0026] The accompanying drawings are for further understanding of embodiments of the present invention. These drawings show embodiments and are used in connection with the description of the basic modes and concepts of the present invention.

[0027] Many other embodiments and the above-described advantages are apparent from the drawings. The elements shown are not necessarily shown to scale with each other.

Brief Description of the Drawings

[0028] [Figure 1] It is a schematic flowchart showing an embodiment of the method for generating a vector representation of a time-series sensor data stream. [Figure 2] It is a schematic flowchart showing an embodiment of the method for preprocessing a time-series sensor data stream for a method of generating a vector representation of a time-series sensor data stream. [Figure 3] It is a schematic block diagram showing an embodiment of the apparatus for generating a vector representation of a time-series sensor data stream.

[0029] In each drawing, identical or functionally equivalent elements, components, or parts are denoted by the same reference numeral unless otherwise specified. [Modes for carrying out the invention]

[0030] This method will be explained below with reference to the respective diagrams.

[0031] Figure 1 shows a schematic flowchart of one embodiment of this method for generating a vector representation of a time-series sensor data stream.

[0032] The method includes S1 supplying at least one time-series sensor data stream 200 having multiple channels 202, S2a dividing at least one time-series sensor data stream 200 into multiple channels 202, S2b dividing the time-series sensor data stream 200 of each channel 202 into multiple time sections 204, and S3 associating at least one position embedding 205 with each of the multiple time sections 204.

[0033] Furthermore, the method includes S4 applying a first machine learning algorithm 206 to multiple channels 202 of at least one time-series sensor data stream 200 to generate a vector representation of the time-series sensor data stream including position embeddings 205 for each of multiple time sections 204; S5 associating at least one channel-specific and sensor-specific embedding 207 with the multiple channels 202 of the time-series sensor data stream; and S6 applying a second machine learning algorithm 208 to the multiple channels 202 of the time-series data stream to generate a vector representation 216 of combined embeddings of at least one channel-specific and sensor-specific embedding 207 for the multiple channels 202 of the time-series sensor data stream.

[0034] At least one position embedding 205 indicates the position within a sequence of time sections 204 for each of the multiple time sections 204.

[0035] At least one channel-specific and sensor-specific embedding 207 for multiple channels 202 of the time-series sensor data stream includes the channel name, the location of the sensor supplying the time-series sensor data stream 200 in an object or person, and / or the sensor type.

[0036] The time-series sensor data stream 200 is supplied from the inertial measurement unit 210, where each channel contains a single-degree-of-freedom time-series sensor data stream 200, specifically acceleration or angular velocity in the x, y, or z direction, and the time-series sensor data streams of multiple channels 202 overlap in time.

[0037] The first machine learning algorithm 206 and the second machine learning algorithm 208 are each composed of a converter-encoder model.

[0038] Figure 2 shows a schematic flowchart of one embodiment of the present method for preprocessing a time-series sensor data stream to generate a vector representation of the time-series sensor data stream.

[0039] The method includes supplying a plurality of time-series sensor data streams S1', wherein each sensor supplying the time-series sensor data stream 200 has a different frequency S1', and in particular resampling the plurality of time-series sensor data streams to a uniform frequency S2a', or dividing the plurality of time-series sensor data streams into a plurality of time sections S2b', such that one of the plurality of time-series sensor data streams has a uniform frequency.

[0040] The division of multiple time-series sensor data streams into multiple time sections S2b' is performed using defined time windows and embeddings in each time section, specifically using layers of a convolutional neural network and subsequent global pooling layers in the time dimension, so that identical dimensions are formed.

[0041] Reference numeral 212 denotes each divided time section. Reference numeral 214 denotes the method described in Figure 1, and reference numeral 216 denotes the output of the method in Figure 1, i.e., the vector representation of the time-series sensor data stream.

[0042] Figure 3 shows a schematic block diagram of one embodiment of the apparatus 500 for generating a vector representation of a time-series sensor data stream. The apparatus includes at least one sensor 502 configured to supply at least one time-series sensor data stream 200 having multiple channels 202.

[0043] Furthermore, the device includes means 504 for dividing at least one time-series sensor data stream 200 into a plurality of channels 202 in S2a, means 506 for dividing the time-series sensor data stream 200 of each channel 202 into a plurality of time sections 204 in S2b, means 508 for associating at least one position embedding 205 with each of the plurality of time sections 204 in S3, and means 510 for applying a first machine learning algorithm 206 to the plurality of channels 202 of at least one time-series sensor data stream 200 in S4 to generate a vector representation of the time-series sensor data stream including the position embedding 205 for each of the plurality of time sections 204.

[0044] Furthermore, the device includes means 512 for associating at least one channel-specific and sensor-specific embedding 207 with a plurality of channels 202 of a time-series sensor data stream in S5, and means 514 for applying a second machine learning algorithm 208 to the plurality of channels 202 of the time-series sensor data stream in order to generate a vector representation 216 of the combined embeddings of at least one channel-specific and sensor-specific embedding 207 for the plurality of channels 202 of the time-series sensor data stream.

Claims

1. A computer-implemented method for generating a vector representation of a time-series sensor data stream, Step (S1) of supplying at least one time-series sensor data stream (200) having multiple channels (202), The steps include: dividing the at least one time-series sensor data stream (200) into a plurality of channels (202) (S2a), and dividing the time-series sensor data stream (200) of each channel (202) into a plurality of time sections (204) (S2b), Step (S3) of associating at least one positional embedding (205) with each of a plurality of time sections (204), Step (S4) of applying a first machine learning algorithm (206) to a plurality of channels (202) of the at least one time-series sensor data stream (200) in order to generate a vector representation of the at least one time-series sensor data stream (200) including position embeddings (205) for each of the plurality of time sections (204), Step (S5) of associating at least one channel-specific and sensor-specific embedding (207) with multiple channels (202) of the at least one time-series sensor data stream (200), Step (S6) to apply a second machine learning algorithm (208) to the multiple channels (202) of the at least one time-series sensor data stream (200) in order to generate a vector representation (216) of the combined embeddings (207) of the multiple channels (202) of the at least one time-series sensor data stream (200), Computerized methods, including those mentioned above.

2. The computer-implemented method according to claim 1, wherein the at least one position embedding (205) indicates a position in a sequence of the plurality of time sections (204) for each of the plurality of time sections (204).

3. The computer-implemented method according to claim 1 or 2, wherein at least one channel-specific and sensor-specific embedding (207) for a plurality of channels (202) of the at least one time-series sensor data stream (200) includes a channel name, the location of the sensor supplying the time-series sensor data stream (200) in an object or person, and / or the type of sensor.

4. The computer-implemented method according to any one of claims 1 to 3, wherein the at least one time-series sensor data stream (200) is supplied from an inertial measurement unit (210), and each channel includes a single-degree-of-freedom time-series sensor data stream (200), particularly acceleration or angular velocity in the x, y, or z direction, and the time-series sensor data streams of the plurality of channels (202) overlap in time.

5. The computer-implemented method according to any one of claims 1 to 4, wherein the first machine learning algorithm (206) and the second machine learning algorithm (208) are each composed of a converter-encoder model.

6. A method for preprocessing a time-series sensor data stream for a method of generating a vector representation of a time-series sensor data stream according to any one of claims 1 to 5, Step (S1') of supplying multiple time-series sensor data streams (200), wherein each sensor supplying the time-series sensor data streams (200) has a different frequency; In particular, the steps include: resampling the plurality of time-series sensor data streams at a uniform frequency by interpolation (S2a'), or dividing the plurality of time-series sensor data streams into a plurality of time sections such that one of the plurality of time-series sensor data streams has a uniform frequency (S2b'), A method that includes this.

7. The step (S2b') of dividing the plurality of time-series sensor data streams into plurality of time sections is performed using a defined time window and embeddings in each time section, in particular using layers of a convolutional neural network and subsequent global pooling layers in the time dimension, so that identical dimensions are formed. The computer-implemented method according to claim 6.

8. A computer program, wherein the computer program, when executed on a computer, includes program code for carrying out at least a part of the method according to any one of claims 1 to 5.

9. A computer-readable data carrier comprising program code for a computer program to implement at least a part of the method described in any one of claims 1 to 5 when executed on a computer.

10. A device (500) for generating a vector representation of a time-series sensor data stream, A system comprising at least one sensor (502) configured to supply at least one time-series sensor data stream (200) having multiple channels (202), Means (504) for dividing the at least one time-series sensor data stream (200) into a plurality of channels (202) (S2a), and means (506) for dividing the at least one time-series sensor data stream (200) of each channel (202) into a plurality of time sections (204) (S2b), Means (508) for associating at least one position embedding (205) with each of the plurality of time sections (204) (S3), Means (510) for applying a first machine learning algorithm (206) to multiple channels (202) of at least one time series sensor data stream (200) in order to generate a vector representation of the time series sensor data stream including position embeddings (205) for each of the multiple time sections (204), Means (512) for associating at least one channel-specific and sensor-specific embedding (207) with a plurality of channels (202) of the time-series sensor data stream (S5), Means (514) for applying a second machine learning algorithm (208) to the multiple channels (202) of the time-series sensor data stream in order to generate a vector representation (216) of the combined embeddings (207) of at least one channel-specific and sensor-specific embeddings (207) of the multiple channels (202) of the time-series sensor data stream, A device (500) equipped with the following.

Images