Sensor data feature extraction method based on self-supervised learning

Abstract

The invention discloses a sensor data feature extraction method based on self-supervised learning. The method is suitable for various sensor data including but not limited to IMU, temperature, pressure, vision and the like. The method comprises the following steps: firstly, acquiring data from different sensors, and preprocessing the data; then, mask processing is carried out on the preprocessed sensor data; then, constructing a task model; the task model comprises a data compression module, an encoder and a decoder. The data compression module performs compression and dimension raising on input sensor data, the encoder is used for extracting high-dimensional features of the sensor data, and the decoder decodes the high-dimensional features; and finally, inputting the sensor data subjected to mask processing into a task model, and outputting reconstructed data by the model. The reconstructed data is compared with the preprocessed data, training loss is calculated, a task model is trained through self-supervised learning, and the high-dimensional features of the sensor data are extracted through a trained encoder. According to the method, characterization learning of the intrinsic and basic features of the sensor data is enhanced through the mask, coding and reconstruction data, the feature extraction precision is improved, the dependence on the marked data is reduced, and the method is suitable for data processing tasks of various sensors.

Description

Technical Field

[0001] The present invention belongs to the field of sensor data processing technology, and specifically relates to a sensor data feature extraction method based on self-supervised learning. The method is applicable to various types of sensor data processing, including but not limited to inertial measurement units, temperature sensors, pressure sensors, visual sensors, optical sensors, etc. Background Art

[0002] Sensor data plays a vital role in modern automation systems, especially in the fields of unmanned driving, industrial automation, health monitoring, etc. The real-time data provided by sensors is the key to supporting system decision-making and operation. In order to extract effective features from these sensor data and process them effectively, it is usually necessary to preprocess the data and extract features to provide accurate information for subsequent tasks (such as positioning, environmental perception, status monitoring, etc.).

[0003] Adaptive feature extraction methods are of great significance in sensor data processing. Traditional feature extraction methods, such as linear transformation and nonlinear fitting, are mainly suitable for low-dimensional and simple sensor data processing, but when processing high-dimensional and complex multi-sensor data, they often face difficulties in capturing subtle changes and complex features. Deep learning-based methods can automatically extract potential effective features through inference models, achieving significant performance improvements. However, most existing deep learning methods rely on supervised learning and require a large amount of labeled data for model training, which not only increases the burden of data annotation, but also limits the generalization ability of the model because the annotation information required for different tasks and scenarios is different.

[0004] In addition, although traditional deep learning models, such as convolutional neural networks (CNN) and recurrent neural networks (RNN), have been successful in processing image and time series data, they are often unable to fully explore and utilize the detailed features in the data due to the limitations of the network structure when processing complex, multi-dimensional sensor data, resulting in poor performance in some practical applications.

[0005] In view of the above problems, the present invention proposes a sensor data feature extraction method based on self-supervised learning. Through self-supervised learning, the method can automatically learn and extract features from data of different sensors without supervised labeled data, thereby improving the generalization ability and accuracy of the model, and is particularly suitable for the fusion and processing of multiple types of sensor data. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention aims to provide a sensor data feature extraction method based on self-supervised learning, which can effectively process and extract data features from various types of sensors (such as IMU, visual sensors, temperature sensors, pressure sensors, optical sensors, etc.), thereby improving the system's environmental perception ability, positioning accuracy and state monitoring accuracy.

[0007] The present invention solves the technical problem by adopting the following technical solution:

[0008] A sensor data feature extraction method based on self-supervised learning, characterized in that the method comprises the following steps:

[0009] Step S1: Acquire data from at least one sensor; perform appropriate preprocessing on the sensor data according to the types of different sensors (such as IMU, visual sensor, temperature sensor, pressure sensor, etc.); the preprocessing steps may include data denoising, time alignment, spatial calibration, normalization, etc.

[0010] Step S2: Mask the preprocessed sensor data. The purpose of masking is to allow the model to learn how to restore the missing parts when part of the data is missing through self-supervised learning, so as to extract effective feature information. The specific steps are as follows:

[0011] S2.1: For any sensor data sequence, generate a random number θ∈[0,1). If the random number θ is greater than or equal to the mask setting probability, no masking is performed; if the random number θ is less than the mask setting probability, multiple data sequences are randomly selected from the sensor data sequence group;

[0012] S2.2: For any selected sensor data sequence, generate a random number s∈[0, L), and use the position corresponding to the random number s in the data sequence as the starting point of the mask processing; according to the predetermined mask length (a random number l is selected from the Gaussian distribution that satisfies the probability of successful mask setting as the mask length, and the cutoff point of the mask processing is e=s+l), set the data at each position from the starting point to the cutoff point in the data sequence to zero;

[0013] Where L represents the length of the sensor data sequence; the mask length l is less than or equal to the remaining length Ls of the sensor data sequence, and is also less than or equal to the remaining number of masks M max -m,M max Indicates the maximum number of masks, and m indicates the number of masks that have been set;

[0014] S2.3: Traverse all selected data sequences, if the number of masks m is equal to the maximum number of masks M max , then stop mask processing; if the number of masks set m is less than the maximum number of masks Mmax , then repeat steps S2.1 and S2.2 until the number of masks set m equals the maximum number of masks M max .

[0015] Step S3: construct a task model; the task model includes a data compression module, an encoder and a decoder, which are used to learn the feature representation of sensor data: the data compression module compresses and increases the dimension of the input sensor data to extract higher-dimensional features; the encoder is used to extract high-dimensional features from the compressed data and use these features for subsequent tasks; the decoder is used to decode the high-dimensional features into reconstructed sensor data or classification results, depending on the requirements of the task;

[0016] Step S4: Input the masked sensor data into the task model, and the model outputs the reconstructed sensor data; compare the reconstructed sensor data with the preprocessed original sensor data, calculate the training loss using the masked data, and train the task model using self-supervised learning to obtain the trained task model. The encoder of the trained task model is used to extract high-dimensional features of the sensor data.

[0017] Furthermore, the encoder includes a first projection layer, a position encoding layer and an encoding module connected in series, and layer normalization is performed after the first projection layer and the position encoding layer; the encoding module is stacked by multiple encoding units, and each encoding unit includes a multi-head attention layer, a second projection layer and a fully connected layer; after the output of the multi-head attention layer is added to the input, the features obtained after layer normalization are used as the input of the second projection layer; after the output of the second projection layer is added to the input, the features obtained after layer normalization are used as the input of the fully connected layer, and after the output of the fully connected layer is added to the input, the features obtained after layer normalization are used as the output of the encoding unit. The design of the encoder can adapt to the input of various sensor types and extract useful feature information.

[0018] Furthermore, the data compression module includes three convolutional layers, the first two convolutional layers are respectively connected to a ReLU activation function, and the third convolutional layer uses transposed convolution to effectively extract and compress features of different sensor data.

[0019] Furthermore, the preprocessing step includes data enhancement and normalization processing. The sensor data after data enhancement, such as acceleration, angular velocity, etc., are enhanced after noise is introduced. The data after normalization processing can adapt to the scale differences of different types of sensors to ensure the uniformity and consistency of feature extraction.

[0020] x=x0+σ x (1)

[0021] In the formula, x0 represents the original sensor data (such as acceleration, angular velocity, temperature, pressure, etc.), x represents the sensor data after data enhancement, and σ x is the introduced Gaussian white noise.

[0022] Taking the inertial sensor as an example, the normalized acceleration and angular velocity measured by it are expressed as:

[0023]

[0024] In the formula, represents the normalized acceleration, They represent the angular velocities before and after the i-th normalization, k=x, y, z represent the components of the angular velocity on the x, y, and z axes, respectively, and n represents the number of samples.

[0025] Furthermore, when the decoder's task is signal reconstruction, the training loss is calculated by formula (5) during the task model training process;

[0026]

[0027] In the formula, represents the data reconstructed from the high-dimensional features of the sensor data, Dec1(·) represents the decoder for the signal reconstruction and denoising task, E represents the high-dimensional features of the sensor data, represents the Huber loss of the signal reconstruction task, x represents the original data of the masked part of the sensor data, and a i , denote the i-th reconstructed acceleration and the original acceleration, ω i , denote the i-th reconstructed angular velocity and the original angular velocity, respectively, ε is the characteristic parameter of Huber loss, ||·|| 2 represents the bi-norm, |·| represents the absolute value, and ρ represents the Huber loss hyperparameter.

[0028] When the decoder's task is classification, the training loss is calculated by formula (12) during the task model training process;

[0029]

[0030] Where Dec2(·) represents the decoder for classification tasks, represents the predicted label sequence, Y represents the original label sequence, represents the cross entropy loss for the classification task. N is the number of categories, Y i and They represent the probability values ​​of the true label and predicted label of the i-th sample respectively.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Representation learning combining self-supervised learning with masking technology: The present invention enhances the model's ability to represent and learn the intrinsic features of various types of sensor data (such as IMU, temperature, pressure, vision, and other sensor data) by combining self-supervised learning with masking technology. Specifically, masking is used to preprocess the sensor data, and with the help of the encoder and decoder architecture, the model can learn effective feature information when some data is missing. By comparing the masked data with the original data for loss calculation, the model can effectively extract the essential features of the sensor data. This method not only significantly improves the efficiency and accuracy of sensor data processing, but also greatly reduces the dependence on a large amount of labeled data, greatly improves the generalization ability and flexibility of the model, and adapts to the complex characteristics of different sensors.

[0033] 2. Multifunctional model for optimizing sensor data processing: The design of the present invention can be widely applied to different types of sensor data processing. The model can accurately extract key features and show significant advantages in tasks such as noise reduction, situational awareness and pattern recognition. It is especially suitable for embedded mobile devices or those with limited computing resources. Specifically, the following optimization designs are proposed based on the characteristics of different sensor signals:

[0034] Data compression: Considering that sensor signals (such as IMU signals, temperature sensors, pressure sensors, etc.) usually contain high-frequency and long sequences, the present invention effectively compresses long sequence data through a data compression module to ensure that important signal features are retained after compression. This design improves the model's ability to process high-frequency data sequences and is applicable to data from a variety of sensors.

[0035] Lightweight encoder design: Given that sensor data processing often involves running in a resource-constrained environment, the present invention adopts a lightweight encoder design. The encoder is composed of multiple encoding units embedded with an attention mechanism, which can effectively extract high-dimensional features and has low computational complexity, making it suitable for running on low-power, embedded systems.

[0036] ·Multi-task adaptability: In order to enable the extracted high-dimensional features to handle different types of tasks, the model is configured with multiple decoders and different encoders for different tasks. For example, for signal denoising tasks, the decoder focuses on the separation of noise and signal; while in scene recognition and pattern recognition tasks, the decoder is used to classify and identify different scenes or patterns. With this multi-task processing capability, the model can not only adapt to the processing of IMU data, but also process data from different sensors (such as vision, temperature and humidity, pressure, etc.), ensuring flexibility and scalability in a variety of application scenarios.

[0037] 3. Cross-sensor versatility and application scenario expansion: The model design of the present invention has good cross-sensor versatility. It is not only suitable for feature extraction of the listed IMU data, but also can process data from other types of sensors (such as temperature, pressure, optics, vision, etc.). By adapting to the signal characteristics of different sensors, the model can play a role in multiple application scenarios, including but not limited to:

[0038] Autonomous driving and robotic navigation: In autonomous driving and robotic navigation, the model can process data from multiple sensors (such as IMU, visual sensors, radar, etc.) and perform real-time position estimation and environmental perception.

[0039] Industrial equipment monitoring and fault diagnosis: Combined with data from temperature sensors, pressure sensors, etc., the model can be used for equipment status monitoring, fault diagnosis, and preventive maintenance.

[0040] Health monitoring and biosignal analysis: For health monitoring devices, the model can process data from physiological sensors (such as heart rate, blood sugar, body temperature, etc.) to provide accurate health monitoring and prediction functions.

[0041] Environmental monitoring: Combining multiple environmental sensor data (such as temperature, humidity, gas concentration, etc.), the model of the present invention can perform environmental monitoring and analysis, supporting scenarios such as smart cities and agricultural monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the overall flow chart;

[0043] Figure 2 is the flow chart of sensor data processing;

[0044] Figure 3 It is a structural diagram of the encoder of the task model. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present application is not limited thereto.

[0046] The present invention provides a sensor data feature extraction method based on self-supervised learning (hereinafter referred to as method, see Figures 1 to 3 ), including the following steps:

[0047] Step S1: Acquire sensor data and perform preprocessing;

[0048] Acquire data from different types of sensors and preprocess the data. Preprocessing includes operations such as data enhancement and normalization:

[0049] Data enhancement: In order to improve the model's ability to process sensor data in complex environments, random noise is added for data enhancement. Data enhancement not only enriches the diversity of training data and improves the model's ability to identify the core features of sensor data, but also enables the model to effectively identify and process background noise, making it more robust in processing various noises that may occur in the real world. It also helps to reduce the overfitting phenomenon of the model and improves the model's generalization ability on unknown data. The process can be expressed as:

[0050] x=x0+σ x (1)

[0051] In the formula, x0 represents the original sensor data (such as acceleration, angular velocity, temperature, pressure, etc.), x represents the sensor data after data enhancement, and σ x is the introduced Gaussian white noise.

[0052] Normalization: Since the measurement values ​​of different types of sensors have different scales, normalization is performed to adapt the data to model training. Normalization can unify the data of different sampling points to a similar scale, which helps to speed up gradient descent and improve the convergence speed of the model. Common normalization methods include maximum / minimum value, mean / variance and other methods. These methods will cause the loss of distribution information and may have a negative impact on the data interval generally represented. Therefore, the present invention performs a simple and effective normalization process on the sensor measurement data to reduce the range difference without seriously changing the distribution of the data itself. For example, for acceleration and angular velocity data, the present invention normalizes the acceleration and angular velocity through equations (2) and (3):

[0053]

[0054] In the formula, represents the normalized acceleration, Respectively represent the angular velocity before and after normalization of the i-th one, k = x, y, z represent the components of the angular velocity on the x, y, and z axes, and n represents the number of samples. Normalized data helps to accelerate model training and improve convergence speed.

[0055] Step S2: performing mask processing on the preprocessed sensor data;

[0056] S2.1: For any sensor data sequence group (including IMU data, temperature data, pressure data, etc.), generate a random number θ∈[0,1). If the random number θ is greater than or equal to the mask setting probability P m , then the data sequence group will not be masked; if the random number θ is less than the mask setting probability P m, then perform mask processing on the data sequence. Perform mask processing of step S2.2 on a plurality of data sequences selected randomly;

[0057] S2.2: For each selected data sequence, generate a random number s∈[0,L) and select the starting point of the mask processing. The mask length is selected by a random number that satisfies the Gaussian distribution. The cutoff point of the mask processing is e=s+l, and then the data is set to zero to form mask data;

[0058] S2.3: Traverse all selected data sequences, if the number of masks set reaches the maximum number of masks M max , then stop the masking process. Otherwise, continue to execute steps S2.1 and S2.2 until the maximum number of masks is reached.

[0059] Step S3: construct a task model; the task model includes a data compression module, an encoder and a decoder; the data compression module compresses and increases the dimension of the input sensor data, the encoder is used to extract the high-dimensional features of the data, and the decoder decodes the high-dimensional features to further complete tasks such as signal noise reduction, scene or pattern perception and recognition, and different tasks use different decoders;

[0060] The data compression module includes three convolutional layers. The first two convolutional layers use ReLU activation functions respectively. The first convolutional layer is used for data compression to ensure that the output sequence length remains unchanged and is sufficient to extract diverse features from the sensor data without being too complicated. The second convolutional layer is used for data dimensionality increase. The third convolutional layer is a transposed convolutional layer, which is used for data shaping processing.

[0061] The input of the encoder is the sensor data output by the data compression module. The encoder includes a first projection layer, a position encoding layer and an encoding module connected in series. The first projection layer and the position encoding layer are followed by layer normalization. The first projection layer realizes data mapping, the layer normalization layer is used for data normalization, and the position encoding embeds the data into the vector space. The encoding module is composed of a plurality of stacked encoding units (four encoding units are included in this embodiment). The output of the previous encoding unit is the input of the next encoding unit. The input of the first encoding unit is the output of the position encoding layer. All encoding units have the same structure and share parameters. The encoding unit includes a multi-head attention layer, a second projection layer and a fully connected layer. After the output of the multi-head attention layer is added to the input, the features obtained by layer normalization are used as the input of the second projection layer. After the output of the second projection layer is added to the input, the features obtained by layer normalization are used as the input of the fully connected layer. After the output of the fully connected layer is added to the input, the features obtained by layer normalization are used as the output of the encoding unit. The specific operation of the encoder is shown in equations (4) to (8):

[0062] Proj1=QW1 (4)

[0063] MAtt=MHA(LN2(LN1(Proj1)+Pos)) (5)

[0064] Proj2=LN3(LN2(LN1(Proj1)+Pos)+MAtt)W2 (6)

[0065] L1=LN6(W(LN4(Proj2+LN1(Proj1)))+LN2(Pos1)) (7)

[0066] E=Encoder(Q) (8)

[0067] Where Proj1 and Proj2 represent the outputs of the first and second projection layers respectively, Q represents the input of the encoder, W1 and W2 represent the projection matrices, Pos represents the output of the position encoding layer, MHA(·) represents the multi-head attention operation, MAtt represents the output of the multi-head attention layer, LN1~LN6 are layer normalization operations, W represents the fully connected layer operation, L1 is the output of the encoding unit, Encoder(·) is the encoder; E is the output of the encoder, that is, the high-dimensional features of the processed sensor data;

[0068] Step S4: Use self-supervised learning to train the task model. Specifically, the masked sensor data is input into the task model, the high-dimensional features are extracted through the encoder, and the data is input into the decoder for data reconstruction. The reconstructed data is compared with the original data, the loss is calculated, and the model parameters are optimized until the loss converges to obtain the trained task model; the encoder of the trained task model is used to extract the high-dimensional features of the sensor data.

[0069] The processed high-dimensional features of sensor data can be further used for tasks such as signal denoising, scene or pattern perception and recognition. Different tasks use different decoders and loss functions. When used for signal denoising tasks, the decoder's task is signal reconstruction, and the original sensor data is used as the training label. During the task model training process, the loss is calculated through the Huber loss function of formula (10) to enhance the robustness of the training;

[0070]

[0071] In the formula, represents the data reconstructed from the high-dimensional features of the sensor data, Dec1(·) represents the decoder for the signal reconstruction and denoising task, E represents the high-dimensional features of the sensor data, represents the Huber loss of the signal reconstruction task, x represents the original data of the masked part of the sensor data, and a i , denote the i-th reconstructed acceleration and the original acceleration, ω i , denote the i-th reconstructed angular velocity and the original angular velocity, respectively, ε is the characteristic parameter of Huber loss, ||·|| 2 represents the bi-norm, |·| represents the absolute value, and ρ represents the Huber loss hyperparameter.

[0072] When the high-dimensional features of sensor data are used for scene or pattern perception and recognition tasks, the decoder's task is classification. The scene or pattern category is used as the training label. During the task model training process, the training loss is calculated using the cross entropy loss function of formula (12) to enhance the robustness of the training.

[0073]

[0074] Where Dec2(·) represents the decoder for classification tasks, represents the predicted label sequence, Y represents the original label sequence, represents the cross entropy loss for the classification task. N is the number of categories, Y i and They represent the probability values ​​of the true label and predicted label of the i-th sample respectively.

[0075] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A sensor data feature extraction method based on self-supervised learning, characterized in that: The method comprises the following steps: Step S1: Acquire sensor data and pre-process the sensor data; Step S2: performing mask processing on the preprocessed sensor data; S2.1: For any sensor data sequence group, generate a random number θ∈[0,1). If the random number θ is greater than or equal to the mask setting probability, no masking is performed; if the random number θ is less than the mask setting probability, multiple sensor data sequences are randomly selected from the sensor data sequence group; S2.2: For any selected sensor data sequence, generate a random number s∈[0,L), and use the position corresponding to the random number s in the sensor data sequence as the starting point of the mask processing; select a random number l from the Gaussian distribution that satisfies the probability of successful mask setting as the mask length, and the cutoff point of the mask processing is e=s+l; set the data at each position from the starting point to the cutoff point in the sensor data sequence to zero; Where L represents the length of the sensor data sequence; the mask length l is less than or equal to the remaining length Ls of the sensor data sequence, and is also less than or equal to the remaining number of masks M max -m,M max Indicates the maximum number of masks, and m indicates the number of masks that have been set; S2.3: Traverse all selected sensor data sequences, if the number of masks m is equal to the maximum number of masks M max , then stop mask processing; if the number of masks set m is less than the maximum number of masks M max , then repeat steps S2.1 and S2.2 until the number of masks set m equals the maximum number of masks M max ; Step S3: constructing a task model; the task model includes a data compression module, an encoder and a decoder, the data compression module compresses and increases the dimension of the input sensor data, the encoder is used to extract high-dimensional features of the sensor data, and the decoder decodes the high-dimensional features of the sensor data; Step S4: input the masked sensor data into the task model, and the model outputs the reconstructed sensor data; compare the reconstructed sensor data with the preprocessed sensor data, calculate the training loss using the masked sensor data, train the task model using self-supervised learning, and obtain the trained task model; use the encoder of the trained task model to extract high-dimensional features of the sensor data.

2. The sensor data feature extraction method based on self-supervised learning according to claim 1 is characterized in that: The encoder includes a first projection layer, a position encoding layer and an encoding module connected in series, and layer normalization is performed after the first projection layer and the position encoding layer; the encoding module is stacked by multiple encoding units, and each encoding unit includes a multi-head attention layer, a second projection layer and a fully connected layer; after the output of the multi-head attention layer is added to the input, the features obtained after layer normalization are used as the input of the second projection layer; after the output of the second projection layer is added to the input, the features obtained after layer normalization are used as the input of the fully connected layer, and after the output of the fully connected layer is added to the input, the features obtained after layer normalization are used as the output of the encoding unit.

3. The sensor data feature extraction method based on self-supervised learning according to claim 1 or 2, characterized in that: The data compression module includes three convolutional layers, the first two convolutional layers are respectively connected to a ReLU activation function, and the third convolutional layer adopts transposed convolution.

4. The sensor data feature extraction method based on self-supervised learning according to claim 1, characterized in that: The preprocessing includes data enhancement and normalization. The acceleration and angular velocity after data enhancement are expressed as: x=x0+σ x (1) In the formula, x0 represents the original sensor data (such as acceleration, angular velocity, temperature, pressure, etc.), x represents the sensor data after data enhancement, and σ x is the introduced Gaussian white noise; The normalized acceleration and angular velocity are expressed as: In the formula, represents the normalized acceleration, They represent the angular velocities before and after the i-th normalization, k=x, y, z represent the components of the angular velocity on the x, y, and z axes, respectively, and n represents the number of samples.

5. The sensor data feature extraction method based on self-supervised learning according to claim 1 or 4, characterized in that: When the decoder's task is signal reconstruction, the training loss is calculated by formula (11) during the task model training process; In the formula, represents the data reconstructed from the high-dimensional features of the sensor data, Dec1(·) represents the decoder for the signal reconstruction and denoising task, E represents the high-dimensional features of the sensor data, represents the Huber loss of the signal reconstruction task, x represents the original data of the masked part of the sensor data, and a i , denote the i-th reconstructed acceleration and the original acceleration, ω i , denote the i-th reconstructed angular velocity and the original angular velocity, respectively, ε is the characteristic parameter of Huber loss, ||·|| 2 represents the bi-norm, |·| represents the absolute value, and ρ represents the Huber loss hyperparameter; When the decoder's task is classification, the training loss is calculated by formula (12) during the task model training process; Where Dec2(·) represents the decoder for classification tasks, represents the predicted label sequence, Y represents the original label sequence, represents the cross entropy loss for the classification task; N is the number of categories, Y i and They represent the probability values ​​of the true label and predicted label of the i-th sample respectively.

Images