Blind person walking direction identification method based on multi-sensor data fusion

Abstract

According to the blind person walking direction identification method based on multi-sensor data fusion, the characteristics of different sensor data are extracted by using the double-branch convolutional neural network, and corresponding information in different characteristic graphs is extracted by using the attention-based position information mining module and the attention-based content information mining module, so that the blind person walking direction identification method based on multi-sensor data fusion is realized. And finally, the extracted information is fused through an information mining and fusion module, so that complementary information among different sensor data is fully utilized, the effectiveness and robustness of a classification model can be effectively improved through comprehensive information mining and fusion of the different sensor data, and more excellent recognition precision is realized.

Description

Technical Field

[0001] The invention relates to the field of walking direction recognition, and in particular to a walking direction recognition method for blind people based on multi-sensor data fusion. Background Art

[0002] Traditional gait recognition methods mainly rely on cameras to collect gait data. However, this data collection method has some disadvantages when used for walking direction estimation. First, the data collected by the camera is easily affected by light, occlusion, and angle changes, which makes it ineffective in some complex or dynamic environments. Second, it is also challenging to continuously capture the gait of visually impaired people through cameras. Inertial sensors, ECG sensors, and EEG sensors can be attached to specific parts of the human body for data collection. Therefore, in recent years, researchers have conducted a lot of research on walking direction recognition based on wearable sensors. However, most of the current research relies on single-modal sensors for walking direction estimation, which limits accuracy and stability. Summary of the invention

[0003] The present invention aims to provide a method for identifying the walking direction of a blind person based on multi-sensor data fusion, which can improve the recognition accuracy of the walking direction of a blind person which is limited by single-modal data.

[0004] In order to achieve the above object, the present invention is implemented through the following technical solutions: Collect action data through data receiving nodes, upload and collect the action data, and perform pre-processing; The preprocessed inertial data and plantar pressure data are converted into grayscale images and divided into training sets and test sets. The training set is used as training data for parameter training to obtain a walking direction recognition model. Inputting the test set data into the walking direction recognition model to obtain a recognition result; The motion data includes inertial data and plantar pressure data; The walking direction recognition model includes an attention-based position information mining module PIM, an attention-based content information mining module CIM, and an information mining and fusion module IMF. After the motion data is input, a dual-branch convolutional neural network is used to extract the feature maps of inertial data and plantar pressure data respectively to obtain inertial feature maps and pressure feature maps. The attention-based position information mining module PIM extracts position information, the attention-based content information mining module CIM extracts content information, and the information mining and fusion module IMF fuses and outputs the extracted position information and content information.

[0005] Furthermore, the dual-branch convolutional neural network includes a first branch convolutional neural network for extracting inertial data and a second branch convolutional neural network for extracting plantar pressure data. The inertial data feature map extracted by the first layer of the convolutional neural network in the first branch convolutional neural network is recorded as , the inertial data feature map extracted by the second layer of convolutional neural network is recorded as , the plantar pressure data feature map extracted by the first layer of the convolutional neural network in the second branch convolutional neural network is recorded as , the plantar pressure data feature map extracted by the second layer of convolutional neural network is recorded as , and The dimension of ,in, represents the number of samples, Indicates the number of channels, represents the height of the feature map, Indicates the width of the feature map; When fusing the location information and content information, the inertial data feature map extracted by the first layer of convolutional neural network is also added. And plantar pressure data characteristic diagram .

[0006] Furthermore, the attention-based position information mining module PIM extracts position information including multi-scale convolution and global pooling operations, which can be expressed as:

[0007]

[0008] in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents the global average pooling operation, Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

[0009] The attention-based position information mining module PIM extracts position information and further includes: nonlinear modeling of the position information extracted and fused by multi-scale convolution, expressed as:

[0010] in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in represents the compression rate and expansion rate hyperparameters in the PIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , location information.

[0011] Furthermore, the attention-based content information mining module CIM extracts content information including multi-scale convolution and global pooling operations, which can be expressed as:

[0012]

[0013] in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents a point-by-point convolution operation that changes the channel to 1. Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

[0014] Furthermore, the content information extraction module CIM based on attention further includes: nonlinear modeling of the content information extracted and fused by multi-scale convolution, which is expressed as:

[0015] in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in are the hyperparameters representing the compression rate and expansion rate in the CIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , content information.

[0016] Furthermore, inertial data is obtained through an inertial sensor placed at the ankle of a human body, and plantar pressure data is obtained through a plantar pressure insole. The collected motion data is uploaded to a host computer data processing platform with the help of a receiving node. The receiving node includes an inertial sensor node and a pressure sensor node. Each inertial sensor node includes an acceleration sensor and a gyroscope, and each pressure sensor node includes a flexible film pressure sensor. For each data receiving node, the collected motion data includes acceleration, angular velocity and magnetic field angle information of the x-axis, y-axis and z-axis. The acceleration signal is recorded as a 3D vector ,in , and are the acceleration forces acting along the x, y and z axes respectively; each gyroscope signal is a 3D vector ,in , and Represents the rotation rate around the x, y and z axes respectively; the magnetic field angle signal can be expressed as a 3D vector ,in , and Respectively represent the components of the magnetic field angle signal in the x-axis, y-axis and z-axis directions; a nine-channel data matrix is ​​obtained , Indicates the number of data receiving nodes, where each element is a 9-dimensional vector

[0017] in , , and is the acceleration force acting on the kth data receiving node along the x, y and z axes, , and is the rotation rate of the kth data receiving node along the x, y and z axes, , and are the components of the magnetic field angle signal of the kth data receiving node in the x-axis, y-axis and z-axis directions.

[0018] Furthermore, inertial data and plantar pressure data were collected for four walking modes: straight walking, curved walking, left-leaning walking, and right-leaning walking. The straight walking was taken as the standard route, walking on the left side of the standard route was left-leaning walking, and walking on the right side of the standard route was right-leaning walking. Walking left-leaning and right-leaning walking simulated the process of walking deviating from the straight line, and curved walking simulated the process of walking after deviating from the straight line. The recognition results of the binary classification experiment are divided into two walking directions: straight and deviated. The straight walking direction is the straight result, and the rest are deviated results; The recognition results of the four-category experiment are divided into straight, deviated, and already deviated. Walking in a straight line belongs to the straight result, walking to the left and walking to the right belong to the deviated results, and walking in a curved shape belongs to the already deviated result.

[0019] Furthermore, the preprocessing of the inertial data and the plantar pressure data includes: The Kalman dynamic filtering algorithm is used to denoise the inertial data; Downsampling and interpolation algorithms are used to align the data lengths of inertial data and plantar pressure data; Segment the continuous gait signal (inertial data and plantar pressure data after data length alignment).

[0020] Furthermore, the sliding window technology is used to realize the gait signal segmentation, and a repetition rate of 50% and a window width of 1 second are selected for division. Each sample data contains at least one to two gait cycles, and real-time continuous gait is obtained.

[0021] The advantages of the present invention are: Since the information in multi-source sensor data is very rich, it is impossible to obtain all the effective information in the data by only using multi-layer convolution to extract the feature map of the data. The present invention designs an information fusion module based on the PIM module and the CIM module to make full use of the complementary information between different sensor data and to mine more effective information in the feature map; In order to achieve adaptive learning of the mined content information, nonlinear modeling is performed on it, and two bottleneck functions are used to compress and restore the dimensions, obtain the weights of adaptive learning, and obtain more discriminative content information; While fusing the mined location information with the content information, the lower-level features extracted by the first layer of convolution are also introduced. The low-level features are invariant to small-scale changes, rotations, and translations in the input data. The introduction of low-level features helps to reduce the degradation of the connection matrix caused by the layer depth, which helps to improve the generalization performance of the model. A person may not have a clear gait cycle in certain situations. For example, an irregular gait may occur when turning during walking. Therefore, the present invention does not use the gait cycle detection preprocessing method, but uses the sliding window technology to achieve segmentation, so as to obtain a real-time and continuous gait, which is not limited by the gait cycle and is more in line with the real scene. The repetition rate is set to 50% to effectively avoid data loss at the edge of the window, so as to fully retain all information. The walking direction recognition method for the blind based on inertial sensors and pressure insoles provided by the present invention utilizes a dual-branch convolutional neural network to extract features of different sensor data, uses position information mining and content information mining based on an attention mechanism to extract corresponding information in different feature maps, and finally fuses the extracted information through an information mining and fusion module to fully utilize the complementary information between different sensor data. The present invention can effectively improve the effectiveness and robustness of the classification model and achieve better recognition accuracy by performing comprehensive information mining and fusion on different sensor data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of a method for identifying walking direction of a blind person based on multi-sensor data fusion according to the present invention; Figure 2 It is a schematic diagram of the attention-based location information mining module of the present invention; Figure 3 Schematic diagram of the attention-based content information mining module of the present invention. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0024] Example 1 This embodiment proposes a method for identifying the walking direction of a blind person based on multi-sensor data fusion.

[0025] Step 1: Collect inertial data and plantar pressure data.

[0026] N wearable inertial sensor nodes are bound to N parts of the human leg, and a pair of pressure insoles are placed in the test shoes. Each inertial sensor node includes a three-axis accelerometer and a three-axis gyroscope, and each pressure sensor node is a flexible film pressure sensor; N sensor nodes are used to collect data on different walking styles of people.

[0027] The walking movements of the subjects are collected. Due to the particularity of the blind and visually impaired patients, blindfolding operations are designed for subjects with normal vision to simulate the walking conditions of blind individuals for data collection. This embodiment collects four walking methods: straight walking, curved walking, left-leaning walking, and right-leaning walking. Among them, straight walking is the standard route, walking on the left side of the standard route is left-leaning walking, and right-leaning is walking on the right side of the standard route. The collected motion data is uploaded to the host computer data processing platform with the help of the receiving node. Each inertial sensor node mainly includes an acceleration sensor and a gyroscope, and each pressure sensor node includes a flexible film pressure sensor.

[0028] Left deviation and right deviation simulate the process of walking away from a straight line, while curved walking simulates the process of walking away from a straight line. For each data receiving node, the collected motion data includes acceleration, angular velocity and magnetic field angle information of the x-axis, y-axis and z-axis, and their curves have periodic changes. The acceleration signal is recorded as a 3D vector ,in , and are the acceleration forces acting along the x, y and z axes respectively; each gyroscope signal is a 3D vector ,in , and Represents the rotation rate around the x, y and z axes respectively; the magnetic field angle signal can be expressed as a 3D vector ,in , and Respectively represent the components of the magnetic field angle signal in the x-axis, y-axis and z-axis directions; a nine-channel data matrix is ​​obtained , Indicates the number of data receiving nodes, where each element is a 9-dimensional vector

[0029] in , , and is the acceleration force acting on the kth data receiving node along the x, y and z axes, , and is the rotation rate of the kth data receiving node along the x, y and z axes, , and are the components of the magnetic field angle signal of the kth data receiving node in the x-axis, y-axis and z-axis directions.

[0030] Step 2: Preprocess action data.

[0031] For the motion data collected by each inertial sensor node, the sensor's built-in Kalman dynamic filter is used. The data waveform collected by the pressure sensor is very smooth, so no denoising is performed. When collecting data, the sampling frequencies of different sensors should be the same. If they are different, downsampling and interpolation techniques can be used to align the lengths of different sensor data. Then the sensor data is normalized and the sensor data of the same type is spliced ​​and fused.

[0032] Step 3: Use sliding window segmentation technology to perform window segmentation on the data obtained in step 2 to obtain real-time and continuous gait window data.

[0033] Each window data should contain at least one or two gait cycles so that each cycle has complete features. Therefore, this embodiment selects a repetition rate of 50% and a window width of 1 second for division. The repetition rate is set to 50% to avoid data loss at the edge of the window to fully retain all information. According to our tests, most people need about 0.8-1 seconds to complete a gait cycle, so the window width is set to 1 second.

[0034] Step 4: Visualize the window data obtained in step 3, that is, convert it into a grayscale image to facilitate the subsequent input of sensor data into the model.

[0035] Step 5: Divide the grayscale image obtained in step 4 into a training set and a test set. The training set is used as training data for parameter training and modeling, thereby obtaining a corresponding blind person's walking direction recognition model. When the test set data is input into the corresponding model, the corresponding recognition result can be obtained.

[0036] Example 2 This embodiment proposes a walking direction recognition model for recognizing multiple directions of a human body when walking.

[0037] First, a wireless transmission system consisting of two nine-axis IMU sensors (WIT Inc., CHN) and a pair of plantar pressure insoles was used to collect walking data. Each inertial sensor node integrates high-precision ICM42605 (accelerometer, gyroscope) and MMC3630 (magnetometer), with the characteristics of small size and low power consumption. Due to the different performances of different parts of the body, resulting in different accelerations and angular velocities, two inertial sensors were placed at the ankles of the human body to obtain inertial data. Its sampling frequency is 50Hz, and the data is wirelessly transmitted to the upper position via HC06-HID (Bluetooth receiver) to receive and process the nine-axis data. The plantar pressure insole (ZNX-01) is built into the sole to obtain plantar pressure information. Each insole has eight plantar pressure flexible film sensors specially designed to obtain pressure information at different plantar positions. The sampling frequency of the plantar pressure flexible film sensor is 1024Hz and is directly connected to the laptop through the USB port. Due to the particularity of the blind and visually impaired patients, blindfolding operations were designed for subjects with normal vision during the collection process to simulate the walking conditions of blind individuals for data collection. Four walking modes were collected: straight walking, curved walking, left-leaning walking, and right-leaning walking. The straight walking was the standard route, walking on the left side of the standard route was left-leaning walking, and right-leaning walking was walking on the right side of the standard route. In general, the purpose of the present invention is to determine whether the current walking direction is a straight line. Left deviation and right deviation simulate the process of walking away from a straight line, while curved walking simulates the process of walking after deviating from a straight line. Therefore, in the subsequent experiments, this embodiment also conducted two groups of experiments: a two-classification experiment to identify whether the walking direction is straight or deviated; a four-classification experiment to determine whether the participant is walking straight, deviating, or has deviated. In this embodiment, a total of 20 experimenters were recruited, with an average age of 25 years old, and the collection environment was a fixed length of 20m indoor road. The collection of each set of data involves the recording of the above four specified walking modes, each lasting 1 minute, and a total of 4 sets of data were collected.

[0038] Since the acquired gait data is a raw signal, the data needs to be preprocessed to improve its quality. This section uses the Kalman dynamic filtering algorithm built into the sensor to denoise the inertial data. The data waveform collected by the pressure sensor is very smooth, so no denoising is performed. Since the sampling frequencies of the inertial sensor and the pressure sensor are different, in order to facilitate subsequent data processing, before this, the data lengths of the inertial and pressure data are first aligned by downsampling and interpolation methods. In addition, in order to facilitate the extraction of the features of the collected inertial data and plantar pressure data, the continuous gait signal needs to be segmented. There are two main types of segmentation methods: one is segmentation based on the gait cycle, and the other is segmentation using sliding window technology. An important factor to consider is that there may be no clear gait cycle in certain circumstances, such as irregular gaits when turning during walking. Therefore, this embodiment decides not to use the method of gait cycle detection preprocessing, and uses sliding window technology to achieve segmentation, so as to obtain real-time and continuous gait, which is not limited by the gait cycle and is more in line with the real scene situation. Each sample data should contain at least one or two gait cycles so that each cycle has complete features. Therefore, this embodiment selects a repetition rate of 50% and a window width of 1 second for division. The repetition rate is set to 50% to avoid data loss at the edge of the window so as to fully retain all information. Subsequently, these data window matrices are normalized and visualized as images.

[0039] The model algorithm proposed in this embodiment is called a comprehensive information mining and fusion algorithm. The model mainly includes three modules. Please refer to Figure 1 , attention-based position information mining (PIM) module, attention-based content information mining (CIM) module and information fusion module.

[0040] First, a dual-branch CNN is used to extract the feature maps of inertial data and plantar pressure data, respectively, and the feature maps extracted by the first layer of CNN are obtained. , and extracted by the second layer of CNN , , and The dimensions are set to ,in, represents the number of samples, Indicates the number of channels, represents the height of the feature map, It represents the width of the feature map. The one with subscript I indicates that the feature map is extracted from inertial data, and the one with subscript P indicates that the feature map is extracted from plantar pressure data.

[0041] The IMF module receives the feature map extracted by CNN. and ,This paper chooses to extract location information and content information from each feature at the same time. In this module, multi-feature IPM and multi-feature ICM modules are designed to mine the location information and content information respectively. and The input is sent to the multi-feature IPM and multi-feature ICM modules for further feature information mining and fusion to make full use of the complementary information between different sensor data.

[0042] For PIM modules, please refer to Figure 2 In order to fully mine and adapt the key position information in the learning habitual data and the plantar pressure data and obtain the key feature expression, this embodiment proposes a multi-feature position information mining module based on the inertial feature map and the pressure feature map. Use multi-scale convolution to extract features from different sensor data to extract information with different receptive fields. This operation can be expressed as:

[0043]

[0044] Right now,

[0045] in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents the global average pooling operation, Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

[0046] In order to achieve adaptive learning of the mined location information, the present invention performs nonlinear modeling on the location information. First, the location information is reduced in dimension, and two bottleneck functions are used to compress and restore the dimension of the location information to obtain a weight that can be adaptively learned, thereby making the location information more discriminative. The specific operation can be expressed as:

[0047] in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in represents the compression rate and expansion rate hyperparameters in the PIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , location information.

[0048] CIM module please refer to Figure 3 In order to fully mine and adaptively learn the important content information in the habitual feature map and the pressure feature map and obtain the key feature expression, a content information mining module is proposed. Similar to the location information mining module, in order to obtain information of different receptive fields, multi-scale convolution is used to extract features from different sensor data. The specific operation can be expressed as:

[0049]

[0050] in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents a point-by-point convolution operation that changes the channel to 1. Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

[0051] In order to achieve adaptive learning of the mined content information, nonlinear modeling is performed on it. Two bottleneck functions are used to compress and restore the dimensions, obtain the weights of adaptive learning, and obtain more discriminative content information. The specific operation can be expressed as:

[0052] in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in are the hyperparameters representing the compression rate and expansion rate in the CIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , content information.

[0053] Figure 1 The invention also includes an information mining and fusion (IMF) module: Since the information in multi-source sensor data is very rich, it is impossible to obtain all the valid information in the data by only using multi-layer convolution to extract the feature map of the data. Therefore, in order to mine as much valid information as possible from the feature map, the present invention designs an information fusion module based on the PIM module and the CIM module. The IMF module can fuse the mined location information with the content information, and also add the inertial data feature map extracted by the first layer of the convolutional neural network. And plantar pressure data characteristic diagram This is because low-level features have a certain invariance to small-scale changes, rotations, and translations in the input data. Introducing low-level features and helping to reduce the degradation of the connection matrix caused by layer depth can help improve the generalization performance of the model.

[0054] This embodiment mainly uses the five-fold cross validation method to evaluate the algorithm, selects the Adam optimizer and sets the learning rate to 0.001. The detailed parameter settings of the backbone network are shown in Table 1. PyTorch is used for implementation, and the number of training is set to 100. The four types of data collected are experimented using two classification methods: one is to set the recognition type to straight walking, curved walking, left-leaning walking, and right-leaning walking according to the detailed classification method, which is referred to as the four-classification experiment in the following; the other is to set the recognition type to straight walking and deviating walking according to a relatively simple classification method, which is referred to as the two-classification experiment in the following.

[0055] Table 1 Specific parameters of the CNN used

[0056] Table 2 and Table 3 respectively give the classification accuracy obtained by the four-classification experiment and the two-classification experiment under different recognition methods. From the results, it can be seen that in both the four-classification experiment and the two-classification experiment, the method proposed in the present invention can obtain the highest recognition accuracy.

[0057] Table 2 Classification accuracy obtained by different recognition methods in the four-classification experiment

[0058] Table 3 Classification accuracy obtained by different recognition methods in the binary classification experiment

[0059] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying walking direction of the blind based on multi-sensor data fusion, characterized in that: Includes steps: Collect action data through data receiving nodes, upload and collect the action data, and perform pre-processing; The preprocessed inertial data and plantar pressure data are converted into grayscale images and divided into training sets and test sets. The training set is used as training data for parameter training to obtain a walking direction recognition model. Inputting the test set data into the walking direction recognition model to obtain a recognition result; The motion data includes inertia data and plantar pressure data; The walking direction recognition model includes an attention-based position information mining module PIM, an attention-based content information mining module CIM, and an information mining and fusion module IMF. When the motion data is input, a dual-branch convolutional neural network is used to extract feature maps of inertial data and plantar pressure data respectively to obtain an inertial feature map and a pressure feature map. The attention-based position information mining module PIM extracts position information, the attention-based content information mining module CIM extracts content information, and the information mining and fusion module IMF fuses and outputs the extracted position information and content information.

2. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 1, characterized in that: The dual-branch convolutional neural network includes a first branch convolutional neural network for extracting inertial data and a second branch convolutional neural network for extracting plantar pressure data. The inertial data feature map extracted by the first layer of the convolutional neural network in the first branch convolutional neural network is recorded as , the inertial data feature map extracted by the second layer of convolutional neural network is denoted as , the plantar pressure data feature map extracted by the first layer of the convolutional neural network in the second branch convolutional neural network is recorded as , the plantar pressure data feature map extracted by the second layer of convolutional neural network is recorded as , and The dimension of ,in, represents the number of samples, Indicates the number of channels, represents the height of the feature map, Indicates the width of the feature map; When fusing the location information and content information, the inertial data feature map extracted by the first layer of convolutional neural network is also added. And plantar pressure data characteristic diagram .

3. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 2, characterized in that: The attention-based position information mining module PIM extracts position information including multi-scale convolution and global pooling operations, which can be expressed as: in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents the global average pooling operation, Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

4. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 3 is characterized in that: The attention-based position information mining module PIM extracts position information and further includes: nonlinear modeling of the position information extracted and fused by multi-scale convolution, expressed as: in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in represents the compression rate and expansion rate hyperparameters in the PIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , location information.

5. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 2, characterized in that: The attention-based content information mining module CIM extracts content information including multi-scale convolution and global pooling operations, which can be expressed as: in, Represents the feature map extracted from the sensor data Use multi-scale convolution to extract information, , They represent convolutional layers with different receptive fields, represents a point-by-point convolution operation that changes the channel to 1. Represents the combined operation of multiple functions. represents element-by-element addition, represents a join operation along the first dimension, Representation feature map and The position information is extracted and fused through multi-scale convolution. The dimension is .

6. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 5, characterized in that: The attention-based content information mining module CIM extracts content information and further includes: nonlinear modeling of content information extracted and fused by multi-scale convolution, expressed as: in, Indicates that The dimension from becomes , Indicates that The dimension from becomes , and are two fully connected layers, , ,in are the hyperparameters representing the compression rate and expansion rate in the CIM module, using ReLU and Sigmoid functions as and , represents element-wise multiplication, Represents the fusion of feature maps obtained through adaptive learning , content information.

7. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 1, characterized in that: The inertial data is obtained by an inertial sensor placed at the ankle of a human body, and the plantar pressure data is obtained by a plantar pressure insole; The data receiving node includes an inertial sensor node and a pressure sensor node, each inertial sensor node includes an acceleration sensor and a gyroscope, and each pressure sensor node includes a flexible film pressure sensor; For each data receiving node, the collected motion data includes acceleration, angular velocity and magnetic field angle information of the x-axis, y-axis and z-axis. The acceleration signal is recorded as a 3D vector ,in , and are the acceleration forces acting along the x, y and z axes respectively; each gyroscope signal is a 3D vector ,in , and Represents the rotation rate around the x, y and z axes respectively; the magnetic field angle signal can be expressed as a 3D vector ,in , and Respectively represent the components of the magnetic field angle signal in the x-axis, y-axis and z-axis directions; a nine-channel data matrix is ​​obtained , Indicates the number of data receiving nodes, where each element is a 9-dimensional vector in , , and is the acceleration force acting on the kth data receiving node along the x, y and z axes, , and is the rotation rate of the kth data receiving node along the x, y and z axes, , and are the components of the magnetic field angle signal of the kth data receiving node in the x-axis, y-axis and z-axis directions.

8. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 1, characterized in that: The recognition results of the binary classification experiment are divided into two walking directions: straight and deviated; The recognition results of the four-category experiment are straight walking, curved walking, walking to the left, and walking to the right. Walking in a straight line is the standard route, walking on the left side of the standard route is walking to the left, and walking on the right side of the standard route is walking to the right. Walking to the left and walking to the right simulate the process of deviating from a straight line, and curved walking simulates the process of walking after deviating from a straight line.

9. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 1, characterized in that: The pre-processing comprises: The Kalman dynamic filtering algorithm is used to denoise the inertial data; Downsampling and interpolation algorithms are used to align the data lengths of inertial data and plantar pressure data; Segment the continuous gait signal.

10. The method for identifying walking direction of the blind based on multi-sensor data fusion according to claim 9, characterized in that: The sliding window technology is used to achieve gait signal segmentation. A repetition rate of 50% and a window width of 1 second are selected for segmentation. Each sample data contains at least one or two gait cycles to obtain real-time continuous gait.

Images