An infrastructure state recognition method based on IMU data

Abstract

The embodiment of the present application provides a kind of infrastructure state identification method based on IMU data, the method comprises: the IMU data that inertia measurement unit is collected is acquired and is deployed on infrastructure, the feature extractor is extracted from the IMU data with the feature vector corresponding to the infrastructure;The reference feature vector of the plurality of states that the infrastructure is preset is carried out similarity calculation with the feature vector corresponding to the infrastructure extracted, and similarity result is obtained;According to similarity result, the state that infrastructure is currently located is determined, the technical scheme of the embodiment of the present application is determined according to the similarity result that the feature vector of the infrastructure extracted and the reference feature vector of the plurality of states that the infrastructure is preset is carried out similarity calculation, the state of infrastructure, it can be widely applied in any infrastructure and identify any infrastructure state specified by user, and it is high and small in cost.

Description

Technical Field

[0001] This invention relates to Internet of Things (IoT) technology, specifically to the field of digital signal processing, and more specifically, to an infrastructure status identification method based on IMU data. Background Technology

[0002] With the rapid development of IoT technology, infrastructure status identification is ubiquitous in daily life, especially in smart homes, industrial internet, fire safety, and smart elderly care, where its importance is self-evident. However, infrastructure inspection often requires customized embedded sensors and identification algorithms for specific use cases, leading to increased costs and barriers between different applications, which greatly hinders the widespread adoption of intelligent sensing technology. Against this backdrop, many sensing technologies have been proposed, including contact and non-contact methods.

[0003] Non-contact sensing technology uses radio frequency technologies such as radar, RFID, and UWB to collect echo signals carrying the state characteristics of the detected object by means of backscattering and multipath effects of radio waves on the detected object, thereby identifying the object's state. However, this method is often limited by problems such as scene specificity, high cost, and difficulty in deployment, and is only implemented in specific scenarios or cases, and has not been widely commercialized.

[0004] Contact sensing technology is mostly based on inertial navigation units (INS), which are characterized by low power consumption and low cost due to their use of MEMS (Micro-Electromechanical Systems) technology. INS typically integrates accelerometers, gyroscopes, and magnetometers, and can characterize the attitude and orientation changes of the detected facility. They are widely used to detect object states, such as home status detection, facility attitude detection, vehicle collision detection, and fall detection. Their detection techniques mostly rely on sensor data to calculate attitude and orientation, identifying states through changes in intensity or direction. However, this computation is complex and inefficient. With the development of machine learning technology, more universal machine learning methods have gradually evolved, moving from traditional methods that extract semantic features such as attitude and orientation to iterative training of parameterized models based on specific scenarios to extract non-semantic features for state recognition. However, this machine learning-based state recognition relies on labeled data specific to a particular task, exhibiting poor transferability between tasks. If covering multi-task scenarios, the excessive number of parameters leads to excessive computational costs.

[0005] Based on the above analysis, existing methods for detecting facility status using non-contact sensing technology are often limited by scenario specificity, resulting in low versatility, high cost, and difficulty in deployment. While contact sensing technology for facility status detection involves computationally complex methods for calculating attitude and orientation based on sensor data, machine learning-based status recognition relies on task-specific labeled data, leading to poor transferability and further reducing versatility, while also incurring excessive computational costs during the transfer process. Therefore, there is an urgent need for a highly versatile infrastructure status recognition system with low computational cost. Summary of the Invention

[0006] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide an infrastructure status identification method based on IMU data.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] According to a first aspect of the present invention, an infrastructure state identification method based on IMU data is provided. The method includes: acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure; extracting a feature vector corresponding to the infrastructure from the IMU data using a feature extractor; performing a similarity calculation between the extracted feature vector corresponding to the infrastructure and a reference feature vector of multiple preset states for the infrastructure to obtain a similarity result; and determining the current state of the infrastructure based on the similarity result.

[0009] In some embodiments of the present invention, the feature extractor is obtained through the following training method: acquiring a constructed neural network classification model, which includes a feature extractor and a classifier; acquiring a first training set, the first training set including multiple first sample data and a label corresponding to each first sample data, the first sample data being IMU human motion data collected when a human performs a certain action, and the label indicating the human motion category corresponding to the first sample data; training the neural network classification model using the first training set to identify the human motion category based on the first sample data, wherein the feature extractor is trained to extract features from the first sample data in the first training set to obtain a feature vector, the classifier is trained to output the human motion category based on the extracted feature vector, a first loss is determined based on the label of the first sample data and the output human motion category using the additive angle margin loss function, and the parameters of the feature extractor and the classifier are updated based on the first loss.

[0010] In some embodiments of the present invention, the feature extractor includes a first branch network, a second branch network, a cascaded layer, a Dropout layer, and a fully connected layer. The first branch network includes a multi-layer convolutional neural network, and the second branch network includes a gated recurrent neural network. The extracted feature vector is obtained as follows: The IMU data is convolved using the multi-layer convolutional neural network in the first branch network to extract a first feature; the IMU data is processed using the gated recurrent neural network in the second branch network to extract a second feature; the first and second features are cascaded using the cascaded layer to obtain a first hidden layer feature vector; the first hidden layer feature vector is randomly deactivated using the Dropout layer to obtain a second hidden layer feature vector; and the second hidden layer feature vector is processed using the fully connected layer to obtain the extracted feature vector.

[0011] In some embodiments of the present invention, the feature extractor is obtained through the following training method: acquiring a constructed autoencoder, which includes a feature extractor and a decoder, both of which include multi-layer convolutional neural networks; acquiring a second training set, which includes multiple second sample data, each of which is a segment of IMU data collected by an inertial measurement unit deployed on the infrastructure at predetermined time intervals; training the feature extractor to encode features in the second sample data using the second training set to obtain feature vectors; decoding the feature vectors using the decoder to obtain decoded data; determining a second loss based on the mean squared error loss function according to the second sample data and the decoded data; and updating the parameters of the feature extractor and the decoder according to the second loss.

[0012] In some embodiments of the present invention, the feature extractor includes a low-pass filter and a statistical feature module, wherein the extracted feature vector is obtained by: filtering the IMU data collected by the inertial measurement unit deployed on the infrastructure using the low-pass filter to obtain IMU signal features; performing statistical processing on the IMU data collected by the inertial measurement unit deployed on the infrastructure using the statistical feature module to obtain statistical features, wherein the statistical features include mean, variance, zero-crossing rate, skewness, kurtosis, and upper quartile; wherein the feature extractor concatenates the IMU signal features and the statistical features to obtain the extracted feature vector.

[0013] In some embodiments of the present invention, for different types of infrastructure, there are two or more preset reference feature vectors corresponding to the type of infrastructure. Each type of infrastructure is one of the following: door, window, robotic arm, drawer, bed, curtain, water pipe, printer, fitness equipment, vent, and toilet.

[0014] In some embodiments of the present invention, the current state of the infrastructure is determined in the following manner: each time, the similarity result between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of one of the multiple preset states is compared with a preset threshold. When the similarity result is greater than the preset threshold, the preset state corresponding to the similarity result is taken as the current state of the infrastructure; or the similarity is calculated between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of each of the multiple preset states to obtain multiple similarity results. The preset state corresponding to the highest similarity result is selected from the multiple similarity results as the current state of the infrastructure.

[0015] In some embodiments of the present invention, the reference feature vector for each of the preset multiple states includes a reference IMU signal feature and a reference statistical feature for the corresponding state. The state of the infrastructure is determined as follows: the IMU signal feature obtained by filtering using a dynamic time warping similarity matching algorithm is compared with the reference IMU signal feature of the preset corresponding state to obtain a signal feature similarity result; the obtained statistical feature is compared with the reference statistical feature of the preset corresponding state to obtain a statistical feature similarity result; when both the signal feature similarity result and the statistical feature similarity result are greater than a preset threshold, the state of the infrastructure is determined to be the preset corresponding state.

[0016] According to a second aspect of the present invention, an infrastructure status identification system based on IMU data is provided. The system is deployed on various infrastructures and includes: a sensor data acquisition module for acquiring IMU data of the infrastructure collected by inertial measurement units deployed on the infrastructure; a microcontroller module for determining the current status of the infrastructure based on the IMU data of the infrastructure according to any one of the methods of the first aspect of the present invention; and a communication module for transmitting the current infrastructure status determined by the microcontroller module to a user for status monitoring.

[0017] According to a third aspect of the present invention, an infrastructure status identification system based on IMU data is provided. The system is deployed on a smart gateway, a local host, a fog computing service platform, or a cloud computing service platform. The infrastructure includes a sensor data acquisition module and a communication module. The sensor data acquisition module acquires IMU data collected by inertial measurement units deployed on the infrastructure. The communication module transmits the acquired infrastructure IMU data to the system. The system includes: a microcontroller module for determining the current state of the infrastructure based on the infrastructure IMU data according to any one of the methods of the first aspect of the present invention; and a communication module for transmitting the current infrastructure state determined by the microcontroller module to a user for status monitoring.

[0018] According to a fourth aspect of the present invention, an infrastructure status identification and alarm device based on IMU data is provided, comprising: a sensor data acquisition module for acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure; a microcontroller module for determining the current status of the infrastructure based on the IMU data of the infrastructure according to any one of the methods of the first aspect of the present invention; and an early warning module for judging whether the status of the infrastructure is abnormal according to the current status of the infrastructure, and issuing an alarm when the status is abnormal.

[0019] According to a fifth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to implement the steps of the method according to any one of the first aspects of the present invention by executing the executable instructions.

[0020] Compared with the prior art, the advantages of the present invention are as follows:

[0021] In this invention's infrastructure status identification method, firstly, inertial measurement units (IMUs) can be quickly deployed on infrastructure at low cost, directly acquiring IMU data collected by the IMUs on the infrastructure and extracting feature vectors from the IMU data to quickly identify the status of the corresponding infrastructure when monitoring and control are required. Secondly, multiple preset reference feature vectors corresponding to different infrastructure states are provided, which are not limited by the specificity of the scene or facility, offering high flexibility. The extracted infrastructure feature vectors are compared with the preset reference feature vectors for similarity calculation, resulting in low computational cost. Finally, based on the similarity results, the degree of similarity between the extracted infrastructure feature vectors and the reference feature vectors can be directly determined. This method involves few parameters and efficiently and quickly determines the current status of the infrastructure, facilitating real-time monitoring of the infrastructure. This status identification method can be widely applied to any infrastructure and can identify any infrastructure state specified by the user. It has high universality and low cost, and can be applied to various home appliances and industrial production machinery to monitor and control the status of the corresponding infrastructure. Attached Figure Description

[0022] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0023] Figure 1 This is a flowchart of an infrastructure status identification method based on IMU data according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of a first feature extractor according to an embodiment of the present invention;

[0025] Figure 3This is a schematic diagram illustrating the process of extracting features and classifying input IMU human motion data by training a neural network classification model according to an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of an autoencoder comprising a feature extractor and a decoder constructed according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram illustrating the structural principle of a third feature extractor according to an embodiment of the present invention;

[0028] Figure 6 This is a schematic diagram of the IMU signal feature sequence obtained by aligned filtering and the reference IMU signal feature sequence according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of a deployment scheme for an infrastructure status identification system according to an embodiment of the present invention;

[0030] Figure 8 This is a schematic diagram of a deployment scheme for an infrastructure status identification system according to another embodiment of the present invention;

[0031] Figure 9 This is a schematic diagram of the deployment scheme of an infrastructure status identification system according to another embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0033] As mentioned in the background section, existing methods for detecting facility status using non-contact sensing technology have low versatility, high cost, and are difficult to deploy, while contact sensing technology for detecting facility status is computationally complex and difficult to implement.

[0034] To address the aforementioned issues, the inventors objectively analyzed the limitations of existing non-contact sensing technologies for detecting facility status, which often suffer from low versatility, high cost, and difficulty in deployment due to scenario-specificity. They proposed a contact sensor technology, deploying inertial measurement units (IMUs) on the infrastructure to collect IMU data corresponding to changes in infrastructure status. This reduces deployment difficulty and cost. Furthermore, existing contact sensing technologies for facility status detection involve computationally complex methods for calculating attitude and orientation based on sensor data for status recognition. Machine learning-based status recognition relies on labeled data specific to the task, resulting in poor transferability, low versatility, and excessive computational costs during migration. Therefore, the inventors pre-set multiple baseline feature vectors for the infrastructure's various states. Simultaneously, a feature extractor extracts the corresponding feature vectors from the IMU data collected from the infrastructure. The extracted feature vectors are then compared with the pre-set baseline feature vectors for the infrastructure's various states to determine the current state of the infrastructure, significantly improving computational efficiency and reducing computational costs.

[0035] Based on the above analysis, according to an embodiment of the present invention, an infrastructure status identification method based on IMU data is proposed, see [link to relevant documentation]. Figure 1In step S1, IMU data collected by inertial measurement units deployed on the infrastructure is acquired. A feature extractor is used to extract the feature vector corresponding to the infrastructure from the IMU data. Inertial measurement units can be quickly deployed on infrastructure at low cost. The IMU data collected by the inertial measurement units on the infrastructure is directly acquired, and the feature vector of the IMU data is extracted to quickly identify the state when it is necessary to monitor and control the state of the corresponding infrastructure. In step S2, the extracted feature vector corresponding to the infrastructure is compared with the baseline feature vectors of multiple states preset for the infrastructure to obtain the similarity result. This invention presets multiple baseline feature vectors for the corresponding infrastructure, that is, different infrastructures can preset the baseline feature vectors corresponding to the possible states of the facility, which is not limited by the specificity of the scene facility and has high flexibility. The similarity calculation between the extracted feature vector of the infrastructure and the preset baseline feature vector has low computation cost. In step S3, the current state of the infrastructure is determined according to the similarity result. The similarity result can directly determine the degree of similarity between the extracted feature vector corresponding to the infrastructure and the baseline feature vector. It involves few parameters and efficiently and quickly determines the state of the infrastructure, making it easy to realize real-time monitoring of the infrastructure. The status recognition method of this embodiment can be widely applied to any infrastructure and can identify the status of any infrastructure specified by the user. It has high versatility and low cost. For example, it can be applied to various home appliances and industrial production machinery and equipment to realize the monitoring and control of the status of the corresponding infrastructure.

[0036] According to one embodiment of the present invention, in the infrastructure status identification method of the present invention, inertial measurement units (IMUs) are installed and deployed on the infrastructure by means of attachment, pasting, embedding, etc. The inertial measurement unit is also called an inertial navigation unit. The inertial measurement unit typically integrates three types of sensors: accelerometer, gyroscope, and magnetometer. Therefore, the IMU data collected by the inertial measurement unit generally includes accelerometer data, gyroscope data, and magnetometer data. Depending on the different types of infrastructure, the IMU data collected by the inertial measurement unit in the present invention may include accelerometer data, gyroscope data, and magnetometer data, or any one or any two of these three types of data. For example, it may only include accelerometer data, or only include accelerometer data and gyroscope data, to be applicable to different types of infrastructure.

[0037] According to one embodiment of the present invention, an infrastructure state identification method based on IMU data is provided. This method involves deploying an inertial measurement unit (IMU) on the infrastructure and having the IMU collect IMU data of the infrastructure when it is in a certain state. The state of the infrastructure is then identified based on the IMU data collected by the IMU. The identification method includes the following steps S1, S2, and S3. To better understand the present invention, each step is described in detail below with reference to specific embodiments.

[0038] In step S1, IMU data collected by inertial measurement units deployed on the infrastructure is acquired, and feature vectors corresponding to the infrastructure are extracted from the IMU data using a feature extractor.

[0039] According to one embodiment of the present invention, after acquiring IMU data collected by inertial measurement units deployed on infrastructure, any one of the following three examples of feature extractors can be used to extract feature vectors corresponding to the infrastructure from the IMU data. The three examples of feature extractors are described below:

[0040] Example 1

[0041] See Figure 2 , Figure 2 The diagram illustrates the structure of a first type of feature extractor. This feature extractor includes a first branch network, a second branch network, a cascaded layer, a Dropout layer (i.e., a random dropout layer), and a fully connected layer. The first branch network comprises a multi-layer convolutional neural network; in this invention, the first branch network uses a three-layer convolutional neural network. The second branch network comprises a gated recurrent neural network. The first feature extractor extracts the feature vector corresponding to the infrastructure from the IMU data by: performing convolution processing on the IMU data using the multi-layer convolutional neural network in the first branch network to extract a first feature; processing the IMU data using the gated recurrent neural network in the second branch network to extract a second feature; cascading the first and second features using the cascaded layer to obtain a first hidden layer feature vector; randomly deactivating the first hidden layer feature vector using the Dropout layer to obtain a second hidden layer feature vector; and processing the second hidden layer feature vector using the fully connected layer to obtain the extracted feature vector. The feature extractor of this invention includes a first branch network and a second branch network, which adopts a dual-stream heterogeneous structure to ensure the accuracy of feature extraction. At the same time, a Dropout layer is used after the two branch networks and the cascaded layer to randomly deactivate the feature vector of the first hidden layer, ensuring the generalization of the feature extractor.

[0042] According to one embodiment of the present invention, the feature extractor is a trained feature extractor, which is obtained through the following steps a1, a2, and a3:

[0043] In step a1, the constructed neural network classification model is obtained, which includes a feature extractor and a classifier. The structure of the feature extractor in this neural network classification model is as follows: Figure 2 The structure of the first feature extractor in China.

[0044] In step a2, a first training set is obtained. The first training set includes multiple first sample data and a label corresponding to each first sample data. The first sample data is IMU human motion data collected when a human performs a certain action. The label indicates the human motion category corresponding to the first sample data.

[0045] Because infrastructure is designed for specific state change scenarios, and datasets for infrastructure within the network are scarce and of limited quality, this invention uses a dataset of human atomic movements to train a neural network classification model to learn the representations of sensors during continuous posture changes. Therefore, according to one embodiment of the invention, the first training set uses the UTD-MHAD dataset (Multimodal Human Motion Recognition Dataset), which contains 27 different human atomic movements. This dataset essentially includes the posture changes of inertial sensors (accelerometers, gyroscopes, and magnetometers) in various directions. This invention utilizes existing datasets, eliminating the need for extensive data collection for specific infrastructure and its state scenarios, thus reducing costs and improving efficiency. Furthermore, human motion data is inherently more complex than infrastructure state changes, resulting in a neural network classification model trained on human motion data exhibiting better generalization ability. The trained feature extractor obtained from this neural network classification model can better extract feature vectors from IMU data and perform state recognition based on the extracted feature vectors, significantly improving the accuracy of infrastructure state recognition.

[0046] In step a3, the neural network classification model is trained using the first training set to identify human action categories based on the first sample data. (See [link to previous step]). Figure 3 , Figure 3 This diagram illustrates the process of feature extraction and classification of input IMU human motion data for training a neural network classification model. The training feature extractor extracts features from the first sample data (i.e., IMU human motion data) in the first training set to obtain feature vectors. The training classifier outputs the human motion category based on the extracted feature vectors. The first loss is determined based on the label of the first sample data and the output human motion category using the additive angle margin loss function. The parameters of the feature extractor and the classifier are updated based on the first loss.

[0047] According to one embodiment of the present invention, the formula for calculating the additive angular margin loss function is as follows:

[0048]

[0049] Where, loss Arcface Let N represent the first loss, N be the total amount of the first sample data used in training, s be the scaling factor used to control the length of the feature vector, and θ be the scaling factor. yi The feature vector output by the first sample data in the FC (fully connected) layer and y i The angle between the weight vectors of the FC (fully connected) layer is y. i Let represent the label of the i-th first sample data, m be the margin, which is used to increase the inter-class distance after training to improve the generalization ability of the neural network classification model for unseen sample data, and n represent θ. j This represents the angle between the feature vector output by the data corresponding to category j in the FC (fully connected layer) and the weight vector corresponding to category j in the FC (fully connected layer), where j represents a category other than human action. Training stops and the parameters of the feature extractor and classifier are updated when the preset number of iterations is reached or the constructed neural network classification model converges, resulting in a trained feature extractor.

[0050] Example 2

[0051] See Figure 4 , Figure 4 This is a schematic diagram of the structure of an autoencoder including a feature extractor and a decoder. Both the feature extractor and decoder include multi-layer convolutional neural networks. In this invention, both the feature extractor and decoder are configured with three-layer convolutional neural networks. After training, the feature extractor in the autoencoder is used to extract the feature vector corresponding to the infrastructure from the IMU data. Specifically, during the training of the feature extractor in the autoencoder, the IMU data is input into the feature extractor for feature encoding to obtain the feature vector. The decoder decodes the feature vector to obtain the data restored by the decoder. Based on the original input data and the decoded data, a second loss is calculated using the mean squared error loss function, and the autoencoder parameters are updated. The feature extractor in the autoencoder can be obtained through the following training steps b1, b2, and b3:

[0052] In step b1, the constructed autoencoder is obtained, which includes a feature extractor and a decoder, both of which include multi-layer convolutional neural networks.

[0053] In step b2, a second training set is obtained, which includes multiple second sample data, each of which is a segment of IMU data collected by an inertial measurement unit deployed on the infrastructure at predetermined time intervals.

[0054] In step b3, the feature extractor is trained using the second training set to encode the features of the second sample data to obtain a feature vector. The feature vector is then decoded by the decoder to obtain decoded data. Based on the mean squared error loss function, the second loss is determined according to the second sample data and the decoded data, and the parameters of the feature extractor and the decoder are updated according to the second loss.

[0055] According to an embodiment of the present invention, the formula for calculating the Mean Squared Error Loss function is as follows:

[0056]

[0057] Where, loss MSE This represents the second loss, where N is the total amount of the second sample data used in training, and y i For the i-th IMU data segment input, The decoder then reconstructs the decoded data output by the decoder. Training continues until a preset number of iterations is reached or the constructed autoencoder converges. At this point, the parameters of the feature extractor and decoder are updated, resulting in a trained second feature extractor.

[0058] Example 3

[0059] See Figure 5 , Figure 5 This is a schematic diagram illustrating the structural principle of the third type of feature extractor. The feature extractor includes a low-pass filter and a statistical feature module. The extracted feature vector is obtained as follows: The low-pass filter is used to filter IMU data collected by inertial measurement units deployed on the infrastructure to obtain IMU signal features; the statistical feature module performs statistical processing on the IMU data collected by the inertial measurement units deployed on the infrastructure to obtain statistical features, including mean, variance, zero-crossing rate, skewness, kurtosis, and upper quartiles; the feature extractor then concatenates the IMU signal features and statistical features to obtain the extracted feature vector. This third type of feature extractor in the present invention concatenates the IMU signal features obtained by filtering the original IMU data and the statistical features obtained by performing feature statistics on the original IMU data to obtain a feature vector that includes multiple data features reflecting the original IMU data, ensuring the accuracy of infrastructure status identification.

[0060] In step S2, the similarity between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of the various states preset for the infrastructure is calculated to obtain the similarity result.

[0061] According to one embodiment of the present invention, for different types of infrastructure, there are two or more preset reference feature vectors corresponding to each type of infrastructure. Each type of infrastructure is one of the following: door, window, robotic arm, drawer, bed, curtain, water pipe, printer, fitness equipment, vent, and toilet. Illustratively, the preset states corresponding to the doors, windows, robotic arms, drawers, beds, curtains, water pipes, printers, fitness equipment, vents, and toilets all include open and closed states. For example, the states of a door are open and closed, the states of a robotic arm are power-on and power-off, and the states of a curtain are unfolded and retracted. Each state corresponding to each type of infrastructure can be preset according to user needs; that is, the user configures the corresponding state reference feature vector for that infrastructure themselves. It should be understood that, depending on the needs of different implementers, the preset states of some infrastructures can be customized and adjusted, and are not necessarily limited to the preset states given in the foregoing illustrative embodiment. For example, the preset states for a robotic arm include raising, lowering, moving left, moving right, moving forward, moving backward, and rotating; the preset states for a bed include moving left, moving right, raising the bed frame, and lowering the bed frame; the preset states for a printer include printing and scanning; and the preset states for a toilet include flushing, opening the lid, and closing the lid.

[0062] According to one embodiment of the present invention, the execution actions of the same state of infrastructure may have deviations in speed and completion degree, and there are also deviations between the execution actions of different states. Therefore, regardless of whether the user wants to monitor several states or different execution actions of the same state, multiple reference feature vectors can be recorded. When it is necessary to detect different completion degrees, different completion speeds, etc. of the same state, multiple reference feature vectors need to be recorded, and these multiple base station feature vectors all represent the same state. For example, the preset states of a robotic arm include raising and lowering states, and the speed of raising and lowering can be fast or slow. Therefore, reference feature vectors at different speeds can be preset for the raising state of the infrastructure. Each reference feature vector at each speed indicates that the robotic arm is in the raising state, further improving the versatility and accuracy of the infrastructure state identification method.

[0063] According to one embodiment of the present invention, the method of calculating the similarity between the extracted feature vector corresponding to the infrastructure and the benchmark feature vectors of multiple preset states for the infrastructure includes: Method 1: Calculating the similarity between the extracted feature vector corresponding to the infrastructure and the benchmark feature vector of one of the multiple preset states for the infrastructure, obtaining a similarity result. Based on the similarity result, determining whether the extracted feature vector is similar to the benchmark feature vector of that state. If similar, determining that the current state of the infrastructure corresponds to the benchmark feature vector similar to its feature vector, and stopping the similarity calculation between the extracted feature vector and the benchmark feature vectors of other states in the multiple states. If not similar, continuing the similarity calculation between the extracted feature vector and the benchmark feature vectors of other states in the multiple states until a benchmark feature vector similar to the feature vector corresponding to the infrastructure is matched, thereby determining the state of the infrastructure. This method calculates and compares the extracted feature vector corresponding to the infrastructure with the benchmark feature vectors of multiple preset states one by one. When two feature vectors are similar, the calculation and comparison are stopped, which reduces the computational load to a certain extent and improves the recognition efficiency. Method 2 involves calculating the similarity between the extracted feature vector corresponding to the infrastructure and the baseline feature vector for each of the preset states for that infrastructure. Multiple similarity results are obtained, each representing the degree of similarity between the extracted feature vector and the baseline feature vector for each preset state. From these multiple similarity results, the baseline feature vector with the highest similarity to the extracted feature vector is selected to determine the infrastructure's state. This method compares the extracted feature vector with the baseline feature vectors for all preset states, selecting the baseline feature vector with the highest similarity, thus improving the accuracy of infrastructure state identification. It should be understood that the above two methods are for illustrative purposes only, and users can choose according to their actual needs.

[0064] According to one embodiment of the present invention, the similarity calculation employs methods including but not limited to calculating the cosine distance, Euclidean distance, Manhattan distance, Pearson correlation coefficient, or Spearman correlation coefficient between the extracted feature vector of the infrastructure and the baseline feature vector of the corresponding preset state for the infrastructure. A similarity result representing the degree of similarity between the two feature vectors is obtained based on the calculated cosine distance, Euclidean distance, Manhattan distance, Pearson correlation coefficient, or Spearman correlation coefficient. The similarity calculation also includes directly performing similarity matching calculations on sensor data using methods such as dynamic time warping similarity matching algorithms. For example, the third feature extractor given in Example 3 extracts the feature vector corresponding to the infrastructure from the collected IMU data. The extracted feature vector includes IMU signal features and statistical features. The baseline feature vector for each of the multiple preset states for the infrastructure includes the baseline IMU signal features and baseline statistical features for the corresponding state. The dynamic time warping similarity matching algorithm is used to calculate the similarity between the IMU signal features extracted in Example 3 and the baseline IMU signal features, and the cosine similarity algorithm is used to calculate the similarity between the statistical features extracted in Example 3 and the baseline statistical features. When users have low requirements for the accuracy of the detection action, the current infrastructure status can be determined when either of the two similarity results reaches the threshold condition set by the user.

[0065] In step S3, the current state of the infrastructure is determined based on the similarity results.

[0066] According to an embodiment of the present invention, the method for determining the current state of the infrastructure based on the similarity calculation in step S2 is as follows: each time, the similarity result between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of one of the preset multiple states is compared with a preset threshold. When the similarity result is greater than the preset threshold, the preset state corresponding to the similarity result is taken as the current state of the infrastructure. Specifically, when the similarity result is greater than a preset threshold, the extracted feature vector is considered similar to the baseline feature vector of the corresponding state; otherwise, the two feature vectors are not similar. For example, if the preset states for infrastructure include state one, state two, and state three, the extracted feature vector corresponding to the infrastructure is compared with the preset baseline feature vectors of the various states one by one. That is, the cosine similarity calculation is performed between the extracted feature vector and the baseline feature vector of state one, and the similarity result is 0.8. The preset threshold is set to 0.95, so the extracted feature vector is not similar to the baseline feature vector of state one. The similarity calculation is then performed between the extracted feature vector and the baseline feature vector of state two, and the similarity result is 0.96. 0.96 is greater than the preset threshold of 0.95, which means that the extracted feature vector is similar to the baseline feature vector of state two. The calculation and comparison with the baseline feature vector corresponding to state three is stopped, and state two is taken as the current state of the infrastructure. The preset threshold is derived from empirical settings that achieve the best recognition effect for the current state of the infrastructure. In actual use, users can set the threshold to match different recognition accuracies. For example, when anomaly detection requires higher accuracy, a higher threshold needs to be set to ensure that the actual infrastructure state and the expected state are highly matched, which provides high flexibility.

[0067] According to one embodiment of the present invention, the method for determining the current state of the infrastructure based on the similarity calculation in step S2 is as follows: The extracted feature vector corresponding to the infrastructure is compared with the baseline feature vector of each of the preset multiple states to obtain multiple similarity results. The preset state corresponding to the highest similarity result is selected from the multiple similarity results as the current state of the infrastructure. Taking the feature vector obtained by the first feature extractor given in Example 1 above as an example, the feature vector corresponding to the infrastructure obtained by the first feature extractor is compared with the baseline feature vector of each of the preset multiple states for the infrastructure to perform cosine similarity calculation. There are a total of 6 preset states. The similarity between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of each of the 6 preset states is calculated, resulting in 6 similarity results: 0.5, 0.6, 0.4, 0.97, 0.91, and 0.8. The baseline feature vector corresponding to the highest similarity result of 0.97 is selected, and the current state of the infrastructure is determined to be the state indicated by the baseline feature vector. It should be understood that the present invention does not limit the method of determining the current state of the infrastructure, and users can choose the appropriate method to determine the current state of the infrastructure according to their needs.

[0068] According to an embodiment of the present invention, taking the feature vector obtained by the third feature extractor given in Example 3 above as an example, the similarity result obtained by the dynamic time warping similarity matching algorithm and the method of determining the state of infrastructure based on the similarity result include the following steps c1, c2 and c3:

[0069] In step c1, the dynamic time warping similarity matching algorithm is used to calculate the similarity between the filtered IMU signal features and the preset corresponding state reference IMU signal features to obtain the signal feature similarity result.

[0070] illustrative, see Figure 6 , Figure 6 A schematic diagram of the IMU signal feature sequence obtained by alignment filtering and the reference IMU signal feature sequence. For the sensor sequence A generated by the user-input target action and the sensor sequence B acquired at a certain time, when using dynamic time warping for similarity calculation, assuming the lengths of the IMU signal feature sequence Q and the reference IMU signal feature sequence C are n and m respectively, Q = q1,…,q i ,…,q n q1 is the first point in sequence Q, q i It is the i-th point in sequence Q, q n It is the nth point of sequence Q, C = c1,…,c j ,…,c m c1 is the first point in sequence C, cj It is the j-th point in sequence C, c m Let be the m-th point in sequence C. Construct an n*m matrix to align the two sequences. For example, each matrix element (i,j) represents the aligned point q. i and c j d(q) i ,c j ) represents the q corresponding to matrix element (i,j) i and c j The distance between two points, that is, the similarity between every point in sequence Q and every point in sequence C, is generally calculated using Euclidean distance, d(q i ,c j )=(q i -c j ) 2 The smaller the distance, the higher the similarity result; the larger the distance, the lower the similarity result. Based on the Euclidean distance corresponding to each element in the calculated matrix, a dynamic programming approach is used to find a continuous and monotonic shortest cumulative distance path in the matrix. The sum of the paths is the dynamic time-warped distance between the two feature sequences, which represents the final calculated signal feature similarity result.

[0071] In step c2, the obtained statistical features are compared with the preset baseline statistical features of the corresponding state to calculate the similarity, thus obtaining the statistical feature similarity result. The cosine distance between the obtained statistical features and the preset baseline statistical features of the corresponding state is calculated using cosine similarity. The larger the cosine distance, the higher the statistical feature similarity result; the smaller the cosine distance, the lower the statistical feature similarity result.

[0072] In step c3, when both the signal feature similarity result and the statistical feature similarity result are greater than a preset threshold, the infrastructure status is determined to be the preset corresponding status. This is configured so that the current infrastructure status is only determined when both the signal feature similarity result and the statistical feature similarity result reach the user-defined threshold conditions, thus improving the accuracy of status identification.

[0073] According to an embodiment of the present invention, an infrastructure status identification system based on IMU data is provided. The system is deployed on various infrastructures and includes: a sensor data acquisition module for acquiring IMU data of the infrastructure collected by inertial measurement units deployed on the infrastructure; a microcontroller module for determining the current status of the infrastructure based on the IMU data of the infrastructure according to the infrastructure status identification method of the above embodiment; and a communication module for transmitting the current infrastructure status determined by the microcontroller module to the user for status monitoring.

[0074] illustrative, see Figure 7 , Figure 7 This diagram illustrates a deployment scheme for an infrastructure status identification system, employing an Nrf52833 microcontroller with Bluetooth communication capabilities as the embedded chip. The embedded chip comprises three modules: a sensor data acquisition module, a microcontroller module, and a communication module. The algorithm (i.e., the infrastructure status identification method in the above embodiment) is deployed at the edge, specifically within the embedded chip. The embedded chip is packaged into a specific product form and fixed to the infrastructure, such as doors, windows, or industrial robotic arms, using adhesive or other methods. Users configure various baseline feature vectors for different infrastructure states based on the desired monitoring status. During infrastructure status monitoring, the sensor data acquisition module acquires IMU data collected by the inertial measurement unit (IMU) on the infrastructure. The microcontroller module performs feature extraction and similarity calculation on the IMU data to determine the current infrastructure state. The communication module transmits the current infrastructure state and also transmits bidirectional control commands. These bidirectional control commands primarily include remote invocation and configuration commands, which enable the server (such as a cloud service) to invoke functions within the embedded chip. Embedded chips can also remotely invoke data operations in servers (such as cloud services), provide user alerts for infrastructure anomalies via cloud services, and issue alarms for dangerous infrastructure conditions perceived at the edge. Specifically, model parameter quantization technology is used to deploy the model (i.e., the feature extractor in this invention). This involves using a strategy to convert model parameters from floating-point values ​​to fixed-point values, thereby reducing the model's parameter space. For example, the model quantization measurement uses the min-max quantization method, an offline quantization method implemented after model training. This method maps the model's floating-point parameter range layer by layer to an 8-bit fixed-point parameter range of -128 to 128, thus compressing the model to approximately 100KB in size, which is then deployed on the microcontroller module of the Nrf52833 embedded chip.

[0075] According to another embodiment of the present invention, an infrastructure status identification system based on IMU data is provided. The system is deployed on a smart gateway, a local host, a fog computing service platform, or a cloud computing service platform. A sensor data acquisition module and a communication module are provided in the infrastructure. The sensor data acquisition module is used to acquire IMU data collected by inertial measurement units deployed on the infrastructure. The communication module is used to transmit the acquired infrastructure IMU data to the system. The system includes: a microcontroller module, used to determine the current status of the infrastructure based on the infrastructure IMU data according to the infrastructure status identification method of the above embodiment; and a communication module, used to transmit the current infrastructure status determined by the microcontroller module to the user for status monitoring.

[0076] illustrative, see Figure 8 , Figure 8 This diagram illustrates another deployment scheme for an infrastructure status identification system. In this system, the algorithm (i.e., the infrastructure status identification method in the above embodiments) is deployed within a microcontroller module on a fog server (a regional server or a cluster of regional server nodes) or a cloud server (cloud service). This example involves installing an embedded chip on infrastructure equipped with an inertial measurement unit (IMU). The embedded chip is packaged into a specific product form and fixed to the infrastructure (e.g., doors, windows, industrial robotic arms) using adhesive or other methods. The embedded chip includes a sensor data acquisition module and a communication module. The user configures various reference feature vectors for different infrastructure states based on the desired monitoring status. The IMU then acquires the data. During infrastructure status monitoring, the sensor data acquisition module obtains the IMU data in real time and transmits it to the fog server or cloud server. The microcontroller module on the corresponding server performs feature extraction and similarity calculation to determine the infrastructure status. The determined current infrastructure status is then transmitted to the user via the communication module on the server for status monitoring.

[0077] illustrative, see Figure 9 , Figure 9 This diagram illustrates another deployment scheme for an infrastructure status identification system. The algorithm (i.e., the infrastructure status identification method in the above embodiment) is deployed in the microcontroller module of the gateway. An embedded chip at the edge acquires IMU data collected by the inertial measurement unit and uploads it to the gateway, which then performs local real-time infrastructure status identification. The edge-end embedded chip deployed in this example includes two modules: a sensor data acquisition module and a communication module. The embedded development board is packaged into a specific product form and fixed to the infrastructure, such as doors, windows, or industrial robotic arms, using adhesive or other fixing methods. The inertial measurement unit acquires IMU data. During infrastructure status monitoring, the sensor data acquisition module acquires IMU data in real time and transmits it to the gateway. The corresponding microcontroller module in the gateway performs feature extraction and similarity calculation to obtain the infrastructure status, which is then uploaded to a cloud server or fed back to the embedded chip. The determined current infrastructure status is then transmitted to the user via the cloud server for status monitoring.

[0078] According to an embodiment of the present invention, an infrastructure status identification and alarm device based on IMU data is provided, comprising: a sensor data acquisition module for acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure; a microcontroller module for determining the current status of the infrastructure based on the IMU data of the infrastructure according to the infrastructure status identification method of the above embodiment; and an early warning module for judging whether the status of the infrastructure is abnormal according to the current status of the infrastructure, and issuing an alarm when the status is abnormal.

[0079] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0080] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0081] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0082] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for infrastructure status identification based on IMU data, characterized in that, The method includes: Acquire IMU data collected by inertial measurement units deployed on the infrastructure, and use a feature extractor to extract the feature vector corresponding to the infrastructure from the IMU data; The similarity between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of the various states preset for the infrastructure is calculated to obtain the similarity result. Based on the similarity results, determine the current state of the infrastructure; For different types of infrastructure, there are two or more preset baseline feature vectors corresponding to each type of infrastructure. When it is necessary to detect different completion levels or different completion speeds of the same state, multiple baseline feature vectors need to be entered, and these multiple baseline feature vectors all represent the same state. Each type of infrastructure is one of the following: door, window, robotic arm, drawer, bed, curtain, water pipe, printer, fitness equipment, vent, and toilet.

2. The method according to claim 1, characterized in that, The feature extractor is obtained through the following training method: Obtain the constructed neural network classification model, which includes a feature extractor and a classifier; Obtain the first training set, which includes multiple first sample data and a label corresponding to each first sample data. The first sample data is IMU human motion data collected when a human performs a certain action, and the label indicates the human motion category corresponding to the first sample data. The neural network classification model is trained using the first training set to identify human action categories based on the first sample data. Specifically, the feature extractor is trained to extract features from the first sample data in the first training set to obtain feature vectors. The classifier is trained to output human action categories based on the extracted feature vectors. The first loss is determined based on the additive angle margin loss function according to the labels of the first sample data and the output human action categories. The parameters of the feature extractor and the classifier are updated based on the first loss.

3. The method according to claim 2, characterized in that, The feature extractor includes a first branch network, a second branch network, cascaded layers, a Dropout layer, and a fully connected layer. The first branch network comprises a multi-layer convolutional neural network, and the second branch network comprises a gated recurrent neural network. The extracted feature vector is obtained through the following method: The IMU data is convolved using a multi-layer convolutional neural network in the first branch network to extract the first feature. The gated recurrent neural network in the second branch network is used to process the IMU data and extract the second feature; The first and second features are concatenated using a cascaded layer to obtain the first hidden layer feature vector; The first hidden layer feature vector is randomly deactivated using the Dropout layer to obtain the second hidden layer feature vector; The feature vector of the second hidden layer is processed using a fully connected layer to obtain the extracted feature vector.

4. The method according to claim 1, characterized in that, The feature extractor is obtained through the following training method: Obtain the constructed autoencoder, which includes a feature extractor. Both the decoder and the feature extractor consist of multi-layer convolutional neural networks; Acquire a second training set, which includes multiple second sample data, each of which is a segment of IMU data collected by an inertial measurement unit deployed on the infrastructure at predetermined time intervals; The feature extractor is trained using the second training set to encode features in the second sample data to obtain feature vectors. The feature vectors are then decoded using a decoder to obtain decoded data. The second loss is determined based on the mean squared error loss function, using the second sample data and the decoded data. The parameters of the feature extractor and decoder are then updated based on the second loss.

5. The method according to claim 1, characterized in that, The feature extractor includes a low-pass filter and a statistical feature module, wherein the extracted feature vector is obtained in the following manner: The IMU signal characteristics are obtained by filtering the IMU data collected by the inertial measurement units deployed on the infrastructure using a low-pass filter. The statistical feature module performs statistical processing on the IMU data collected by the inertial measurement units deployed on the infrastructure to obtain statistical features, including mean, variance, zero-crossing rate, skewness, kurtosis, and upper quartile. The feature extractor concatenates IMU signal features and statistical features to obtain the extracted feature vector.

6. The method according to claim 1, characterized in that, The current state of the infrastructure is determined as follows: Each time, the similarity result between the extracted feature vector corresponding to the infrastructure and the baseline feature vector of one of the preset multiple states is compared with a preset threshold. When the similarity result is greater than the preset threshold, the preset state corresponding to the similarity result is taken as the current state of the infrastructure. or The extracted feature vector corresponding to the infrastructure is compared with the baseline feature vector of each of the preset multiple states to obtain multiple similarity results. The preset state with the highest similarity result is selected from the multiple similarity results as the current state of the infrastructure.

7. The method according to claim 5, characterized in that, The baseline feature vector for each of the preset multiple states includes the baseline IMU signal features and baseline statistical features for the corresponding state. The state of the infrastructure is determined as follows: The similarity between the filtered IMU signal features obtained by the dynamic time warping similarity matching algorithm and the preset corresponding state reference IMU signal features is calculated to obtain the signal feature similarity result. The obtained statistical features are compared with the baseline statistical features of the corresponding predefined states to calculate the similarity, and the statistical feature similarity results are obtained. When both the signal feature similarity result and the statistical feature similarity result are greater than the preset threshold, the state of the infrastructure is determined to be the preset corresponding state.

8. An infrastructure status identification system based on IMU data, the system being deployed on various infrastructures, characterized in that, The system includes: The sensor data acquisition module is used to acquire IMU data of the infrastructure collected by the inertial measurement unit deployed on the infrastructure; A microcontroller module is configured to determine the current state of the infrastructure based on IMU data of the infrastructure according to any one of claims 1-7; The communication module is used to transmit the current infrastructure status determined by the microcontroller module to the user for status monitoring.

9. An infrastructure status identification system based on IMU data, characterized in that, The system is deployed on a smart gateway, local host, fog computing service platform, or cloud computing service platform. The infrastructure includes a sensor data acquisition module and a communication module. The sensor data acquisition module acquires IMU data collected by the inertial measurement unit deployed on the infrastructure, and the communication module transmits the acquired infrastructure IMU data to the system. The system includes: A microcontroller module is configured to determine the current state of the infrastructure based on the infrastructure IMU data according to any one of claims 1-7; The communication module is used to transmit the current infrastructure status determined by the microcontroller module to the user for status monitoring.

10. An infrastructure status identification and alarm device based on IMU data, characterized in that, include: The sensor data acquisition module is used to acquire IMU data collected by the inertial measurement units deployed on the infrastructure; A microcontroller module is configured to determine the current state of the infrastructure based on IMU data of the infrastructure according to any one of claims 1-7; The early warning module is used to determine whether the infrastructure is in an abnormal state based on its current status, and to issue an alarm when the state is abnormal.

11. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

12. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

Images