An infrastructure state recognition method and system based on inertial sensor data

Abstract

The application provides a kind of infrastructure state identification method and system based on inertial sensor data, the method comprises: obtaining the sensing data collected by the inertial sensor deployed on the infrastructure and the previous state sequence of the infrastructure before the sensing data is collected;Determine the current state of the infrastructure according to the sensing data and the previous state sequence using a trained state identification model, wherein the trained state identification model includes two prediction modes, the first prediction mode is to predict according to the sensing data and the corresponding previous state sequence, and the second prediction mode is to predict according to the mask value of the sensing data and the corresponding previous state sequence;Determine whether the current state of the infrastructure needs to be calibrated using a calibrator, and if so, update the current state of the infrastructure with the state of the infrastructure predicted by the trained state identification model in the second prediction mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital signal processing, specifically to the field of combination of Internet of Things and neural network, and more particularly to an infrastructure state recognition and system based on inertial sensor data. BACKGROUND

[0002] With the rapid development of Internet of Things technology, the application of infrastructure state recognition can be seen everywhere in life, especially in the fields of smart home, industrial internet, fire safety, and smart elderly care. However, the detection of infrastructure often requires customized embedded sensors and recognition algorithms for specific use scenarios, resulting in increased costs and barriers between sub-applications, which greatly hinders the popularization of intelligent sensing technology.

[0003] In this context, many sensing technologies have been proposed, including contact sensing technology and non-contact sensing technology.

[0004] Non-contact sensing technology uses radar, radio frequency identification (RFID), ultra-wideband (UWB), and other radio frequency technologies to collect echo signals carrying state characteristics of the detected object through backscattering and multipath effects of radio waves on the detected object, to realize the recognition of the state of the object. However, this technology has the problems of being limited by scene specificity, high cost, and difficulty in deployment, and can only be implemented in specific scenarios, making it difficult to be applied on a large scale.

[0005] Contact sensing technology is mostly based on inertial sensors, which have the characteristics of low power consumption and low cost due to the use of MEMS technology. Inertial sensors usually integrate accelerometers, gyroscopes, and magnetometers, or integrate accelerometers and gyroscopes. They can represent the attitude change and direction of the detected facility and are widely used for detecting object states, such as home state detection, facility attitude detection, vehicle impact detection, and fall detection. The detection technology is mostly based on sensor data to calculate the attitude and direction, and the state is recognized through changes in intensity or direction.

[0006] With the development of machine learning technology, more versatile machine learning methods have been gradually evolved, from traditional extraction of semantic attitude and direction features to parameterized model iteration training of non-semantic features for state recognition according to specific scenarios. However, the sensing data of inertial sensors has limitations, and the accuracy of some detected states is not high when the state of the infrastructure is determined only based on the current sensing data of the infrastructure. SUMMARY

[0007] Therefore, the purpose of the present application is to overcome the defects of the prior art and provide an infrastructure state recognition method and system based on inertial sensor data.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] According to a first aspect of the present application, an infrastructure state recognition method based on inertial sensor data is provided, comprising: acquiring sensor data collected by an inertial sensor deployed on an infrastructure and a sequence of previous states of the infrastructure before the sensor data is collected; determining the current state of the infrastructure according to the sensor data and the sequence of previous states using a trained state recognition model, wherein the trained state recognition model includes two prediction modes, a first prediction mode in which prediction is performed according to sensor data and a corresponding sequence of previous states, and a second prediction mode in which prediction is performed according to a mask value of sensor data and a corresponding sequence of previous states; determining whether the current state of the infrastructure needs to be calibrated using a calibrator, and if so, updating the current state of the infrastructure with the state of the infrastructure predicted by the trained state recognition model in the second prediction mode.

[0010] Optionally, the trained state recognition model is trained in the following manner: a first training set is acquired, which includes a plurality of samples and a corresponding label for each sample, each sample including sensor data of an infrastructure and a sequence of previous states of the infrastructure before the sensor data is collected, and each label indicating the annotated state of the corresponding infrastructure when the sensor data contained in the sample is collected; a constructed state recognition model is acquired, which includes a feature extractor for extracting a feature vector from the sensor data and the sequence of previous states, and a prediction layer for recognizing the state of the infrastructure according to the feature vector; and the state recognition model is trained in the first prediction mode or the second prediction mode alternately using the first training set.

[0011] Optionally, when the state recognition model is trained in the first prediction mode, the sensor data of the infrastructure in the sample and the corresponding sequence of previous states are input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined based on the output state of the infrastructure and the corresponding label; when the state recognition model is trained in the second prediction mode, the sensor data of the infrastructure in the sample and the mask value of the corresponding sequence of previous states are input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined based on the output state of the infrastructure and the corresponding label.

[0012] Optionally, the feature extractor comprises: a first branch for extracting features from three-axis acceleration data in the sensor data to obtain acceleration features; a second branch for extracting features from three-axis angular velocity data in the sensor data to obtain angular velocity features; a previous sequence branch for extracting features from the previous state sequence to obtain previous state features; and a fusion layer for fusing the acceleration features, the angular velocity features and the previous state features to obtain the feature vector.

[0013] Optionally, the first branch performs acceleration fusion on the three-axis acceleration data through a three-channel 1D convolution layer and filters the fusion result using a mean operator to obtain the acceleration features; and the second branch performs angular velocity fusion on the three-axis angular velocity data through a single-layer RNN network and filters the fusion result using a Sobel operator to obtain the angular velocity features.

[0014] Optionally, the feature extractor comprises: a first branch for extracting features from three-axis acceleration data in the sensor data to obtain acceleration features; a second branch for extracting features from three-axis angular velocity data in the sensor data to obtain angular velocity features; a third branch for extracting features from three-axis magnetometer data in the sensor data to obtain magnetic force features; and a fusion layer for fusing the acceleration features, the angular velocity features, the magnetic force features and the previous state features to obtain the feature vector.

[0015] Optionally, the feature extractor comprises: a backbone neural network configured to extract intermediate features in the form of a multi-dimensional vector from the sensor data using an Inception network; and an attention module based on the previous state sequence, comprising: a fully connected layer configured to extract an attention value of each dimension of the intermediate features from the previous state sequence, and a fusion layer configured to weight a value of each dimension of the intermediate features according to the attention value of each dimension to obtain a weighted intermediate feature, and fuse the weighted intermediate feature and the intermediate feature to obtain the feature vector.

[0016] Optionally, the trained calibrator is trained in the following manner: using the trained state recognition model to output a confidence of each state of the infrastructure in the second prediction mode for each sample in the first training set, extracting a confidence of a labeled state from the confidence of each state as a second label of the sample according to a first label corresponding to the sample; and obtaining a calibrator, the calibrator being a regression model, and using the sensor data of the infrastructure in the sample of the first training set and the corresponding second label to perform regression training on the calibrator.

[0017] Optionally, when the confidence output by the trained calibrator on the sensor data is greater than or equal to a confidence threshold, it is determined that the current state of the infrastructure needs to be calibrated.

[0018] Optionally, at least two preset states corresponding to different kinds of infrastructure are provided, and each kind of infrastructure is one of a door, a window, a water pipe, a drawer, a bed, a curtain, a ventilation port, an electrical appliance, fitness equipment, an industrial production device, and a printer.

[0019] According to a second aspect of the present application, there is provided an infrastructure state monitoring system, comprising: one or more information collection devices, each of which is capable of being deployed on an infrastructure to be monitored and comprises an inertia sensor-based data collection module and a communication module; and a state monitoring device configured to interact with the communication module of the information collection device to obtain sensor data of the infrastructure collected by the inertia sensor of the corresponding infrastructure, and to identify the state of one or more infrastructures to be monitored based on an initialized sequence of previous states and the method of the first aspect.

[0020] According to a third aspect of the present application, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is configured to store executable instructions; and the one or more processors are configured to implement the steps of the method of the first aspect by executing the executable instructions.

[0021] Compared with the prior art, the present application has the following advantages:

[0022] The embodiment of the present application not only acquires sensor data, but also acquires a sequence of previous states of the infrastructure before the sensor data is collected, and determines the current state of the infrastructure according to the sensor data and the sequence of previous states by using a trained state recognition model; because the sequence of previous states of the infrastructure before the sensor data is collected is added, the accuracy of infrastructure state prediction can be improved. At the same time, the inventor has found through testing that although the accuracy of infrastructure state prediction is improved in some cases after the sequence of previous states is added, in a small number of cases, state prediction only relying on sensor data is more accurate. Therefore, the trained state recognition model comprises two prediction modes, a first prediction mode in which prediction is performed according to sensor data and a corresponding sequence of previous states, and a second prediction mode in which prediction is performed according to sensor data and a mask value of a corresponding sequence of previous states; in addition, a calibrator is provided for determining whether the current state of the infrastructure needs to be calibrated according to the sensor data, and if so, the state of the infrastructure predicted by the trained state recognition model in the second prediction mode is used to update the current state of the infrastructure. Because prediction in the second prediction mode is performed according to sensor data and a mask value of a corresponding sequence of previous states (equivalent to shielding the influence of the sequence of previous states), the state predicted only according to sensor data can be calibrated without changing the structure of the trained state recognition model or setting another model, thereby further improving the overall accuracy of infrastructure state recognition. Attached Figure Description

[0023] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0024] Figure 1 This is a schematic diagram illustrating the operating principle of the state recognition model and calibrator according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a feature extractor according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of another embodiment of the feature extractor according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of a state recognition model according to an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram illustrating a principle of a calibrator according to an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram illustrating another embodiment of the calibrator according to an embodiment of the present invention;

[0030] Figure 7 This is an application diagram according to an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram illustrating another application according to an embodiment of the present invention;

[0032] Figure 9 This is another application diagram according to an embodiment of the present invention;

[0033] Figure 10 This is a schematic diagram showing the results of an ablation experiment according to an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0035] As mentioned in the background section, inertial sensor data has limitations. When determining the state of infrastructure solely based on current infrastructure sensor data, the accuracy of the detected state at certain moments is not high. To address these issues, see [link to relevant documentation]. Figure 1This invention not only acquires sensor data but also acquires the preceding state sequence of the infrastructure before the sensor data was acquired. A trained state recognition model is then used to determine the current state of the infrastructure based on the sensor data and the preceding state sequence. Because the preceding state sequence of the infrastructure before the sensor data was acquired is added, the accuracy of infrastructure state prediction can be improved. However, the inventors found through testing that while adding the preceding state sequence sometimes improves the accuracy of infrastructure state prediction, in some cases, relying solely on sensor data for state prediction is more accurate. Therefore, this invention includes two prediction modes for the trained state recognition model: a first prediction mode based on sensor data and the corresponding preceding state sequence, and a second prediction mode based on the mask value of the sensor data and the corresponding preceding state sequence. Additionally, a calibrator is provided to determine whether the current state of the infrastructure needs calibration based on the sensor data. If so, the current state of the infrastructure is updated by the state predicted by the trained state recognition model in the second prediction mode. Since the second prediction mode makes predictions based on sensor data and the mask value of the corresponding preceding state sequence (which is equivalent to shielding the influence of the preceding state sequence), the state prediction based solely on sensor data can be calibrated without changing the structure of the trained state recognition model or setting up a separate model, thereby further improving the overall accuracy of infrastructure state recognition.

[0036] To illustrate the technical solution of this invention, the following description will cover the aspects of inertial sensors, state recognition models, calibrators, and corresponding application scenarios.

[0037] I. Sensor Data

[0038] According to one embodiment of the present invention, the sensing data utilized in this embodiment is sensing data acquired by an inertial sensor. The inertial sensor can be an existing sensor, such as a six-axis sensor or a nine-axis sensor. A six-axis sensor includes a three-axis accelerometer and a three-axis angular velocity sensor. A nine-axis sensor includes a three-axis accelerometer, a three-axis angular velocity sensor, and a three-axis magnetometer. The sensor can be selected as needed. For example, for infrastructure where component movement causes a change in state, a three-axis accelerometer and a three-axis angular velocity sensor can be used; for example, doors, windows, water pipes, drawers, beds, vents, curtains, etc. As another example, for infrastructure where the state changes due to whether it is powered on or off, a nine-axis sensor can be used, and the magnetometer data in the sensing data can be given greater weight (e.g., signal amplification); for example, electrical appliances, fitness equipment, industrial production equipment, and printers. It should be understood that implementers can adaptively select the appropriate inertial sensor based on experience, cost, or other factors, and this embodiment does not impose any limitations in this regard. For example, if the implementer believes that a product needs to be produced that can be used regardless of whether it is placed on an infrastructure with or without electricity, then a nine-axis sensor can be set in the product. If the inertial sensor of the product is used on a window to determine whether the window is open or closed, the sensor data can be partially utilized, such as by not using or shielding the data from the magnetometer. If it is placed on an appliance to determine whether the appliance is open or closed, the nine-axis sensor data can be fully utilized.

[0039] II. State Recognition Model

[0040] The following explanation will focus on the states, structures, datasets, and model training aspects of the state recognition model.

[0041] 2.1 States identified by the state recognition model

[0042] According to one embodiment of the present invention, the state to be identified by the state recognition model needs to be determined based on the infrastructure and the corresponding state of the infrastructure that the implementer needs to identify. The infrastructure can be one of the following: door, window, water pipe, drawer, bed, curtain, vent, electrical appliance, fitness equipment, industrial production device, and printer. Different infrastructures can be configured with corresponding state recognition models for state recognition. Illustratively, in some scenarios, the identification can be of the open or closed state of the corresponding infrastructure given here. For example, whether a window is open or closed; whether a water pipe is open (corresponding to a state where water flows through the pipe) or closed (corresponding to a state where water does not flow through the pipe). Of course, the implementer can also define other implementation methods to identify other states. For example, the industrial production device can be in a normal operating or abnormal state. Furthermore, the implementer can define more granular states; for example, for fitness equipment, the implementer can set the states to be identified as: open (unused), open (used), and closed.

[0043] 2.2 Structure of the State Recognition Model

[0044] According to one embodiment of the present invention, the state recognition model includes a feature extractor for extracting feature vectors from sensor data and preceding state sequences, and a prediction layer for recognizing the state of infrastructure based on the feature vectors.

[0045] According to one embodiment of the present invention, the feature extractor includes: a first branch for extracting features from triaxial acceleration data in sensing data to obtain acceleration features; a second branch for extracting features from triaxial angular velocity data in sensing data to obtain angular velocity features; a preceding branch for extracting features from a preceding state sequence to obtain preceding state features; and a fusion layer for fusing the acceleration features, angular velocity features, and preceding state features to obtain a feature vector. Optionally, both the first branch and the second branch in this embodiment can employ a CNN network for data processing.

[0046] The inventors improved the feature processor for six-axis inertial sensor processing through testing to achieve better feature extraction results. According to one embodiment of the invention, see... Figure 2 The first branch passes through a three-channel 1D convolutional layer on the three axes (e.g., corresponding to...). Figure 2 The acceleration data (X, Y, Z axes) shown are fused to obtain the total acceleration direction, and the fused acceleration data is filtered using a mean operator (a low-pass filtering method) to obtain the result of the first branch (i.e., acceleration features); the second branch processes the three axes (e.g., corresponding to...) using a single-layer RNN network. Figure 2 The X, Y, and Z axis angular velocity data shown are fused to obtain attitude information, and the fused angular velocity data is filtered using the Sobel operator (a high-pass filtering method) to obtain the result of the second branch processing (i.e., angular velocity features). This embodiment achieves at least the following beneficial technical effects: It utilizes the attitude information carried by acceleration and the attitude information carried by gyroscopes for complementary fusion, enabling the extraction of motion attitude-related features of infrastructure using a small number of trainable parameters, exhibiting extremely lightweight characteristics compared to traditional deep learning methods.

[0047] According to another embodiment of the present invention, the feature extractor includes: a first branch for processing triaxial acceleration data in the sensing data; a second branch for processing triaxial angular velocity data in the sensing data; a third branch for processing triaxial magnetometer data in the sensing data; and a fusion layer for fusing the results of the processing by the first, second, and third branches to obtain a feature vector. In this embodiment, the first, second, and third branches can all employ a three-channel 1D convolutional layer (or a stacked structure of 1D convolutional layers and pooling layers) to process the triaxial data respectively. Alternatively, the first and second branches can employ... Figure 2 The structure shown has a third branch that uses a three-channel 1D convolutional layer (or a stacked structure of 1D convolutional layers and pooling layers) to process the data on the three axes respectively.

[0048] Other structures can also be used to further improve the feature extraction effect, such as adding an attention mechanism to the feature extractor to improve the accuracy of state recognition. According to one embodiment of the present invention, see... Figure 3 The feature extractor includes: a backbone neural network that uses an Inception network to extract intermediate features in the form of multi-dimensional vectors from sensor data; and an attention module based on a preceding state sequence, comprising: a fully connected layer for extracting attention values ​​for each dimension of the intermediate features from the preceding state sequence, and a fusion layer for weighting the values ​​of each dimension of the intermediate features according to the attention values ​​of each dimension, and fusing the weighted intermediate features and the intermediate features to obtain a feature vector. The Inception network in this embodiment can use Inception V1, V2, V3, or V4 models. In addition, other structures can be added to form different implementations. For example, one, two, or more fully connected layers can be added between the fusion layer and the prediction layer to perform feature processing on the output feature vector of the fusion layer to obtain the final feature vector. This embodiment achieves at least the following beneficial technical effects: by using the preceding state sequence to calculate attention values ​​to weight the intermediate features, the model can pay more attention to certain features based on the preceding states, and finally perform the final infrastructure state inference, thereby improving the accuracy of the model in state recognition.

[0049] In addition to the embodiments described above, other structures can also be used to implement the feature extractor. According to one embodiment of the present invention, see... Figure 4 The feature extractor includes a hybrid encoding layer for hybrid encoding of the sensor data and the preceding state sequence to obtain hybrid features, and passes the hybrid features through a convolutional layer and a pooling layer (or multiple stacked convolutional and pooling layers). Figure 4 (Not shown) and after processing by the fully connected layer, the feature vector is obtained.

[0050] According to one embodiment of the present invention, the prediction layer can adopt an existing structure, for example, the prediction layer includes a fully connected layer and a softmax function. The fully connected layer of the prediction layer is used to adjust the dimension of the feature vector to obtain predicted features, and the softmax function is used to output the confidence scores of each state of the infrastructure based on the predicted features. Alternatively, the prediction layer can use only the softmax function (in this case, a fully connected layer needs to be set in the feature extractor to adjust the vector dimension of the feature vector, so that it is suitable for directly using the softmax function to make predictions based on the feature vector output by the feature extractor).

[0051] Furthermore, any state recognition model formed by combining the above embodiments has two prediction modes. In the first prediction mode, prediction is performed based on sensor data and the corresponding preceding state sequence. In the second prediction mode, prediction is performed based on sensor data and the mask value of the corresponding preceding state sequence. The mask value of the preceding state sequence can be implemented by adding a corresponding mask layer to the preceding state sequence before the feature extractor. The mask layer contains a mask vector with the same length as the preceding state sequence. The product of each dimension of the preceding state sequence and each dimension of the mask vector is used as the actual input preceding state sequence to the feature extractor. In the first prediction mode, each dimension of the mask vector is 1, meaning the preceding state sequence is not masked. In the second prediction mode, each dimension of the mask vector is 0, meaning the preceding state sequence is masked, thus obtaining the mask value of the preceding state sequence. Alternatively, this mask layer can also be set in the feature extractor. Alternatively, an additional zeroing module can be set to zero the corresponding preceding state sequence when the state recognition model needs to predict the state of the infrastructure in the second prediction mode.

[0052] 2.3 Dataset

[0053] Before a state recognition model can be put into use, it needs to be trained using a dataset.

[0054] According to one embodiment of the present invention, the dataset includes multiple samples and a label corresponding to each sample. Each sample includes sensing data of the infrastructure and a sequence of preceding states of the infrastructure before the sensing data was collected. Each label indicates the labeled state of the corresponding infrastructure when the sensing data contained in its sample was collected. The preceding state sequence may include preceding states at one or more time points. For example, assuming the preceding state sequence requires 3 (or other values, such as 6, 9, 20, etc.) preceding states, the preceding state sequence can be 0,0,1; where 0 can represent on, 1 represents off, or vice versa. The dataset can be created by the implementer or use a dataset created by others.

[0055] Once the dataset is created, as those skilled in the art know, it can be divided into a training set, a validation set, and a test set in a preset ratio (e.g., 7:2:1); or it can be divided into a training set and a test set in a preset ratio (e.g., 8:2). Here, the training set is the first training set used to train the state recognition model.

[0056] 2.4 Model Training

[0057] According to one embodiment of the present invention, the trained state recognition model is trained as follows: obtaining a first training set; obtaining the constructed state recognition model; and training the state recognition model alternately in a first prediction mode or a second prediction mode using the first training set. Preferably, when training the state recognition model in the first prediction mode, the sensing data of the infrastructure in the sample and the corresponding preceding state sequence are input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined by the output state of the infrastructure and the corresponding label. When training the state recognition model in the second prediction mode, the mask value of the sensing data of the infrastructure in the sample and the corresponding preceding state sequence is input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined by the output state of the infrastructure and the corresponding label. Illustratively, the model is trained for 1 epoch (2 epochs, 3 epochs, etc.) in the first prediction mode using the first training set, and then trained for 1 epoch (2 epochs, 3 epochs, etc., where epoch refers to an epoch) in the second prediction mode. After a preset number of epochs (e.g., 1000 epochs, 2000 epochs, or 10000 epochs) or when a preset convergence criterion is reached, the training ends. As an illustration, the samples in the first training set can be divided into multiple batches to train the state recognition model separately.

[0058] Of course, the structures of the parameters that need to be updated can differ between the first and second prediction modes. For example, for Figure 3 The structure shown allows for supervised training of state recognition during training, starting with a masked preceding state (i.e., all preceding state sequences are 0, the attention mechanism is disabled, corresponding to the training of the second prediction mode). This involves updating the parameters of the backbone neural network and the attention module. After the backbone neural network converges, the parameters of the backbone neural network are fixed, and the attention mechanism is trained again based on the output of the backbone neural network to the training data and the preceding state sequence (corresponding to the training of the first prediction mode). This involves updating the parameters of the attention module without updating the parameters of the backbone neural network.

[0059] III. Calibrator

[0060] According to one embodiment of the present invention, the calibrator can also be implemented using a regression model (such as a neural network like a single-layer, two-layer, or multi-layer perceptron, or a traditional non-neural network regression analysis model). For example, the calibrator can employ a regression model based on random forests, gradient boosting trees, multi-layer perceptrons, or various deep learning models. See also Figure 5 The calibrator determines whether the current state of the infrastructure needs calibration based on the sensor data. If so, the current state of the infrastructure is updated by the state prediction of the infrastructure by the trained state recognition model in the second prediction mode (i.e., based on the mask value of the preceding state and the sensor data). Optionally, the trained calibrator is trained as follows: the trained state recognition model outputs the confidence level of each state of the infrastructure for each sample in the first training set in the second prediction mode; the confidence level of the labeled state (i.e., the confidence level of the correct state) is extracted from the confidence level of each state according to the first label corresponding to the sample as the second label of the sample; the calibrator is obtained, which is a regression model, and the calibrator is trained by regression using the sensor data of the infrastructure in the samples of the first training set and the corresponding second label. Wherein, when the confidence level of the trained calibrator outputting the sensor data is greater than or equal to a confidence threshold (e.g., 80%, 85%, 90%, 95%), it is determined that the current state of the infrastructure needs calibration. Since this calibrator is trained to identify the confidence level for correctly predicting a state under the second prediction mode, if its output confidence level is greater than or equal to a confidence threshold, it indicates that using only sensor data for prediction can achieve a relatively high accuracy rate. Therefore, calibration is performed to improve the prediction accuracy in this situation. The calibrator can adaptively calibrate the infrastructure state identification results to improve the accuracy of the state identification model during continuous operation and prevent error accumulation.

[0061] Alternatively, implementers may define other rules to determine whether calibration is required. For example, calibration is required when the confidence level of the trained calibrator in the output of the sensor data is greater than or equal to a confidence threshold, and the confidence level in the output of the sensor data is greater than the maximum value among the confidence levels of each state output by the discriminator.

[0062] In addition, other methods can be used to implement the calibrator; for example, the calibrator may not be based on training. See one embodiment of the invention. Figure 6 The calibrator is configured to acquire the confidence level of each state identified by the state recognition model, and determine whether the maximum value of the confidence level of each state is less than or equal to a preset minimum threshold. If so, calibration is performed.

[0063] In addition, in some cases, the recognition accuracy is often high, and it is not necessary to perform calibration for every state recognition. That is, the execution frequency of the calibrator can be the same as or different from the frequency of the state recognition model's state recognition (for example, the calibrator performs state calibration once after the state recognition model performs 2, 3 or 4 recognitions). The difference in the execution frequency between the two can be determined according to the actual deployment accuracy and power consumption requirements.

[0064] IV. Application Scenarios

[0065] According to an embodiment of the present invention, an infrastructure state identification method based on inertial sensor data is provided, comprising: acquiring sensing data collected by inertial sensors deployed on the infrastructure and a sequence of previous states of the infrastructure before the sensing data was collected;

[0066] A trained state recognition model is used to determine the current state of the infrastructure (corresponding to a first prediction mode) based on the sensor data and the preceding state sequence. The trained state recognition model includes two prediction modes: a first prediction mode based on the sensor data and the corresponding preceding state sequence, and a second prediction mode based on the sensor data and the mask value of the corresponding preceding state sequence. A calibrator is used to determine whether the current state of the infrastructure needs calibration based on the sensor data. If so, the current state of the infrastructure is updated by the prediction of the infrastructure in the second prediction mode by the trained state recognition model; otherwise, the current state of the infrastructure determined based on the first prediction mode is retained. Furthermore, the corresponding preceding state sequence is updated based on the determined current state of the infrastructure. This embodiment achieves at least the following beneficial technical effects: this embodiment infers the infrastructure state based on the current sensor data (e.g., real-time acquisition, instant acquisition) and the preceding state sequence, which can improve the accuracy of the inference.

[0067] According to one embodiment of the present invention, an infrastructure status monitoring system is provided, comprising: one or more information acquisition devices, each information acquisition device being deployable on the infrastructure to be monitored and including an inertial sensor-based data acquisition module and a communication module; and a status monitoring system configured to interact with the communication module of the information acquisition devices to obtain sensing data of the infrastructure collected by the corresponding infrastructure inertial sensors, and to identify the status of each infrastructure to be monitored based on an initialized pre-state sequence using an infrastructure status identification method based on inertial sensor data. The initialized pre-state sequence can be obtained through an initialization method desired by the implementer, such as random initialization, or first collecting multiple consecutive states of the infrastructure to set the initialized pre-state sequence, etc.

[0068] According to one embodiment of the present invention, see Figure 7The functional modules (corresponding algorithms) of the infrastructure status recognition method based on inertial sensor data can be deployed at the edge. The algorithm is deployed in an embedded chip (or embedded device) at the edge, using the Nrf52833, a microcontroller with Bluetooth communication capabilities. Model parameter quantization technology is used to complete the model deployment, that is, a certain strategy is employed to convert the model's parameters from floating-point values ​​to fixed-point values, thereby reducing the model's parameter space. In this embodiment, the model quantization measurement uses the min-max quantization method, an offline quantization method implemented after model training. This method maps the model's floating-point parameter range layer by layer to an 8-bit fixed-point parameter range of -128 to 128, thereby compressing the model to approximately 100KB in size, which is then deployed on the Nrf52833 chip for inference. The embedded chip deployed at the edge in this embodiment includes three modules: an inertial navigation sensor module, a microcontroller module, and a two-way communication module. After being packaged into a specific product form, the embedded chip is fixed to the infrastructure, such as doors, windows, or industrial robotic arms, using adhesive or other fixing methods. After the user configures the target monitoring action according to the process, the inertial navigation sensor module collects the sensing data, the microcontroller module executes the infrastructure status identification method based on the inertial sensor data, and the two-way communication module interacts with the cloud service to obtain the current infrastructure status and control command values ​​to complete the corresponding control operation or parameter configuration.

[0069] Of course, the edge device can also be responsible only for the data acquisition process. See another embodiment of the invention. Figure 8 The algorithm is deployed on a server in the fog or cloud, and sensor data from the inertial navigation unit is collected by an embedded chip at the edge and uploaded to the fog or cloud server for real-time infrastructure status identification. In this embodiment, the edge embedded chip includes two modules: an inertial navigation sensor module and a two-way communication module. The embedded chip is packaged into a specific product form and fixed to the infrastructure, such as doors, windows, or industrial robotic arms, using adhesive or other fixing methods. The embedded chip is only responsible for the sensor data acquisition process. The status identification algorithm can be deployed in the cloud (e.g., cloud services) or on the fog (e.g., regional servers). Of course, it can also be deployed elsewhere, for example, see [link to relevant documentation]. Figure 9 Status recognition algorithms can also be deployed on gateways (such as smart gateways). The intelligent network can identify the status of the infrastructure and report it to the cloud service. Users can then obtain the status of the various infrastructure components they want to monitor from the cloud service.

[0070] To verify the calibrator's effectiveness, the inventors also conducted ablation experiments on the status recognition of infrastructure such as doors (on / off), refrigerators (on / off), water usage (on / off of water pipes or faucets), monitors (on / off), curtains (on / off), and toilets (whether they are in use or not). The results of the ablation experiments are as follows: Figure 10 As shown, the accuracy was improved in all infrastructure status recognition scenarios after using the calibrator, especially in the status recognition scenarios of infrastructure such as monitors, curtains and toilets, where the accuracy improvement was significant.

[0071] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0072] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0073] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0074] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for infrastructure condition identification based on inertial sensor data, characterized in that, include: Acquire sensing data collected by inertial sensors deployed on the infrastructure and the preceding state sequence of the infrastructure before the sensing data was collected; The current state of the infrastructure is determined by using a trained state recognition model based on the sensor data and the preceding state sequence. The trained state recognition model includes two prediction modes: a first prediction mode that predicts based on the sensor data and the corresponding preceding state sequence, and a second prediction mode that predicts based on the sensor data and the mask value of the corresponding preceding state sequence. The first prediction mode is used by default to predict the current state of the infrastructure. The calibrator is used to determine whether the current state of the infrastructure needs calibration. If so, the current state of the infrastructure is updated by the state of the infrastructure predicted by the trained state recognition model in the second prediction mode. If not, the current state of the infrastructure determined based on the first prediction mode is retained.

2. The method according to claim 1, characterized in that, The trained state recognition model is trained in the following manner: Obtain the first training set, which includes multiple samples and labels corresponding to each sample. Each sample includes the sensing data of the infrastructure and the preceding state sequence of the infrastructure before the sensing data was collected. Each label indicates the labeled state of the infrastructure when the sensing data contained in its sample was collected. A constructed state recognition model is obtained, the state recognition model including a feature extractor for extracting feature vectors from sensor data and preceding state sequences, and a prediction layer for identifying the state of infrastructure based on the feature vectors; The state recognition model is trained alternately using the first training set in either the first prediction mode or the second prediction mode.

3. The method according to claim 2, characterized in that, in, When training the state recognition model in the first prediction mode, the sensing data of the infrastructure in the sample and the corresponding preceding state sequence are input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined by the output state of the infrastructure and the corresponding label. When training the state recognition model in the second prediction mode, the sensor data of the infrastructure in the sample and the mask value of the corresponding preceding state sequence are input into the state recognition model, the state of the infrastructure is output, and the parameters of the state recognition model are updated based on the loss value determined by the output state of the infrastructure and the corresponding label.

4. The method according to claim 3, characterized in that, The feature extractor includes: The first branch is used to extract features from triaxial acceleration data in sensor data to obtain acceleration features. The second branch is used to extract features from the triaxial angular velocity data in the sensing data to obtain the angular velocity features. Used to extract features from the preceding state sequence to obtain the preceding state features of the preceding branch; A fusion layer used to fuse acceleration features, angular velocity features, and preceding state features to obtain feature vectors.

5. The method according to claim 4, characterized in that, The first branch performs acceleration fusion on the triaxial acceleration data through a three-channel 1D convolutional layer and filters the acceleration fusion result using a mean operator to obtain acceleration features; The second branch performs angular velocity fusion on the triaxial angular velocity data through a single-layer RNN network and filters the fused angular velocity results using the Sobel operator to obtain angular velocity features.

6. The method according to claim 3, characterized in that, The feature extractor includes: The first branch is used to extract features from triaxial acceleration data in sensor data to obtain acceleration features. The second branch is used to extract features from the triaxial angular velocity data in the sensing data to obtain the angular velocity features. The third branch is used to extract features from triaxial magnetometer data in the sensor data to obtain magnetic features; A fusion layer used to obtain feature vectors from acceleration features, angular velocity features, magnetic force features, and preceding state features.

7. The method according to claim 3, characterized in that, The feature extractor includes: The backbone neural network uses the Inception network to extract intermediate features in the form of multi-dimensional vectors from the sensor data; The attention module based on the preceding state sequence includes: Fully connected layers for extracting attention values ​​for each dimension of intermediate features from the preceding state sequence, and A fusion layer is used to obtain weighted intermediate features by weighting the values ​​of each dimension of the intermediate features based on the attention value of each dimension, and then fusing the weighted intermediate features and the intermediate features to obtain a feature vector.

8. The method according to claim 2, characterized in that, The calibrator was trained as follows: Using the trained state recognition model, the confidence level of each state of the infrastructure is output for each sample in the first training set in the second prediction mode. The confidence level of the labeled state is extracted from the confidence level of each state according to the first label corresponding to the sample as the second label of the sample. A calibrator is obtained, which is a regression model. The calibrator is trained by regression using the sensing data of the infrastructure in the samples of the first training set and the corresponding second label.

9. The method according to claim 8, characterized in that, When the confidence level of the calibrator's output of the sensor data is greater than or equal to the confidence threshold, it is determined that the current state of the infrastructure requires calibration.

10. The method according to any one of claims 1-9, characterized in that, For different types of infrastructure, there are at least two preset states corresponding to each type of infrastructure. Each type of infrastructure is one of the following: door, window, water pipe, drawer, bed, curtain, vent, electrical appliance, fitness equipment, industrial production device and printer.

11. An infrastructure condition monitoring system, characterized in that, include: One or more information acquisition devices, each of which can be deployed on the infrastructure that needs to be monitored and includes an inertial sensor-based data acquisition module and a communication module; A condition monitoring device is configured to interact with the communication module of the information acquisition device to obtain sensing data of the infrastructure collected by the corresponding infrastructure inertial sensor, and to identify the condition of one or more infrastructures to be monitored based on an initialized pre-state sequence using the method of any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 10.

13. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.

Images