Device status identification method, device, electronic device and storage medium

Abstract

The present invention provides a device state recognition method, apparatus, electronic device and storage medium, wherein the method comprises: extracting features from the audio data of the device to be tested to obtain audio features, and extracting features from the image data of the device to be tested to obtain image features; inputting the audio features and the image features into a device state recognition model to obtain a state recognition result output by the device state recognition model. The present invention uses audio features and image features to recognize the device state at the same time, so that the model can capture more comprehensive and richer information about the device state, thereby improving the accuracy and robustness of recognition. By introducing a global attention module and a local attention module into the model, a multiple attention mechanism is added, which can effectively remove redundant information, and improve the representation capability of features from three levels: full time domain, full frequency domain and local details, further improving the accuracy and generalization capability of recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a device state recognition method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of industrial intelligent manufacturing technology, industries such as manufacturing and energy are gradually moving towards automation and intelligence. In this process, the stable operation of industrial equipment is crucial to ensuring production efficiency and safe production. Equipment failure will not only lead to production line stagnation and economic losses, but may also cause safety hazards.

[0003] At present, the technology of equipment status recognition has made significant progress, among which sound-based machine recognition technology has become a common method. There are differences in the sounds emitted by equipment in normal and abnormal states. Through the feature extraction and analysis of these sounds by artificial intelligence technologies such as deep learning, the equipment status can be effectively judged. However, this solution still has some defects: First, it is highly dependent on the amount of labeled training data. If the training data is insufficient, the recognition effect of the model will be significantly reduced. Secondly, the source of training data is single, and the model is prone to overfitting, resulting in poor recognition effect when facing data from unseen scenes. Therefore, how to improve the accuracy and generalization ability of equipment status recognition under the condition of limited labeled data is a problem that needs to be solved urgently. Summary of the invention

[0004] The present invention provides a device state recognition method, an apparatus, an electronic device and a storage medium, which are used to solve the defects of insufficient accuracy and generalization ability of device state recognition in related technologies.

[0005] The present invention provides a device status identification method, comprising: Performing feature extraction on audio data of the device under test to obtain audio features, and performing feature extraction on image data of the device under test to obtain image features; Inputting the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0006] According to a device state recognition method provided by the present invention, the device state recognition model further includes a convolution module, a mask module, an encoding module and a classification module, and the audio feature and the image feature are input into the device state recognition model to obtain a state recognition result output by the device state recognition model, including: Based on the global attention module, weighting the audio features and the image features in time dimension and frequency dimension to obtain global features; Based on the convolution module, feature extraction, feature transformation and feature segmentation are performed on the global features to obtain multiple feature blocks; Based on the local attention module, weighting the multiple feature blocks by block dimensions to obtain local features; Based on the mask module, masking the local features, and inputting the masked features into the encoding module to obtain encoding results; The encoding result is linearly transformed, and the transformed result is input into the classification module to obtain the state recognition result.

[0007] According to a device state recognition method provided by the present invention, the global attention module includes a time attention module and a frequency attention module, and based on the global attention module, the audio features and the image features are weighted in the time dimension and the frequency dimension to obtain global features, including: Inputting the audio feature and the image feature into the time attention module to obtain a time dimension attention vector, and multiplying the time dimension attention vector with the audio feature and the image feature to obtain a time domain feature; Inputting the audio feature and the image feature into the frequency attention module to obtain a frequency dimension attention vector, and multiplying the frequency dimension attention vector with the audio feature and the image feature to obtain a frequency domain feature; The time domain feature and the frequency domain feature are fused to obtain the global feature.

[0008] According to a device state recognition method provided by the present invention, the block dimension weighting of the multiple feature blocks is performed based on the local attention module to obtain local features, including: Input the multiple feature blocks into the local attention module to obtain a block-dimensional attention vector; The block-dimensional attention vector is multiplied by the multiple feature blocks to obtain the local features.

[0009] A device status identification method provided by the present invention also includes: Based on the block-dimensional attention vector, determining the attention weight corresponding to each feature block; Determining a key feature block from the feature blocks based on the attention weights corresponding to the feature blocks; The audio data is analyzed based on the time range and frequency range of the key feature block.

[0010] According to a device state recognition method provided by the present invention, the method extracts features from audio data of the device under test to obtain audio features, and extracts features from image data of the device under test to obtain image features, including: The audio data is sampled to a preset size by bilinear interpolation to obtain initial audio features, and the initial audio features are mean-variance regularized to obtain the audio features; The image data is sampled to a preset size through bilinear interpolation to obtain initial image features, and the initial image features are mean-variance regularized to obtain the image features.

[0011] According to a device state recognition method provided by the present invention, the training step of the device state recognition model includes: Acquire an initial model and collect unlabeled sample device data, wherein the sample device data includes sample audio data and sample image data of the sample device, and the initial model includes a global attention module, a convolution module, a local attention module, a mask module, an encoding module, and a decoding module; Performing feature extraction, feature alignment and feature regularization on the sample audio data and the sample image data to obtain sample audio features and sample image features; Inputting the sample audio features and the sample image features into the initial model to obtain the reconstructed features output by the initial model; Determine a reconstruction loss based on the difference between the reconstructed feature and the sample audio feature and the sample image feature, and apply the reconstruction loss to perform parameter iteration on the initial model to obtain a pre-trained state recognition model; The structure of the pre-trained state recognition model is adjusted, and the parameters of the adjusted model are fine-tuned based on sample device data with category labels to obtain the device state recognition model. The adjusted model is obtained by replacing the decoding module in the pre-trained state recognition model with a linear layer and a classification head.

[0012] The present invention also provides a device for identifying a device state, comprising: A feature extraction unit is used to perform feature extraction on the audio data of the device under test to obtain audio features, and to perform feature extraction on the image data of the device under test to obtain image features; A state recognition unit, used for inputting the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned device status identification methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the device status identification method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the device status identification method described above is implemented.

[0016] The device state recognition method, device, electronic device and storage medium provided by the present invention can enable the model to capture more comprehensive and richer information about the device state by extracting features from the audio data and image data of the device to be tested, and inputting the extracted audio features and image features into the device state recognition model, thereby improving the accuracy and robustness of recognition. By introducing a global attention module and a local attention module into the model, a multiple attention mechanism is added, which can effectively remove redundant information and improve the representation ability of features from three levels: the full time domain, the full frequency domain and the local details, so that the model can better focus on key information and improve the accuracy and generalization ability of recognition. In addition, the present invention proposes an innovative sound-image fusion training strategy, which can significantly improve the feature representation ability of the model by training with sample audio features and sample image features at the same time, thereby improving the accuracy of device state recognition, and only needs to use sample audio features and sample image features to train the model in a self-supervised manner, which greatly reduces the cost and difficulty of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the process of device status identification in the prior art; Figure 2 It is a flow chart of the device status identification method provided by the present invention; Figure 3 Schematic diagram of the principle of the SE channel attention module provided by the present invention; Figure 4 is a schematic diagram of key feature blocks in an image provided by the present invention; Figure 5 It is a flow chart of the equipment state recognition model training provided by the present invention; Figure 6 It is a structural schematic diagram of the initial model provided by the present invention; Figure 7 It is a structural schematic diagram of the device state identification device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The stable operation of industrial equipment has become the cornerstone of ensuring production efficiency and safe production. Once the equipment fails, it will not only cause the production line to stagnate, resulting in huge economic losses, but may also cause a series of safety hazards. Therefore, it is particularly important to monitor and accurately identify the status of industrial equipment in real time.

[0021] At present, the equipment status recognition technology has made significant progress, among which the sound-based machine recognition technology has become a common means. There are significant differences in the sounds emitted by the equipment in normal and abnormal states. Using artificial intelligence technologies such as deep learning to extract and analyze the features of these sounds can effectively determine the operating status of the equipment. Figure 1 It is a flow chart of device status recognition in the prior art, such as Figure 1 As shown in the figure, the specific process includes preparing device sound training data with normal and abnormal category labels, extracting acoustic features such as MFCC (Mel Frequency Cepstral Coefficient), and then using a deep neural network model for supervised training. In the actual application test process, the sound data of the device under test is collected and the acoustic features are extracted. Through the trained model, the extracted features can be used for reasoning to obtain the posterior probability of each category. The category label corresponding to the maximum probability is taken as the model prediction category result, and the state category of the device under test can be obtained.

[0022] However, although this deep learning-based device status recognition solution performs well when there are sufficient labeled training samples and can solve the problem of fitting complex distributed data, it still has certain defects. First, the solution is highly dependent on the amount of labeled training data. If the training data is insufficient, the recognition effect of the model will be significantly reduced and cannot meet the needs of practical applications. Secondly, the source of training data is relatively single, and the model is prone to overfitting, resulting in poor recognition effect when facing data from unseen scenes, which limits the generalization ability of the model and reduces its applicability in complex and changing industrial environments.

[0023] In this regard, the present invention provides a method for identifying device status, which does not rely on a large amount of labeled sample data. Instead, it only needs a large amount of unlabeled data (including audio data and image data) from various types of industrial equipment during operation, so as to pre-train a highly robust sound-image fusion base model in a self-supervised manner. The model can then be fine-tuned in combination with a small amount of labeled device data to perform device status identification, thereby overcoming the above-mentioned defects.

[0024] Figure 2 is a flow chart of the device status identification method provided by the present invention, such as Figure 2 As shown, the method includes: Step 210 , extracting features from the audio data of the device under test to obtain audio features, and extracting features from the image data of the device under test to obtain image features.

[0025] Specifically, the device under test refers to the target device that needs to be identified in terms of status. These devices may belong to manufacturing, energy or other industrial fields, and their operating status is crucial to production efficiency and safety. The audio data of the device under test refers to the sound signals generated by the device during operation. These sound signals may contain rich information about the operating status of the device, such as vibration, friction, wear, etc. By collecting and analyzing these audio data, useful features can be extracted for device status identification. Here, audio features refer to key information extracted from the audio data of the device under test that can reflect the status of the device. For example, audio features may be STFT (Short-Time Fourier Transform) features.

[0026] The image data of the device under test refers to the visual information of the device obtained through a camera or other image acquisition device. These image data may contain information such as the appearance, structure, and motion status of the device, which helps to understand the status of the device more comprehensively. By collecting and analyzing the image data of the device under test, image features can be extracted from it. Here, image features refer to key visual information extracted from the image data of the device under test that can reflect the status of the device. These features may include parameters such as image texture, color, and shape.

[0027] Step 220, inputting the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0028] Specifically, after the audio features and image features are extracted, these features can be simultaneously input into the device state recognition model, and the state of the device can be recognized through the model, so as to obtain the state recognition result output by the model. Here, the device state recognition model is a model based on machine learning or deep learning technology, which can extract useful information from the input audio features and image features for judging the state of the device. The state recognition result output by the model refers to the conclusion about the state of the device drawn by the device state recognition model after processing the input audio features and image features. The result can be a classification label for the device being in a normal state, an abnormal state, or other specific state, or it can be a quantitative indicator or probability distribution of the device state, which is not specifically limited in the embodiments of the present invention.

[0029] It is understandable that before executing step 220, the device state recognition model can be pre-trained, and the model can be trained in the following way: first, a large amount of unlabeled data from various types of industrial equipment is collected, including sample audio data and sample image data, and features are extracted from these data to obtain sample audio features and sample image features. Considering that the distribution of sample audio features and sample image features is quite different, and the model is difficult to converge, these features can be size-aligned and regularized to satisfy the normal distribution; then, a suitable initial model is selected, for example, the model can adopt the MAE (Masked AutoEncoder) structure, and the regularized sample audio features and sample image features are input into the initial model, and pre-trained in a self-supervised manner; then, the pre-trained model is fine-tuned in combination with a small amount of labeled device data to obtain the device state recognition model. It should be noted that the specific training and fine-tuning process of the device state recognition model will be described in detail through the following embodiments.

[0030] The device status recognition model is trained by a large number of sample audio features and sample image features, and can identify the feature differences of the device in normal or abnormal state. After the model pre-training and fine-tuning steps, the device status recognition model is obtained, and then the model can be applied to device status recognition.

[0031] Furthermore, a global attention module and a local attention module are introduced into the structure of the device state recognition model, wherein the global attention module is responsible for extracting global information from the input features, that is, weighting the time dimension and the frequency dimension to obtain global features. By further extracting, transforming and segmenting the global features, multiple feature blocks can be obtained, and the local attention module is used to further process these feature blocks, that is, weighting these feature blocks by block dimensions to obtain local features. Here, global features refer to information extracted from input features that can reflect the overall operating status of the device. Local features refer to information extracted from feature blocks that can reflect the local operating status of the device.

[0032] It should be understood that weighting the time dimension and frequency dimension refers to performing different weighting processing on the input audio features and image features in the time dimension and frequency dimension to emphasize or suppress certain key information. This weighting processing helps the model better capture the useful information in the original signal and improve the accuracy of recognition. Block dimension weighting refers to performing different weighting processing on feature blocks to emphasize or suppress certain local information. This weighting processing helps the model better capture the local feature differences of the device in different states and improve the accuracy of recognition.

[0033] The method provided by the embodiment of the present invention can enable the model to capture more comprehensive and richer information about the device status by extracting features from the audio data and image data of the device to be tested respectively, and inputting the extracted audio features and image features into the device status recognition model, thereby improving the accuracy and robustness of recognition. By introducing a global attention module and a local attention module into the model, a multiple attention mechanism is added, which can effectively remove redundant information and improve the representation ability of features from three levels: full time domain, full frequency domain and local details, so that the model can better focus on key information and improve the accuracy and generalization ability of recognition. In addition, the present invention proposes an innovative sound-image fusion training strategy, which can significantly improve the feature representation ability of the model by training with sample audio features and sample image features at the same time, thereby improving the accuracy of device status recognition, and only needs to use sample audio features and sample image features to train the model in a self-supervised manner, which greatly reduces the cost and difficulty of data annotation.

[0034] Based on any of the above embodiments, step 210 specifically includes: Step 211: Sample the audio data to a preset size through bilinear interpolation to obtain initial audio features, and perform mean-variance regularization on the initial audio features to obtain the audio features.

[0035] Step 212: Sample the image data to a preset size through bilinear interpolation to obtain initial image features, and perform mean variance regularization on the initial image features to obtain the image features.

[0036] Specifically, considering the large difference in the distribution of audio features and image features, when extracting audio features and image features, the features can be size-aligned and regularized. For image data, bilinear interpolation can be used to upsample or downsample to a preset size (e.g. Here, the image data is upsampled or downsampled to a preset size through bilinear interpolation, mainly to align the sizes of audio features and image features while retaining important features in the image as much as possible.

[0037] For audio data, features can be extracted first to obtain STFT features, and then all STFT features can be upsampled or downsampled to a preset size (for example, The size of the audio data is used as the initial audio feature. Here, in the process of extracting the audio data to obtain the STFT feature, the Fourier transform window length n_fft in the parameters is set to 512 points, the frame length win_length is set to 20ms (converted to the corresponding number of points according to the sampling rate), and the frame shift hop_length is set to 10ms. Assuming that the audio length is audio_length, the extracted STFT feature size is (1+n_fft / 2, 1+audio_length / / hop_length), that is, (257, 1+audio_length / / hop_length). Because 257 and 224 are close in value, the Fourier transform window length n_fft selects 512 points.

[0038] Since the distribution of STFT features and initial image features is quite different, the mean and variance of these two types of features can be regularized separately so that they both satisfy the normal distribution. The specific regularization method is as follows: Regularized STFT feature = (STFT feature - mean of all STFT features) / standard deviation of all STFT features; Regularized image features = (initial image features - mean of all initial image features) / standard deviation of all initial image features; The above regularized STFT features are the final audio features, and the regularized image features are the final image features.

[0039] It should be understood that during the model training process, the sample audio features and sample image features also need to be regularized. This is because they come from different modalities and have large distribution differences. Direct fusion may make the model difficult to train and converge slowly. Feature regularization helps solve the problem of training difficulties caused by feature distribution differences in multimodal training tasks.

[0040] Based on any of the above embodiments, the device state recognition model further includes a convolution module, a mask module, an encoding module and a classification module. Accordingly, step 220 specifically includes: Step 221: Based on the global attention module, the audio features and the image features are weighted in terms of time dimension and frequency dimension to obtain global features.

[0041] Specifically, the global attention module is used to extract global information from the input audio features and image features. The module can include a time attention module and a frequency attention module. The working principles of these two modules are similar to the SE (Squeeze-and-Excitation) module, but the time attention module (referred to as the TimeSE module) focuses on the importance of the input features in the time dimension, and the frequency attention module (referred to as the FreqSE module) focuses on the importance of the input features in the frequency dimension. Here, the SE module is a channel attention module that enables the model to pay more attention to important channel features by explicitly modeling the interdependence between feature channels.

[0042] Figure 3 Schematic diagram of the principle of the SE channel attention module provided by the present invention, such as Figure 3 As shown, the input is The characteristic of size, F sq is the global average pooling operation, F ex is two linear layers. After two linear layers, F ex Will output a The attention vector of size , multiplying the vector by the feature point of the original input, can get a new Size characteristics.

[0043] Accordingly, step 221 specifically includes: Step 2211, input the audio features and the image features into the time attention module to obtain a time dimension attention vector, and multiply the time dimension attention vector with the audio features and the image features to obtain a time domain feature.

[0044] Specifically, the principle of the TimeSE module is similar to that of the SE channel attention module, but it focuses on the time dimension rather than the channel dimension. Specifically, the TimeSE module performs global average pooling on the input features (including audio features and image features) in the time dimension to generate a vector that describes the characteristics of the time dimension. This vector then passes through one or more linear layers (usually two layers) to learn and output a time dimension attention vector, each element of which represents the importance or weight of the corresponding time point in the task. Finally, the time dimension attention vector is dot-multiplied with the original input feature in the time dimension to obtain the time domain feature.

[0045] Step 2212: input the audio features and the image features into the frequency attention module to obtain a frequency dimension attention vector, and multiply the frequency dimension attention vector with the audio features and the image features to obtain a frequency domain feature.

[0046] Specifically, the working principle of the FreqSE module is similar to that of the TimeSE module, but it focuses on the importance of the input features in the frequency dimension. The frequency dimension is an important feature when processing signals such as images and audio. The FreqSE module performs global average pooling on the input features in the frequency dimension to generate a vector that describes the characteristics of the frequency dimension. This vector is then passed through one or more linear layers to learn and output a frequency-dimensional attention vector, each element of which represents the importance or weight of the corresponding frequency in the task. Finally, the frequency-dimensional attention vector is dot-multiplied with the original input features in the frequency dimension to emphasize or suppress different frequency components to obtain frequency domain features.

[0047] Step 2213: Fusing the time domain features and the frequency domain features to obtain the global features.

[0048] Specifically, after obtaining the time domain features and the frequency domain features, the time domain features and the frequency domain features can be fused by direct concatenation or weighted fusion to obtain the global features. For example, the time domain features output by the TimeSE module and the frequency domain features output by the FreqSE module are added to obtain the global features.

[0049] Step 222: Based on the convolution module, perform feature extraction, feature transformation and feature segmentation on the global features to obtain multiple feature blocks.

[0050] Specifically, the convolution module can adopt the CNN (Convolutional Neural Networks) structure. The output of the TimeSE module is added to the output of the FreqSE module and then sent to the convolution module to obtain multiple feature blocks (i.e., Patch Embeddings) output by the module. Here, the convolution module uses CNN to further extract and transform the global features. Specifically, the input global features are processed through a series of convolutional layers, pooling layers, activation functions, etc. to extract useful features and project them into a low-dimensional or high-dimensional space. Finally, the projected features are segmented. For example, the features can be segmented into 196 Patch blocks of different sizes, that is, multiple feature blocks are obtained.

[0051] Step 223: Based on the local attention module, the multiple feature blocks are weighted by block dimension to obtain local features.

[0052] Specifically, the principle of the local attention module (PatchSE module for short) is also similar to the SE channel attention module, but it is applied to Patch Embeddings (i.e., multiple feature blocks) instead of the entire feature map.

[0053] Further, step 223 specifically includes: Step 2231, input the multiple feature blocks into the local attention module to obtain a block dimension attention vector; Step 2232, multiply the block-dimensional attention vector by the multiple feature blocks to obtain the local feature.

[0054] Specifically, through the PatchSE module, each feature block is globally averaged and pooled. Here, since each feature block is already a vector, this step can be to average each element of the vector, or to retain the integrity of the vector and proceed directly to the next step. It is also possible to perform some form of aggregation (such as summation, averaging, or maximum pooling) on ​​each feature block to obtain a global descriptor representing the feature block. Next, one or two linear layers (or fully connected layers) are used to learn the nonlinear relationship between feature blocks. These linear layers map the input global descriptor to a new space, where the weight (or attention score) of each feature block is calculated, and a weight vector (i.e., the attention vector at the Patch level, also called the block-dimensional attention vector) with the same number of input feature blocks is output. Each element in this vector corresponds to the attention score of a patch in the input Patch Embeddings. Finally, the block-dimensional attention vector is multiplied by multiple feature blocks of the original input to obtain local features.

[0055] Step 224: mask the local features based on the mask module, and input the masked features into the encoding module to obtain encoding results.

[0056] Step 225, performing a linear transformation on the encoding result, and inputting the transformed result into the classification module to obtain the state recognition result.

[0057] Specifically, the local features output by the PatchSE module are input into the mask module (i.e., the Mask module), which is masked to obtain the masked features. Here, the local features are still for multiple feature blocks, so a part of them can be masked, for example, 70% can be masked, and then the masked features are output to the encoding module for encoding. The encoding module can adopt a Transformer structure. After being processed by the encoding module, the encoding results output by the module are averaged and sent to a linear layer for linear transformation. Subsequently, the transformed results are input into the classification module for classification and recognition, and the state recognition results of the device under test can be obtained.

[0058] Based on any of the above embodiments, the method further includes: Based on the block-dimensional attention vector, determining the attention weight corresponding to each feature block; Determining a key feature block from the feature blocks based on the attention weights corresponding to the feature blocks; The audio data is analyzed based on the time range and frequency range of the key feature block.

[0059] Specifically, the PatchSE module processes the input Patch Embeddings (i.e., multiple feature blocks, i.e., small blocks or regions in the feature map) and outputs a patch-level attention vector (i.e., a block-dimensional attention vector). Each element in this vector represents the attention weight of the corresponding feature block, reflecting the importance of the feature block to the final classification or prediction task of the model. Therefore, based on the block-dimensional attention vector, the attention weight corresponding to each feature block can be determined.

[0060] Since each feature block represents an area in the time-frequency feature map, by comparing the attention weights corresponding to each feature block, the area in the original feature map that contributes more to the model classification results, that is, the key feature block, can be determined. Here, the key feature block refers to those areas in the input features that contribute more to the model classification or prediction results. By comparing the attention weights of each feature block in the block-dimensional attention vector output by the PatchSE module, those feature blocks with higher weights can be identified, and these feature blocks are the key feature blocks.

[0061] Figure 4 is a schematic diagram of key feature blocks in an image provided by the present invention, such as Figure 4 As shown, the yellow area is the feature area that contributes the most to the model classification as "dog" among all the areas in the original image, that is, the key feature block.

[0062] After determining the key feature blocks, the time range and frequency range of these feature blocks can be obtained, which is helpful for analyzing the signal characteristics of the original audio. Here, in order to obtain the time range and frequency range of the key feature blocks, it is first necessary to clarify the correspondence between Patch Embeddings and the original audio signal. Usually, Patch Embeddings are obtained by extracting features from the original audio signal, transforming features, and dividing it into multiple small blocks (i.e., Patches). Each Patch corresponds to a specific area in the original signal, which is clearly defined in time and frequency. Once it is determined which Patches are key feature blocks (i.e., Patches with higher attention weights), the time range and frequency range of the key feature blocks can be determined by finding the positions of these Patches in the original signal.

[0063] After obtaining the time range and frequency range of the key feature blocks, the original audio signal can be analyzed more deeply. For example, the signal features in these key areas can be extracted and further processed or analyzed. These features can include the amplitude, phase, spectrum, etc. of the signal, which can reveal the characteristics of the signal at different times and frequencies. By analyzing these features, the key information in the original audio signal can be more deeply understood, thereby improving the performance of the model or increasing the accuracy of classification.

[0064] In the embodiment of the present invention, by focusing on the areas that contribute more to the classification or prediction tasks, the model can use the input information more efficiently, thereby improving the accuracy of classification or the performance of prediction.

[0065] Based on any of the above embodiments, Figure 5 is a flow chart of the device state recognition model training provided by the present invention, such as Figure 5 As shown, the training steps of the device state recognition model include: Step 510, obtain an initial model and collect unlabeled sample device data, wherein the sample device data includes sample audio data and sample image data of the sample device, and the initial model includes a global attention module, a convolution module, a local attention module, a mask module, an encoding module and a decoding module.

[0066] Specifically, before model training, a basic model architecture, namely the initial model, must be selected first. The model architecture includes a global attention module, a convolution module, a local attention module, a mask module, an encoding module, and a decoding module. These modules work together to process multimodal data (i.e., audio data and image data). In the early stages of self-supervised learning, a large amount of unlabeled sample audio and sample image data is collected, which will be used in the pre-training stage of the model to learn the intrinsic structure and representation of the data.

[0067] Step 520: Perform feature extraction, feature alignment and feature regularization on the sample audio data and the sample image data to obtain sample audio features and sample image features.

[0068] Specifically, preliminary feature extraction is performed on sample audio data and sample image data. Since audio and image data come from different modalities, they need to be processed within an aligned framework to ensure that audio and image features correspond to each other in subsequent processing. Due to the large difference in the distribution of audio features and image features, in order to avoid difficulty in model training convergence, the extracted features can be regularized.

[0069] Step 530: input the sample audio features and the sample image features into the initial model to obtain the reconstructed features output by the initial model.

[0070] Specifically, Figure 6 It is a structural schematic diagram of the initial model provided by the present invention, such as Figure 6As shown in the figure, the initial model adopts the MAE structure, and adds the TimeSE module and the FreqSE module after the feature input, and adds the PatchSE module after the Patch Embeddings. By adding multiple attention mechanisms, the representation ability of features can be improved from the three levels of full time domain, full frequency domain and local details. Specifically, the global attention module includes the TimeSE module and the FreqSE module. The TimeSE module is used to process the input features to obtain the attention vector of the time dimension and multiply it with the input features; the FreqSE module is used to process the input features to obtain the attention vector of the frequency dimension and multiply it with the input features. The outputs of the TimeSE module and the FreqSE module are added and sent to the CNN module (i.e., the convolution module) to obtain PatchEmbeddings, and then the attention vector at the patch level is obtained through the PatchSE module, and it is multiplied with the original PatchEmbeddings. Subsequently, the output of the PatchSE module is masked by the Mask module (i.e., the mask module), and the masked features are sent to the Encoder module (i.e., the encoding module), which adopts the Transformer structure. The encoding module converts the masked features into a more compact representation, while the decoder module attempts to reconstruct the original features from this compact representation. The decoder module uses a CNN structure. By comparing the difference between the reconstructed features and the original features, the model can be trained to learn an effective representation of the data.

[0071] Step 540, based on the difference between the reconstructed feature and the sample audio feature and the sample image feature, determine the reconstruction loss, apply the reconstruction loss, perform parameter iteration on the initial model, and obtain a pre-trained state recognition model.

[0072] Specifically, the reconstruction loss is calculated based on the difference between the reconstructed features and the sample audio features and sample image features. A certain distance metric (such as mean square error) can be used to quantify the reconstruction loss. The reconstruction loss is applied to the parameters of the initial model, and optimization algorithms such as gradient descent can be used to minimize the reconstruction loss, thereby updating the weights and bias parameters of the model. After multiple iterations, the model gradually learns to extract useful features from the input data and generate accurate reconstruction features. At this point, the model has a preliminary state recognition capability.

[0073] Step 550, adjust the structure of the pre-trained state recognition model, and fine-tune the parameters of the adjusted model based on the sample device data with category labels to obtain the device state recognition model. The adjusted model is obtained by replacing the decoding module in the pre-trained state recognition model with a linear layer and a classification head.

[0074] Specifically, after the model pre-training is completed, the decoding module of the pre-trained state recognition model can be removed, and only the previous part can be retained. After taking the average of the output of the encoding module, it is sent to a linear layer and connected to the classification head for classification tasks. Specifically, in the model fine-tuning stage, sample device data (including audio and image data) of the target scene with category labels can be collected, and the cross entropy loss can be used as the optimization target to fine-tune the parameters of the adjusted model. The cross entropy loss measures the difference between the probability distribution of the model output and the true category label. After fine-tuning, the model can accurately identify the state of the device and output the corresponding category label. At this point, the model is ready for use in practical applications.

[0075] Based on any of the above embodiments, the embodiment of the present invention provides an industrial equipment status recognition method, which does not rely on a large amount of labeled training data, but only needs to collect a large amount of unlabeled data (which can be audio of various sampling rates) from various types of industrial equipment during operation, and a large amount of image data (such as ImageNet dataset), that is, a highly robust sound-image fusion base model can be pre-trained in a self-supervised manner, and a small amount of labeled target scene industrial equipment data can be used to fine-tune the model, so as to recognize the status of the equipment under test. The overall process of the method is as follows: S1, Feature Extraction S11, collect a large amount of unlabeled industrial equipment audio data with unlimited sampling rate, and download the ImageNet dataset; S12, for images in the ImageNet dataset, upsample or downsample to The size of , as an image feature; S13, extract STFT features from audio data, upsample or downsample all original STFT features to The size of the audio feature is obtained; S14, performing mean-variance regularization on the audio features and the image features respectively to obtain regularized audio features and image features.

[0076] S2, model pre-training S21, after the regularized audio features and image features are shuffled, they are randomly composed into sample features and sent to the network training. It should be noted that the network model adopts the MAE structure, which divides the features into 196 The size of the patch is 70% masked. The encoder uses the Transformer structure and the decoder uses the CNN structure. The training loss function is the reconstruction loss, and the MSE loss (mean square error loss) is used to calculate the reconstruction error between the reconstructed features output by the decoder and the input features; S22, input the sample features into the TimeSE module, extract the time dimension attention vector, and multiply the time dimension attention vector and the original sample features; input the sample features into the FreqSE module, extract the frequency dimension attention vector, and multiply the frequency dimension attention vector and the original sample features; add the output of the TimeSE module and the output of the FreqSE module and send them to the CNN module to obtain Patch Embeddings; then pass the PatchSE module to obtain the patch level attention vector and multiply it with the original Patch Embeddings. Mask the output of the PatchSE module, input the masked features into the Encoder module, and send the output of the module into the Decoder module. According to the difference between the output result of the Decoder module and the sample features, determine the reconstruction loss, apply the reconstruction loss, iterate the model parameters, and obtain the pre-trained state recognition model.

[0077] S3, model fine-tuning S31, remove the Decoder module in the pre-trained state recognition model, take the average of the output of the Encoder module of the model, send it to the linear layer, and connect it to the classification head; S32 uses a small amount of target scene industrial equipment data with category labels and uses cross entropy loss to fine-tune the model parameters to obtain an equipment state recognition model.

[0078] S4, Model Application After fine-tuning is completed, the obtained device state recognition model can be used to identify the state of the device under test. Specifically, feature extraction is performed on the audio data and image data of the device under test to obtain audio features and image features, and these features are input into the device state recognition model to obtain the state recognition result output by the model.

[0079] In addition, because each patch represents an area in the time-frequency feature map, the attention vector of the PatchSE module is used to compare the attention weights corresponding to each patch in the vector, so as to analyze the areas in the original features that contribute more to the model classification results, and obtain the time range and frequency range of such areas, which is helpful for analyzing the signal characteristics of the original audio.

[0080] The method provided by the embodiment of the present invention has the following advantages: (1) A pre-training strategy for sound-image fusion is proposed, which uses both audio feature data and image feature data for training to improve the feature representation capability of the pre-trained model. In addition, a size alignment method and feature regularization method for sound and image features are proposed to adapt to the above pre-training strategy.

[0081] (2) The TimeSE, FreqSE and PatchSE modules are introduced into the model, and a multi-attention mechanism is added to effectively remove redundant information, improve the feature representation ability from three levels: full time domain, full frequency domain and local details, and improve the robustness of the pre-trained model.

[0082] (3) By comparing the attention weights corresponding to each patch in the attention vector of the PatchSE module, we can analyze the areas in the original features that contribute more to the model classification results, and obtain the time range and frequency range of such areas, which is helpful for analyzing the key feature areas of the original audio signal.

[0083] Based on any of the above embodiments, Figure 7 is a schematic diagram of the structure of the device state identification device provided by the present invention, such as Figure 7 As shown, the device comprises: The feature extraction unit 710 is used to perform feature extraction on the audio data of the device under test to obtain audio features, and to perform feature extraction on the image data of the device under test to obtain image features; A state recognition unit 720, configured to input the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0084] The device provided by the embodiment of the present invention can enable the model to capture more comprehensive and richer information about the device status by extracting features from the audio data and image data of the device to be tested, respectively, and inputting the extracted audio features and image features into the device status recognition model, thereby improving the accuracy and robustness of recognition. By introducing a global attention module and a local attention module into the model, a multiple attention mechanism is added, which can effectively remove redundant information and improve the representation ability of features from three levels: the full time domain, the full frequency domain, and local details, so that the model can better focus on key information and improve the accuracy and generalization ability of recognition. In addition, the present invention proposes an innovative sound-image fusion training strategy, which can significantly improve the feature representation ability of the model by simultaneously using sample audio features and sample image features for training, thereby improving the accuracy of device status recognition, and only needs to use sample audio features and sample image features to train the model in a self-supervised manner, which greatly reduces the cost and difficulty of data annotation.

[0085] Based on any of the above embodiments, the device state recognition model further includes a convolution module, a mask module, an encoding module and a classification module. Accordingly, the state recognition unit 720 specifically includes: A global attention unit, configured to weight the audio features and the image features in terms of time and frequency dimensions based on the global attention module to obtain global features; A convolution unit, used for performing feature extraction, feature transformation and feature segmentation on the global feature based on the convolution module to obtain a plurality of feature blocks; A local attention unit, configured to perform block dimension weighting on the plurality of feature blocks based on the local attention module to obtain local features; A masking unit, configured to mask the local features based on the masking module, and input the masked features into the encoding module to obtain an encoding result; The classification unit is used to perform linear transformation on the encoding result and input the transformed result into the classification module to obtain the state recognition result.

[0086] Based on any of the above embodiments, the global attention module includes a time attention module and a frequency attention module, and the global attention unit specifically includes: A time attention subunit, configured to input the audio feature and the image feature into the time attention module to obtain a time dimension attention vector, and multiply the time dimension attention vector with the audio feature and the image feature to obtain a time domain feature; A frequency attention subunit, configured to input the audio feature and the image feature into the frequency attention module to obtain a frequency dimension attention vector, and multiply the frequency dimension attention vector with the audio feature and the image feature to obtain a frequency domain feature; The fusion subunit is used to fuse the time domain feature and the frequency domain feature to obtain the global feature.

[0087] Based on any of the above embodiments, the local attention unit is specifically used for: Input the multiple feature blocks into the local attention module to obtain a block-dimensional attention vector; The block-dimensional attention vector is multiplied by the multiple feature blocks to obtain the local features.

[0088] Based on any of the above embodiments, the device further includes a key area analysis unit, and the key area analysis unit is used to: Based on the block-dimensional attention vector, determining the attention weight corresponding to each feature block; Determining a key feature block from the feature blocks based on the attention weights corresponding to the feature blocks; The audio data is analyzed based on the time range and frequency range of the key feature block.

[0089] Based on any of the above embodiments, the feature extraction unit 710 is specifically used for: The audio data is sampled to a preset size by bilinear interpolation to obtain initial audio features, and the initial audio features are mean-variance regularized to obtain the audio features; The image data is sampled to a preset size through bilinear interpolation to obtain initial image features, and the initial image features are mean-variance regularized to obtain the image features.

[0090] Based on any of the above embodiments, the device further includes a model training unit, and the model training unit is used to: Acquire an initial model and collect unlabeled sample device data, wherein the sample device data includes sample audio data and sample image data of the sample device, and the initial model includes a global attention module, a convolution module, a local attention module, a mask module, an encoding module, and a decoding module; Performing feature extraction, feature alignment and feature regularization on the sample audio data and the sample image data to obtain sample audio features and sample image features; Inputting the sample audio features and the sample image features into the initial model to obtain the reconstructed features output by the initial model; Determine a reconstruction loss based on the difference between the reconstructed feature and the sample audio feature and the sample image feature, and apply the reconstruction loss to perform parameter iteration on the initial model to obtain a pre-trained state recognition model; The structure of the pre-trained state recognition model is adjusted, and the parameters of the adjusted model are fine-tuned based on sample device data with category labels to obtain the device state recognition model. The adjusted model is obtained by replacing the decoding module in the pre-trained state recognition model with a linear layer and a classification head.

[0091] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the device state recognition method, which includes: extracting features from the audio data of the device to be tested to obtain audio features, and extracting features from the image data of the device to be tested to obtain image features; inputting the audio features and the image features into the device state recognition model to obtain the state recognition result output by the device state recognition model; the device state recognition model is obtained by training based on sample audio features and sample image features, and the device state recognition model includes a global attention module and a local attention module, the global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features, and the local attention module is used to weight multiple feature blocks in the block dimension to obtain local features, and the multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0092] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0093] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the device state recognition method provided by the above methods, which method includes: performing feature extraction on the audio data of the device to be tested to obtain audio features, and performing feature extraction on the image data of the device to be tested to obtain image features; inputting the audio features and the image features into a device state recognition model to obtain a state recognition result output by the device state recognition model; the device state recognition model is obtained by training based on sample audio features and sample image features, and the device state recognition model includes a global attention module and a local attention module, the global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features, and the local attention module is used to weight multiple feature blocks in the block dimension to obtain local features, and the multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0094] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the device state recognition method provided by the above-mentioned methods, the method comprising: performing feature extraction on the audio data of the device to be tested to obtain audio features, and performing feature extraction on the image data of the device to be tested to obtain image features; inputting the audio features and the image features into a device state recognition model to obtain a state recognition result output by the device state recognition model; the device state recognition model is obtained by training based on sample audio features and sample image features, the device state recognition model comprises a global attention module and a local attention module, the global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features, the local attention module is used to weight multiple feature blocks in the block dimension to obtain local features, the multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0096] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A device status identification method, characterized in that: include: Performing feature extraction on audio data of the device under test to obtain audio features, and performing feature extraction on image data of the device under test to obtain image features; Inputting the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple feature blocks are obtained based on feature transformation and feature segmentation of the global features.

2. The device status identification method according to claim 1, characterized in that: The device state recognition model further includes a convolution module, a mask module, an encoding module and a classification module. The audio feature and the image feature are input into the device state recognition model to obtain a state recognition result output by the device state recognition model, including: Based on the global attention module, weighting the audio features and the image features in time dimension and frequency dimension to obtain global features; Based on the convolution module, feature extraction, feature transformation and feature segmentation are performed on the global features to obtain multiple feature blocks; Based on the local attention module, weighting the multiple feature blocks by block dimensions to obtain local features; Based on the mask module, masking the local features, and inputting the masked features into the encoding module to obtain encoding results; The encoding result is linearly transformed, and the transformed result is input into the classification module to obtain the state recognition result.

3. The device status identification method according to claim 2, characterized in that: The global attention module includes a time attention module and a frequency attention module. Based on the global attention module, the audio features and the image features are weighted in the time dimension and the frequency dimension to obtain global features, including: Inputting the audio feature and the image feature into the time attention module to obtain a time dimension attention vector, and multiplying the time dimension attention vector with the audio feature and the image feature to obtain a time domain feature; Inputting the audio feature and the image feature into the frequency attention module to obtain a frequency dimension attention vector, and multiplying the frequency dimension attention vector with the audio feature and the image feature to obtain a frequency domain feature; The time domain feature and the frequency domain feature are fused to obtain the global feature.

4. The device status identification method according to claim 2, characterized in that: The method of weighting the multiple feature blocks by block dimensions based on the local attention module to obtain local features includes: Input the multiple feature blocks into the local attention module to obtain a block-dimensional attention vector; The block-dimensional attention vector is multiplied by the multiple feature blocks to obtain the local features.

5. The device status identification method according to claim 4, characterized in that: Also includes: Based on the block-dimensional attention vector, determining the attention weight corresponding to each feature block; Determining a key feature block from the feature blocks based on the attention weights corresponding to the feature blocks; The audio data is analyzed based on the time range and frequency range of the key feature block.

6. The device status identification method according to claim 1, characterized in that: The step of extracting features from the audio data of the device under test to obtain audio features, and extracting features from the image data of the device under test to obtain image features, comprises: The audio data is sampled to a preset size by bilinear interpolation to obtain initial audio features, and the initial audio features are mean-variance regularized to obtain the audio features; The image data is sampled to a preset size through bilinear interpolation to obtain initial image features, and the initial image features are mean-variance regularized to obtain the image features.

7. The device status identification method according to any one of claims 1 to 6, characterized in that: The training steps of the device state recognition model include: Acquire an initial model and collect unlabeled sample device data, wherein the sample device data includes sample audio data and sample image data of the sample device, and the initial model includes a global attention module, a convolution module, a local attention module, a mask module, an encoding module, and a decoding module; Performing feature extraction, feature alignment and feature regularization on the sample audio data and the sample image data to obtain sample audio features and sample image features; Inputting the sample audio features and the sample image features into the initial model to obtain the reconstructed features output by the initial model; Determine a reconstruction loss based on the difference between the reconstructed feature and the sample audio feature and the sample image feature, and apply the reconstruction loss to perform parameter iteration on the initial model to obtain a pre-trained state recognition model; The structure of the pre-trained state recognition model is adjusted, and the parameters of the adjusted model are fine-tuned based on sample device data with category labels to obtain the device state recognition model. The adjusted model is obtained by replacing the decoding module in the pre-trained state recognition model with a linear layer and a classification head.

8. A device for identifying device status, characterized in that: include: A feature extraction unit is used to perform feature extraction on the audio data of the device under test to obtain audio features, and to perform feature extraction on the image data of the device under test to obtain image features; A state recognition unit, used for inputting the audio feature and the image feature into a device state recognition model to obtain a state recognition result output by the device state recognition model; The device state recognition model is obtained by training based on sample audio features and sample image features. The device state recognition model includes a global attention module and a local attention module. The global attention module is used to weight the audio features and the image features in the time dimension and the frequency dimension to obtain global features. The local attention module is used to weight multiple feature blocks in the block dimension to obtain local features. The multiple sample feature blocks are obtained based on feature transformation and feature segmentation of the global features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the device status identification method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the device status identification method according to any one of claims 1 to 7 is implemented.

Images