Device breakage detection method and apparatus, electronic device, and storage medium

Abstract

The application provides a device breakage detection method, device, electronic device and storage medium. The method comprises the following steps: collecting data of a device to be detected, wherein the data comprises sound data and image data, and the sound data is obtained by collecting knocking sound of the device to be detected; and performing breakage detection on the device to be detected based on the collected data to obtain a device detection result. The method, device, electronic device and storage medium provided by the application perform breakage detection on the device to be detected based on the collected data comprising sound data and image data, thereby improving the accuracy of device breakage detection, and the sound data is obtained by collecting knocking sound of the device to be detected, thereby further improving the accuracy of device breakage detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of device detection, and in particular to a device rupture detection method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid development of technology, the requirement for device quality detection is higher and higher. Device detection is related to factory operation safety, personal safety and other aspects, so it is necessary to detect the device. In particular, it is necessary to detect the rupture of the tank device.

[0003] At present, device rupture detection is mostly obtained by measuring instruments to measure the liquid level, pressure and other measurement parameters of the tank device, so as to manually judge the device detection result based on the measurement parameters. However, these measurement parameters cannot accurately reflect whether the tank is ruptured, and they are too dependent on expert experience and are subjective, resulting in reduced accuracy of device rupture detection and high labor costs. At the same time, manual judgment cannot give real-time device detection results, resulting in high delay of device rupture detection and being unable to timely make early warning treatment of the device detection result. SUMMARY

[0004] The present application provides a device rupture detection method, device, electronic device and storage medium to solve the defects of low accuracy, high delay and high labor cost of device rupture detection in the prior art, and to realize automatic device rupture detection with high accuracy.

[0005] The present application provides a device rupture detection method, comprising:

[0006] determining the collection data of the device to be detected, the collection data comprising sound data and image data, the sound data being obtained by collecting the knocking sound of the device to be detected;

[0007] based on the collection data, performing rupture detection on the device to be detected to obtain a device detection result.

[0008] According to the device rupture detection method provided by the present application, the collection data is used to perform rupture detection on the device to be detected to obtain a device detection result, which comprises:

[0009] based on a feature extraction model, performing feature extraction on the sound data to obtain a first feature extraction vector, and performing feature extraction on the image data to obtain a second feature extraction vector;

[0010] based on a device detection model, performing rupture detection on the first feature extraction vector and the second feature extraction vector to obtain a device detection result;

[0011] The feature extraction model is obtained based on unsupervised training of first sample collection data, and the first sample collection data includes first sample sound data and first sample image data.

[0012] The device detection model is obtained based on second sample collection data and sample device detection results corresponding to the second sample collection data, and the second sample collection data includes second sample sound data and second sample image data.

[0013] According to the device rupture detection method provided by the application, the feature extraction model is trained based on the following steps:

[0014] The feature vector of the first sample sound data is subjected to mask processing to obtain a mask position feature and an unmasked position feature.

[0015] The feature extraction model is trained based on the first encoding vector and the maximum class center vector.

[0016] The second encoding vector is subjected to clustering processing to obtain a clustering result, and a maximum class center vector of the second encoding vector is determined based on the clustering result.

[0017] The feature extraction model is trained based on the similarity between the first encoding vector and the maximum class center vector.

[0018] According to the device rupture detection method provided by the application, the feature vector is determined based on the following steps:

[0019] The first sample sound data is encoded to obtain an encoding vector.

[0020] The encoding vector is fused with a corresponding preset sound modal vector to obtain the feature vector, and the preset sound modal vector is used to represent sound modal information.

[0021] According to the device rupture detection method provided by the application, the loss function of the device detection model includes a classification loss function, and the second sample collection data includes normal sample data and abnormal sample data.

[0022] The classification loss function is determined based on a classification loss, and the classification loss includes a first classification loss and a second classification loss.

[0023] The first sample equipment detection result is obtained by performing breakage detection on a third feature extraction vector based on the equipment detection model, the second sample equipment detection result is obtained by performing breakage detection on a fourth feature extraction vector based on the equipment detection model, the third feature extraction vector is obtained by performing feature extraction on the normal sample data based on the feature extraction model, and the fourth feature extraction vector is obtained by performing feature extraction on the abnormal sample data based on the feature extraction model.

[0024] According to the equipment breakage detection method provided by the application, the classification loss further includes a third classification loss and a fourth classification loss;

[0025] The third classification loss is determined based on a third sample equipment detection result and a sample equipment detection result corresponding to the normal sample data, and the fourth classification loss is determined based on a fourth sample equipment detection result and a sample equipment detection result corresponding to the abnormal sample data.

[0026] The third sample equipment detection result is obtained by performing breakage detection on a fifth feature extraction vector based on the equipment detection model, and the fourth sample equipment detection result is obtained by performing breakage detection on a sixth feature extraction vector based on the equipment detection model, the fifth feature extraction vector is obtained by performing mask processing on the third feature extraction vector, and the sixth feature extraction vector is obtained by performing mask processing on the fourth feature extraction vector.

[0027] According to the equipment breakage detection method provided by the application, the loss function further includes a contrast loss function;

[0028] The contrast loss function is determined based on a first similarity and a second similarity, the first similarity is determined based on the similarity between normal features, and the second similarity is determined based on the similarity between normal features and abnormal features, the normal features include normal sample features and normal sample mask features, and the abnormal features include abnormal sample features and abnormal sample mask features.

[0029] The normal sample features are obtained by encoding the third feature extraction vector based on the equipment detection model, the abnormal sample features are obtained by encoding the fourth feature extraction vector based on the equipment detection model, the normal sample mask features are obtained by encoding the fifth feature extraction vector based on the equipment detection model, and the abnormal sample mask features are obtained by encoding the sixth feature extraction vector based on the equipment detection model.

[0030] According to the equipment breakage detection method provided by the application, the loss function further includes a reconstruction loss function;

[0031] The reconstruction loss function is determined based on a third similarity and a fourth similarity, wherein the third similarity is determined based on a similarity between a normal sample feature and a normal sample mask feature, and the fourth similarity is determined based on a similarity between an abnormal sample feature and an abnormal sample mask feature;

[0032] The normal sample feature is obtained by encoding the third feature extraction vector based on the device detection model, the abnormal sample feature is obtained by encoding the fourth feature extraction vector based on the device detection model, the normal sample mask feature is obtained by encoding the fifth feature extraction vector based on the device detection model, and the abnormal sample mask feature is obtained by encoding the sixth feature extraction vector based on the device detection model.

[0033] According to a device rupture detection method provided by the present invention, the device rupture detection is performed on the first feature extraction vector and the second feature extraction vector based on the device detection model to obtain a device detection result, including:

[0034] Based on the device detection model, performing crack detection on the first feature extraction vector and the second feature extraction vector to obtain a first device detection result, and performing crack detection on the first mask feature vector and the second mask feature vector to obtain a second device detection result, where the first mask feature vector is obtained by masking the first feature extraction vector, and the second mask feature vector is obtained by masking the second feature extraction vector;

[0035] The device detection result is determined based on similarities between the first device detection result and a plurality of preset device detection results, and similarities between the second device detection result and a plurality of preset device detection results.

[0036] According to a device rupture detection method provided by the present invention, the device rupture detection is performed on the first feature extraction vector and the second feature extraction vector based on the device detection model to obtain a device detection result, and then further includes:

[0037] When the device detection result has not been manually confirmed and the confidence level of the device detection result is greater than a preset confidence threshold, training the device detection model based on the collected data and the device detection result;

[0038] When the device detection result is manually confirmed and the device detection result is a correct result, training the device detection model based on the collected data and the device detection result;

[0039] In a case where the device detection result is confirmed by a human being and the device detection result is an error result, the device detection model is trained based on the collection data and a sample device detection result corresponding to the collection data.

[0040] The application further provides a device breakage detection device, comprising:

[0041] A determination module is configured to determine collection data of a to-be-detected device, wherein the collection data comprises sound data and image data, and the sound data is obtained by collecting knocking sound of the to-be-detected device.

[0042] A detection module is configured to perform breakage detection on the to-be-detected device based on the collection data, and obtain a device detection result.

[0043] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the device breakage detection method according to any one of the above embodiments when executing the program.

[0044] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the device breakage detection method according to any one of the above embodiments.

[0045] The device breakage detection method, device, electronic device and storage medium provided by the application perform breakage detection on a to-be-detected device based on collection data comprising sound data and image data, thereby improving the accuracy of device breakage detection, and the sound data is obtained by collecting knocking sound of the to-be-detected device, thereby further improving the accuracy of device breakage detection. Meanwhile, the device detection result is determined by the detection device, which saves labor cost compared with manually judging the device detection result, and the device detection result can be given in real time and automatically, thereby reducing the delay of device breakage detection and realizing automatic device breakage detection. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0047] Figure 1 One of the flowcharts of the device breakage detection method provided by the present application;

[0048] Figure 2 The second flowchart of the device breakage detection method provided by the present application;

[0049] Figure 3 Figure 3 is a flowchart of a device rupture detection method provided by the present application;

[0050] Figure 4 Figure 4 is a structural diagram of a device rupture detection apparatus provided by the present application;

[0051] Figure 5 Figure 5 is a structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0053] The present application proposes the following embodiments. Figure 1 Figure 1 is a flowchart of a device rupture detection method provided by the present application, which comprises the following steps: Figure 1

[0054] In step 110, collection data of a device to be detected is determined, the collection data comprising sound data and image data, the sound data being obtained by collecting knocking sound of the device to be detected.

[0055] Here, the device to be detected is a device that needs to be detected for rupture. For example, the device to be detected is a tank device, and rupture detection needs to be performed on the tank device.

[0056] Here, the sound data is obtained by collecting sound of knocking the device to be detected, and further, the sound data is obtained by collecting sound of knocking the device to be detected by a fixing device. The image data can comprise images of different screenshots of the device to be detected.

[0057] In an embodiment, the collection data is preprocessed to obtain preprocessed collection data. The preprocessing manner can comprise, but is not limited to, noise reduction, enhancement, windowing, etc.

[0058] In step 120, rupture detection is performed on the device to be detected based on the collection data to obtain a device detection result.

[0059] Specifically, rupture detection is performed on a feature extraction vector of the collection data based on a device detection model to obtain a device detection result. The specific structure of the device detection model can be set according to actual needs, and the present application does not make a specific limitation on this. ​

[0060] In an embodiment, the device detection result includes a plurality of categories of detection results, and the plurality of categories of detection results are divided into normal detection results and abnormal detection results.

[0061] The device rupture detection method provided by the embodiment of the present application is based on the collected data including sound data and image data to perform rupture detection on the device to be detected, thereby improving the accuracy of the device rupture detection, and the sound data is obtained by collecting the knocking sound of the device to be detected, thereby further improving the accuracy of the device rupture detection. Meanwhile, the device detection result is determined by the detection device, compared with the artificial judgment of the device detection result, the labor cost is saved, and the device detection result can be given in real time and automatically, thereby reducing the delay of the device rupture detection and realizing the automatic device rupture detection.

[0062] Based on the above embodiment, Figure 2 For the flowchart of the device rupture detection method provided by the present application, as shown in Figure 2 The step 120 includes:

[0063] In step 121, the sound data is subjected to feature extraction based on a feature extraction model to obtain a first feature extraction vector, and the image data is subjected to feature extraction based on the feature extraction model to obtain a second feature extraction vector.

[0064] Here, the feature extraction model is used to extract features from the collected data to obtain a feature extraction vector corresponding to the collected data. The specific structure of the feature extraction model can be set according to actual needs, and the embodiment of the present application does not make specific limitations.

[0065] In some embodiments, the collected data is encoded to obtain an encoded vector, and the encoded vector is subjected to feature extraction based on a feature extraction model to obtain a feature extraction vector. The collected data includes a plurality of modalities of data, i.e., includes sound data and image data, and based on this, the plurality of modalities of data are respectively encoded to obtain an encoded vector corresponding to each modality. Based on the feature extraction model, each encoded vector is subjected to feature extraction to obtain a feature extraction vector corresponding to each modality.

[0066] In an embodiment, the image data can be subjected to local binary feature coding to obtain an encoded vector corresponding to the image modality, and the sound data can be subjected to Fourier transform coding to obtain an encoded vector corresponding to the sound modality, and the encoded vector corresponding to the sound modality is a frequency domain feature vector.

[0067] In an embodiment, the corresponding preset modality vector is fused with each coding vector respectively to obtain a feature vector corresponding to each modality; and the feature extraction model is used to perform feature extraction on each feature vector to obtain a feature extraction vector corresponding to each modality. The preset modality vector is used to represent modality information. The fusion manner of the coding vector and the preset modality vector can be a concatenation manner, and can also be other fusion manners, which are not limited in the embodiment of the present application.

[0068] In an embodiment, the preset modality vector can be obtained by using onehot coding. For example, the image modality vector can be [1, 0], and the sound modality vector can be [0, 1].

[0069] It can be understood that the coding vector is fused with the corresponding preset modality vector, so that the feature extraction model can distinguish between different modalities when performing feature extraction, thereby improving the performance of the feature extraction model.

[0070] The feature extraction model is obtained by unsupervised training based on first sample collection data, and the first sample collection data includes first sample sound data and first sample image data.

[0071] Here, the first sample collection data corresponds to the above-mentioned collection data. The first sample collection data includes sufficient data to train the feature extraction model of the collection data.

[0072] In some embodiments, the first sample collection data is coded to obtain a coding vector, and the feature extraction model is trained based on the coding vector. The first sample collection data includes collection data of multiple modalities, i.e., first sample sound data and first sample image data, based on which the collection data of multiple modalities is coded respectively to obtain a coding vector corresponding to each modality; and the feature extraction model is trained based on the coding vector corresponding to each modality.

[0073] In an embodiment, the first sample image data can be locally binary feature coded to obtain an image modality corresponding coding vector; and the first sample sound data can be Fourier transform coded to obtain a sound modality corresponding coding vector, which is a frequency domain feature vector.

[0074] In an embodiment, the corresponding preset modality vector is fused with each coding vector respectively to obtain a feature vector corresponding to each modality; and the feature extraction model is trained based on the feature vector corresponding to each modality. The preset modality vector is used to represent modality information. The fusion manner of the coding vector and the preset modality vector can be a concatenation manner, and can also be other fusion manners, which are not limited in the embodiment of the present application.

[0075] In an embodiment, the preset modality vector can be obtained by onehot encoding. For example, the image modality vector can be [1, 0], and the sound modality vector can be [0, 1].

[0076] It can be understood that the encoding vector is fused with the corresponding preset modality vector, so that the feature extraction model can distinguish between different modalities when performing feature extraction, thereby improving the performance of the feature extraction model.

[0077] In step 122, the first feature extraction vector and the second feature extraction vector are subjected to breakage detection based on a device detection model to obtain a device detection result.

[0078] Here, the device detection model is used to perform breakage detection on the first feature extraction vector and the second feature extraction vector to obtain a device detection result corresponding to the collection data.

[0079] In an embodiment, the first feature extraction vector is subjected to mask processing to obtain a first mask feature vector, and the second feature extraction vector is subjected to mask processing to obtain a second mask feature vector; the first feature extraction vector and the second feature extraction vector are subjected to breakage detection based on the device detection model to obtain a first device detection result, and the first mask feature vector and the second mask feature vector are subjected to breakage detection to obtain a second device detection result.

[0080] In another embodiment, the first feature extraction vector and the second feature extraction vector are respectively subjected to mask processing to obtain a first mask feature vector and a second mask feature vector; the first mask feature vector and the second mask feature vector are subjected to breakage detection based on the device detection model to obtain a device detection result.

[0081] In an embodiment, the device detection result includes a plurality of category detection results, and the plurality of category detection results are divided into normal detection results and abnormal detection results.

[0082] The device detection model is trained based on second sample collection data and sample device detection results corresponding to the second sample collection data, and the second sample collection data includes second sample sound data and second sample image data.

[0083] Here, the second sample collection data can be the same as or different from the above-mentioned first sample collection data. The second sample collection data corresponds to the above-mentioned collection data.

[0084] In some embodiments, the second sample collection data is encoded to obtain an encoded vector, the encoded vector is feature extracted based on a feature extraction model to obtain a feature extraction vector, and the device detection model is trained based on the feature extraction vector. The second sample collection data includes collection data of multiple modalities, i.e., includes second sample sound data and second sample image data. Based on this, the collection data of multiple modalities is respectively encoded to obtain an encoded vector corresponding to each modality. The feature extraction model is used to perform feature extraction on each encoded vector to obtain a feature extraction vector corresponding to each modality. The device detection model is trained based on the feature extraction vector.

[0085] In an embodiment, the second sample image data can be locally binary feature encoded to obtain an encoded vector corresponding to the image modality. The second sample sound data can be Fourier transform encoded to obtain an encoded vector corresponding to the sound modality, which is a frequency domain feature vector.

[0086] In an embodiment, each encoded vector is fused with a corresponding preset modality vector to obtain a feature vector corresponding to each modality. The feature extraction model is used to perform feature extraction on each feature vector to obtain a feature extraction vector corresponding to each modality. The device detection model is trained based on the feature extraction vector corresponding to each modality. The preset modality vector is used to represent modality information. The fusion manner of the encoded vector and the preset modality vector can be a concatenation manner, and can also be other fusion manners, which are not limited in the embodiments of the present application.

[0087] In an embodiment, the onehot (one-hot) encoding can be used to obtain the preset modality vector. For example, the image modality vector can be [1, 0], and the sound modality vector can be [0, 1].

[0088] It can be understood that the encoded vector is fused with the corresponding preset modality vector, so that the feature extraction model can distinguish between modalities when performing feature extraction, thereby improving the performance of the feature extraction model and improving the accuracy of the device detection model.

[0089] In an embodiment, the first feature extraction vector and the second feature extraction vector are respectively subjected to mask processing to obtain a first mask feature vector and a second mask feature vector. The device detection model is trained based on the first feature extraction vector, the second feature extraction vector, the first mask feature vector, and the second mask feature vector.

[0090] The device rupture detection method provided by the embodiment of the present application extracts features of collected data of a to-be-detected device based on a feature extraction model, obtains a first feature extraction vector and a second feature extraction vector, and improves the feature extraction accuracy of the feature extraction model, which is obtained based on unsupervised training of first sample collected data, and saves labor costs, because the unsupervised training does not require manual label marking. The device detection model is used to detect the rupture of the first feature extraction vector and the second feature extraction vector, and obtain a device detection result. The accuracy of the first feature extraction vector and the second feature extraction vector is improved, which improves the rupture detection accuracy of the device detection model. The device detection model is obtained based on second sample collected data and a sample device detection result corresponding to the second sample collected data, which improves the rupture detection accuracy of the device detection model. Compared with manual judgment of the device detection result, the embodiment of the present application saves labor costs, and can give the device detection result in real time and automatically based on the device detection model, thereby reducing the delay of device detection and realizing automatic device rupture detection.

[0091] Based on any of the above embodiments, Figure 3 For the flowchart of the device rupture detection method provided by the present application, Figure 3 As shown in the figure, the feature extraction model is trained based on the following steps:

[0092] In step 310, the feature vector of the first sample sound data is subjected to mask processing to obtain a mask position feature and an unmasked position feature.

[0093] Here, the feature vector is used to represent the information of the first sample sound data. Specifically, the first sample sound data is encoded to obtain the feature vector.

[0094] In an embodiment, the first sample sound data can be subjected to Fourier transform encoding to obtain a feature vector corresponding to a sound mode, and the feature vector corresponding to the sound mode is a frequency domain feature vector.

[0095] Here, the mask processing method can be set according to actual needs, for example, random mask, nonlinear activation function Relu, dropout mechanism, etc.

[0096] In step 220, the feature extraction model is used to encode the mask position feature to obtain a first encoding vector, and encode the unmasked position feature to obtain a second encoding vector.

[0097] Specifically, the encoding layer of the feature extraction model is used to extract high-dimensional information from the mask position feature and the unmasked position feature to obtain the first encoding vector and the second encoding vector.

[0098] ​At step 230, the second encoding vectors are clustered to obtain a clustering result, and a maximum class center vector of the second encoding vectors is determined based on the clustering result.

[0099] Here, the clustering manner can be set according to actual needs, for example, a hierarchical clustering algorithm, a partition clustering algorithm, a fuzzy clustering algorithm, a density-based clustering algorithm, etc.

[0100] Here, the maximum class center vector is a center vector of a maximum class in the clustering result, the maximum class includes the most second encoding vectors, and the center vector is a cluster center vector of the maximum class.

[0101] In an embodiment, the second encoding vectors are hierarchically clustered to obtain a clustering result, a maximum class of the second encoding vectors is determined based on the clustering result, and a maximum class center vector of the maximum class is determined.

[0102] At step 240, the feature extraction model is trained based on the similarity between the first encoding vectors and the maximum class center vector.

[0103] Here, the similarity can be calculated according to various similarity algorithms, for example, the similarity can be calculated by a cosine distance similarity algorithm.

[0104] Specifically, the feature extraction model is trained based on a loss function of the feature extraction model. The loss function is determined based on the similarity between the first encoding vectors and the maximum class center vector. The number of the first encoding vectors can be multiple, and based on this, the loss function is determined based on multiple similarities between the multiple first encoding vectors and the maximum class center vector. For example, the loss function is as follows:

[0105]

[0106] In the formula, M represents the number of the first encoding vectors, f1 represents the maximum class center vector, f i represents the i-th first encoding vector, D(f1, f i ) represents the cosine distance between the i-th first encoding vector and the maximum class center vector, that is, the greater the cosine distance, the smaller the similarity, and the greater the loss.

[0107] Furthermore, the steps for training the feature extraction model based on the first sample image data are as follows: masking the feature vectors of the first sample image data to obtain masked position features and unmasked position features; encoding the masked position features based on the feature extraction model to obtain a first encoding vector, and encoding the unmasked position features to obtain a second encoding vector; clustering the second encoding vector to obtain a clustering result, and determining the maximum cluster center vector of the second encoding vector based on the clustering result; and training the feature extraction model based on the similarity between the first encoding vector and the maximum cluster center vector. These specific training steps are identical to those for training the first sample sound data and will not be detailed here.

[0108] The device rupture detection method provided by an embodiment of the present invention performs masking processing on the feature vector of the first sample sound data to obtain masked position features and unmasked position features; based on the feature extraction model, the masked position features are encoded to obtain a first coding vector, and the unmasked position features are encoded to obtain a second coding vector; the second coding vector is clustered to obtain a clustering result, and the maximum class center vector of the second coding vector is determined based on the clustering result; based on the similarity between the first coding vector and the maximum class center vector, the feature extraction model is trained, thereby realizing unsupervised training without the need for manual labeling, thereby saving labor costs, and shortening the distance between the first coding vector and the maximum class center vector, so that the feature extraction model can extract effective high-dimensional features, thereby improving the feature extraction accuracy of the feature extraction model.

[0109] Based on any of the above embodiments, in this method, the feature vector is determined based on the following steps:

[0110] Encoding the first sample sound data to obtain an encoding vector;

[0111] The encoding vector is fused with a corresponding preset sound modal vector to obtain the feature vector, where the preset sound modal vector is used to represent the sound modal information.

[0112] Exemplarily, the first sample sound data may be subjected to Fourier transform encoding to obtain an encoding vector corresponding to the sound mode, where the encoding vector corresponding to the sound mode is a frequency domain feature vector.

[0113] Here, the encoding vector and the preset sound modality vector may be fused in a concatenated manner, or in other fusion manners, which is not specifically limited in the embodiment of the present invention.

[0114] In one embodiment, onehot encoding may be used to obtain the preset sound modality vector. For example, the preset sound modality vector may be [0, 1].

[0115] In addition, the feature vector of the first sample image data is determined based on the following steps: encoding the first sample image data to obtain an encoding vector; and fusing the encoding vector with a corresponding preset image modality vector to obtain the feature vector, the preset image modality vector being used to represent image modality information. The specific steps are the same as the corresponding steps of the first sample sound data, which will not be repeated here.

[0116] The device rupture detection method provided by the embodiments of the present application fuses the encoding vector with the corresponding preset sound modality vector, so that the feature extraction model can distinguish between different modalities in advance when performing feature extraction, thereby improving the performance of the feature extraction model.

[0117] Based on any of the above embodiments, in the method, the loss function of the device detection model includes a classification loss function, the second sample collection data includes normal sample data and abnormal sample data; the classification loss function is determined based on a classification loss, the classification loss includes a first classification loss and a second classification loss, the first classification loss is determined based on a first sample device detection result and a sample device detection result corresponding to the normal sample data, and the second classification loss is determined based on a second sample device detection result and a sample device detection result corresponding to the abnormal sample data; the first sample device detection result is obtained by performing rupture detection on a third feature extraction vector based on the device detection model, the second sample device detection result is obtained by performing rupture detection on a fourth feature extraction vector based on the device detection model, the third feature extraction vector is obtained by performing feature extraction on the normal sample data based on the feature extraction model, and the fourth feature extraction vector is obtained by performing feature extraction on the abnormal sample data based on the feature extraction model.

[0118] Here, the normal sample data is collection data of a device without problems, and the abnormal sample data is collection data of a device with problems. In an embodiment, the sample device detection result includes a plurality of categories of detection results, and the plurality of categories of detection results are divided into normal detection results and abnormal detection results. Based on this, the normal sample data is sample data labeled with a normal detection result label, and the abnormal sample data is sample data labeled with an abnormal detection result label.

[0119] Here, the sample device detection result corresponding to the normal sample data includes a normal detection result label and a device detection result label. The sample device detection result corresponding to the abnormal sample data includes an abnormal detection result label and a device detection result label.

[0120] Here, the classification loss function is determined based on the loss value of the classification loss. That is, the classification loss function is determined based on the first classification loss, or the classification loss function is determined based on the second classification loss. The classification loss function is used to guide the device detection model to correctly predict the sample category.

[0121] In an embodiment, the classification loss function is a cross-entropy loss function, based on which, based on the first sample equipment detection result and the sample equipment detection result corresponding to the normal sample data, the loss value of the classification loss function is determined; based on the second sample equipment detection result and the sample equipment detection result corresponding to the abnormal sample data, the loss value of the classification loss function is determined.

[0122] In an embodiment, the equipment detection model includes an encoding layer and a classification layer, based on which, the third feature extraction vector is encoded to obtain a normal sample feature, and the fourth feature extraction vector is encoded to obtain an abnormal sample feature; based on the classification layer, the normal sample feature is classified and predicted to obtain the first sample equipment detection result, and the abnormal sample feature is classified and predicted to obtain the second sample equipment detection result.

[0123] In some embodiments, the normal sample data is encoded to obtain a first target encoding vector, and the abnormal sample data is encoded to obtain a second target encoding vector, based on the feature extraction model, the first target encoding vector is feature-extracted to obtain a third feature extraction vector, and the second target encoding vector is feature-extracted to obtain a fourth feature extraction vector.

[0124] The second sample collection data includes collection data of multiple modalities, i.e., including second sample sound data and second sample image data, based on which, the collection data of multiple modalities is respectively encoded to obtain the first target encoding vector and the second target encoding vector corresponding to each modality; based on the feature extraction model, the first target encoding vector corresponding to each modality is feature-extracted to obtain the second feature extraction vector corresponding to each modality, and the second target encoding vector corresponding to each modality is feature-extracted to obtain the fourth feature extraction vector corresponding to each modality.

[0125] In an embodiment, the first target encoding vector and the second target encoding vector corresponding to each modality are respectively fused with the corresponding preset modality vector to obtain the first feature vector and the second feature vector corresponding to each modality; the first feature vector corresponding to each modality is feature-extracted to obtain the third feature extraction vector corresponding to each modality, and the second feature vector corresponding to each modality is feature-extracted to obtain the fourth feature extraction vector corresponding to each modality. The preset modality vector is used to represent modality information. The fusion mode of the encoding vector and the preset modality vector can be a concatenation mode, and of course can also be other fusion modes, which are not limited in the embodiments of the present application.

[0126] In an embodiment, the preset modality vector can be obtained by onehot (one-hot) encoding. For example, the image modality vector can be [1, 0], and the sound modality vector can be [0, 1].

[0127] In some embodiments, the loss function further comprises a contrastive loss function; the contrastive loss function is determined based on a first similarity and a second similarity, the first similarity is determined based on a similarity between normal sample features, and the second similarity is determined based on a similarity between the normal sample features and abnormal sample features; the normal sample features are obtained by encoding the third feature extraction vectors based on the device detection model, and the abnormal sample features are obtained by encoding the fourth feature extraction vectors based on the device detection model.

[0128] The contrastive loss function is used to guide the similarity between the normal sample features obtained by the device detection model to be greater than the similarity between the normal sample features and the abnormal sample features obtained by the device detection model, i.e., to guide the first similarity to be greater than the second similarity.

[0129] In an embodiment, the contrastive loss function can be a triplet loss. For example, the contrastive loss function is as follows:

[0130] L tri = max (‖f a -f a ‖-‖f a -f b ‖+ margin, 0) ;

[0131] In the formula, L tri represents the contrastive loss function, f a represents the normal sample features, f b represents the abnormal sample features, ‖f a -f a ‖ represents the first similarity, ‖f a -f b ‖ represents the second similarity, and margin represents a constant greater than 0.

[0132] The device rupture detection method provided by the embodiments of the present application includes normal sample data and abnormal sample data in the second sample collection data, thereby training the device detection model based on the classification loss function, and further guiding the device detection model to correctly predict the sample category, i.e., effectively and accurately distinguishing the normal sample data and the abnormal sample data, and further improving the detection accuracy of the device detection model.

[0133] In any of the above embodiments, the classification loss further includes a third classification loss and a fourth classification loss; the third classification loss is determined based on a third sample equipment detection result and a sample equipment detection result corresponding to the normal sample data, and the fourth classification loss is determined based on a fourth sample equipment detection result and a sample equipment detection result corresponding to the abnormal sample data; the third sample equipment detection result is obtained by performing breakage detection on the fifth feature extraction vector based on the equipment detection model, and the fourth sample equipment detection result is obtained by performing breakage detection on the sixth feature extraction vector based on the equipment detection model; the fifth feature extraction vector is obtained by performing mask processing on the third feature extraction vector, and the sixth feature extraction vector is obtained by performing mask processing on the fourth feature extraction vector.

[0134] Specifically, the classification loss function is determined based on the third classification loss, or the classification loss function is determined based on the fourth classification loss.

[0135] In an embodiment, the classification loss function is a cross-entropy loss function, based on which, the loss value of the classification loss function is determined based on the third sample equipment detection result and the sample equipment detection result corresponding to the normal sample data, and the loss value of the classification loss function is determined based on the fourth sample equipment detection result and the sample equipment detection result corresponding to the abnormal sample data.

[0136] Here, the mask processing method can be set according to actual needs, for example, random mask, nonlinear activation function Relu, dropout mechanism, etc.

[0137] In an embodiment, the equipment detection model includes an encoding layer and a classification layer, based on which, the fifth feature extraction vector is encoded to obtain normal sample mask features, and the sixth feature extraction vector is encoded to obtain abnormal sample mask features; based on the classification layer, the normal sample mask features are classified and predicted to obtain the third sample equipment detection result, and the abnormal sample mask features are classified and predicted to obtain the fourth sample equipment detection result.

[0138] The device rupture detection method provided by the embodiment of the application further includes a third classification loss and a fourth classification loss, the third classification loss is determined based on a third sample device detection result and a sample device detection result corresponding to normal sample data, the fourth classification loss is determined based on a fourth sample device detection result and a sample device detection result corresponding to abnormal sample data, the third sample device detection result is obtained by performing device detection on a fifth feature extraction vector based on the device detection model, the fourth sample device detection result is obtained by performing device detection on a sixth feature extraction vector based on the device detection model, the fifth feature extraction vector is obtained by performing mask processing on the third feature extraction vector, and the sixth feature extraction vector is obtained by performing mask processing on the fourth feature extraction vector, so that features irrelevant to the classification of the device detection model are filtered out, and the detection accuracy of the device detection model is further improved, and the anti-interference performance of the device detection model is improved.

[0139] Based on any of the above embodiments, the loss function further includes a contrast loss function; the contrast loss function is determined based on a first similarity and a second similarity, the first similarity is determined based on the similarity between normal features, the second similarity is determined based on the similarity between normal features and abnormal features, the normal features include normal sample features and normal sample mask features, and the abnormal features include abnormal sample features and abnormal sample mask features; the normal sample features are obtained by encoding the third feature extraction vector based on the device detection model, the abnormal sample features are obtained by encoding the fourth feature extraction vector based on the device detection model, the normal sample mask features are obtained by encoding the fifth feature extraction vector based on the device detection model, and the abnormal sample mask features are obtained by encoding the sixth feature extraction vector based on the device detection model.

[0140] The contrast loss function is used to guide the similarity between normal features obtained by encoding based on the device detection model to be greater than the similarity between normal features and abnormal features obtained by encoding based on the device detection model, that is, to guide the first similarity to be greater than the second similarity.

[0141] In an embodiment, the contrast loss function can be a triplet loss. For example, the contrast loss function is as follows:

[0142] L tri = max (‖f a -f a ‖-‖f a -f b ‖+margin, 0);

[0143] In the formula, L trirepresents a contrastive loss function, f a represents a normal feature, f b represents an abnormal feature, ||f a -f a ||represents a first similarity, ||f a -f b ||represents a second similarity, and margin represents a constant greater than 0.

[0144] The device rupture detection method provided by the embodiments of the present application trains a device detection model based on a contrastive loss function, so that normal features are closer to each other and normal features are more distant from abnormal features, thereby improving the robustness and accuracy of the device detection model.

[0145] Based on any of the above embodiments, in the method, the loss function further includes a reconstruction loss function; the reconstruction loss function is determined based on a third similarity and a fourth similarity, the third similarity is determined based on a similarity between a normal sample feature and a normal sample mask feature, and the fourth similarity is determined based on a similarity between an abnormal sample feature and an abnormal sample mask feature; the normal sample feature is obtained by encoding the third feature extraction vector based on the device detection model, the abnormal sample feature is obtained by encoding the fourth feature extraction vector based on the device detection model, the normal sample mask feature is obtained by encoding the fifth feature extraction vector based on the device detection model, and the abnormal sample mask feature is obtained by encoding the sixth feature extraction vector based on the device detection model.

[0146] The reconstruction loss function is used to guide the normal sample feature and the normal sample mask feature obtained by encoding the device detection model to be consistent, and is used to guide the abnormal sample feature and the abnormal sample mask feature obtained by encoding the device detection model to be consistent, that is, to guide the third similarity to be as small as possible, and to guide the fourth similarity to be as small as possible.

[0147] In an embodiment, the reconstruction loss function can be a mse loss (mean square loss function). For example, the reconstruction loss function is as follows:

[0148] L r =||f1-f2||+||f3-f4||;

[0149] In the formula, L r represents a reconstruction loss function, f1 represents a normal sample feature, f2 represents a normal sample mask feature, f3 represents an abnormal sample feature, f4 represents an abnormal sample mask feature, ||f1-f2|| represents a third similarity, and ||f3-f4|| represents a fourth similarity.

[0150] The device rupture detection method provided by the embodiments of the present application trains a device detection model based on a reconstruction loss function, so that normal sample features and normal sample mask features tend to be consistent, and abnormal sample features and abnormal sample mask features tend to be consistent, thereby improving the robustness and accuracy of the device detection model.

[0151] Based on any of the above embodiments, the step 122 includes:

[0152] Based on the device detection model, rupture detection is performed on the first feature extraction vector and the second feature extraction vector to obtain a first device detection result, and rupture detection is performed on the first mask feature vector and the second mask feature vector to obtain a second device detection result, the first mask feature vector being obtained by performing mask processing on the first feature extraction vector, and the second mask feature vector being obtained by performing mask processing on the second feature extraction vector.

[0153] Based on the similarity of the first device detection result and a plurality of preset device detection results, and the similarity of the second device detection result and the plurality of preset device detection results, the device detection result is determined.

[0154] In an embodiment, the first device detection result includes a plurality of category detection results, and the second device detection result includes a plurality of category detection results, the plurality of category detection results being divided into normal detection results and abnormal detection results.

[0155] Here, the plurality of preset device detection results are conventional device detection results set in advance. Further, the plurality of preset device detection results are abnormal detection results, so that based on the similarity of the device detection result and the plurality of preset device detection results, it is determined whether the device detection result is an abnormal detection result, and the specific detection result of the device detection result is determined, for example, the plurality of preset device detection results are a registered abnormal sample library.

[0156] Specifically, based on the similarity of the first device detection result and the plurality of preset device detection results, a preset device detection result corresponding to a maximum similarity is determined, and based on the preset device detection result corresponding to the maximum similarity, a first similarity score is determined; based on the similarity of the second device detection result and the plurality of preset device detection results, a preset device detection result corresponding to a maximum similarity is determined, and based on the preset device detection result corresponding to the maximum similarity, a second similarity score is determined; the first similarity score and the second similarity score are fused to obtain a fused similarity score; and based on the fused similarity score, the device detection result is determined.

[0157] The preset device detection result and the similarity score have a mapping relationship, which can be set in advance, for example, the mapping relationship is taken as a hyperparameter of the model.

[0158] The fusion manner can include but is not limited to addition, weighted aggregation, etc. In an embodiment, the first similarity score and the second similarity score are weighted aggregated based on a first weight of the first similarity score and a second weight of the second similarity score, and the first weight and the second weight can be set according to actual needs.

[0159] In an embodiment, when the fusion similarity score is greater than a preset threshold, it is determined that the equipment detection result is an abnormal detection result; and when the fusion similarity score is less than the preset threshold, it is determined that the equipment detection result is a normal detection result. The preset threshold can be set according to actual needs.

[0160] The equipment rupture detection method provided by the embodiment of the application is based on the equipment detection model to perform rupture detection on the feature extraction vector to obtain a first equipment detection result, and perform rupture detection on the mask feature vector to obtain a second equipment detection result, so as to determine the equipment detection result based on the similarity of the first equipment detection result and a plurality of preset equipment detection results and the similarity of the second equipment detection result and the plurality of preset equipment detection results. Since the mask processing can filter unnecessary features, the accuracy of the second equipment detection result is improved, and the accuracy of the equipment detection result is improved based on the similarity of the first equipment detection result and the similarity of the second equipment detection result, which further improves the accuracy of the equipment rupture detection.

[0161] Based on any of the above embodiments, after the step 122, the method further includes:

[0162] In a case where the equipment detection result is not manually confirmed and the confidence of the equipment detection result is greater than a preset confidence threshold, the equipment detection model is trained based on the collected data and the equipment detection result;

[0163] In a case where the equipment detection result is manually confirmed and the equipment detection result is a correct result, the equipment detection model is trained based on the collected data and the equipment detection result;

[0164] In a case where the equipment detection result is manually confirmed and the equipment detection result is an incorrect result, the equipment detection model is trained based on the collected data and a sample equipment detection result corresponding to the collected data.

[0165] It should be noted that the equipment detection result can be manually confirmed to confirm whether the equipment detection result is a correct result.

[0166] Here, the first case, in the case that the device detection result is not manually confirmed, and the confidence of the device detection result is greater than the pre-set confidence threshold, the device detection model is trained based on the collection data and the sample device detection result corresponding to the collection data. The specific training method refers to the above embodiments, which will not be repeated here. The sample device detection result corresponding to the collection data is the device detection result. The pre-set confidence threshold can be set according to actual needs.

[0167] Here, the second case, in the case that the device detection result is manually confirmed, and the device detection result is a correct result, the device detection model is trained based on the collection data and the sample device detection result corresponding to the collection data. The specific training method refers to the above embodiments, which will not be repeated here. The sample device detection result corresponding to the collection data is the device detection result.

[0168] Here, the third case, in the case that the device detection result is manually confirmed, and the device detection result is an incorrect result, the device detection model is trained based on the collection data and the sample device detection result corresponding to the collection data. The specific training method refers to the above embodiments, which will not be repeated here. The sample device detection result corresponding to the collection data can be a sample label obtained by manual annotation.

[0169] It can be understood that the device detection result of the first case and the second case is accurate, so that the device detection result can be directly determined as the sample device detection result, and then manual annotation is not required, saving labor cost; the device detection result of the third case is inaccurate, so that the sample device detection result is obtained by manual annotation, and then the training accuracy of the device detection model is ensured.

[0170] In some embodiments, the fourth case, in the case that the device detection result is not manually confirmed, and the confidence of the device detection result is less than or equal to the pre-set confidence threshold, the device detection model is not trained based on the collection data.

[0171] In some embodiments, a first quantity of the collected data belonging to the first case, a second quantity of the collected data belonging to the second case, and a third quantity of the collected data belonging to the third case are determined; a first target quantity is determined based on a product of the first quantity and a first weighted weight, a second target quantity is determined based on a product of the second quantity and a second weighted weight, and a third target quantity is determined based on a product of the third quantity and a third weighted weight; first target collected data is determined from the collected data belonging to the first case based on the first target quantity, the device detection model is trained based on the first target collected data, second target collected data is determined from the collected data belonging to the second case based on the second target quantity, the device detection model is trained based on the second target collected data, and third target collected data is determined from the collected data belonging to the third case based on the third target quantity, the device detection model is trained based on the third target collected data. The first weighted weight, the second weighted weight, and the third weighted weight are all less than 1.

[0172] In an embodiment, the first weighted weight is less than the second weighted weight, considering that the manually confirmed collected data has a greater contribution to the model training than the unconfirmed collected data; and the second weighted weight is less than the third weighted weight, considering that the collected data corresponding to the error result has a greater contribution to the model training than the collected data corresponding to the correct result.

[0173] The device rupture detection method provided by the embodiments of the present application can train the device detection model based on the collected data in the application process of the device detection model, continuously improve the performance of the device detection model, further improve the device rupture detection accuracy of the device detection model, and enable the device detection model to train the model based on the collected data when facing time-varying, collected data type change, and other interference factors, thereby improving the robustness, accuracy, and anti-interference of the device detection model, and improving the generalization performance of the device detection model to new data and new environments.

[0174] Based on any of the above embodiments, after the step 130, the method further includes:

[0175] The feature extraction model is trained based on the collected data. The specific training method is referred to the above embodiments, which will not be repeated here.

[0176] The device rupture detection method provided in the embodiments of the present application can also train the feature extraction model based on the collected data during the application of the feature extraction model, so as to continuously improve the performance of the feature extraction model, thereby further improving the accuracy of the feature extraction vector, and enabling the feature extraction model to face interference factors such as time variation and changes in the types of collected data, and enabling the feature extraction model to train the model based on the collected data, thereby improving the robustness, accuracy and anti-interference performance of the feature extraction model, and improving the generalization performance of the feature extraction model to new data and new environments.

[0177] The device rupture detection device provided in the present application is described below, and the device rupture detection device described below can be referred to in correspondence with the device rupture detection method described above.

[0178] Figure 4 The device rupture detection device provided in the present application is described below, and the device rupture detection device described below can be referred to in correspondence with the device rupture detection method described above. Figure 4 As shown in the structural schematic diagram of the device rupture detection device provided in the present application,

[0179] The determination module 410 is configured to determine collected data of a to-be-detected device, the collected data including sound data and image data, and the sound data being obtained by collecting knocking sound of the to-be-detected device.

[0180] The detection module 420 is configured to perform rupture detection on the to-be-detected device based on the collected data, to obtain a device detection result.

[0181] The device rupture detection device provided in the embodiments of the present application performs rupture detection on a to-be-detected device based on collected data including sound data and image data, thereby improving the accuracy of device rupture detection, and the sound data is obtained by collecting knocking sound of the to-be-detected device, thereby further improving the accuracy of device rupture detection. Meanwhile, the device detection result is determined by the detection device, which saves labor cost compared with manually judging the device detection result, and can give the device detection result in real time and automatically, thereby reducing the delay of device rupture detection and realizing automatic device rupture detection.

[0182] Based on any of the above embodiments, the detection module 420 includes:

[0183] The feature extraction unit is configured to perform feature extraction on the sound data based on a feature extraction model, to obtain a first feature extraction vector, and perform feature extraction on the image data, to obtain a second feature extraction vector.

[0184] The rupture detection unit is configured to perform rupture detection on the first feature extraction vector and the second feature extraction vector based on a device detection model, to obtain a device detection result.

[0185] The feature extraction model is obtained based on unsupervised training of first sample collection data, and the first sample collection data includes first sample sound data and first sample image data.

[0186] The device detection model is obtained based on second sample collection data and sample device detection results corresponding to the second sample collection data, and the second sample collection data includes second sample sound data and second sample image data.

[0187] Based on any of the above embodiments, the device further includes a model training module, which includes:

[0188] The mask processing unit is configured to perform mask processing on the feature vector of the first sample sound data to obtain a mask position feature and an unmasked position feature.

[0189] The feature encoding unit is configured to encode the mask position feature based on the feature extraction model to obtain a first encoding vector, and encode the unmasked position feature to obtain a second encoding vector.

[0190] The vector clustering unit is configured to perform clustering processing on the second encoding vector to obtain a clustering result, and determine a maximum class center vector of the second encoding vector based on the clustering result.

[0191] The model training unit is configured to train the feature extraction model based on the similarity between the first encoding vector and the maximum class center vector.

[0192] Based on any of the above embodiments, the mask processing unit is further configured to:

[0193] Encode the first sample sound data to obtain an encoding vector.

[0194] Fuse the encoding vector with a corresponding preset sound modality vector to obtain the feature vector, and the preset sound modality vector is used to represent sound modality information.

[0195] Based on any of the above embodiments, the loss function of the device detection model includes a classification loss function, and the second sample collection data includes normal sample data and abnormal sample data; the classification loss function is determined based on a classification loss, and the classification loss includes a first classification loss and a second classification loss, the first classification loss is determined based on a first sample device detection result and a sample device detection result corresponding to the normal sample data, and the second classification loss is determined based on a second sample device detection result and a sample device detection result corresponding to the abnormal sample data; the first sample device detection result is obtained by performing breakage detection on a third feature extraction vector based on the device detection model, the second sample device detection result is obtained by performing breakage detection on a fourth feature extraction vector based on the device detection model, the third feature extraction vector is obtained by performing feature extraction on the normal sample data based on the feature extraction model, and the fourth feature extraction vector is obtained by performing feature extraction on the abnormal sample data based on the feature extraction model.

[0196] Based on any of the above embodiments, the classification loss further includes a third classification loss and a fourth classification loss; the third classification loss is determined based on a third sample device detection result and a sample device detection result corresponding to the normal sample data, and the fourth classification loss is determined based on a fourth sample device detection result and a sample device detection result corresponding to the abnormal sample data; the third sample device detection result is obtained by performing breakage detection on a fifth feature extraction vector based on the device detection model, and the fourth sample device detection result is obtained by performing breakage detection on a sixth feature extraction vector based on the device detection model, the fifth feature extraction vector is obtained by performing mask processing on the third feature extraction vector, and the sixth feature extraction vector is obtained by performing mask processing on the fourth feature extraction vector.

[0197] Based on any of the above embodiments, the loss function further includes a contrast loss function; the contrast loss function is determined based on a first similarity and a second similarity, the first similarity is determined based on a similarity between normal features, and the second similarity is determined based on a similarity between normal features and abnormal features, the normal features include normal sample features and normal sample mask features, and the abnormal features include abnormal sample features and abnormal sample mask features; the normal sample features are obtained by encoding the third feature extraction vector based on the device detection model, the abnormal sample features are obtained by encoding the fourth feature extraction vector based on the device detection model, the normal sample mask features are obtained by encoding the fifth feature extraction vector based on the device detection model, and the abnormal sample mask features are obtained by encoding the sixth feature extraction vector based on the device detection model.

[0198] Based on any of the above embodiments, the loss function further includes a reconstruction loss function; the reconstruction loss function is determined based on a third similarity and a fourth similarity, the third similarity is determined based on a similarity between a normal sample feature and a normal sample mask feature, and the fourth similarity is determined based on a similarity between an abnormal sample feature and an abnormal sample mask feature; the normal sample feature is obtained by encoding the third feature extraction vector based on the equipment detection model, the abnormal sample feature is obtained by encoding the fourth feature extraction vector based on the equipment detection model, the normal sample mask feature is obtained by encoding the fifth feature extraction vector based on the equipment detection model, and the abnormal sample mask feature is obtained by encoding the sixth feature extraction vector based on the equipment detection model.

[0199] Based on any of the above embodiments, the crack detection unit is further configured to:

[0200] Based on the equipment detection model, the first feature extraction vector and the second feature extraction vector are subjected to crack detection to obtain a first equipment detection result, and the first mask feature vector and the second mask feature vector are subjected to crack detection to obtain a second equipment detection result, the first mask feature vector being obtained by mask processing on the first feature extraction vector, and the second mask feature vector being obtained by mask processing on the second feature extraction vector;

[0201] Based on the similarity between the first equipment detection result and a plurality of preset equipment detection results, and the similarity between the second equipment detection result and a plurality of preset equipment detection results, the equipment detection result is determined.

[0202] Based on any of the above embodiments, the device further comprises:

[0203] The model training module is configured to, in a case where the equipment detection result is not manually confirmed and the confidence of the equipment detection result is greater than a preset confidence threshold, train the equipment detection model based on the acquisition data and the equipment detection result;

[0204] The model training module is further configured to, in a case where the equipment detection result is manually confirmed and the equipment detection result is a correct result, train the equipment detection model based on the acquisition data and the equipment detection result;

[0205] The model training module is further configured to, in a case where the equipment detection result is manually confirmed and the equipment detection result is an incorrect result, train the equipment detection model based on the acquisition data and the sample equipment detection result corresponding to the acquisition data.

[0206] Figure 5An example of a schematic diagram of a physical structure of an electronic device is shown in FIG. 1. Figure 5 As shown in FIG. 1, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 can communicate with each other through the communications bus 540. The processor 510 can invoke a logical instruction in the memory 530 to execute a device breakage detection method, which includes determining acquisition data of a device to be detected, the acquisition data including sound data and image data, the sound data being obtained by collecting knocking sound of the device to be detected, and performing breakage detection on the device to be detected based on the acquisition data to obtain a device detection result.

[0207] In addition, the logical instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partly, or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disk, and various other media capable of storing program codes.

[0208] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a device breakage detection method provided by the above-mentioned methods, which includes determining acquisition data of a device to be detected, the acquisition data including sound data and image data, the sound data being obtained by collecting knocking sound of the device to be detected, and performing breakage detection on the device to be detected based on the acquisition data to obtain a device detection result.

[0209] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0210] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0211] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting equipment rupture, characterized in that: The method comprises the following steps: acquiring data of a device to be detected, the data comprising sound data and image data, the sound data being obtained by collecting knocking sound of the device to be detected; performing a crack detection on the device to be detected based on the acquired data to obtain a device detection result; the step of performing a crack detection on the device to be detected based on the acquired data to obtain a device detection result comprises: performing feature extraction on the sound data based on a feature extraction model to obtain a first feature extraction vector, and performing feature extraction on the image data to obtain a second feature extraction vector; performing a crack detection on the first feature extraction vector and the second feature extraction vector based on a device detection model to obtain a device detection result; wherein the feature extraction model is obtained by unsupervised training based on first sample acquisition data, the first sample acquisition data comprising first sample sound data and first sample image data; the device detection model is obtained by training based on second sample acquisition data and sample device detection results corresponding to the second sample acquisition data, the second sample acquisition data comprising second sample sound data and second sample image data; a loss function of the device detection model comprises a classification loss function, and the second sample acquisition data comprises normal sample data and abnormal sample data; the classification loss function is determined based on a classification loss, the classification loss comprising a first classification loss and a second classification loss, the first classification loss being determined based on a first sample device detection result and a sample device detection result corresponding to the normal sample data, and the second classification loss being determined based on a second sample device detection result and a sample device detection result corresponding to the abnormal sample data; the first sample device detection result is obtained by performing a crack detection on a third feature extraction vector based on the device detection model, the second sample device detection result is obtained by performing a crack detection on a fourth feature extraction vector based on the device detection model, the third feature extraction vector being obtained by performing feature extraction on the normal sample data based on the feature extraction model, and the fourth feature extraction vector being obtained by performing feature extraction on the abnormal sample data based on the feature extraction model.

2. The apparatus breakage detection method according to claim 1, characterized in that, the feature extraction model is trained based on the following steps: performing mask processing on a feature vector of the first sample sound data to obtain a mask position feature and an unmasked position feature; encoding the mask position feature based on the feature extraction model to obtain a first encoding vector, and encoding the unmasked position feature to obtain a second encoding vector; performing clustering processing on the second encoding vector to obtain a clustering result, and determining a maximum class center vector of the second encoding vector based on the clustering result; training the feature extraction model based on a similarity between the first encoding vector and the maximum class center vector.

3. The apparatus breakage detection method of claim 2, wherein, the feature vector is determined based on the following steps: encoding the first sample sound data to obtain an encoding vector; Fusing a preset sound modality vector corresponding to the coding vector to obtain the feature vector, the preset sound modality vector being used to represent sound modality information.

4. The apparatus breakage detection method of claim 1, wherein The classification loss further includes a third classification loss and a fourth classification loss. The third classification loss is determined based on a third sample equipment detection result and a sample equipment detection result corresponding to the normal sample data, and the fourth classification loss is determined based on a fourth sample equipment detection result and a sample equipment detection result corresponding to the abnormal sample data. The third sample equipment detection result is obtained by performing breakage detection on a fifth feature extraction vector based on the equipment detection model, the fourth sample equipment detection result is obtained by performing breakage detection on a sixth feature extraction vector based on the equipment detection model, the fifth feature extraction vector is obtained by performing mask processing on the third feature extraction vector, and the sixth feature extraction vector is obtained by performing mask processing on the fourth feature extraction vector.

5. The apparatus breakage detection method according to claim 4, characterized in that, The loss function further includes a contrast loss function; The contrast loss function is determined based on a first similarity and a second similarity, the first similarity being determined based on a similarity between normal features, and the second similarity being determined based on a similarity between normal features and abnormal features, the normal features including normal sample features and normal sample mask features, and the abnormal features including abnormal sample features and abnormal sample mask features. The normal sample features are obtained by encoding the third feature extraction vector based on the equipment detection model, the abnormal sample features are obtained by encoding the fourth feature extraction vector based on the equipment detection model, the normal sample mask features are obtained by encoding the fifth feature extraction vector based on the equipment detection model, and the abnormal sample mask features are obtained by encoding the sixth feature extraction vector based on the equipment detection model.

6. The apparatus breakage detection method according to claim 4, wherein, The loss function further includes a reconstruction loss function; The reconstruction loss function is determined based on a third similarity and a fourth similarity, the third similarity being determined based on a similarity between normal sample features and normal sample mask features, and the fourth similarity being determined based on a similarity between abnormal sample features and abnormal sample mask features. The normal sample features are obtained by encoding the third feature extraction vector based on the equipment detection model, the abnormal sample features are obtained by encoding the fourth feature extraction vector based on the equipment detection model, the normal sample mask features are obtained by encoding the fifth feature extraction vector based on the equipment detection model, and the abnormal sample mask features are obtained by encoding the sixth feature extraction vector based on the equipment detection model.

7. The apparatus breakage detection method of claim 1, wherein The breakage detection on the first feature extraction vector and the second feature extraction vector based on the equipment detection model to obtain the equipment detection result includes: The first feature extraction vector and the second feature extraction vector are subjected to breakage detection based on a device detection model to obtain a first device detection result, and the first mask feature vector and the second mask feature vector are subjected to breakage detection to obtain a second device detection result, the first mask feature vector being obtained by performing mask processing on the first feature extraction vector, and the second mask feature vector being obtained by performing mask processing on the second feature extraction vector; The device detection result is determined based on similarities between the first device detection result and a plurality of preset device detection results, and similarities between the second device detection result and the plurality of preset device detection results.

8. The apparatus breakage detection method of claim 1, wherein, The device detection model is used to perform breakage detection on the first feature extraction vector and the second feature extraction vector to obtain a device detection result, and then further includes: In a case where the device detection result is not manually confirmed and a confidence degree of the device detection result is greater than a preset confidence threshold, the device detection model is trained based on the collected data and the device detection result; In a case where the device detection result is manually confirmed and the device detection result is a correct result, the device detection model is trained based on the collected data and the device detection result; In a case where the device detection result is manually confirmed and the device detection result is an incorrect result, the device detection model is trained based on the collected data and a sample device detection result corresponding to the collected data.

9. A device for detecting equipment rupture, characterized in that: It includes: A determination module is configured to determine collected data of a device to be detected, the collected data including sound data and image data, the sound data being obtained by collecting knocking sound of the device to be detected; A detection module is configured to perform breakage detection on the device to be detected based on the collected data to obtain a device detection result; The collected data is used to perform breakage detection on the device to be detected to obtain a device detection result, including: A feature extraction model is used to perform feature extraction on the sound data to obtain a first feature extraction vector, and perform feature extraction on the image data to obtain a second feature extraction vector; A device detection model is used to perform breakage detection on the first feature extraction vector and the second feature extraction vector to obtain a device detection result; The feature extraction model is obtained based on unsupervised training of first sample collected data, the first sample collected data including first sample sound data and first sample image data; The device detection model is obtained based on second sample collected data and a sample device detection result corresponding to the second sample collected data, the second sample collected data including second sample sound data and second sample image data; The loss function of the device detection model includes a classification loss function, and the second sample collected data includes normal sample data and abnormal sample data; The classification loss function is determined based on the classification loss, the classification loss includes a first classification loss and a second classification loss, the first classification loss is determined based on the first sample device detection result and the sample device detection result corresponding to the normal sample data, and the second classification loss is determined based on the second sample device detection result and the sample device detection result corresponding to the abnormal sample data; The first sample device detection result is obtained based on the device detection model performing rupture detection on the third feature extraction vector, the second sample device detection result is obtained based on the device detection model performing rupture detection on the fourth feature extraction vector, the third feature extraction vector is obtained based on the feature extraction model performing feature extraction on the normal sample data, and the fourth feature extraction vector is obtained based on the feature extraction model performing feature extraction on the abnormal sample data.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the device rupture detection method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, the device rupture detection method according to any one of claims 1 to 8 is implemented.

Images