State recognition method, system and equipment of industrial equipment and medium

Abstract

The invention discloses a state recognition method and system for industrial equipment, equipment and a medium, and the method comprises the steps: obtaining multi-modal original data, a preset industrial multi-modal prompt template, an industrial large language model and a historical vector data set of the industrial equipment; performing multi-modal feature extraction on the multi-modal original data to obtain multi-modal feature data; according to an industrial multi-modal prompt template, performing prompt word generation on the multi-modal feature data to obtain a multi-modal prompt word; according to the multi-modal cue words and the industrial large language model, enhanced retrieval matching processing is conducted on the historical vector data set, a state recognition result of the industrial equipment is obtained, and the state recognition result is the equipment state corresponding to the multi-modal historical feature vector with the highest similarity with the multi-modal feature data in the historical vector data set. According to the method, the timeliness of state recognition of the industrial equipment can be effectively improved, and the false alarm rate is reduced. The invention relates to the technical field of industrial data processing.

Description

Technical Field

[0001] The present application relates to the technical field of industrial data processing, and in particular to a method, system, device and medium for identifying the state of industrial equipment. Background Art

[0002] In modern industrial production, the identification of the real-time status of industrial equipment is crucial to the operation and maintenance of industrial equipment. It is mainly through real-time data collection and analysis to identify the operating status of industrial equipment.

[0003] At present, the traditional method of state recognition of industrial equipment mainly collects a single type of state information of industrial equipment in real time through sensors, and then retrieves and compares the single type of state information based on manual experience to derive the operating status of the industrial equipment. This method relies heavily on manual inspections, and has a slow response to state recognition of industrial equipment. It often fails to detect state changes of industrial equipment in a timely manner, and its timeliness is unsatisfactory, with a high false alarm rate.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] In order to solve at least one of the above technical problems, the present application provides a method, system, device and medium for state identification of industrial equipment, wherein the method can effectively improve the timeliness of state identification of industrial equipment and reduce the false alarm rate.

[0006] According to a first aspect of the present application, a method for identifying a state of an industrial device is provided, comprising:

[0007] Acquire multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model, and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment and a device state corresponding to each of the multimodal historical feature vectors;

[0008] Performing multimodal feature extraction on the multimodal raw data to obtain multimodal feature data;

[0009] Generating prompts for the multimodal feature data according to the industrial multimodal prompt template to obtain multimodal prompts;

[0010] According to the multimodal prompt and the industrial large language model, the historical vector data set is enhanced by retrieval and matching processing to obtain a state recognition result of the industrial equipment. The state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data.

[0011] In some embodiments, the multimodal raw data includes image unimodal data, audio unimodal data, and text unimodal data, and the multimodal feature extraction of the multimodal raw data to obtain the multimodal feature data includes:

[0012] Performing image unimodal feature extraction on the image unimodal data to obtain image unimodal feature data;

[0013] Performing audio unimodal feature extraction on the audio unimodal data to obtain audio unimodal feature data;

[0014] Performing text unimodal feature extraction on the text unimodal data to obtain text unimodal feature data;

[0015] According to the image unimodal feature data and the audio unimodal feature data, multimodal feature fusion is performed on the text unimodal feature data to obtain the multimodal feature data.

[0016] In some embodiments, the step of extracting image unimodal features from image unimodal data to obtain image unimodal feature data includes:

[0017] Get a pre-trained image extraction model;

[0018] Inputting the image unimodal data into the pre-trained image extraction model to extract image features, thereby obtaining the image unimodal feature data;

[0019] The step of extracting audio unimodal features from the audio unimodal data to obtain audio unimodal feature data includes:

[0020] Convert the audio unimodal data into a spectrogram to obtain audio spectrum data;

[0021] Performing cepstrum feature analysis on the audio spectrum data to obtain the audio unimodal feature data;

[0022] The step of extracting text unimodal features from the text unimodal data to obtain text unimodal feature data includes:

[0023] Performing word frequency analysis on the text unimodal data to obtain a plurality of word frequency weights, each of the word frequency weights corresponding to a text word in the text unimodal data;

[0024] According to the word frequency weight, the text unimodal data is word embedded to obtain the text unimodal feature data.

[0025] In some embodiments, the state identification method further includes:

[0026] Acquire original image data, original audio data, and original text data about industrial equipment;

[0027] Performing image preprocessing on the original image data to obtain the image unimodal data, and performing audio preprocessing on the original audio data to obtain the audio unimodal data;

[0028] Performing text preprocessing on the original text data to obtain the text unimodal data;

[0029] The image unimodal data, the audio unimodal data and the text unimodal data are integrated to obtain the multimodal original data.

[0030] In some embodiments, performing image preprocessing on the original image data to obtain the image single-modality data includes:

[0031] Performing image filtering on the original image data to obtain intermediate image data;

[0032] Performing image normalization on the intermediate image data to obtain the image single-modal data;

[0033] The performing audio preprocessing on the original audio data to obtain the audio single-modal data includes:

[0034] Performing audio denoising on the original audio data to obtain intermediate audio data;

[0035] Dividing the intermediate audio data into audio frames to obtain the audio unimodal data;

[0036] The performing text preprocessing on the original text data to obtain the text unimodal data includes:

[0037] Performing text segmentation on the original text data to obtain intermediate text data;

[0038] The intermediate text data is screened for text words to obtain the text unimodal data.

[0039] In some embodiments, the enhanced retrieval and matching processing is performed on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set with the highest similarity to the multimodal feature data, including:

[0040] Performing query encoding on the multimodal prompt to obtain a multimodal query vector;

[0041] According to the multimodal query vector, vector matching is performed on all the multimodal historical feature vectors to obtain a target historical feature vector, where the target historical feature vector is a multimodal historical feature vector having the highest vector similarity with the multimodal query vector;

[0042] The multimodal prompt and the device state corresponding to the target historical feature vector are input into the industrial large language model to generate the state, and the state recognition result is obtained.

[0043] In some embodiments, the state identification method further includes:

[0044] According to the state recognition result, the multimodal feature data is state-verified to obtain a state verification result, wherein the state verification result is used to indicate whether the state recognition result is an accurate device state corresponding to the multimodal feature data;

[0045] If the state verification result is that the state identification result is an accurate device state corresponding to the multimodal feature data, then the feature vector of the historical vector data set is updated according to the multimodal feature data and the state identification result to obtain an updated historical vector data set.

[0046] According to a second aspect of the present application, a state identification system for industrial equipment is provided, comprising:

[0047] A first processing unit is used to obtain multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment and a device state corresponding to each of the multimodal historical feature vectors;

[0048] A second processing unit is used to perform multimodal feature extraction on the multimodal raw data to obtain multimodal feature data;

[0049] A third processing unit, configured to generate a prompt for the multimodal feature data according to the industrial multimodal prompt template to obtain a multimodal prompt;

[0050] The fourth processing unit is used to perform enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data.

[0051] According to a third aspect of the present application, a computer device is provided, comprising:

[0052] at least one processor;

[0053] at least one memory for storing at least one program;

[0054] When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of the above aspects.

[0055] According to a fourth aspect of the present application, a computer-readable storage medium is provided, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the method described in any of the above aspects.

[0056] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0057] The present application provides a state recognition method, system, device and medium for industrial equipment, wherein the state recognition method obtains multimodal original data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model and a historical vector data set, wherein the historical vector data set records a number of multimodal historical feature vectors of the industrial equipment, and the equipment state corresponding to each of the multimodal historical feature vectors; performs multimodal feature extraction on the multimodal original data to obtain multimodal feature data; generates prompts for the multimodal feature data according to the industrial multimodal prompt template to obtain multimodal prompts; performs enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompts and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data. This state recognition method is based on feature extraction of multimodal raw data of industrial equipment, which can improve the recognition accuracy of state recognition of industrial equipment and reduce the false alarm rate; in addition, this state recognition method also uses an industrial large language model to enhance the retrieval of multimodal prompts and historical vector data, which can discover the state changes of industrial equipment more promptly and effectively improve the timeliness of state recognition of industrial equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a method for identifying the state of industrial equipment provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of an optional flow chart of a method for identifying the state of industrial equipment provided in an embodiment of the present application;

[0060] Figure 3 A detailed flow chart of step A2 provided in an embodiment of the present application;

[0061] Figure 4 A detailed flow chart of step A3 provided in an embodiment of the present application;

[0062] Figure 5 A detailed flow chart of step S120 provided in an embodiment of the present application;

[0063] Figure 6 A detailed flow chart of step B3 provided in an embodiment of the present application;

[0064] Figure 7 A detailed flow chart of step S140 provided in an embodiment of the present application;

[0065] Figure 8 Another optional flow chart of a method for identifying the state of industrial equipment provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of a framework of a state identification system for industrial equipment provided in an embodiment of the present application;

[0067] Fig.10 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0069] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0071] At present, the traditional method of state recognition of industrial equipment mainly collects a single type of state information of industrial equipment in real time through sensors, and then retrieves and compares the single type of state information based on manual experience to derive the operating status of the industrial equipment. This method relies heavily on manual inspections, and has a slow response to state recognition of industrial equipment. It often fails to detect state changes of industrial equipment in a timely manner, and its timeliness is unsatisfactory, with a high false alarm rate.

[0072] In view of this, the embodiments of the present application provide a method, system, device and medium for state recognition of industrial equipment, wherein the state recognition method is based on feature extraction of multimodal original data of industrial equipment, specifically by extracting features of image, audio and text type operating state data of industrial equipment, which is beneficial to the recognition accuracy of state recognition of industrial equipment and reduces the false alarm rate; in addition, the state recognition method also enhances the retrieval of multimodal prompts and historical vector data through an industrial large language model, which can discover the state changes of industrial equipment more timely and effectively improve the timeliness of state recognition of industrial equipment.

[0073] The embodiments of the present application provide a method, system, device, and medium for identifying the state of industrial equipment, which can be specifically described through the following embodiments. First, a method for identifying the state of industrial equipment in an embodiment of the present application is described.

[0074] The state identification method of industrial equipment provided in the embodiment of the present application can be applied to industrial production application scenarios. In industrial production application scenarios, the user of industrial equipment can identify the operating status of several industrial equipment used by the method provided in the embodiment of the present application, and can more timely discover the state changes of industrial equipment, effectively improve the timeliness of state identification of industrial equipment, and improve the recognition accuracy of state identification of industrial equipment, and reduce the false alarm rate.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] Reference Figure 1 , Figure 1 It is an optional flow chart of a method for identifying the state of industrial equipment provided in an embodiment of the present application, which may include but is not limited to steps S110 to S140.

[0077] Step S110: acquiring multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model, and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment, and a device state corresponding to each of the multimodal historical feature vectors;

[0078] In the embodiments of the present application, industrial equipment may be equipment in industrial production scenarios, which mainly include processing equipment (such as lathes, milling machines, grinders, 3D printers, etc.), automation equipment (such as robotic arms, CNC machine tools, etc.), separation equipment (daily centrifuges), etc. The embodiments of the present application do not limit the specific equipment types of industrial equipment.

[0079] It can be understood that the multimodal raw data of industrial equipment can be image unimodal data, audio unimodal data and text unimodal data about the industrial equipment, among which the image unimodal data can be the real-time operation diagram of the industrial equipment, the audio unimodal data can be the sound generated during the real-time operation of the industrial equipment, and the text unimodal data can be the report message text produced by the industrial equipment during the real-time operation.

[0080] It should be noted that the industrial multimodal prompt template can be pre-built based on the prompt project, which is used to generate prompts corresponding to the multimodal original data of the industrial equipment. There are many specific implementation methods of the industrial multimodal prompt template, which will not be repeated in this application; the industrial large language model can be a large language model (Large Language Model, LLM) used in the field of industrial production. The industrial large language model can be obtained by fine-tuning the pre-trained large language model based on the industrial data training set, or it can be directly trained on the initialized large language model based on the industrial data training set. In addition, the historical vector data set in the embodiment of the present application can be stored in the industrial system database, which records the multimodal historical feature vectors and corresponding equipment status of each industrial equipment. The industrial system database can update the historical vector data set based on the status recognition results received in real time.

[0081] Reference Figure 2 In some embodiments, the state identification method further includes:

[0082] A1. Obtaining original image data, original audio data, and original text data about industrial equipment;

[0083] A2. performing image preprocessing on the original image data to obtain the image unimodal data, and performing audio preprocessing on the original audio data to obtain the audio unimodal data;

[0084] A3, performing text preprocessing on the original text data to obtain the text unimodal data;

[0085] A4. Integrate the image unimodal data, the audio unimodal data and the text unimodal data to obtain the multimodal original data.

[0086] In an embodiment of the present application, for a certain industrial equipment, a real-time operation image of the industrial equipment can be captured by a camera device (such as a camera), and the captured real-time operation image can be determined as original image data, and the sound generated by the industrial equipment during the real-time operation process can be recorded by an audio acquisition device (such as a microphone); in addition, the original reported message text of the industrial equipment during the real-time operation process can be determined as original text data.

[0087] It can be understood that after obtaining the original image data, original audio data and original text data of the industrial equipment, the original image data, original audio data and original text data can be preprocessed respectively to obtain image unimodal data, audio unimodal data and text unimodal data, and then the image unimodal data, audio unimodal data and text unimodal data are integrated and spliced ​​to obtain multimodal original data of the industrial image.

[0088] Reference Figure 3 Further, the step A2, performing image preprocessing on the original image data to obtain the image single-modality data, includes:

[0089] A21, performing image filtering on the original image data to obtain intermediate image data;

[0090] A22, performing image normalization on the intermediate image data to obtain the image single-modality data;

[0091] The step A2, performing audio preprocessing on the original audio data to obtain the audio unimodal data, includes:

[0092] A23, performing audio denoising on the original audio data to obtain intermediate audio data;

[0093] A24, dividing the intermediate audio data into audio frames to obtain the audio single-mode data;

[0094] In an embodiment of the present application, step A21 may be to perform image denoising on the original image data. Specifically, Gaussian filtering may be used to reduce the noise present in the original image data, thereby obtaining intermediate image data. Then, image normalization is performed on the intermediate image data, and the pixel value of each image pixel point in the intermediate image data is scaled within the range of [0, 1] to obtain image unimodal data.

[0095] It can be understood that step A23 can be audio denoising for the original audio data, which can first be spectral analysis of the original audio data to obtain the original audio data in the frequency domain, then estimate the noise power spectrum of the original audio data in the frequency domain, and subtract the noise power spectrum from the original audio power spectrum corresponding to the original audio data in the frequency domain, and then obtain the intermediate audio data through inverse transformation; step A24 can be dividing the audio signal in the intermediate audio data into several audio division frames, and then determining all the obtained audio division frames as audio single modal data.

[0096] Reference Figure 4 The step A3, performing text preprocessing on the original text data to obtain the text unimodal data, includes:

[0097] A31, performing text segmentation on the original text data to obtain intermediate text data;

[0098] A32. Screen the intermediate text data for words to obtain the text unimodal data.

[0099] In an embodiment of the present application, step A31 may be to obtain a preset dictionary, match each word in the original text data with the dictionary, and determine several words that are successfully matched as intermediate text data; or, step A31 may also input the original text data into a text segmentation model for segmentation to obtain intermediate text data, and the specific text segmentation model may be any one of an N-gram model, a hidden Markov (HMM) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, etc. The examples of this application are for illustration only and are not intended to limit this application. In addition, text word screening may be to remove at least one of stop words, prepositions, conjunctions, special characters, etc. in the intermediate text data, and then determine the intermediate text data after removal as text unimodal data.

[0100] Step S120, performing multimodal feature extraction on the multimodal original data to obtain multimodal feature data;

[0101] In the embodiment of the present application, step S120 may be to extract feature data of each modality of the multimodal original data, and integrate the extracted feature data of each modality to obtain multimodal feature data.

[0102] Reference Figure 5 In some embodiments, the step S120, the multimodal original data includes image unimodal data, audio unimodal data and text unimodal data, and the multimodal feature extraction of the multimodal original data to obtain the multimodal feature data includes:

[0103] B1. performing image unimodal feature extraction on the image unimodal data to obtain image unimodal feature data;

[0104] B2. extracting audio unimodal features from the audio unimodal data to obtain audio unimodal feature data;

[0105] B3, performing text unimodal feature extraction on the text unimodal data to obtain text unimodal feature data;

[0106] B4. Perform multimodal feature fusion on the text unimodal feature data according to the image unimodal feature data and the audio unimodal feature data to obtain the multimodal feature data.

[0107] In an embodiment of the present application, multimodal original data can be modally divided based on data modality to obtain image unimodal data, audio unimodal data and text unimodal data; then, the modal features of the image unimodal data, audio unimodal data and text unimodal data are extracted respectively to obtain image unimodal feature data, audio unimodal feature data and text unimodal feature data.

[0108] It can be understood that after obtaining the image unimodal feature data, audio unimodal feature data and text unimodal feature data, step B4 can first be to obtain the image feature weights corresponding to the image unimodal feature data, the audio feature weights corresponding to the audio unimodal feature data and the text feature weights corresponding to the text unimodal feature data; then, based on the image feature weights, audio feature weights and text feature weights, the image unimodal feature data, the audio unimodal feature data and the text unimodal feature data are weighted averaged and fused to obtain multimodal feature data. The specific values ​​of the image feature weights, audio feature weights and text feature weights can be flexibly set according to actual conditions.

[0109] Furthermore, the step B1, extracting image unimodal features from the image unimodal data to obtain image unimodal feature data, includes:

[0110] B11. Obtain a pre-trained image extraction model;

[0111] B12, inputting the image unimodal data into the pre-trained image extraction model to extract image features, thereby obtaining the image unimodal feature data;

[0112] The step B2, extracting audio unimodal features from the audio unimodal data to obtain audio unimodal feature data, comprises:

[0113] B21, performing spectrum conversion on the audio single modal data to obtain audio spectrum data;

[0114] B22, performing cepstrum feature analysis on the audio spectrum data to obtain the audio unimodal feature data;

[0115] Reference Figure 6 , the step B3, extracting text unimodal features from the text unimodal data to obtain text unimodal feature data, includes:

[0116] B31, performing word frequency analysis on the text unimodal data to obtain a plurality of word frequency weights, each of the word frequency weights corresponding to a text word in the text unimodal data;

[0117] B32. According to the word frequency weight, word embedding is performed on the text unimodal data to obtain the text unimodal feature data.

[0118] In an embodiment of the present application, the pre-trained image extraction model may be a pre-trained VGG-16 (Visual Geometry Group 16-layer network) model, and step B12 may be inputting the image unimodal data into the VGG-16 model, and extracting the image features in the image unimodal data through a series of convolutional layers and fully connected layers in the VGG-16 model to obtain the image unimodal feature data.

[0119] It can be understood that the spectrogram conversion of audio unimodal data can be to perform Fourier transform on the audio unimodal data, thereby obtaining audio spectrum data by mapping the audio unimodal data from the time domain to the frequency domain; then, calculating the Mel-frequency cepstral coefficients (MFCCs) corresponding to the audio spectrum data and forming audio unimodal feature data in the form of feature vectors based on the calculated Mel-frequency cepstral coefficients.

[0120] It should be noted that step B31 can be based on TF-IDF (term frequency–inverse document frequency) technology to calculate the word frequency weight of each text word in the text unimodal data; then based on all the word frequency weights, the word embedding model (nomic-embed-text) is used to convert the text words in the text unimodal data into text unimodal feature data in the form of feature vectors.

[0121] Step S130: generating a prompt for the multimodal feature data according to the industrial multimodal prompt template to obtain a multimodal prompt;

[0122] In an embodiment of the present application, step S130 may be to perform prompt encapsulation on the multimodal feature data based on the industrial multimodal prompt template, so as to obtain a multimodal prompt in the form of a prompt that can be understood by the industrial large language model. Specifically, the multimodal feature data may be added to the corresponding position of the industrial multimodal prompt template to obtain the multimodal prompt.

[0123] Step S140: performing enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment.

[0124] In an embodiment of the present application, the industrial large language model can be generated by enhanced retrieval based on multimodal prompts and historical vector data sets to obtain the state recognition result of the industrial equipment. The state recognition result is the equipment state corresponding to the multimodal historical feature vector with the highest similarity to the multimodal feature data among all multimodal historical feature vectors in the historical vector data set.

[0125] Reference Figure 7 In some embodiments, the step S140 performs enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set with the highest similarity to the multimodal feature data, including:

[0126] C1. performing query encoding on the multimodal prompt to obtain a multimodal query vector;

[0127] C2. According to the multimodal query vector, vector matching is performed on all the multimodal historical feature vectors to obtain a target historical feature vector, where the target historical feature vector is a multimodal historical feature vector having the highest vector similarity with the multimodal query vector;

[0128] C3. Inputting the multimodal prompt and the device state corresponding to the target historical feature vector into the industrial large language model to generate the state and obtain the state recognition result.

[0129] In the embodiment of the present application, the multimodal prompt can first be vector-encoded to obtain a multimodal query vector corresponding to the multimodal prompt. Then, step C2 can be based on the multimodal query vector, and the vector similarity matching can be performed with each multimodal historical feature vector in the historical vector data set. The specific similarity matching method can be a matching method based on cosine similarity, so as to obtain the vector similarity corresponding to each multimodal historical feature vector; then, the vector similarity corresponding to each multimodal historical feature vector is screened, so as to screen out the multimodal historical feature vector with the highest vector similarity, and determine the multimodal historical feature vector as the target historical feature vector.

[0130] It can be understood that after obtaining the target historical feature vector, the target historical feature vector and the multimodal prompt can be context-fused to obtain an extended multimodal prompt, and then the extended multimodal prompt is input into the industrial large language model for text generation to obtain the state recognition result.

[0131] Reference Figure 8 , further, the state identification method also includes:

[0132] D1. According to the state recognition result, the multimodal feature data is state-verified to obtain a state verification result, wherein the state verification result is used to indicate whether the state recognition result is an accurate device state corresponding to the multimodal feature data;

[0133] D2. If the state verification result is that the state identification result is an accurate device state corresponding to the multimodal feature data, then based on the multimodal feature data and the state identification result, the feature vector of the historical vector data set is updated to obtain an updated historical vector data set.

[0134] In an embodiment of the present application, after obtaining the state recognition result, the state verification of the multimodal feature data can also be performed according to the state recognition result. The state verification can specifically be to obtain the actual operating state of the industrial equipment corresponding to the multimodal feature data, and then generate the state verification result based on the actual operating state and the state recognition result. Specifically, when the actual operating state is consistent with the state recognition result, a state verification result can be generated that represents the state recognition result as the accurate device state corresponding to the multimodal feature data; or, when the actual operating state is inconsistent with the state recognition result, a state verification result can be generated that represents the state recognition result as not being the accurate device state corresponding to the multimodal feature data.

[0135] It can be understood that when the state verification result is that the state identification result is not the accurate device state corresponding to the multimodal feature data, then no operation can be performed on the multimodal feature data and the corresponding state identification result; or, when the state verification result is that the state identification result is the accurate device state corresponding to the multimodal feature data, then the multimodal feature data and the corresponding state identification result can be added to the historical vector data set, so as to achieve continuous updating of the historical vector data set and obtain an updated historical vector data set.

[0136] Fig. 9 A schematic diagram of a framework of a state identification system for industrial equipment provided in an embodiment of the present application includes:

[0137] The first processing unit 810 is used to obtain multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment and a device state corresponding to each of the multimodal historical feature vectors;

[0138] The second processing unit 820 is used to extract multimodal features from the multimodal raw data to obtain multimodal feature data;

[0139] The third processing unit 830 is used to generate a prompt for the multimodal feature data according to the industrial multimodal prompt template to obtain a multimodal prompt;

[0140] The fourth processing unit 840 is used to perform enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data.

[0141] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present application includes:

[0143] at least one processor 980;

[0144] At least one memory 920, used to store at least one program;

[0145] When the at least one program is executed by the at least one processor 980, the at least one processor 980 implements the methods described in the foregoing embodiments.

[0146] An embodiment of the present application also provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor 980, it is used to implement the methods described in the above embodiments.

[0147] Fig.10 Specifically, the computer device may be a user terminal or a server.

[0148] The present application embodiment takes the computer device as a user terminal as an example, and the details are as follows:

[0149] like Fig.10 As shown, the computer device 900 may include an RF (Radio Frequency) circuit 910, a memory 920 including one or more computer-readable storage media, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a WiFi module 970, a processor 980 including one or more processing cores, and a power supply 990. Those skilled in the art will appreciate that Fig.10 The device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0150] The RF circuit 910 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 980 for processing; in addition, the data related to the uplink is sent to the base station. Generally, the RF circuit 910 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a user identity module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. In addition, the RF circuit 910 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0151] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device 900 (such as audio data, a phone book, etc.), etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 920 may also include a memory controller to provide the processor 980 and the input unit 930 with access to the memory 920. Although Fig.10 The RF circuit 910 is shown, but it is understandable that it is not an essential component of the device 900 and can be omitted as required without changing the essence of the invention.

[0152] The input unit 930 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 932 and other input devices 931. The touch-sensitive surface 932, also known as a touch display screen or a touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 932), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 932 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 980, and can receive and execute commands sent by the processor 980. In addition, the touch-sensitive surface 932 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch-sensitive surface 932, the input unit 930 may further include other input devices 931. Specifically, the other input devices 931 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0153] The display unit 940 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device 900, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 940 may include a display panel 941. Optionally, the display panel 941 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 932 may be covered on the display panel 941. When the touch-sensitive surface 932 detects a touch operation on or near it, it is transmitted to the processor 980 to determine the type of touch event. Subsequently, the processor 980 provides a corresponding visual output on the display panel 941 according to the type of touch event. Although in Fig.10 In the embodiment, the touch-sensitive surface 932 and the display panel 941 are implemented as two independent components to implement input and output functions, but in some embodiments, the touch-sensitive surface 932 and the display panel 941 can be integrated to implement input and output functions.

[0154] The computer device 900 may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 941 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 941 and / or the backlight when the device 900 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the device 900, they will not be repeated here.

[0155] The audio circuit 960, the speaker 961, and the microphone 962 can provide an audio interface between the user and the device 900. The audio circuit 960 can transmit the electrical signal converted from the received audio data to the speaker 961, which is converted into a sound signal for output; on the other hand, the microphone 962 converts the collected sound signal into an electrical signal, which is received by the audio circuit 960 and converted into audio data, and then the audio data is processed by the output processor 980 and sent to another control device through the RF circuit 910, or the audio data is output to the memory 920 for further processing. The audio circuit 960 may also include an earplug jack to provide communication between an external headset and the device 900.

[0156] The device 900 can transmit information with the wireless transmission module set on the fighting device through the WiFi module 970.

[0157] The processor 980 is the control center of the device 900. It uses various interfaces and lines to connect various parts of the entire control device. It executes various functions of the device 900 and processes data by running or executing software programs and / or modules stored in the memory 920, and calling data stored in the memory 920, so as to monitor the control device as a whole. Optionally, the processor 980 may include one or more processing cores; optionally, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 980.

[0158] The device 900 also includes a power supply 990 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 980 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 990 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0159] Although not shown, the device 900 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0160] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the method described in the above embodiments.

[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0162] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0167] The step numbers in the above method embodiment are only provided for the convenience of explanation and description, and no limitation is imposed on the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0168] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for identifying the state of industrial equipment, characterized in that: include: Acquire multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model, and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment and a device state corresponding to each of the multimodal historical feature vectors; Performing multimodal feature extraction on the multimodal raw data to obtain multimodal feature data; Generating prompts for the multimodal feature data according to the industrial multimodal prompt template to obtain multimodal prompts; According to the multimodal prompt and the industrial large language model, the historical vector data set is enhanced by retrieval and matching processing to obtain a state recognition result of the industrial equipment. The state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data.

2. The state recognition method according to claim 1, characterized in that: The multimodal original data includes image unimodal data, audio unimodal data and text unimodal data, and the multimodal feature extraction is performed on the multimodal original data to obtain multimodal feature data, including: Performing image unimodal feature extraction on the image unimodal data to obtain image unimodal feature data; Performing audio unimodal feature extraction on the audio unimodal data to obtain audio unimodal feature data; Performing text unimodal feature extraction on the text unimodal data to obtain text unimodal feature data; According to the image unimodal feature data and the audio unimodal feature data, multimodal feature fusion is performed on the text unimodal feature data to obtain the multimodal feature data.

3. The state recognition method according to claim 2, characterized in that: The step of extracting image unimodal features from image unimodal data to obtain image unimodal feature data includes: Get a pre-trained image extraction model; Inputting the image unimodal data into the pre-trained image extraction model to extract image features, thereby obtaining the image unimodal feature data; The step of extracting audio unimodal features from the audio unimodal data to obtain audio unimodal feature data includes: Performing a spectrogram conversion on the audio unimodal data to obtain audio spectrum data; Performing cepstrum feature analysis on the audio spectrum data to obtain the audio unimodal feature data; The step of extracting text unimodal features from the text unimodal data to obtain text unimodal feature data includes: Performing word frequency analysis on the text unimodal data to obtain a plurality of word frequency weights, each of the word frequency weights corresponding to a text word in the text unimodal data; According to the word frequency weight, the text unimodal data is word embedded to obtain the text unimodal feature data.

4. The state recognition method according to claim 2, characterized in that: The state recognition method further includes: Acquire original image data, original audio data, and original text data about industrial equipment; Performing image preprocessing on the original image data to obtain the image unimodal data, and performing audio preprocessing on the original audio data to obtain the audio unimodal data; Performing text preprocessing on the original text data to obtain the text unimodal data; The image unimodal data, the audio unimodal data and the text unimodal data are integrated to obtain the multimodal original data.

5. The state recognition method according to claim 4, characterized in that: The performing image preprocessing on the original image data to obtain the image single-modality data includes: Performing image filtering on the original image data to obtain intermediate image data; Performing image normalization on the intermediate image data to obtain the image single-modal data; The performing audio preprocessing on the original audio data to obtain the audio single-modal data includes: Performing audio denoising on the original audio data to obtain intermediate audio data; Dividing the intermediate audio data into audio frames to obtain the audio unimodal data; The performing text preprocessing on the original text data to obtain the text unimodal data includes: Performing text segmentation on the original text data to obtain intermediate text data; The intermediate text data is screened for text words to obtain the text unimodal data.

6. The state recognition method according to claim 1, characterized in that: According to the multimodal prompt and the industrial large language model, the historical vector data set is enhanced for retrieval and matching processing to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector with the highest similarity to the multimodal feature data in the historical vector data set, including: Performing query encoding on the multimodal prompt to obtain a multimodal query vector; According to the multimodal query vector, vector matching is performed on all the multimodal historical feature vectors to obtain a target historical feature vector, where the target historical feature vector is a multimodal historical feature vector having the highest vector similarity with the multimodal query vector; The multimodal prompt and the device state corresponding to the target historical feature vector are input into the industrial large language model to generate the state, and the state recognition result is obtained.

7. The state recognition method according to claim 1, characterized in that: The state recognition method further includes: According to the state recognition result, the multimodal feature data is state-verified to obtain a state verification result, wherein the state verification result is used to indicate whether the state recognition result is an accurate device state corresponding to the multimodal feature data; If the state verification result is that the state identification result is an accurate device state corresponding to the multimodal feature data, then the feature vector of the historical vector data set is updated according to the multimodal feature data and the state identification result to obtain an updated historical vector data set.

8. A state recognition system for industrial equipment, characterized in that: include: A first processing unit is used to obtain multimodal raw data of industrial equipment, a preset industrial multimodal prompt template, an industrial large language model and a historical vector data set, wherein the historical vector data set records a plurality of multimodal historical feature vectors of the industrial equipment and a device state corresponding to each of the multimodal historical feature vectors; A second processing unit is used to perform multimodal feature extraction on the multimodal raw data to obtain multimodal feature data; A third processing unit, configured to generate a prompt for the multimodal feature data according to the industrial multimodal prompt template to obtain a multimodal prompt; The fourth processing unit is used to perform enhanced retrieval and matching processing on the historical vector data set according to the multimodal prompt and the industrial large language model to obtain a state recognition result of the industrial equipment, wherein the state recognition result is the equipment state corresponding to the multimodal historical feature vector in the historical vector data set that has the highest similarity with the multimodal feature data.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images