State monitoring method and device based on multi-modal feature fusion, equipment and medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a state monitoring method and device based on multi-modal feature fusion, equipment and a medium, which comprises the following steps: acquiring an original audio signal and performing multi-scale convolution processing and gate processing to generate an audio feature; acquiring text description information of a device running state and performing adaptive transformer coding processing to generate a text feature; adopting a contrast learning strategy to jointly train the audio feature and the text feature to generate a shared embedding space model; extracting a real-time audio feature of a monitored device, and performing similarity matching on the real-time audio feature and candidate text features in a preset text feature library; and determining the device running state according to a matching result. The application realizes semantic alignment of the audio and the text through the shared embedding space, makes the device state recognition free from dependence on a large number of labeled samples, and still has good recognition effect and generalization ability under a low sample or zero sample condition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a state monitoring method, apparatus, device, and storage medium based on multimodal feature fusion. Background Technology

[0002] Against the backdrop of the rapid intelligent development of new energy vehicle production lines, equipment condition monitoring plays an increasingly prominent role in ensuring production continuity and product quality. Currently, many factories still rely primarily on traditional vibration sensors or manual inspections for the operation and maintenance monitoring of critical equipment. However, vibration sensors have weak detection capabilities when facing certain non-structural faults (such as gas leaks, non-contact friction, and early abnormal noises), especially lacking sufficient response to acoustic anomalies that only manifest in specific frequency bands. In addition, large-scale deployment of vibration sensors involves high installation and maintenance costs, limiting their promotion in multiple equipment and scenarios. Manual inspections are even less able to meet the real-time and full-process visualization management requirements of new energy production lines, suffering from problems such as response lag, subjective judgment, and limited coverage.

[0003] With the application of deep learning technology, intelligent fault identification based on acoustic signals has become a research hotspot in recent years. Some existing solutions utilize models such as convolutional neural networks and recurrent neural networks to process and identify audio signals from industrial sites. However, these methods generally rely on a large number of structurally complete and high-quality labeled fault audio samples as training sets to achieve ideal recognition accuracy. In practical applications, due to the diverse types of equipment and rapid operating pace in new energy production lines, some faults occur with extremely low probabilities, making it difficult and costly to collect high-coverage audio samples, which severely restricts the training efficiency and generalization ability of the models. Especially in the case of "zero samples" or "few samples," existing models struggle to effectively judge unseen abnormal patterns.

[0004] Furthermore, the model design of traditional acoustic monitoring methods is often highly task-dependent, meaning that a separate monitoring model needs to be built for each device, and different fault types also require parameter tuning and adaptation. This approach lacks a unified, highly scalable architecture, which is not conducive to cross-device and cross-scenario deployment and makes it difficult to cope with the dynamic and ever-changing production environment and complex and diverse operating states of new energy vehicle factories.

[0005] In summary, existing acoustic monitoring technologies still have significant limitations, including strong data dependence, insufficient generalization ability, and poor cross-device flexibility, making it difficult to support efficient perception and accurate identification of multi-source heterogeneous faults in complex production lines. These issues have become key technological bottlenecks that urgently need to be overcome in the acoustic monitoring of equipment in intelligent manufacturing systems. Summary of the Invention

[0006] The main objective of this invention is to provide a state monitoring method, device, equipment, and storage medium based on multimodal feature fusion, aiming to solve the technical problems of existing technologies that rely on single-modal data, resulting in poor anti-interference ability, lack of cross-modal semantic association, and inability to achieve high-precision equipment state identification in complex industrial noise environments.

[0007] To achieve the above objectives, the present invention provides a state monitoring method based on multimodal feature fusion, comprising: The original audio signal is acquired, and multi-scale convolution and gating processing is performed on the original audio signal to generate audio features; Obtain textual description information of the device's operating status, and perform adaptive transformer encoding on the textual description information to generate textual features; A contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model. Acquire the audio signal to be monitored from the monitored device, and extract real-time audio features from the audio signal to be monitored; The shared embedding space model is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library, and the operating status of the monitored device is determined based on the similarity matching results.

[0008] Furthermore, to achieve the above objectives, the present invention provides a state monitoring device based on multimodal feature fusion, comprising: The audio feature extraction module is used to acquire the original audio signal and perform multi-scale convolution and gating processing on the original audio signal to generate audio features; The text semantic encoding module is used to obtain text description information of the device's operating status and perform adaptive transformer encoding processing on the text description information to generate text features; The contrastive learning training module is used to jointly train the audio features and the text features using a contrastive learning strategy to generate a shared embedding space model. The real-time audio feature extraction module is used to acquire the audio signal to be monitored from the monitored device and extract real-time audio features from the audio signal to be monitored. The status matching and recognition module is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library through the shared embedding space model, and determine the operating status of the monitored device based on the similarity matching results.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a state monitoring program based on multimodal feature fusion stored in the memory and executable on the processor, wherein when the state monitoring program based on multimodal feature fusion is executed by the processor, it implements the steps of the state monitoring method based on multimodal feature fusion as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a state monitoring program based on multimodal feature fusion, wherein the state monitoring program based on multimodal feature fusion, when executed by a processor, implements the steps of the state monitoring method based on multimodal feature fusion as described above.

[0011] Beneficial Effects: This invention relates to the field of artificial intelligence technology and discloses a state monitoring method, apparatus, device, and medium based on multimodal feature fusion. The method includes: acquiring raw audio signals and performing multi-scale convolution and gating processing to generate audio features; acquiring textual description information of the device's operating status and performing adaptive transformer encoding processing to generate textual features; employing a contrastive learning strategy to jointly train the audio and textual features to generate a shared embedding space model; extracting real-time audio features of the monitored device and performing similarity matching with candidate textual features in a preset textual feature library; and determining the device's operating status based on the matching results. This invention, by constructing a shared embedding space model of audio and textual features, achieves state recognition in multi-device scenarios without requiring a large number of labeled fault audio samples. By combining real-time audio features with preset textual features for matching, it can accurately identify unseen fault states under zero-sample or low-sample conditions, thereby improving the generalization ability and practical deployment efficiency of acoustic monitoring systems. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a state monitoring method based on multimodal feature fusion according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the state monitoring method based on multimodal feature fusion according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the state monitoring device based on multimodal feature fusion of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The state monitoring method based on multimodal feature fusion provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can acquire the raw audio signal from the user terminal and perform multi-scale convolution and gating processing to generate audio features; acquire text description information of the device's operating status and perform adaptive transformer encoding processing to generate text features; use a contrastive learning strategy to jointly train the audio and text features to generate a shared embedding space model; extract the real-time audio features of the monitored device and perform similarity matching with candidate text features in a preset text feature library, determining the device's operating status based on the matching results. This invention, by constructing a shared embedding space model of audio and text features, achieves status recognition in multi-device scenarios without requiring a large number of labeled fault audio samples. By combining real-time audio features with preset text features for matching, it can accurately identify unseen fault states under zero-sample or low-sample conditions, thereby improving the generalization ability and actual deployment efficiency of the acoustic monitoring system. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the state monitoring method based on multimodal feature fusion provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] like Figure 2 As shown, the state monitoring method based on multimodal feature fusion proposed in this invention includes the following steps: S10, acquire the original audio signal, and perform multi-scale convolution and gating processing on the original audio signal to generate audio features; In this embodiment, the acquisition of the raw audio signal is accomplished through an audio acquisition device, typically deployed around the device's operating area and equipped with a high-sensitivity microphone array to capture the device's operating sounds under actual conditions. The audio signal is acquired in the form of a time-domain waveform, generally in linear pulse code modulation (PCM) format, with a configurable sampling rate of 16kHz, 32kHz, or higher to ensure complete preservation of high-frequency features. The data obtained in this step is the unprocessed raw time-domain waveform, serving as the input basis for subsequent feature extraction.

[0017] After the raw audio signal is acquired, it needs to be adapted to the time-frequency domain structure to facilitate subsequent feature extraction operations. First, a short-time Fourier transform can be performed to form a two-dimensional time-frequency spectrum matrix. The horizontal axis represents time frames, and the vertical axis represents frequency distribution. Each point in the matrix represents the energy distribution within a specific time window and frequency band. This structure can be used as input to subsequent multi-scale convolutional networks. By setting different window lengths and overlap ratios, the balance between frequency resolution and time resolution can be adjusted. For example, a window length of 512 points and a stride of 256 points is a basic method for achieving time-frequency signal deconstruction.

[0018] The convolutional processing stage is based on a multi-scale convolutional neural network with a parallel structure. This network contains multiple convolutional branches, each using convolutional kernels of different sizes, such as 3×3, 5×5, and 7×7, to extract local structural features of the audio at different scales. Smaller kernels are used to capture local details of rapidly changing or high-frequency oscillations in the audio signal, while larger kernels are used to perceive rhythmic, periodic, or wide-bandgap features. For example, the 3×3 convolutional branch responds to device oscillation edges and impulsive signals, while the 7×7 convolutional branch identifies low-frequency resonances or long-period anomalies. All convolutional branches can use the same number of channels, ensuring uniform fusion across the channel dimension.

[0019] The multi-scale feature maps output by the parallel convolutional network are concatenated and then passed to the gating processing module. The gating process uses a gating mechanism to dynamically modulate the multi-channel feature maps. Each channel generates a separate set of gating weight vectors, with weights limited to the range [0,1], representing the importance or confidence contribution of the current channel to the overall task. Weight generation can be achieved through single-layer or multi-layer perceptron networks, or by using an attention mechanism to generate response weights dependent on the local structure of the feature map. Channel weights are applied to the original feature map through a Hadamard product to selectively enhance or suppress channels. Paths with zero channel weights correspond to suppressed interference signal channels, such as background noise or interference responses from non-target devices. The Hadamard product preserves the size and semantic structure of the original feature map, facilitating contextual continuity in subsequent processing stages.

[0020] The gated multi-channel feature maps undergo cross-layer fusion. The fusion structure used is residual connections, with either skip-joining or weighted stacking. This unifies the feature maps from shallow, mid-layer, and deep layers into a single fusion output. Shallow outputs retain edge features and short-term variations, mid-layer fusion encodes mid-term temporal relationships, and deep outputs aggregate high-order semantic information. Residual connections avoid feature degradation in deep networks and enhance the information representation capability of the feature stream. The final fusion output is the audio feature, providing a unified embedding representation for subsequent semantic matching models and contrastive learning.

[0021] In practical deployment scenarios, microphone arrays of varying numbers and directions can be configured based on the sound source characteristics of the equipment. For example, for enclosed chassis-type equipment, proximity microphones can be used, while for open production line equipment, a combination of highly directional array microphones can be used to eliminate background noise. The microphone sampling accuracy can be adjusted according to actual signal-to-noise ratio requirements. For instance, when detecting abnormal sounds from high-speed rotating components, the sampling rate can be increased to 48kHz.

[0022] In the convolution module, the size and number of convolutional kernels and the network depth can be flexibly configured. For example, in edge deployment environments that require higher real-time performance, shallow convolutional structures with two branches (3×3 and 5×5) can be used to reduce inference time; while in cloud training scenarios where higher detection accuracy is required, the number of convolutional branches and channels can be increased, and more scale convolutional kernels can be connected in parallel to extract fine-grained and global features.

[0023] In the gating processing module, the channel weight generation strategy can be adapted to attention weights or a gating strategy based on local maximum response values, or feature-selective activation can be performed using the SE module structure (Squeeze-and-Excitation). The number of channels and gating resolution can be adjusted according to the device type. For example, for multi-source dense devices, the number of channels can be configured from 64 to 128 to facilitate feature separation.

[0024] Residual connections can be implemented by adding and fusing feature maps, or by concatenating features and then fusing them through 1×1 convolutions, thus unifying the semantics and scale of multi-level features. In terms of the fusion structure, the number of fusion nodes and the connection positions can be configured as needed. If short-term high-frequency faults are a concern, shallow feature preservation can be enhanced; if periodic low-frequency waveforms are a concern, deep feature paths can be weighted.

[0025] Example Description: In a new energy vehicle assembly workshop, a microphone array is deployed above a lithium battery welding workstation to collect audio signals during the welding process at a sampling rate of 16kHz. The collected signals are fed into a short-time Fourier transform module to generate a time-frequency spectrum matrix, which is then input into a multi-scale convolutional network containing three convolutional branches: 3×3, 5×5, and 7×7. A gating module generates a weight vector for each channel, which is then weighted using a Hadamard product to suppress background equipment noise and ventilation noise. The multi-layer outputs are fused via residual connections to ultimately generate a 128-dimensional audio feature vector for embedding spatial learning and equipment status recognition. In scenarios where a slight howling sound is heard at the weld joint, this processing flow can accurately extract the difference signal, successfully identify the abnormal state, and trigger an early warning display and recording.

[0026] This embodiment extracts the temporal structure at different frequency dimensions through multi-scale convolution, combines a gating mechanism to suppress background interference, and further employs cross-layer residual connections to enhance expressive power. This enables the output audio features to accurately represent the operating state of the target device in complex acoustic backgrounds, while also exhibiting higher robustness and generalization ability. This processing method significantly improves the feature's response to early, subtle faults and maintains high discriminative power even under low signal-to-noise ratio or high dynamic interference conditions.

[0027] S20, obtain text description information of the device's operating status, and perform adaptive transformer encoding on the text description information to generate text features; In this embodiment, the textual description information of the equipment's operating status is typically obtained from manual fault reporting records, operator comments, maintenance log systems, or real-time human-machine interaction input. This information appears in natural language form, usually including structured short sentences (such as "a high-frequency sharp whistling sound occurred") or long text records (such as "a continuous high-frequency noise occurred during battery module packaging"). To eliminate non-target semantic interference, semantic cleaning and syntactic parsing of the text content are required. First, a hierarchical lexical filtering mechanism is applied. This mechanism, based on industrial-specific dictionaries and statistical strategies (such as TF-IDF or BM25), breaks down the text into the smallest semantic units, namely lexical units or sub-tokens, and removes stop words, common descriptive adverbs, and other components with low information load, retaining only keywords that are directional to fault type, equipment location, and timing description.

[0028] The text after hierarchical lexical filtering is input into the adaptive transformer module, which typically employs a modified Transformer structure for multi-layer semantic encoding. The transformer uses a multi-head self-attention mechanism to model the weights of fully connected lexical units, capturing the contextual relationships between keywords. To enhance the model's sensitivity to key fault description lexical units, a pulsed attention enhancement mechanism is introduced. When the model detects that the current lexical sequence contains preset high-priority fault keywords (such as "abnormal noise," "high temperature," and "air leakage"), the row or column corresponding to that keyword in the weighted matrix of the corresponding attention head will be amplified by an additional factor, forming a semantic pulse effect, thus occupying a higher representational importance in the vector space. The determination of high-priority keywords is based on a human experience database and historical fault scenario frequency statistics, and can be continuously updated through a semi-supervised approach.

[0029] The context-aware vector sequence output through the attention mechanism is transformed into primary semantic vectors by a multi-layer feedforward network. Then, feature compression and spatial mapping are performed using a learnable projection matrix within an adaptive transformer, aligning the dimensional structure of the text features with that of the audio features. This projection matrix is ​​obtained through training and optimization; its structure is typically a linear transform or a multi-layer perceptron with a non-linear activation function, used to eliminate dimensionality inconsistencies between different modalities. The final output text features are fixed-length embedding vectors with the same dimensionality as the audio features, which can be used for similarity matching and contrastive learning in the embedding space.

[0030] In practical applications, the input format of text description information can be diversified. For industrial MES system access scenarios, text is input through structured fields, such as combinations of fields like fault type, sound description, and location of occurrence. For natural language input interfaces, a speech recognition system can be used to achieve speech-to-text conversion and subsequent processing. During the dictionary selection process, a multi-level dictionary structure can be constructed: the first layer contains general fault terms for equipment, the second layer contains professional terms for specific process equipment, and the third layer contains user-defined terms for specific scenarios, achieving cross-domain transferability.

[0031] The adaptive transformer encoding module can employ BERT or a modified TinyTransformer structure, compressing the number of parameters as needed to adapt to edge deployments. For resource-constrained scenarios, only a 2-4 layer Transformer encoder structure can be retained to reduce the number of attention heads, while adjusting the hidden dimension to compress the model size. Impulsive attention enhancement mechanisms can be implemented by superimposing high-priority keyword location information weights into the attention score, or by directly enhancing the attention intensity of specific location channels using a location mask matrix.

[0032] The projection matrix can be designed as a single-layer linear transformation matrix or a two-layer perceptron structure, where the embedding dimension is consistent with the audio feature dimension, such as 128-dimensional or 256-dimensional. For scenarios where training with few sample texts is insufficient, text augmentation strategies (such as random replacement and synonym expansion) can be used to enhance robustness.

[0033] Example Description: In a new energy vehicle assembly line, after detecting an assembly anomaly in a battery module, the operator inputs the text "A sharp whistling sound exists inside the assembled module." The system retains the keywords "whistling sound" and "module" through text segmentation and domain dictionary matching, identifying "whistling sound" as a high-priority fault keyword. During the adaptive transformer encoding process, the word corresponding to "whistling sound" is assigned a high weight in the attention matrix. The resulting text feature vector is highly similar to the collected high-frequency abnormal sound wave audio feature vector in the shared embedding space, thus supporting accurate identification of equipment status and alarm triggering. This mechanism achieves semantic enhancement of text-to-audio perception, effectively improving the correlation between unstructured descriptions and structured acoustic features.

[0034] This embodiment improves the model's ability to extract key fault semantics from natural language input by performing hierarchical lexical filtering and adaptive transformer encoding on text description information. Furthermore, it significantly enhances the responsiveness to high-priority anomaly descriptions through a pulsed attention enhancement mechanism. The learnable projection matrix further achieves structural alignment between text and audio features in a shared space, thus ensuring consistency in multimodal semantic expression. The overall design improves the modeling ability for unstructured text semantics and the adaptation effect of contrastive learning in situations lacking large amounts of labeled text data.

[0035] S30, a contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model; In this embodiment, a contrastive learning strategy is employed to jointly train audio and text features. The aim is to construct a shared embedding space that aligns features from two heterogeneous modalities, bringing audio and text features corresponding to the same device state closer together, while feature pairs from different states are further apart. First, positive and negative sample pairs are constructed for training. Positive sample pairs consist of audio and text features from the same device state, such as a sound wave clip and a text description of the same abnormal event. Negative sample pairs consist of heterogeneous modal combinations from different states or device conditions. These positive and negative sample pairs are typically constructed as a relation matrix or batch comparison structure, used as input to the contrastive loss function.

[0036] In the feature alignment stage, audio and text features are simultaneously input into a shared embedding space model for similarity learning. The shared embedding space model comprises two main branches: an audio feature sub-network and a text feature sub-network. These two features have already been generated through corresponding processing steps in previous stages, so their structures are not expanded further in this stage. After processing by their respective sub-networks, the two features are mapped to the same vector space, outputting fixed-length vector representations. The outputs of these two sub-networks, after normalization, can be used to calculate cross-modal cosine similarity scores, measuring the semantic closeness of the two modalities in the shared space.

[0037] To optimize the structure of this embedding space, a contrastive loss function is introduced during training, such as the sample pair loss function built based on the InfoNCE framework. This function enhances the model's ability to aggregate semantically consistent samples by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. A temperature adjustment parameter is introduced into the formula to control the sensitivity to negative samples, adjusting the model's learning intensity for difficult-to-classify samples. To enhance training stability, sample sampling strategies, such as hard negative sample mining or weighted sampling, can also be introduced to improve the model's generalization ability in multi-device types and complex sound environments.

[0038] After training, the learned shared embedding space model can be fed new audio and text features during the inference phase to achieve cross-modal similarity matching and device status determination. The model possesses end-to-end optimization capabilities, allowing for further incremental training in the future based on sample feedback.

[0039] During implementation, the training of the shared embedding space model can be deployed on a cloud platform with ample computing resources, with the input being a batch dataset of historical audio-text sample pairs. Audio and text features are obtained through an offline preprocessing process, and each pair of samples is labeled to indicate whether it originates from the same device state, in order to construct supervised labels. The training process can employ gradient descent optimization algorithms, combined with batch normalization and dropout mechanisms to prevent overfitting.

[0040] The shared embedding space can be constructed by having two sub-network structures share parameters, or by using a non-shared architecture to optimize the mapping paths for audio and text modalities separately. The contrastive loss function can be implemented in various ways, such as SimCLR or NT-Xent, where audio serves as the anchor and text as the comparison object; a bidirectional structure can also be used, allowing the two modalities to serve as mutual comparison references. Furthermore, in edge device deployment scenarios, the trained shared embedding space model can be compressed, such as through knowledge distillation or low-rank decomposition, to reduce deployment overhead.

[0041] For training scenarios with small sample sizes, data augmentation strategies can be introduced, including audio reverberation, cropping, and semantically similar text replacement, to expand the coverage of training samples. For training data with imbalanced classes, class reweighting coefficients can be added to the loss function to reduce the bias of master frequency faults on model training.

[0042] Example Explanation: In historical samples collected from new energy vehicle assembly lines, a fault scenario of "continuous abnormal noise during battery pack packaging" includes multiple audio segments and multiple text descriptions. Positive and negative sample pairs are constructed, where positive samples represent the audio features and corresponding text descriptions for this event, and negative samples represent the audio and text for other events. During training, the model continuously adjusts the embedding vectors, causing the text embedding vectors for the "continuous abnormal noise" category to gradually approach the corresponding audio embedding vectors, while moving away from the embedding points corresponding to other text descriptions such as "normal operation" or "start-up vibration." After several iterations, the model learns to aggregate the semantically descriptive text of "packaging abnormality" and the audio features that actually generate high-frequency noise into the same semantic cluster, providing a unified and compact feature representation basis for subsequent matching judgments.

[0043] This embodiment employs a contrastive learning strategy to jointly train audio and text features, significantly improving the model's ability to discriminate in unknown fault scenarios without relying on fully labeled categories. The shared embedding space model constructs a unified vector space that can simultaneously accommodate speech and text semantics, ensuring high consistency in the multimodal representation of device states, thereby enhancing the robustness and generalization ability of state determination. Even in cases of uneven sample distribution or limited sample counts, the introduction of negative sample structures and contrast mechanisms can maintain recognition accuracy and reduce reliance on large-scale labeled data.

[0044] S40, acquire the audio signal to be monitored from the monitored device, and extract real-time audio features from the audio signal to be monitored; In this embodiment, acquiring the monitored audio signal from the monitored device refers to continuously collecting raw sound wave data generated by the device during operation using acoustic acquisition devices deployed at key locations, such as directional microphone arrays or anti-interference industrial microphones. This signal is generally a continuous audio stream in the time domain, and the sampling frequency is selected based on the specific device characteristics, commonly ranging from 16kHz to 96kHz, ensuring coverage of various fault characteristic frequency bands, from low-frequency mechanical resonance to high-frequency friction. The device coverage should be based on the distribution of its moving parts, working cavities, and shielding structures, with pickup nodes rationally arranged to ensure the representativeness and consistency of the acquired signal.

[0045] After acquiring the signal, a joint mechanism of time-frequency analysis and feature modeling is needed to extract discriminative real-time audio features. First, the original time-domain audio stream is preprocessed, including silence removal, bandpass filtering, and energy normalization, to remove invalid background signals and low-frequency drift. Then, the audio signal is mapped into a two-dimensional time-frequency spectrum using Short-Time Fourier Transform (STFT) or continuous wavelet transform, constructing a feature space interwoven with the frequency and time axes, which serves as the input for subsequent feature extraction.

[0046] The atlas is input into a multi-scale feature extraction network composed of multi-branch convolutional structures. Different kernel sizes are used to model local short-period textures (such as high-frequency impacts) and long-period trends (such as low-frequency vibrations). A gating mechanism is introduced during feature extraction to calculate the gating weight vector channel by channel, combined with Hadamard product to suppress irrelevant channel noise and enhance the feature dimensions related to faults. To further improve robustness, a residual structure can be introduced to aggregate multi-level semantic features.

[0047] The goal of this stage is not only to extract spectral information but also to achieve real-time performance; therefore, the model structure must possess low-latency inference capabilities. The feature extraction results form a fixed-length vector in the output layer, representing the audio semantic embedding representation under the current window, which serves as the basic input for subsequent state determination or similarity matching.

[0048] In practical deployments, the audio signal acquisition module can be integrated onto edge computing nodes, utilizing low-power ARM chips or embedded GPUs for local signal processing. The acquisition time window can be set to a sliding window of 1 to 3 seconds, with an audio feature update performed after each acquisition to meet near real-time response requirements. The feature extraction network is deployed as a lightweight convolutional neural structure, employing depthwise separable convolutions or a hybrid convolutional attention module to reduce computational load.

[0049] In high-interference scenarios, such as production line environments where multiple devices operate in parallel, an adaptive noise estimation module can be added to dynamically strip away periodic background interference patterns identified in the spectrum, thereby highlighting the characteristics of the target device. Furthermore, the feature output can be calibrated to ensure consistent feature distribution across different device acquisition channels, facilitating cross-device applications within a shared embedding space.

[0050] The sampling frequency, number of convolutional layers, and number of channels can be adjusted in different application environments. For devices dominated by high-frequency, short-lived abnormal signals, such as stamping machinery, the receptive field and frame rate density of the high-frequency branch can be increased; for devices dominated by abnormal signals such as continuous friction or fan howling, the number of convolutional layers of the low-frequency structure can be increased to enhance temporal stability modeling.

[0051] Example Description: In a new energy vehicle battery packaging production line, a microphone array is deployed next to an automated tightening workbench. When stripping or squealing occurs during bolt tightening, the acquisition system captures the signal at a sampling rate of 44.1 kHz per second, generates a 64×128 time-frequency spectrum through short-time Fourier transform, and inputs it into a multi-scale convolutional neural network. The model automatically identifies abrupt frequency bands in the high-frequency branches and suppresses low-energy segments through a gating mechanism. The audio feature vector extracted within the current 3-second window is then sent to a comparison matching system to identify the fault type corresponding to the anomaly, enabling simultaneous execution of edge reasoning and visual alarms.

[0052] This embodiment extracts real-time audio features from the raw audio signals during equipment operation, enabling the acquisition of high temporal resolution and high frequency coverage operational status information without relying on additional physical sensors. This is suitable for dynamic and complex industrial environments. Employing multi-scale modeling and gating mechanisms, it effectively suppresses background noise and non-fault signal interference while retaining key fault response components. This processing method supports continuous inference, possesses high efficiency and low latency, and is adaptable to edge deployment, achieving real-time status awareness.

[0053] S50, using the shared embedding space model, the real-time audio features are matched with each group of candidate text features in the preset text description information library for similarity, and the operating status of the monitored device is determined based on the similarity matching results.

[0054] In this embodiment, the matching operation using a shared embedding space model refers to calculating the similarity between the audio features to be identified and the pre-constructed set of text features within a unified feature representation space, and completing the state mapping based on the similarity ranking results. The shared embedding space model is obtained through joint training, and its purpose is to project semantic information from different modalities (audio and text) into a structurally and distributed vector space, ensuring that audio and text features describing the same device state are geometrically close to each other. Sharing is guaranteed through loss function optimization; that is, the feature vectors of positively matched pairs are closer in this space, while non-matching pairs are pushed further apart.

[0055] Real-time audio features are fixed-length vectors generated by the monitored device at the current moment, representing the acoustic characterization of the current operating state. The pre-built text description information library is a set of text features constructed in advance. Each text description corresponds to a typical device state or fault type, usually derived from manual annotation or standard operating instruction documents, and is encoded into a fixed-dimensional embedding representation through an adaptive transformer model. These text features have been aligned into the shared space during the model training phase.

[0056] During similarity matching, candidate text features are first extracted sequentially from the text description information database and then compared with real-time audio features using a one-to-many cosine similarity calculation. Cosine similarity is an indicator calculated based on the angle between vectors, which can eliminate the influence of feature magnitude and only focus on directional similarity, making it suitable for measuring the semantic alignment between audio and text. All calculation results are combined into a matching score, forming a matching result matrix. This matrix serves as the decision-making basis; the text feature corresponding to the highest score is considered to be semantically closest to the current audio feature, and thus mapped to the corresponding device operating state.

[0057] State mapping goes beyond simple similarity filtering. It can also incorporate device context information, such as the current device type, operating condition, and historical state transition probabilities, to ensure the output results are reasonable in terms of temporal continuity and semantic stability. In practical applications, to improve model robustness, confidence verification is typically performed on multiple high-scoring candidates. If the confidence level is insufficient, manual review can be triggered, or a definitive judgment can be temporarily withheld.

[0058] In industrial deployments, the text description information database can be pre-built in the cloud and regularly maintained and updated based on expert experience or user feedback. Each text, after language cleaning, word segmentation, and structuring, is input into an adaptive transformer model to generate text features, which are then uniformly stored in an embedded database, supporting rapid indexing. Real-time audio features, generated through edge devices, can be compared with all candidate text features in the database locally or remotely for similarity calculation.

[0059] The similarity calculation module can employ a vectorized batch calculation strategy to compute the similarity scores of all candidate texts in parallel at once. To improve response speed, the text feature library can utilize a vector search engine (such as FAISS) to construct an index structure, achieving approximate nearest neighbor search. After calculation, the Top-N candidate texts are extracted by sorting them according to their similarity scores, and the status is determined by combining a set threshold or contextual conditions.

[0060] In multi-device scenarios, to avoid mismatches of text features between different devices, device ID filtering or device domain constraints embedded in vectors need to be introduced. Before the status output, a strategy module based on confidence or stability indicators should be added to determine whether to maintain a delayed status output, enter an observation state, or issue an early warning.

[0061] Example Description: On a new energy vehicle power battery assembly line, a tightening device emits a whistling sound during operation. The edge computing unit extracts the current audio features. This feature is input into a shared embedding space model and matched against multiple fault description features in a text information database, including "motor running normally," "thread jamming," and "drive whistling." The model's output similarity score shows that the audio has the highest cosine similarity to the "drive whistling" text feature, exceeding a preset confidence threshold. Therefore, the system directly determines that the device is in a "drive whistling" state. The system then pushes this status to the interface and triggers the associated audible and visual alarm module.

[0062] This embodiment achieves state recognition without explicit label supervision by performing audio-text similarity matching in a unified feature space, improving the system's ability to perceive unseen anomaly types. The shared embedding space fuses cross-modal information, enabling the system to identify faults with zero samples. Alignment using cosine similarity maintains stability under varying audio source intensities and text description lengths. This approach constructs a reasoning path for state determination without manual label verification, adapting to the operational needs of various operating conditions and equipment types.

[0063] This invention relates to the field of artificial intelligence technology and discloses a state monitoring method, apparatus, device, and medium based on multimodal feature fusion. The method includes: acquiring raw audio signals and performing multi-scale convolution and gating processing to generate audio features; acquiring textual descriptions of the device's operating status and performing adaptive transformer encoding to generate textual features; jointly training the audio and textual features using a contrastive learning strategy to generate a shared embedding space model; extracting real-time audio features of the monitored device and performing similarity matching with candidate textual features in a preset textual feature library; and determining the device's operating status based on the matching results. This invention, by constructing a shared embedding space model of audio and textual features, achieves state recognition in multi-device scenarios without requiring a large number of labeled fault audio samples. By combining real-time audio features with preset textual features for matching, it can accurately identify unseen fault states under zero-sample or low-sample conditions, thereby improving the generalization ability and practical deployment efficiency of acoustic monitoring systems.

[0064] In one embodiment, step S10 includes: S101, acquire the raw audio signal from the target device; S102, perform data enhancement on the original audio signal, the data enhancement including adding random time shift and injecting mixed noise; S103 adds adversarial perturbations to the data-augmented original audio signal to generate the original audio signal after anti-interference training. S104, the original audio signal after anti-interference training is input into a parallel multi-scale convolutional neural network, and multi-scale feature maps are generated through convolutional kernels of different sizes; S105, a gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map; S106, The gated weight vector is weighted channel by channel by the corresponding multi-scale feature map through the Hadamard product to generate the weighted feature map; S107, the weighted feature maps of the multi-level outputs are fused across layers through residual connections to generate the audio features.

[0065] In this embodiment, acquiring the raw audio signal from the target device refers to real-time acquisition of the sound signal emitted by the device of interest under different operating states using a microphone array or sensing components in an actual operating environment. This audio signal is a continuous time-domain waveform without any processing and may contain both valid sound source information generated by the device's own operation and background noise. The quality of the raw signal directly affects the expressive power of subsequent feature extraction; therefore, the integrity and sampling accuracy of the signal must be ensured through reasonable sampling rates (e.g., 16kHz or higher) and dynamic range parameters.

[0066] Data augmentation of the original audio signal aims to improve the model's robustness to input disturbances or environmental changes. Adding random time shifts involves translating the original signal along the time dimension to simulate the uncertainty of the signal's origin. Hybrid noise injection involves superimposing common industrial background noises (such as fan noise, impact noise, and mechanical drag noise) into the original signal at random signal-to-noise ratios, allowing the model to adapt to input distributions under non-ideal acquisition conditions. Noise samples can be derived from historical records or collected on-site and require bandpass filtering and energy normalization to ensure optimal synthesis results.

[0067] Adding adversarial perturbations to enhanced audio signals is a strategy for improving model robustness during the training phase. Adversarial perturbations involve adding small perturbations to the audio signal that are most sensitive to the model's discrimination boundaries. These perturbations are almost imperceptible to the human ear but effectively expose the model's weaknesses in anomalous regions. Implementation typically involves using algorithms such as the FastGradient Sign Method to generate tiny perturbations and superimpose them onto the original signal. This constructs more challenging training samples, forcing the model to learn a more generalizable representation structure.

[0068] The core processing method in the feature extraction stage is to input the anti-interference trained audio signal into a parallel multi-scale convolutional neural network. The multi-scale structure extracts local detail features, structural mid-frequency features, and global change trends along parallel pathways by setting different convolutional kernel sizes (e.g., 3×3, 5×5, 7×7, etc.), thereby constructing a multi-level description that can discern the device status. Each convolutional branch can be composed of a concatenated convolutional layer, a batch normalization layer, and an activation function layer. Finally, the outputs of each branch are concatenated along the channel dimension to form a multi-scale feature map.

[0069] A gating unit is used to generate a gating weight vector for each channel, aiming to give the model the ability to adaptively adjust the importance of different feature channels. The gating unit typically consists of a fully connected layer and a sigmoid activation function. Its input is the feature map representation after global pooling, and its output is a scalar weight for each channel, reflecting the importance of that channel under the current input. The gating weight vector is then multiplied channel-by-channel with the original channel features using a Hadamard product, thereby enhancing useful information and suppressing redundant channels, generating a weighted feature map.

[0070] Residual connections are used to fuse weighted feature maps from multiple layers across different semantic levels, aiming to integrate feature representations from different semantic levels. Residual connections involve adding or concatenating shallow and deep features after aligning their dimensions, which helps preserve structural information in early features and mitigate gradient vanishing. The fused features not only contain local characteristics such as edges and periods from lower layers but also retain abstract patterns and global trends from higher layers, thus better supporting subsequent semantic mapping and state discrimination.

[0071] This embodiment extracts feature information from different frequency bands and structural levels in audio signals through multi-scale convolutional paths, enhancing the ability to distinguish complex equipment status sounds. Data augmentation and adversarial operations improve the model's robustness to background noise and input fluctuations, making it suitable for production environments with multiple types of interference. The gated modulation mechanism introduces a semantic weighting mechanism, improving the sparsity and discriminative power of feature representations. Multi-layer feature fusion further expands the model's perception range of local and global patterns, ensuring the integrity and discriminative quality of input features for downstream tasks.

[0072] In one embodiment, step S20 above includes: S201, Receive natural language text containing a description of the device's operating status; S202, perform hierarchical lexical filtering on the natural language text, filter non-key semantic components based on a predefined industrial domain dictionary, and generate filtered text description information; S203, the filtered text description information is semantically encoded through the multi-head self-attention mechanism of the adaptive transformer. If a preset high-priority fault keyword is detected in the filtered text description information, the pulse attention enhancement module of the adaptive transformer is triggered to strengthen the semantic weight of the word unit corresponding to the high-priority fault keyword and generate a primary semantic vector. S204, the primary semantic vector is mapped to a shared embedding space consistent with the audio feature dimension using the learnable projection matrix of the adaptive transformer to generate the text features.

[0073] In this embodiment, receiving natural language text containing a description of the device's operating status refers to obtaining user-inputted or system-generated language descriptions from the operator interface, automatic data acquisition platform, or device logs. This text can originate from speech recognition results, maintenance records, anomaly reporting systems, etc., and its content typically describes the device's operating status, detected problems, or fault phenomena within a specific time period, such as "the cooling fan is making intermittent sharp whistling noises" or "the motor starts with a low-frequency resonance sound." The diversity of text input determines the robustness requirements of the subsequent semantic processing system.

[0074] Hierarchical lexical filtering of natural language text is the first step in the structured cleaning of the raw text input. Hierarchical processing includes dividing sentences into phrase-level and word-level structures, followed by operations such as part-of-speech tagging and named entity recognition. In the lexical filtering stage, by comparing with a specialized dictionary for the industrial field (containing technical terms, fault categories, and descriptions of high-frequency operational behaviors), stop words, modifiers, and redundant components irrelevant to equipment status are removed, retaining high-semantic-density technical lexical terms. This dictionary can be built by domain experts, supports dynamic updates, and the filtered text description information has a higher fault feature signal ratio.

[0075] The multi-head self-attention mechanism using an adaptive transformer to semantically encode the filtered text description information is the core processing stage for feeding the cleaned text input into the structured embedding model. The multi-head attention mechanism models the contextual dependencies between lexical units through multiple independent attention subspaces, enabling the model to capture various semantic associations simultaneously, such as causal relationships, synonym / contrast relationships, and component-function mappings. Each attention head generates a set of weighted lexical embeddings, which are then concatenated or weighted fused to output a semantic vector representation with consistent dimensions.

[0076] If the filtered text description contains pre-defined high-priority fault keywords, the pulsed attention enhancement module in the adaptive transformer is triggered. This module handles fault types with urgent discrimination needs, such as safety-related or high-impact equipment anomalies. Keywords like "explosive noise," "burning smell," and "strong friction sound" can be pre-defined as high priority. Once these keywords are detected, the model dynamically adjusts the attention weights, focusing more attention on the area surrounding relevant lexical terms, thereby increasing the response strength of the representation to key fault signals. The pulsed enhancement mechanism is typically implemented through positional weighting or lexical confidence gating, modulating its contextual sensitivity within the encoding structure.

[0077] A learnable projection matrix is ​​used to map primary semantic vectors to a shared embedding space with dimensions consistent with the audio features. The aim is to build a unified vector space foundation for subsequent audio-text matching. The projection matrix is ​​a high-dimensional linear transformation structure, and its parameters are optimized together with the audio branch during joint training, aligning the embeddings of different modalities in spatial distribution. The transformed text features retain the original semantics while possessing the dimensionality and structural consistency required for cross-modal comparison. The shared embedding space makes audio and text features of similar states closer in Euclidean space, providing fundamental support for downstream matching and state inference.

[0078] This embodiment extracts discriminative embeddings of device status from unstructured natural language input through a text processing workflow, and enhances industrial semantic saliency through a domain filtering mechanism. An adaptive coding structure combined with a pulse enhancement strategy enables the model to focus on key fault points even when faced with ambiguous or complex expressions, improving expressive power. A projection mechanism aligns vectors with the audio modality, achieving effective fusion of data from different sources within the same discriminative space. This process provides high-quality text features for subsequent multimodal training and reduces the false matching rate.

[0079] In one embodiment, step S30 above includes: S301, The shared embedding space model is pre-trained on the cloud server using historical audio features and corresponding historical text features, wherein the historical audio features are generated by multi-scale convolution processing and gated weighting of the original historical audio signal, and the historical text features are generated by adaptive transformer encoding of historical text descriptions to obtain initial model parameters. S302, Based on the audio features and the text features, construct a matching audio-text positive sample pair relationship matrix and a non-matching audio-text negative sample pair relationship matrix, wherein positive sample pairs represent audio features and text features of the same device state, and negative sample pairs represent audio features and text features of different device states. S303, Based on the positive sample pair relationship matrix and the negative sample pair relationship matrix, calculate the cosine similarity score of the positive sample pair and the negative sample pair; S304, The cosine similarity score is optimized using a contrastive loss function that includes a temperature adjustment parameter. The model’s attention to low-similarity negative samples is controlled by adjusting the temperature adjustment parameter, thereby generating a shared embedding space model. S305, the initial model parameters are sent to the edge computing device, and a real-time positive and negative sample pair relationship matrix is ​​constructed based on the real-time generated audio features and text features, and the shared embedding space model is incrementally fine-tuned and trained. S306, dynamically adjust the sampling weights of positive and negative sample pairs according to equipment type and fault risk level, and perform weighted training on sample pairs of key fault states. S307, Periodically verify the positive and negative sample discrimination ability of the shared embedding space model. When the discrimination is detected to be lower than the threshold, cloud parameter optimization is triggered.

[0080] In this embodiment, pre-training using historical audio features and corresponding historical text features on a cloud server forms the foundation for building multimodal semantic fusion capabilities. Historical audio features are processed through multi-scale convolution to extract responses at different frequency components, and then a gated weighting mechanism is used to adjust channel contributions, generating a high-dimensional representation with time-frequency resolution and semantic distribution. Historical text features are processed by applying hierarchical lexical filtering and adaptive transformer encoding to extract structured embedding vectors highly correlated with the device's operating state. These two embeddings serve as input to the initial training set, completing pre-training in a shared embedding space. The resulting model parameters are used as initial weights for subsequent joint training, improving training convergence speed and semantic alignment capabilities.

[0081] Constructing matching audio-text positive sample pair relationship matrices and non-matching audio-text negative sample pair relationship matrices is a way to structure the training set. Positive sample pairs refer to audio and text features originating from the same device status label, such as "motor starting vibration sound" and "impact vibration at startup" forming a positive pair. Negative sample pairs come from different labels, such as the combination of "fan bearing abnormal noise" and "heating module current abnormality." Organizing these sample pairs in matrix form facilitates the unified construction of the computational graph during the training phase, supporting parallel computation and batch gradient propagation.

[0082] Based on the aforementioned positive and negative sample pair relationship matrix, calculating the corresponding cosine similarity score is the core operation for achieving multimodal contrastive supervision. Cosine similarity measures the degree of similarity between the angles formed by two vectors in a high-dimensional space. Positive sample pairs are expected to receive higher scores, while negative sample pairs receive lower scores, thus enabling the model to learn the discriminative directionality between audio and text.

[0083] Optimizing the similarity score using a contrastive loss function with a temperature-adjusted parameter fine-tunes the gradient structure during training. This temperature-adjusted parameter scales the logits value in the similarity distribution, making the model more sensitive to difficult-to-distinguish negative samples and less concerned with easily distinguishable samples, thus enhancing the model's generalization ability. The entire loss function structure supports many-to-one or many-to-many sample distributions, improving the robustness of semantic embedding learning. In contrastive learning training, low-similarity negative samples refer to text features that exhibit significant semantic differences from the current audio features in the shared embedding space; their similarity scores are usually significantly lower than those of positive or difficult-to-distinguish pairs. Quantitatively, low-similarity negative samples can be defined by calculating the cosine similarity score between audio and text features. Taking a standardized scoring range [-1, 1] as an example, negative samples with a similarity score below 0.2 are generally considered to have significant semantic differences and lack sufficient discriminative challenge, thus being classified as low-similarity negative samples.

[0084] In practical implementation, the low similarity judgment interval can be dynamically determined based on the statistical distribution of the training data. For example, in a training batch, the similarity score distribution of all negative sample pairs can be statistically analyzed, and samples with scores below the bottom 20th percentile of the negative sample similarity distribution can be defined as low-similarity negative samples. This definition is both data-driven and adaptable to changes in similarity distribution at different training stages, thus more stably identifying negative samples that contribute little to the current model's learning. These samples can be assigned a lower gradient contribution in subsequent loss optimization, or under certain strategies, their participation in training can be completely ignored, allowing training resources to be concentrated on those difficult negative samples with similarity scores in the boundary region (such as between 0.4 and 0.7), thereby accelerating model convergence and enhancing the model's discriminative ability at semantic boundaries.

[0085] Distributing initial model parameters to edge computing devices and constructing a real-time positive-negative sample pair relationship matrix based on real-time generated audio and text features is a key step in achieving distributed model deployment and dynamic updates. Edge devices perform incremental fine-tuning training locally to adapt to changes in specific device operating conditions and noise levels, improving recognition accuracy and latency response capabilities. The construction process of the real-time relationship matrix is ​​consistent with that in the cloud, but its samples originate from inputs collected during actual operation.

[0086] Dynamically adjusting the sampling weights of positive and negative sample pairs based on equipment type and fault risk level is a differentiated optimization strategy tailored to the varying state sensitivities of different equipment types in industrial settings. Higher sampling weights can be assigned to high-risk or critical equipment, allowing for repeated learning during training to enhance the ability to identify severe anomalies. Weight adjustments can be parameterized based on indicators such as tag frequency, alarm history, and expert evaluation.

[0087] Periodically validating the model's ability to distinguish between positive and negative samples is a mechanism for continuously monitoring the effectiveness of the model after deployment. By sampling test sample pairs within a set period and calculating their classification boundary confidence, if the confidence level is lower than a preset threshold, it indicates that the model may have problems such as overfitting, failure, or concept drift. This triggers a cloud-based optimization process, which involves re-collecting samples for small-batch retraining or reconstructing the training set for parameter calibration.

[0088] Example Description: In a piece of equipment on a battery assembly line, the system pre-constructs a set of audio-text positive and negative sample pair relationship matrices using manually labeled historical data, and performs shared-space pre-training on a cloud server to generate an initial model. This model is then deployed to edge devices in the field. During equipment operation, when a new audio signal is acquired and a corresponding text description is generated, the system constructs real-time sample pairs, performs edge fine-tuning, and automatically determines that the sample belongs to a novel fault not previously encountered. Because the model has previously optimized its ability to distinguish low-similarity negative samples through temperature adjustment parameters, it can accurately distinguish this sample from the normal state. During periodic validation, a decrease in confidence is detected, and the system automatically sends the sample back to the cloud, triggering model retraining to improve the ability to identify new faults.

[0089] This embodiment employs a contrastive learning strategy to jointly train audio and text features, constructing an alignable shared embedding space model. This enables the system to maintain strong discriminative capabilities even in the absence of sufficient annotations. Through a combination of cloud-based pre-training and edge incremental fine-tuning, the model possesses both global generalization and local adaptability. Dynamic sampling and periodic verification mechanisms further enhance the model's sensitivity to critical faults and long-term stability, achieving a highly efficient closed-loop end-cloud collaborative approach for cross-modal state recognition.

[0090] In one embodiment, step S40 above includes: S401 acquires the operating audio signal of the monitored device through a microphone array and generates a raw time-domain audio stream; S402, Perform a short-time Fourier transform on the original time-domain audio signal to generate a time-frequency spectrum matrix; S403, Multi-scale feature extraction is performed on the time-frequency spectrum matrix through a multi-branch convolutional neural network to generate a multi-scale feature spectrum containing features of different resolutions; S404, a gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map, and the multi-scale feature map is weighted by Hadamard product to generate a weighted feature map. S405, perform temporal anti-interference training on the weighted feature map to generate anti-interference enhanced features; S406, the anti-interference enhancement features of the multi-level output are fused across layers through residual connection to generate the real-time audio features.

[0091] In this embodiment, a microphone array is used to capture the raw acoustic information of the monitored device during operation. Microphone arrays offer higher signal-to-noise ratios and spatial resolution than a single microphone, making them suitable for complex industrial environments with aliasing and significant background interference. The raw audio data is recorded as a time-domain waveform, forming a raw time-domain audio stream. To extract frequency-domain information suitable for model processing, the raw time-domain audio signal is fed into a short-time Fourier transform module, converting the audio sequence into a time-frequency spectrum matrix that reflects the changes in different frequency bands along the time axis, facilitating subsequent frequency-domain feature analysis.

[0092] The generated time-frequency map matrix is ​​then input into a convolutional neural network with parallel branches. Each branch is equipped with convolutional kernels of different sizes (e.g., 3×3, 5×5, 7×7, etc.) to perceive high-frequency local texture, structural mid-frequency variations, and low-frequency global trends, respectively, forming a multi-scale feature map. To enhance the dynamic ability of inter-channel information representation, a gating mechanism is introduced, generating a separate gating weight vector for each channel. The gating weights can be constructed based on a combination of global average pooling and nonlinear activation to reflect the saliency of each channel. The gating weights are then weighted channel-wise using Hadamard product to generate a weighted feature map, thereby suppressing invalid channels and strengthening the representation of key frequency bands.

[0093] Furthermore, to improve the model's robustness to on-site interference noise in actual deployment, temporal anti-interference training is performed on the weighted feature map. Anti-interference training can be conducted through simulated noise injection, background mixing perturbations, or random temporal occlusion, enabling the model to adapt to common industrial background signals such as motor noise, airflow noise, and operational collision sounds. The anti-interference enhancement features are then fused with the previous layer's output through a residual connection mechanism. The residual structure effectively mitigates gradient vanishing and preserves semantic information at different scales. The final fused output is the real-time audio feature, whose expressive power covers both local and global signal variation patterns, while possessing a certain degree of noise resistance and adaptability to diverse environments.

[0094] This embodiment enables efficient structured representation of audio signals in production scenarios with multiple interferences and diverse devices. Multi-scale perception enhances the model's sensitivity to different acoustic modes, the gated modulation mechanism significantly improves the response capability of key sound sources, anti-interference training improves the model's stability under non-ideal acoustic conditions, and the residual fusion mechanism further enhances the ability to integrate multi-level information, thereby generating real-time audio features with high resolution and robustness.

[0095] In one embodiment, step S50 above includes: S501, Load a preset text description information library. Each set of candidate text features in the preset text description information library is generated by an adaptive transformer encoding and bound to a predefined device status label. S502, calculate the cosine similarity score between the real-time audio features and all candidate text features using the shared embedding space model, and generate a similarity matching result matrix; S503, Based on the similarity matching result matrix, filter the candidate text features with the highest similarity score to the real-time audio features, and extract the corresponding device status labels; S504, Based on a predefined fault decision rule base, verify the confidence level of the device status label. If the confidence level exceeds a preset confidence threshold, output the device status; otherwise, trigger the review process. S505, the device status tag is bound to the monitoring timestamp to generate a historical record of device operating status and update it to the cloud database.

[0096] In this embodiment, the operational status identification process is based on a shared embedding space model. First, a pre-built text description information library needs to be loaded. Each candidate description in this library represents a typical equipment operational status, such as "abnormal bearing noise" or "abnormal motor current." The text is encoded using an adaptive transformer to convert it into a structured candidate text feature vector. These feature vectors establish a one-to-one correspondence with the actual equipment status and are identified by equipment status labels, forming a candidate set that structurally supports indexing and semantic mapping.

[0097] During the recognition phase, the audio features generated by the real-time audio feature extraction module at the current moment are input into the shared embedding space model and matched with each feature in the candidate text feature set for similarity. This similarity matching calculation is typically based on the cosine similarity measure in the vector space, generating a two-dimensional similarity matrix, where each element represents the semantic similarity score between the audio feature and a candidate text feature. By traversing the matching result matrix, the system selects the candidate with the highest score and extracts its associated device status label as a preliminary judgment of the operating status at the current monitoring moment.

[0098] To improve the reliability of status recognition, the system introduces a confidence level verification mechanism. This mechanism, based on a predefined fault decision rule base, combines the similarity score of the current candidate label, the historical confidence distribution of the label, and contextual environmental factors to comprehensively evaluate the confidence level of the current status determination. If the confidence level is higher than a set threshold (e.g., ≥0.85), the status label will be confirmed and output; otherwise, the system enters a manual review process, awaiting confirmation, labeling, or intervention from operations and maintenance personnel.

[0099] After the status label is confirmed, the system binds it to the current timestamp to form a complete device operating status record. This record includes fields such as status type, time, similarity score, and confidence level. All records are uploaded and stored in a cloud database for subsequent querying, statistical analysis, and model training updates.

[0100] The construction of the pre-defined text description information base can be combined with equipment operation and maintenance manuals, on-site log texts, and experience documents. High-frequency state phrases are extracted through standardized language processing, and then manually reviewed as candidate entries. Each description is encoded into an embedding vector using an adaptive transformer model. During its training, task alignment loss can be introduced to enhance its comparability with audio features in the shared space. Audio-text similarity calculation can use a standard cosine similarity function, achieving fast matrix multiplication acceleration while maintaining uniform vector dimension processing. The decision rule base can be designed as a threshold-based multi-level confidence filter, considering both the frequency and stability of candidate labels within historical time windows, and dynamically adjusting its output weights. The cloud database can adopt a time-series database architecture, such as InfluxDB or TimescaleDB, to support high-concurrency state writing and real-time analysis.

[0101] This embodiment implements an unsupervised semantic matching-driven acoustic state recognition mechanism. It achieves efficient alignment and judgment through shared representations of text and audio features without requiring the manual construction of a specific classification model. The labeling management of candidate states reduces model deployment complexity, the confidence rule verification mechanism effectively prevents false alarms from low-confidence recognition results, and the binding of states with timestamps forms a complete traceability record, which is helpful for subsequent equipment health assessments and maintenance decisions.

[0102] In one embodiment, after step S50 above, the method further includes: S601, Based on the device status labels and corresponding monitoring timestamps in the operating state, generate a visual classification result of device status through a predefined fault level mapping rule library; S602, In the main monitoring panel of the interactive interface, the device status icon is rendered according to the visualization classification result, and the monitoring timestamp is overlaid and displayed; S603, if the device status is fault or warning status, trigger the audible and visual alarm module, match the corresponding alarm tone and flashing frequency according to the fault level mapping rule library, and generate an alarm event log; S604, the alarm event log is bound to the device operation status history record, stored in the cloud database, and time interval filtering and multi-dimensional statistical chart display are provided in the history query module of the interactive interface; S605 receives user feedback input on alarm events or device status labels, generates a labeled dataset and sends it back to the cloud database to update the parameters of the shared embedded space model.

[0103] In this embodiment, after completing equipment status identification, the system further performs status visualization classification and outputs the results. Equipment status tags and their associated monitoring timestamps constitute the information entry point for this stage. Using a predefined fault level mapping rule base, the system performs level parsing on the status tags, typically mapping them to several visualization levels, such as normal, warning, and fault. This mapping rule base can be built based on historical maintenance experience, manufacturer specifications, or industry standards, with different status tags categorized into different response level visualization classifications.

[0104] The visual classification results will be used for interface display. The system's main monitoring panel will call the graphics rendering module to load the corresponding status icon styles based on the mapping results. For example, a green icon indicates that the device is in normal condition, a yellow icon is used to indicate minor anomalies or potential risks, and a red icon is used to identify significant faults. In addition, a timestamp will be overlaid on the status icons, allowing users to accurately know when the anomaly occurred and improving the timeliness of fault response.

[0105] When a device's status is determined to be faulty or at a warning level, the system will automatically activate the audible and visual alarm module. The alarm module will determine the specific alarm tone frequency and flashing light cycle based on the alarm parameters retrieved from the fault level mapping rule base. For example, a severe fault may trigger a high-frequency continuous beep and rapid red light flashing, while minor anomalies can be configured with intermittent beeping sounds and orange light alerts. An alarm event log will be generated simultaneously upon its occurrence, containing information such as the alarm level, device number, timestamp, alarm parameters, and status label.

[0106] Alarm event logs will be linked to previously generated historical records of device operating status and stored uniformly in a cloud database, forming a queryable and traceable data chain. The system's historical query module supports filtering based on time intervals, allowing users to specify time periods to view alarm and status data. Multi-dimensional statistical charts, such as alarm trend charts, fault level distribution charts, and time comparison bar charts, are generated on the front-end interface to assist maintenance personnel in conducting analysis.

[0107] Users can provide feedback on alarm events or device status labels in the front-end interface, such as confirming whether it is a false alarm, whether it has been handled, or adding notes. This feedback information will be structured and packaged into a labeled dataset and uploaded to a cloud database. The labeled data will participate as incremental samples in the subsequent update process of the shared embedding space model to optimize model parameters and improve the system's robustness in recognizing boundary states, fuzzy labels, or atypical samples.

[0108] In implementation, the fault level mapping rule base can use JSON or a database structure to record the mapping relationship between each status label and alarm level, with parameter fields to configure alarm signal details. Icon rendering can be based on web front-end SVG or Canvas graphics processing components, supporting dynamic status change display. The audible and visual alarm module can control buzzers and LED indicator lights to execute alarm logic via an industrial control board (such as a PLC), and parameter adjustments can be configured through the interface or remotely issued from the rule base. Alarm logs are recorded in a structured log format (such as JSON), with timestamps as the primary key to associate with status history. The query module can provide fast retrieval based on full-text indexing engines such as Elasticsearch, and combine with chart components such as ECharts to complete visualization. User feedback input is connected to the backend system through an access control interface, automatically organized into labeled samples, with the label type aligned with the model training structure for easy retraining.

[0109] Example Description: Industrial-grade microphone arrays are deployed in the body-in-white stamping workshop of new energy vehicles and the automated assembly line of power batteries to continuously collect acoustic data from key equipment such as stamping machines and battery module welding stations. The audio sampling signals are converted into time-frequency spectra through short-time Fourier transform and then input into a multi-branch convolutional network with 3×3, 5×5, and 7×7 kernels to extract multi-scale feature maps. To enhance anti-interference capabilities, gating units are introduced to generate weight vectors, and channel weighting is performed through Hadamard product. Temporal perturbation simulation training is then conducted to improve the system's robustness to features under high-noise backgrounds. Multi-level features are fused in the residual structure to finally extract real-time audio features.

[0110] On the text side, the text information database maintained by operations engineers contains natural language descriptions related to the status of various equipment, such as "insufficient lubrication of the press slide, resulting in a harsh metallic friction sound." The system denoises the text through hierarchical lexical filtering and an industrial-domain dictionary, encodes it using an adaptive transformer, generates corresponding embedding vectors, and uniformly maps them to a shared embedding space dimension consistent with the audio features. To optimize the training of this space, the cloud server first constructs a positive and negative sample pair relationship matrix using historical audio features and text features. Positive sample pairs are speech and text features of the same fault, while negative sample pairs are irrelevant state pairs. A contrastive loss function including temperature parameters is used for model optimization training, and the results are deployed to edge devices.

[0111] During actual operation, the system continuously extracts real-time audio features and calculates a similarity score with each set of candidate text features in a preset text information database. Taking poor lubrication of the press slide as an example, when the similarity is higher than the 0.75 threshold, it is determined that the current state is highly consistent with the description, and the state label and matching confidence score are returned. If the score is in a fuzzy range (such as 0.6-0.75), it is sent to the cloud for fine-grained review.

[0112] The identified status labels are bound to the current timestamp to form a historical record of operational status and synchronized to the cloud database. Simultaneously, the system visualizes the status classification results on the main monitoring interface using fault level mapping rules. Normal / warning / fault statuses are dynamically rendered with green / yellow / red icons respectively, and in severe states, audible and visual alarms and log recording mechanisms are automatically triggered. Maintenance personnel can receive status push notifications via mobile devices and provide feedback on the interface, such as modifying labels or correcting misclassifications. The generated labeled data will be incorporated into the subsequent fine-tuning training of the shared embedding space model, enabling continuous evolution.

[0113] This embodiment achieves rapid visual judgment of abnormal states by mapping status labels to fault levels; the audio-visual linkage mechanism improves the response efficiency to critical faults; the log and time binding mechanism enhances the traceability and structured management capabilities of status data; and the user feedback collection mechanism provides a data source for sustainable optimization of the model, enabling the system to continuously learn and evolve. The overall design significantly enhances the monitoring system's response speed, data transparency, and intelligence level.

[0114] In one embodiment, a state monitoring device based on multimodal feature fusion is provided, which corresponds one-to-one with the state monitoring method based on multimodal feature fusion described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the state monitoring device based on multimodal feature fusion of the present invention. The modules include an audio feature extraction module 10, a text semantic encoding module 20, a contrastive learning training module 30, a real-time audio feature extraction module 40, and a state matching and recognition module 50. Detailed descriptions of each functional module are as follows: The audio feature extraction module 10 is used to acquire the original audio signal and perform multi-scale convolution and gating processing on the original audio signal to generate audio features; The text semantic encoding module 20 is used to acquire text description information of the device's operating status and perform adaptive transformer encoding processing on the text description information to generate text features; The contrastive learning training module 30 is used to jointly train the audio features and the text features using a contrastive learning strategy to generate a shared embedding space model. The real-time audio feature extraction module 40 is used to acquire the audio signal to be monitored from the monitored device and extract real-time audio features from the audio signal to be monitored. The status matching and recognition module 50 is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library through the shared embedding space model, and determine the operating status of the monitored device based on the similarity matching results.

[0115] In one embodiment, the audio feature extraction module 10 is specifically used for: Acquire the raw audio signal from the target device; The original audio signal is subjected to data enhancement, which includes adding random time shifts and injecting mixed noise; Add adversarial perturbations to the original audio signal after data augmentation to generate the original audio signal after anti-interference training; The original audio signal after anti-interference training is input into a parallel multi-scale convolutional neural network, and multi-scale feature maps are generated through convolutional kernels of different sizes. A gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map. The gated weight vector is weighted channel-by-channel by the corresponding multi-scale feature map through the Hadamard product to generate a weighted feature map. The audio features are generated by fusing the weighted feature maps from multiple levels through residual connections.

[0116] In one embodiment, the text semantic encoding module 20 is specifically used for: Receive natural language text containing a description of the device's operating status; The natural language text is subjected to hierarchical lexical filtering, and non-critical semantic components are filtered based on a predefined industrial domain dictionary to generate filtered text description information; The filtered text description information is semantically encoded using the multi-head self-attention mechanism of the adaptive transformer. If a preset high-priority fault keyword is detected in the filtered text description information, the pulse-type attention enhancement module of the adaptive transformer is triggered to strengthen the semantic weight of the word unit corresponding to the high-priority fault keyword and generate a primary semantic vector. The text features are generated by mapping the primary semantic vector to a shared embedding space consistent with the audio feature dimension using the learnable projection matrix of the adaptive transformer.

[0117] In one embodiment, the contrastive learning training module 30 is specifically used for: The shared embedding space model is pre-trained on a cloud server using historical audio features and corresponding historical text features. The historical audio features are generated by multi-scale convolution and gated weighting of the original historical audio signal, and the historical text features are generated by adaptive transformer encoding of historical text descriptions to obtain initial model parameters. Based on the audio features and the text features, a matching audio-text positive sample pair relationship matrix and a non-matching audio-text negative sample pair relationship matrix are constructed, wherein positive sample pairs represent audio features and text features of the same device state, and negative sample pairs represent audio features and text features of different device states. Based on the positive sample pair relationship matrix and the negative sample pair relationship matrix, calculate the cosine similarity score of the positive sample pair and the negative sample pair; The cosine similarity score is optimized using a contrastive loss function that includes a temperature adjustment parameter. By adjusting the temperature adjustment parameter, the model’s attention to low-similarity negative samples is controlled, thereby generating a shared embedding space model. The initial model parameters are sent to the edge computing device, and a real-time positive and negative sample pair relationship matrix is ​​constructed based on the real-time generated audio features and text features. The shared embedding space model is then incrementally fine-tuned and trained. The sampling weights of positive and negative sample pairs are dynamically adjusted according to the equipment type and the level of failure risk, and the sample pairs of key failure states are trained with weighted weights. The ability of the shared embedding space model to distinguish between positive and negative samples is periodically verified. When the discrimination is detected to be lower than the threshold, cloud parameter optimization is triggered.

[0118] In one embodiment, the real-time audio feature extraction module 40 is specifically used for: The operating audio signals of the monitored equipment are acquired using a microphone array to generate a raw time-domain audio stream. Perform a short-time Fourier transform on the original time-domain audio signal to generate a time-frequency spectrum matrix; Multi-scale feature extraction is performed on the time-frequency spectrum matrix using a multi-branch convolutional neural network to generate a multi-scale feature spectrum containing features at different resolutions. A gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map, and the multi-scale feature map is weighted by Hadamard product to generate a weighted feature map. The weighted feature map is subjected to temporal anti-interference training to generate anti-interference enhanced features; The anti-interference enhancement features of the multi-level outputs are fused across layers through residual connections to generate the real-time audio features.

[0119] In one embodiment, the state matching and recognition module 50 is specifically used for: A preset text description information library is loaded. Each set of candidate text features in the preset text description information library is generated by encoding through an adaptive transformer and bound to a predefined device status label. The cosine similarity score between the real-time audio features and all candidate text features is calculated using the shared embedding space model to generate a similarity matching result matrix. Based on the similarity matching result matrix, candidate text features with the highest similarity scores to the real-time audio features are selected, and their corresponding device status labels are extracted. Based on a predefined fault decision rule base, the confidence level of the device status label is verified. If the confidence level exceeds a preset confidence threshold, the device status is output; otherwise, a review process is triggered. The device status tag is bound to the monitoring timestamp to generate a historical record of the device's operating status and update it to the cloud database.

[0120] In one embodiment, the state matching and recognition module 50 is specifically used for: Based on the device status labels and corresponding monitoring timestamps in the operating state, a visual classification result of the device status is generated through a predefined fault level mapping rule library; In the main monitoring panel of the interactive interface, device status icons are rendered based on the visualization classification results, and the monitoring timestamps are overlaid. If the device is in a fault or warning state, the audible and visual alarm module is triggered. According to the fault level mapping rule library, the corresponding alarm tone and flashing frequency are matched, and an alarm event log is generated. The alarm event logs are bound to the device operation status history records, stored in the cloud database, and time interval filtering and multi-dimensional statistical charts are provided in the history query module of the interactive interface; The system receives user feedback on alarm events or device status labels, generates a labeled dataset, and sends it back to the cloud database to update the parameters of the shared embedded space model.

[0121] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a state monitoring method based on multimodal feature fusion on the server side.

[0122] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a state monitoring method based on multimodal feature fusion on the user side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The original audio signal is acquired, and multi-scale convolution and gating processing is performed on the original audio signal to generate audio features; Obtain textual description information of the device's operating status, and perform adaptive transformer encoding on the textual description information to generate textual features; A contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model. Acquire the audio signal to be monitored from the monitored device, and extract real-time audio features from the audio signal to be monitored; The shared embedding space model is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library, and the operating status of the monitored device is determined based on the similarity matching results.

[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The original audio signal is acquired, and multi-scale convolution and gating processing is performed on the original audio signal to generate audio features; Obtain textual description information of the device's operating status, and perform adaptive transformer encoding on the textual description information to generate textual features; A contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model. Acquire the audio signal to be monitored from the monitored device, and extract real-time audio features from the audio signal to be monitored; The shared embedding space model is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library, and the operating status of the monitored device is determined based on the similarity matching results.

[0124] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0125] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0127] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A state monitoring method based on multimodal feature fusion, characterized in that, Includes the following steps: The original audio signal is acquired, and multi-scale convolution and gating processing is performed on the original audio signal to generate audio features; Obtain textual description information of the device's operating status, and perform adaptive transformer encoding on the textual description information to generate textual features; A contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model. Acquire the audio signal to be monitored from the monitored device, and extract real-time audio features from the audio signal to be monitored; The shared embedding space model is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library, and the operating status of the monitored device is determined based on the similarity matching results.

2. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, The original audio signal is acquired, and multi-scale convolution and gating processing is performed on the original audio signal to generate audio features, including: Acquire the raw audio signal from the target device; The original audio signal is subjected to data enhancement, which includes adding random time shifts and injecting mixed noise; Add adversarial perturbations to the original audio signal after data augmentation to generate the original audio signal after anti-interference training; The original audio signal after anti-interference training is input into a parallel multi-scale convolutional neural network, and multi-scale feature maps are generated through convolutional kernels of different sizes. A gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map. The gated weight vector is weighted channel-by-channel by the corresponding multi-scale feature map through the Hadamard product to generate a weighted feature map. The audio features are generated by fusing the weighted feature maps from multiple levels through residual connections.

3. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, Obtain textual description information of the device's operating status, and perform adaptive transformer encoding on the textual description information to generate textual features, including: Receive natural language text containing a description of the device's operating status; The natural language text is subjected to hierarchical lexical filtering, and non-critical semantic components are filtered based on a predefined industrial domain dictionary to generate filtered text description information; The filtered text description information is semantically encoded using the multi-head self-attention mechanism of the adaptive transformer. If a preset high-priority fault keyword is detected in the filtered text description information, the pulse-type attention enhancement module of the adaptive transformer is triggered to strengthen the semantic weight of the word unit corresponding to the high-priority fault keyword and generate a primary semantic vector. The text features are generated by mapping the primary semantic vector to a shared embedding space consistent with the audio feature dimension using the learnable projection matrix of the adaptive transformer.

4. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, A contrastive learning strategy is used to jointly train the audio features and the text features to generate a shared embedding space model, including: The shared embedding space model is pre-trained on a cloud server using historical audio features and corresponding historical text features. The historical audio features are generated by multi-scale convolution and gated weighting of the original historical audio signal, and the historical text features are generated by adaptive transformer encoding of historical text descriptions to obtain initial model parameters. Based on the audio features and the text features, a matching audio-text positive sample pair relationship matrix and a non-matching audio-text negative sample pair relationship matrix are constructed, wherein positive sample pairs represent audio features and text features of the same device state, and negative sample pairs represent audio features and text features of different device states. Based on the positive sample pair relationship matrix and the negative sample pair relationship matrix, calculate the cosine similarity score of the positive sample pair and the negative sample pair; The cosine similarity score is optimized using a contrastive loss function that includes a temperature adjustment parameter. By adjusting the temperature adjustment parameter, the model’s attention to low-similarity negative samples is controlled, thereby generating a shared embedding space model. The initial model parameters are sent to the edge computing device, and a real-time positive and negative sample pair relationship matrix is ​​constructed based on the real-time generated audio features and text features. The shared embedding space model is then incrementally fine-tuned and trained. The sampling weights of positive and negative sample pairs are dynamically adjusted according to the equipment type and the level of failure risk, and the sample pairs of key failure states are trained with weighted weights. The ability of the shared embedding space model to distinguish between positive and negative samples is periodically verified. When the discrimination is detected to be lower than the threshold, cloud parameter optimization is triggered.

5. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, Acquiring the audio signal to be monitored from the monitored device and extracting real-time audio features from the audio signal to be monitored, including: The operating audio signals of the monitored equipment are acquired using a microphone array to generate a raw time-domain audio stream. Perform a short-time Fourier transform on the original time-domain audio signal to generate a time-frequency spectrum matrix; Multi-scale feature extraction is performed on the time-frequency spectrum matrix using a multi-branch convolutional neural network to generate a multi-scale feature spectrum containing features at different resolutions. A gating unit is used to generate a gating weight vector for each channel of the multi-scale feature map, and the multi-scale feature map is weighted by Hadamard product to generate a weighted feature map. The weighted feature map is subjected to temporal anti-interference training to generate anti-interference enhanced features; The anti-interference enhancement features of the multi-level outputs are fused across layers through residual connections to generate the real-time audio features.

6. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, Using the shared embedding space model, the real-time audio features are matched with each group of candidate text features in a preset text description information database based on similarity. Based on the similarity matching results, the operating status of the monitored device is determined, including: A preset text description information library is loaded. Each set of candidate text features in the preset text description information library is generated by encoding through an adaptive transformer and bound to a predefined device status label. The cosine similarity score between the real-time audio features and all candidate text features is calculated using the shared embedding space model to generate a similarity matching result matrix. Based on the similarity matching result matrix, candidate text features with the highest similarity scores to the real-time audio features are selected, and their corresponding device status labels are extracted. Based on a predefined fault decision rule base, the confidence level of the device status label is verified. If the confidence level exceeds a preset confidence threshold, the device status is output; otherwise, a review process is triggered. The device status tag is bound to the monitoring timestamp to generate a historical record of the device's operating status and update it to the cloud database.

7. The state monitoring method based on multimodal feature fusion as described in claim 1, characterized in that, After performing similarity matching between the real-time audio features and each set of candidate text features in the preset text description information database using the shared embedding space model, and determining the operating status of the monitored device based on the similarity matching results, the process further includes: Based on the device status labels and corresponding monitoring timestamps in the operating state, a visual classification result of the device status is generated through a predefined fault level mapping rule library; In the main monitoring panel of the interactive interface, device status icons are rendered based on the visualization classification results, and the monitoring timestamps are overlaid. If the device is in a fault or warning state, the audible and visual alarm module is triggered. According to the fault level mapping rule library, the corresponding alarm tone and flashing frequency are matched, and an alarm event log is generated. The alarm event logs are bound to the device operation status history records, stored in the cloud database, and time interval filtering and multi-dimensional statistical charts are provided in the history query module of the interactive interface; The system receives user feedback on alarm events or device status labels, generates a labeled dataset, and sends it back to the cloud database to update the parameters of the shared embedded space model.

8. A state monitoring device based on multimodal feature fusion, characterized in that, The state monitoring device based on multimodal feature fusion includes: The audio feature extraction module is used to acquire the original audio signal and perform multi-scale convolution and gating processing on the original audio signal to generate audio features; The text semantic encoding module is used to obtain text description information of the device's operating status and perform adaptive transformer encoding processing on the text description information to generate text features; The contrastive learning training module is used to jointly train the audio features and the text features using a contrastive learning strategy to generate a shared embedding space model. The real-time audio feature extraction module is used to acquire the audio signal to be monitored from the monitored device and extract real-time audio features from the audio signal to be monitored. The status matching and recognition module is used to perform similarity matching between the real-time audio features and each set of candidate text features in the preset text description information library through the shared embedding space model, and determine the operating status of the monitored device based on the similarity matching results.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a state monitoring program based on multimodal feature fusion stored in the memory and executable on the processor. When executed by the processor, the state monitoring program based on multimodal feature fusion implements the steps of the state monitoring method based on multimodal feature fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a state monitoring program based on multimodal feature fusion, which, when executed by a processor, implements the steps of the state monitoring method based on multimodal feature fusion as described in any one of claims 1-7.

Images