Conference equipment state monitoring method and system based on multi-modal sensor

Abstract

The invention discloses a conference equipment state monitoring method and system based on a multi-modal sensor, and belongs to the technical field of equipment state monitoring. The method comprises the following steps: acquiring multi-modal sensor data of conference equipment, and generating a standardized sensor feature vector through time-frequency domain feature extraction; fusing by using a preset weighted fusion algorithm to generate a health feature vector set; processing through a pre-trained health assessment model to obtain health state information; and generating a health assessment report according to the pre-constructed knowledge graph. Through multi-modal data fusion and intelligent model analysis, conference equipment state monitoring and evaluation are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of equipment status monitoring, and in particular relates to a conference equipment status monitoring method and system based on a multimodal sensor. Background Art

[0002] With the advancement of intelligent and integrated conferencing equipment, multimodal sensors are increasingly being used in equipment condition monitoring. These sensors can simultaneously acquire multi-dimensional data, such as acoustics, vibration, temperature, and current, providing a rich information foundation for equipment health assessment. Traditional conferencing equipment condition monitoring methods often rely on independent analysis of single-modal sensor data or simple fusion of multimodal data using fixed weights. For example, they rely solely on vibration sensors to detect abnormal operating noises or identify anomalies in a single current parameter based on a preset threshold. These methods lack the ability to extract and dynamically fusion multiple data sources in the time and frequency domains. Existing technologies suffer from the following significant shortcomings: Single-modal data, with its limited monitoring dimension, fails to fully characterize the complex operating conditions of equipment and is prone to missing complex fault signatures; traditional feature extraction methods fail to fully exploit the coupled relationships between data in the time and frequency domains, resulting in the loss of critical fault information; fixed-weight fusion algorithms cannot adapt to changing operating conditions during equipment operation, resulting in insufficient fusion accuracy; and health assessment models lack in-depth analysis of the dynamic characteristics of time series, making it difficult to accurately predict the remaining useful life. This, in turn, leads to delayed maintenance decisions and untimely fault warnings, increasing the risk of equipment downtime and operational management costs. Summary of the Invention

[0003] Based on this, it is necessary to provide a conference equipment status monitoring method and system based on multimodal sensors that can solve the above problems.

[0004] In a first aspect, the present application provides a conference device status monitoring method based on a multimodal sensor, comprising:

[0005] Acquire multimodal sensor data from conference equipment, perform time-frequency domain feature extraction, and generate standardized sensor feature vectors;

[0006] The standardized sensor feature vectors are fused using a preset weighted fusion algorithm to generate a healthy feature vector set;

[0007] Use the pre-trained health assessment model to process the health feature vector set and generate health status information;

[0008] Based on health status information, a health assessment report is generated using a pre-built knowledge graph.

[0009] In one embodiment, performing time-frequency domain feature extraction on multimodal sensor data to generate a standardized sensor feature vector includes:

[0010] Through the hardware clock algorithm, the time sequence of each sensor data in the multimodal sensor data is aligned to obtain the accurate sensor data;

[0011] Using the short-time Fourier transform algorithm, the time-frequency domain joint feature extraction of the time-accurate sensor data is performed to generate a multi-scale time-frequency feature matrix;

[0012] The multi-scale time-frequency feature matrix is ​​subjected to principal component analysis and dimensionality reduction to obtain the standardized sensor feature vector.

[0013] In one embodiment, a preset weighted fusion algorithm is used to fuse the standardized sensor feature vectors to generate a health feature vector set, including:

[0014] Determine the initial weight coefficient of each sensor based on the time-frequency characteristics of the standardized sensor feature vector;

[0015] The confidence level of each sensor feature is calculated through a sliding window mechanism, and the initial weight coefficient is dynamically adjusted based on the confidence level to generate an optimized weight matrix.

[0016] The weight matrix is ​​used to perform weighted fusion on the standardized sensor feature vectors to generate a healthy feature vector. In one embodiment, the preset weighted fusion algorithm is constructed by the following formula:

[0017] F=∑ k∈{a，v，t，c} W k ·F k

[0018]

[0019] Among them, F represents the fused feature vector, F a represents the acoustic eigenvector, F v represents the vibration eigenvector, F t represents the temperature eigenvector, F c represents the current eigenvector, W k Represents the weight coefficient matrix of each mode, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d k Indicates the dimension of the key vector.

[0020] In one embodiment, a pre-trained health assessment model is used to process a health feature vector set to generate health status information, including:

[0021] Using the pre-trained improved Transformer model, a preliminary diagnostic analysis of the health feature vector is performed to obtain preliminary fault classification results;

[0022] Using a pre-trained LSTM network model, the dynamic difference between the time-varying characteristics of the health feature vector set and the preset benchmark value is calculated to generate the remaining service life prediction value;

[0023] The classification results and the remaining useful life prediction values ​​are combined to generate health status information.

[0024] In one embodiment, after generating the health status information, the following steps are included:

[0025] Based on the health status information, the health degree is calculated. If the health degree is less than 85%, a maintenance work order is generated. The maintenance work order contains the identification of the conference equipment, the fault type and the recommended maintenance measures, and is sent to the designated terminal via the network.

[0026] In one embodiment, obtaining multimodal sensor data of a conference device includes:

[0027] Acquire acoustic sensor data from conference equipment. The acoustic sensor's pickup aperture is ≤3mm and its frequency response range is 20Hz-20kHz.

[0028] Acquire the vibration sensor data of the conference equipment. The vibration sensor is a MEMS three-axis accelerometer.

[0029] Obtain infrared thermal imager data from conference equipment. The infrared thermal imager has a resolution of 160×120 and a temperature measurement accuracy of ±1.5°C.

[0030] Acquire the current sensor data of the conference equipment. The current sensor is a Hall effect sensor with a sampling rate of 1kHz.

[0031] In a second aspect, the present application also provides a conference equipment status monitoring system based on a multimodal sensor, comprising:

[0032] The data acquisition and processing module is used to obtain multimodal sensor data from conference equipment, perform time-frequency domain feature extraction, and generate standardized sensor feature vectors;

[0033] A feature weighted fusion module is used to fuse the standardized sensor feature vectors using a preset weighted fusion algorithm to generate a health feature vector set;

[0034] A health status assessment module is used to process a health feature vector set using a pre-trained health assessment model to generate health status information;

[0035] The health report generation module is used to generate a health assessment report based on health status information using a pre-built knowledge graph.

[0036] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned method for monitoring the status of conference equipment based on a multimodal sensor are implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for monitoring the status of conference equipment based on a multimodal sensor.

[0038] The above-mentioned conference equipment status monitoring method and system based on multimodal sensors, computer equipment and storage medium are implemented. By obtaining multimodal sensor data of conference equipment and performing time-frequency domain feature extraction, the limitations of the single dimension of traditional single-modal monitoring are broken through, and the multi-dimensional characteristics of the equipment operation status are fully captured; the standardized feature vectors are dynamically fused using a preset weighted fusion algorithm to solve the problem that traditional fixed weights cannot adapt to changes in working conditions and improve the accuracy of multi-source data fusion; the health feature vector set is processed with the help of a pre-trained health assessment model to achieve in-depth analysis of the equipment status and remaining life prediction, making up for the lack of time series dynamic feature analysis in existing models; health assessment reports are generated based on knowledge graphs to achieve accurate fault warning and maintenance decision support, solving the technical problems of fault information loss, warning lag and high operation and maintenance costs in traditional methods, and improving the comprehensiveness, accuracy and intelligence of equipment status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a conference equipment status monitoring method based on a multimodal sensor according to the present invention;

[0041] Figure 2 This is a structural diagram of a conference equipment status monitoring system based on multimodal sensors of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The implementation environment of this invention includes multimodal sensors and a computing terminal or server. When conference equipment requires real-time status monitoring to prevent interruptions, the multimodal sensors synchronously collect data and transmit it to the computing terminal or server. After time alignment, feature extraction, and weighted fusion, a health assessment model generates status information. This information is then generated through a knowledge graph to generate assessment reports and maintenance work orders. Interaction between the hardware components is achieved through a network.

[0044] In one embodiment, Figure 1 As shown, a method for monitoring the status of conference equipment based on multimodal sensors is provided. This embodiment uses the method applied to a computing terminal as an example. It is understood that the method can also be applied to a server, or to an architecture including a computing terminal and a server, and implemented through the interaction between the computing terminal and the server. In this embodiment, the method includes the following steps:

[0045] S11, obtaining multimodal sensor data of the conference equipment, and performing time-frequency domain feature extraction to generate a standardized sensor feature vector.

[0046] Among them, multimodal sensor data may include a variety of sensors, such as acoustic sensors, accelerometers, infrared thermal imagers, current sensors, etc., to obtain multi-dimensional physical signals such as acoustics, vibration, temperature, and current when the conference equipment is running. For the collected sensor data, different time-frequency domain feature extraction methods (such as short-time Fourier transform) can be used according to its characteristics to extract the corresponding features and capture the transient characteristics and frequency domain distribution of the equipment's operating status. After standardization processing (such as time alignment and feature standardization), a standardized sensor feature vector is generated. Through multi-dimensional physical signal acquisition and time-frequency domain deep feature extraction, the limitations of traditional single-modal monitoring are broken through, laying a data foundation for the accurate assessment of equipment health status.

[0047] S12, using a preset weighted fusion algorithm to fuse the standardized sensor feature vectors to generate a health feature vector set.

[0048] The preset weighted fusion algorithm can include the following four steps: initial weight modeling based on the time-frequency domain distribution characteristics of the standardized sensor feature vector; dynamic calculation of the confidence level of the sensor feature vector (reflecting the reliability of each modal data under the current working conditions); weight optimization and adjustment of the sensor feature vector based on the confidence level (to achieve adaptive adjustment of the weight as the working conditions change); and weighted fusion of the adjusted feature vectors (to improve the comprehensiveness and accuracy of the equipment status representation) to generate a set of health feature vectors. The dynamic weight mechanism solves the problem that traditional fixed fusion algorithms cannot adapt to changes in working conditions, realizes intelligent coupling of multimodal features, and provides high-dimensional and highly reliable feature input for subsequent health assessments.

[0049] S13, using the pre-trained health assessment model to process the health feature vector set and generate health status information. Among them, the pre-trained health assessment model may include a fault classification model that performs cross-modal feature correlation analysis on the health feature vector set, captures the temporal dependency between acoustic, vibration and other features, and identifies coupled fault features such as mechanical abnormal noise and sudden change in vibration amplitude; a life prediction model that dynamically models the time-varying characteristics of the health feature vector, such as constructing a sliding time window, calculating the dynamic difference between the current feature vector and the normal operation benchmark value of the equipment (temperature gradient change rate, current harmonic distortion rate, etc.) to capture the feature degradation trend and generate the remaining service life prediction value. Through the dual-layer model architecture that integrates temporal dynamic analysis and intelligent diagnosis, the fault classification results and the remaining service life prediction value are integrated to generate multi-dimensional health status information, thereby realizing a multi-dimensional assessment of the health status of the equipment.

[0050] S14, based on the health status information, a health assessment report is generated using a pre-built knowledge graph.

[0051] Among them, the pre-built knowledge graph can use the triple (entity-relationship-attribute) model to construct a knowledge graph in the field of conference equipment, covering entities and relationships such as equipment components (such as microphones, projectors), failure modes (such as abnormal current, excessive vibration), causal relationships (such as bearing wear → increased vibration amplitude), and maintenance strategies (such as motor overheating → recommended dust cleaning and heat dissipation); integrating equipment manuals, historical failure cases, expert experience, and real-time monitoring data to form a dynamically updated knowledge network to address the limitations of traditional reports that rely on a single rule base. The fault types (such as motor bearing wear) and degradation indicators (such as remaining life of 72 hours) in the health status information are converted into node entities in the knowledge graph; through the causal relationship chain in the graph (such as abnormal vibration → bearing wear → recommended replacement), the root cause of the fault and the scope of impact are automatically derived, achieving in-depth analysis from phenomenon description to cause tracing. Based on the node attributes of the knowledge graph, a structured health assessment report is generated, which may include: fault overview, standard definitions of associated fault type nodes; cause analysis, traversing the causal chain in the graph to generate a multi-level cause tree; maintenance recommendations, matching specific measures (such as tool models and operating procedures) to maintenance strategy nodes; historical reference, retrieving similar fault case nodes and providing historical processing effect data; multimodal information fusion, embedding visual data such as time-frequency feature maps and infrared thermal images, and achieving intuitive mapping of features, faults, and measures through graph node association to improve report readability. Through the semantic reasoning capability of the knowledge graph, the assessment report can make faults explainable, causes traceable, and maintenance actionable, solving the problem of traditional reports relying on manual experience and lacking systematic analysis. Based on the historical case matching of the graph, data support is provided for maintenance decisions, reducing operation and maintenance costs and improving the level of intelligent equipment management.

[0052] The above-mentioned conference equipment status detection method based on multimodal sensors obtains multimodal sensor data from conference equipment and extracts time-frequency domain features to generate standardized vectors. It then uses a preset weighted fusion algorithm to dynamically fuse multi-source features to generate a health feature vector set. This is then processed with a pre-trained health assessment model to obtain health status information including fault classification and remaining life prediction. A structured assessment report is then generated based on a pre-built knowledge graph. Multimodal data fusion overcomes the limitations of traditional single-modal monitoring, which relies on a single dimension. Joint time-frequency domain feature extraction is used to mine data coupling relationships to avoid loss of key information. A dynamic weighted fusion algorithm is used to adapt to changes in equipment operating conditions and improve fusion accuracy. Classification and prediction models are combined to perform in-depth analysis of dynamic time series features to accurately predict remaining service life. Knowledge graph reasoning is used to generate explainable maintenance recommendations. This addresses the existing issues of missing complex fault features, insufficient fusion accuracy, delayed warnings, and high operation and maintenance costs, effectively improving the comprehensiveness, accuracy, and intelligence of equipment status monitoring.

[0053] In one embodiment, performing time-frequency domain feature extraction on multimodal sensor data to generate a standardized sensor feature vector includes:

[0054] S21, using a hardware clock algorithm, timing-aligning the sensor data in the multimodal sensor data to obtain accurate sensor data;

[0055] S22, using the short-time Fourier transform algorithm, performs joint time-frequency domain feature extraction on the time-accurate sensor data to generate a multi-scale time-frequency feature matrix;

[0056] S23, performing principal component analysis and dimensionality reduction processing on the multi-scale time-frequency feature matrix to obtain a standardized sensor feature vector.

[0057] Specifically, a hardware clock algorithm is used to timestamp and calibrate asynchronously sampled data from acoustic and vibration sensors, eliminating clock offsets and constructing a time-consistent, accurate data sequence, addressing the timing misalignment problem of multimodal signals. Based on the short-time Fourier transform (STFT), a Hamming window function can be used to convert accurate data into a multi-scale time-frequency feature matrix, simultaneously capturing both time-domain transient features (such as impulse response) and frequency-domain steady-state features (such as harmonic distribution). Principal component analysis (PCA) is used to compress the time-frequency matrix, retaining principal components with a cumulative contribution rate of ≥95%, reducing the original dimensional feature vector to ≤256 dimensions. Z-score normalization is then performed to ensure that the eigenvalue distribution satisfies the standard normal distribution of μ = 0 and σ = 1, forming a standardized feature vector with uniform dimensions and consistent dimensions. By combining hardware synchronization with time-frequency analysis, the timing mismatch problem in multimodal data fusion is effectively addressed, enabling efficient characterization and compression of device status features, and providing high-dimensional, orthogonal feature input for subsequent fault diagnosis.

[0058] In one embodiment, a preset weighted fusion algorithm is used to fuse the standardized sensor feature vectors to generate a health feature vector set, including:

[0059] S31, determining an initial weight coefficient of each sensor based on the time-frequency characteristics of the normalized sensor feature vector;

[0060] S32, calculating the confidence level of each sensor feature through a sliding window mechanism, and dynamically adjusting the initial weight coefficient based on the confidence level to generate an optimized weight matrix;

[0061] S33, performing weighted fusion on the standardized sensor feature vectors using a weight matrix to generate a health feature vector.

[0062] For example, according to the distribution characteristics of the normalized sensor feature vector in the time and frequency dimensions (such as the Mel-frequency cepstral coefficient distribution of the acoustic feature, the wavelet packet energy entropy of the vibration feature, etc.), the formula Determine the weight coefficient of each sensor data during initial fusion, where Var(F j ) is the variance of the i-th sensor feature, which quantifies the characterization ability of each modality on the equipment status; a sliding window technique (window length T = 500ms, sliding step T = 500ms) is used to perform time series analysis on the characteristics of each sensor, and a real-time confidence index is generated by calculating the degree of deviation between the current feature and the historical normal operation data. Based on the confidence, the initial weight coefficient is dynamically corrected to form an optimized weight matrix that adapts to the current working conditions; the optimized weight matrix is ​​used to perform weighted fusion of multi-modal standardized feature vectors such as acoustics, vibration, temperature, and current, and the complementary information in the sensor data is integrated into a comprehensive health feature vector, which solves the problem that the traditional fixed weight fusion algorithm cannot adapt to changes in equipment operating conditions and improves the accuracy of multi-source data fusion and equipment status characterization.

[0063] In one embodiment, in S41, the preset weighted fusion algorithm is constructed by the following formula:

[0064] F=∑ k∈{a，v，t，c} W k ·F k

[0065]

[0066] Among them, F represents the fused feature vector, F a represents the acoustic eigenvector, F v represents the vibration eigenvector, F t represents the temperature eigenvector, F c represents the current eigenvector, W kRepresents the weight coefficient matrix of each mode, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d k Indicates the dimension of the key vector.

[0067] Specifically, F k Represent acoustic (F a )、Vibration(F v ), temperature (F t ) and current (F c ) The standardized feature vectors of the four modes are generated by time-frequency domain feature extraction and contain multi-dimensional physical signal characteristics of the equipment operation status. The fused feature vector F integrates multi-modal information through weighted summation to fully characterize the health status of the equipment. Q, K, and V are the query matrix, key matrix, and value matrix respectively, which are based on the Transformer attention mechanism. Among them, Q is used to retrieve the correlation of each modal feature, K corresponds to the key matrix of each modality, and V is the eigenvalue matrix. The dimension of the key vector d k Used to scale the matrix dot product results to avoid the gradient vanishing problem and ensure the stability of weight calculation. Through the Softmax function, the dot product results of the query matrix and the key matrix are normalized into a probability distribution to generate the weight coefficient matrix W of each mode. k , so that the weights have dynamic adaptive characteristics. Through the weighted summation formula, the characteristic vectors of different physical quantities such as acoustics and vibration are nonlinearly fused, breaking through the dimensional limitations of single-modal monitoring and capturing the characteristics of equipment complex faults (such as abnormal vibration accompanied by current fluctuations). The weight calculation formula based on the attention mechanism enables the algorithm to automatically adjust the weights of each mode according to the real-time working conditions: when a certain modal data (such as vibration) has a higher correlation with the fault, the corresponding W k The weight is increased to strengthen the contribution of this feature to health assessment; combined with the feature confidence calculated by the sliding window mechanism, the weight matrix is ​​further optimized to adapt to the operating condition fluctuations during equipment operation.

[0068] In one embodiment, a pre-trained health assessment model is used to process a health feature vector set to generate health status information, including:

[0069] S51, using the pre-trained improved Transformer model, performs preliminary diagnostic analysis on the health feature vector to obtain preliminary fault classification results;

[0070] S52, using a pre-trained LSTM network model, calculates the dynamic difference between the time-varying characteristics of the health feature vector set and the preset benchmark value to generate a remaining service life prediction value;

[0071] S53, generating health status information by combining the classification result and the remaining service life prediction value.

[0072] For example, a pre-trained improved Transformer model uses a self-attention mechanism to capture long-range dependencies between multimodal features (such as the correlation between acoustic anomalies and vibration peaks). Based on the health feature vector, a preliminary classification of possible fault types for conference equipment is performed, resulting in preliminary fault classification results. A pre-trained LSTM network model is used to calculate the dynamic difference between the health feature vector (such as the temperature rise rate and current fluctuation amplitude) and a preset baseline value (the feature threshold under normal operating conditions). The LSTM's gating mechanism memorizes historical patterns of device performance degradation and predicts the remaining service life. The preliminary fault classification results (such as vibration anomalies and suspected fan imbalance) are weighted and fused with the remaining service life prediction value (such as a remaining service life of 300 hours) to form a comprehensive assessment that includes fault type, severity, and maintenance urgency. By synergizing the Transformer's global feature association capabilities with the LSTM's time-series dynamic analysis capabilities, this approach addresses the problems of missed fault features and delayed life prediction in existing technologies, enabling an upgrade from post-diagnosis to pre-emptive warning, significantly enhancing the intelligent level of equipment condition monitoring.

[0073] In one embodiment, after generating the health status information, the following steps are included:

[0074] S61, based on the health status information, calculate the health level. If the health level is less than 85%, generate a maintenance work order. The maintenance work order includes the identification of the conference device, the fault type and the recommended maintenance measures, and sends it to the designated terminal via the network.

[0075] Specifically, a comprehensive assessment model is constructed based on health status information (including fault classification results and remaining useful life predictions) to calculate equipment health indicators. The health calculation integrates multiple parameters, including fault severity, degradation rate, and operating conditions (such as vibration abnormality level and temperature deviation from baseline values), to form a quantitative score ranging from 0 to 100%. This multi-parameter fusion enables precise quantification of health status, providing data support for maintenance decisions. Based on a preset health threshold (85%), when the calculated result falls below this threshold, a maintenance work order is triggered. The work order is logically linked to a pre-built knowledge graph and matched to the corresponding maintenance strategy (such as hardware replacement or parameter adjustment) based on the fault type. This threshold-based automatic triggering mechanism avoids the lag of manual inspections and enables the transition from predictive maintenance to proactive maintenance. A maintenance work order can include three core elements: equipment identification, a unique code (such as MAC address or asset number) to precisely locate the faulty device; fault type: a diagnosis based on the improved Transformer (e.g., current anomaly - power module failure); and recommended maintenance action: a standardized solution linked to the knowledge graph (e.g., replacing capacitor model XXX). Achieve structuring and standardization, eliminate ambiguity in manual records, and improve operation and maintenance efficiency. Work orders can be encrypted and transmitted to designated terminals (such as the operator's mobile app or management backend) via network protocols such as TCP / IP, supporting real-time push notifications and historical work order inquiries. Through a closed-loop process of health measurement - automatic triggering - structured work orders - remote push notifications, this solves the problems of delayed manual judgment and ambiguous maintenance decisions in traditional operation and maintenance, enabling intelligent, precise, and efficient equipment maintenance, significantly reducing downtime risks and operation and maintenance costs.

[0076] In one embodiment, obtaining multimodal sensor data of a conference device includes:

[0077] S71, obtain acoustic sensor data of the conference device. The acoustic sensor has a pickup aperture of ≤3mm and a frequency response range of 20Hz-20kHz.

[0078] S72, obtaining vibration sensor data of the conference device, where the vibration sensor is a MEMS three-axis accelerometer;

[0079] S73, obtain the infrared thermal imager data of the conference equipment. The infrared thermal imager has a resolution of 160×120 and a temperature measurement accuracy of ±1.5°C.

[0080] S74, obtaining current sensor data of the conference device. The current sensor is a Hall effect sensor with a sampling rate of 1 kHz.

[0081] For example, an acoustic sensor with a pickup aperture ≤3mm is used, combined with a wide-frequency response range design of 20Hz-20kHz. A small pickup aperture is used to reduce environmental noise interference, focusing on abnormal noises of the equipment itself (such as fan friction and loose component noise). The wide-frequency response covers the audible range of the human ear and the ultrasonic frequency band, capturing high-frequency abnormal signals such as motor bearing wear; a MEMS three-axis accelerometer is selected as a vibration sensor, and the three-axis design simultaneously monitors the X / Y / Z axis vibration components to comprehensively characterize the three-dimensional vibration characteristics of the equipment (such as multi-directional vibration caused by motor eccentricity). The MEMS process realizes miniaturized integration, adapts to the compact installation space of conference equipment, and has both high sensitivity (μg-level resolution) and anti-interference ability; a 160×120 resolution infrared thermal imager is used, and the temperature measurement accuracy is controlled at ±1.5℃. It can identify temperature gradients as small as 0.1°C on the device surface (such as localized overheating caused by loose solder joints on circuit boards). High-precision temperature measurement eliminates the effects of ambient temperature drift, ensuring reliable temperature anomaly diagnosis (e.g., distinguishing normal heat dissipation from faulty overheating). Using a Hall-effect current sensor with a sampling rate of 1kHz, non-contact measurement eliminates intervening circuit losses, making it suitable for low-power monitoring of conference equipment. The 1kHz sampling rate captures current waveform distortions (such as current fluctuations caused by contactor oxidation) in real time, providing high-frequency data support for electrical fault diagnosis. Through the coordinated optimization of multimodal sensor parameters (acoustic focusing, 3D vibration monitoring, high-precision thermal imaging, and high-frequency current sampling), a three-dimensional monitoring network covering mechanical, electrical, and thermodynamic conditions is constructed, addressing the limitations of traditional single-modal monitoring, which suffers from limited dimensions and missed features. The sensor parameters have been industrially validated, meeting the requirements of miniaturization and low power consumption for conference equipment while improving fault identification through data complementarity (e.g., combined diagnosis of acoustic anomalies and current fluctuations), providing high-quality input for subsequent multi-source data fusion.

[0082] The present invention discloses a conference equipment status monitoring method based on multimodal sensors, which breaks through the limitation of single dimension of traditional single-modal monitoring by acquiring multimodal sensor data of conference equipment and extracting time-frequency domain features, and captures multi-dimensional features of equipment operation status. The hardware clock algorithm is used to align the timing to solve the problem of multimodal signal timing misalignment, and short-time Fourier transform and principal component analysis are used to realize joint feature extraction and dimensionality reduction in the time-frequency domain, thus avoiding the loss of key fault information. The preset weighted fusion algorithm is used to determine the initial weight based on the time-frequency features, and the sliding window mechanism is used to dynamically adjust the weight to solve the problem that traditional fixed weights cannot adapt to changes in working conditions, thereby improving the accuracy of multi-source data fusion. The pre-trained improved Transformer model and LSTM network model are used to realize Fault classification and remaining service life prediction are combined to generate health status information, which makes up for the lack of time series dynamic feature analysis in existing models and realizes a comprehensive evaluation from fault diagnosis to life prediction. Based on the health status information, a pre-built knowledge graph is used to generate a health assessment report containing fault overview, cause analysis, maintenance suggestions, etc., and based on the semantic reasoning of the graph, the fault can be explained, the cause can be traced, and the maintenance can be operated. At the same time, when the health level is lower than the threshold, a maintenance work order containing equipment identification, fault type and recommended measures is automatically generated and sent to the designated terminal, solving the technical problems of fault information loss, early warning lag and high operation and maintenance costs in traditional methods, improving the comprehensiveness, accuracy and intelligence level of equipment status monitoring, and reducing the risk of equipment downtime and operation and maintenance management costs.

[0083] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0084] Based on the same inventive concept, the embodiments of the present application also provide a multimodal sensor-based conference equipment status monitoring system for implementing the aforementioned multimodal sensor-based conference equipment status monitoring method. The implementation solution provided by this system is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the multimodal sensor-based conference equipment status monitoring system provided below can be found in the above-mentioned limitations of the multimodal sensor-based conference equipment status monitoring method, and will not be repeated here.

[0085] In an exemplary embodiment, Figure 2 As shown, a conference equipment status monitoring system based on a multimodal sensor is provided, comprising:

[0086] The data acquisition and processing module 101 is used to obtain multimodal sensor data of the conference equipment, perform time-frequency domain feature extraction, and generate a standardized sensor feature vector;

[0087] The feature weighted fusion module 102 is used to fuse the standardized sensor feature vectors using a preset weighted fusion algorithm to generate a health feature vector set;

[0088] The health status assessment module 103 is used to process the health feature vector set using a pre-trained health assessment model to generate health status information;

[0089] The health report generation module 104 is used to generate a health assessment report based on health status information using a pre-built knowledge graph.

[0090] In one embodiment, the data acquisition and processing module 101 is further configured to:

[0091] Through the hardware clock algorithm, the time sequence of each sensor data in the multimodal sensor data is aligned to obtain the accurate sensor data;

[0092] Using the short-time Fourier transform algorithm, the time-frequency domain joint feature extraction of the time-accurate sensor data is performed to generate a multi-scale time-frequency feature matrix;

[0093] The multi-scale time-frequency feature matrix is ​​subjected to principal component analysis and dimensionality reduction to obtain the standardized sensor feature vector.

[0094] In one embodiment, the feature weighted fusion module 102 is further configured to:

[0095] Determine the initial weight coefficient of each sensor based on the time-frequency characteristics of the standardized sensor feature vector;

[0096] The confidence level of each sensor feature is calculated through a sliding window mechanism, and the initial weight coefficient is dynamically adjusted based on the confidence level to generate an optimized weight matrix.

[0097] The weight matrix is ​​used to perform weighted fusion on the standardized sensor feature vectors to generate a healthy feature vector.

[0098] In one embodiment, the feature weighted fusion module 102 is further configured to construct a preset weighted fusion algorithm using the following formula:

[0099] F=∑ k∈{a，v，t，c} W k ·F k

[0100]

[0101] Among them, F represents the fused feature vector, F a represents the acoustic eigenvector, F v represents the vibration eigenvector, F t represents the temperature eigenvector, F c represents the current eigenvector, W k Represents the weight coefficient matrix of each mode, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d k Indicates the dimension of the key vector.

[0102] In one embodiment, the health status assessment module 103 is further configured to:

[0103] Using the pre-trained improved Transformer model, a preliminary diagnostic analysis of the health feature vector is performed to obtain preliminary fault classification results;

[0104] Using a pre-trained LSTM network model, the dynamic difference between the time-varying characteristics of the health feature vector set and the preset benchmark value is calculated to generate the remaining service life prediction value;

[0105] The classification results and the remaining useful life prediction values ​​are combined to generate health status information.

[0106] In one embodiment, a maintenance work order generation module is further included, which is used to:

[0107] Based on the health status information, the health degree is calculated. If the health degree is less than 85%, a maintenance work order is generated. The maintenance work order contains the identification of the conference equipment, the fault type and the recommended maintenance measures, and is sent to the designated terminal via the network.

[0108] In one embodiment, the data acquisition and processing module 101 is further configured to:

[0109] Acquire acoustic sensor data from conference equipment. The acoustic sensor's pickup aperture is ≤3mm and its frequency response range is 20Hz-20kHz.

[0110] Acquire the vibration sensor data of the conference equipment. The vibration sensor is a MEMS three-axis accelerometer.

[0111] Obtain infrared thermal imager data from conference equipment. The infrared thermal imager has a resolution of 160×120 and a temperature measurement accuracy of ±1.5°C.

[0112] Acquire the current sensor data of the conference equipment. The current sensor is a Hall effect sensor with a sampling rate of 1kHz.

[0113] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the conference device status monitoring method based on a multimodal sensor as described above are implemented.

[0114] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0116] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A conference equipment status monitoring method based on multimodal sensors, characterized in that: The method comprises: Acquire multimodal sensor data from conference equipment, perform time-frequency domain feature extraction, and generate standardized sensor feature vectors; fusing the standardized sensor feature vectors using a preset weighted fusion algorithm to generate a health feature vector set; Processing the health feature vector set using a pre-trained health assessment model to generate health status information; Based on the health status information, a health assessment report is generated using a pre-built knowledge graph.

2. The method according to claim 1, characterized in that The extracting time-frequency domain features from the multimodal sensor data to generate a standardized sensor feature vector includes: Performing time alignment on each sensor data in the multimodal sensor data by using a hardware clock algorithm to obtain accurate sensor data; Using a short-time Fourier transform algorithm, performing time-frequency domain joint feature extraction on the time-accurate sensor data to generate a multi-scale time-frequency feature matrix; The multi-scale time-frequency feature matrix is ​​subjected to principal component analysis and dimensionality reduction processing to obtain a standardized sensor feature vector.

3. The method according to claim 2, characterized in that The step of fusing the standardized sensor feature vectors using a preset weighted fusion algorithm to generate a health feature vector set includes: Determining an initial weight coefficient for each sensor based on the time-frequency characteristics of the standardized sensor feature vector; Calculating the confidence level of each sensor feature through a sliding window mechanism, and dynamically adjusting the initial weight coefficient based on the confidence level to generate an optimized weight matrix; The weight matrix is ​​used to perform weighted fusion on the standardized sensor feature vectors to generate the health feature vector.

4. The method according to claim 1, wherein The preset weighted fusion algorithm is constructed by the following formula: Among them, F represents the fused feature vector, F a represents the acoustic eigenvector, F v represents the vibration eigenvector, F t represents the temperature eigenvector, F c represents the current eigenvector, W k Represents the weight coefficient matrix of each mode, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d k Indicates the dimension of the key vector.

5. The method according to claim 1, wherein The step of processing the health feature vector set using a pre-trained health assessment model to generate health status information includes: Using a pre-trained improved Transformer model, a preliminary diagnostic analysis is performed on the health feature vector to obtain a preliminary fault classification result; Using a pre-trained LSTM network model, the dynamic difference between the time-varying characteristics of the health feature vector set and the preset benchmark value is calculated to generate a remaining service life prediction value; The health status information is generated by combining the classification result and the remaining useful life prediction value.

6. The method according to claim 5, characterized in that After generating the health status information, the following steps are included: Based on the health status information, a health degree calculation is performed. If the health degree is less than 85%, a maintenance work order is generated. The maintenance work order includes the identification of the conference device, the fault type and the recommended maintenance measures, and is sent to a designated terminal via the network.

7. The method according to claim 1, characterized in that The acquiring of multimodal sensor data of the conference device includes: Acquiring acoustic sensor data of the conference device, where the acoustic sensor has a sound pickup aperture of ≤3 mm and a frequency response range of 20 Hz-20 kHz; Acquire vibration sensor data of the conference device, where the vibration sensor is a MEMS three-axis accelerometer; Obtain infrared thermal imager data of the conference equipment, where the infrared thermal imager has a resolution of 160×120 and a temperature measurement accuracy of ±1.5°C; The current sensor data of the conference device is obtained, where the current sensor is a Hall effect sensor with a sampling rate of 1 kHz.

8. A conference equipment status monitoring system based on multimodal sensors, characterized in that: The device comprises: The data acquisition and processing module is used to obtain multimodal sensor data from conference equipment, perform time-frequency domain feature extraction, and generate standardized sensor feature vectors; A feature weighted fusion module, configured to fuse the standardized sensor feature vectors using a preset weighted fusion algorithm to generate a health feature vector set; a health status assessment module, configured to process the health feature vector set using a pre-trained health assessment model to generate health status information; The health report generation module is used to generate a health assessment report based on the health status information using a pre-built knowledge graph.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images