Industrial equipment predictive maintenance system based on multi-modal fusion of voiceprint and thermal imaging

Abstract

This invention relates to the field of predictive maintenance technology, specifically to a predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging. The system includes an acoustic and thermal signal acquisition module, a response feature extraction module, a coupling difference generation module, an anomaly stage identification module, and a maintenance operation triggering module. In this invention, a joint feature sequence is constructed by synchronously acquiring acoustic signature signals and thermal imaging images. Rhythmic differences and abrupt trajectories in amplitude and grayscale trends are extracted. Combined with response intervals and intensity shifts, state change trends are generated. Rotational speed linkage analysis is performed on anomaly stages, and correlated fault sections are identified. Control criteria are extracted from the intersection of image expansion areas and trajectory shift results, generating command information for maintenance execution. This achieves joint identification of anomaly response rhythm, trend shift, and spatial diffusion, enhancing the continuity of fault identification and the relevance of decision-making instructions.

Description

Technical Field

[0001] This invention relates to the field of predictive maintenance technology, and in particular to a predictive maintenance system for industrial equipment based on the fusion of acoustic signature and thermal imaging multimodal methods. Background Technology

[0002] Predictive maintenance technology primarily involves real-time monitoring and health assessment of the operating status of industrial equipment. Core aspects include equipment status perception, fault symptom identification, remaining life prediction, and maintenance strategy formulation. It aims to predict and intervene in potential equipment failures through multi-source information collection and fusion, machine learning model training, feature extraction and analysis, and health status modeling. This technology is widely used in various industrial scenarios such as manufacturing, power, and transportation. Its overall technical system encompasses multiple interdisciplinary fields, including sensing technology, data mining, fault diagnosis, reliability modeling, and prediction algorithms, exhibiting highly intelligent and information-based integrated characteristics. Traditional industrial equipment predictive maintenance systems, on the other hand, collect single-type data generated during equipment operation, such as vibration signals or temperature data, and combine this data with empirically set thresholds or reasoning methods based on a single physical model to determine whether equipment anomalies exist and predict the timeline for potential failures. Vibration data is typically acquired using accelerometers, and feature parameters are extracted using frequency domain analysis methods such as Fourier transform. Then, a classification or regression model based on support vector machines is used for health status assessment and lifespan prediction. In thermal imaging detection, single-frame infrared images are usually compared with temperature gradients to identify abnormal locations, but this method cannot accurately correlate fault types and severity. Acoustic analysis, on the other hand, relies on microphones to collect audio signals, uses Mel-frequency cepstral coefficients to extract acoustic features, and then uses template matching to identify the operating status of specific equipment. However, its analytical scope is limited by environmental noise interference and insufficient feature dimensions.

[0003] Existing predictive maintenance technologies are mostly based on independent analysis of single sensor information. Vibration monitoring relies on frequency domain parameter extraction, which is difficult to reflect dynamic temporal characteristics. Thermal imaging relies only on static image grayscale comparison and cannot identify temperature change trends. Acoustic recognition is limited by simple feature dimensions and is easily affected by background interference, making it difficult to achieve multi-angle state characterization in complex working conditions. It lacks effective utilization of the correlation between multi-source signals. The lack of a time synchronization mechanism between sensor data causes delays or omissions in the capture of abnormal signs. The overall recognition results are limited to static threshold judgment, which is difficult to cover the actual needs of changing operating conditions, affecting the sensitivity of anomaly detection and the timeliness of maintenance decisions. Summary of the Invention

[0004] To address the technical problems existing in the prior art, embodiments of the present invention provide a predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging. On one hand, a predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging is provided, the system comprising: The acoustic and thermal signal acquisition module collects acoustic fingerprint sensing signals during the operation of industrial equipment, performs segmented processing according to fixed time periods, identifies the position of amplitude change in each signal segment, extracts the amplitude change of adjacent points, acquires thermal imaging images corresponding to the time period, locates the gray value distribution area, extracts the gray value change rate between adjacent frames, and constructs the acoustic fingerprint amplitude trend and thermal field temperature rise trend. Based on the acoustic amplitude trend and the thermal field temperature rise trend, the response feature extraction module calculates the difference in signal rise rate in each time period, arranges the difference in time order and performs inter-group comparison to obtain dual-channel comparison results. Based on the dual-channel comparison results, the coupling difference generation module identifies change points in a continuous time period, calculates and classifies the response intervals between change points, and combines the difference in change intensity between adjacent time periods to perform offset amplitude accumulation and construct a difference trajectory. Based on the difference trajectory, the abnormal phase identification module identifies continuous offset time periods, performs synchronous judgment by comparing the equipment speed change information within the time period, filters time periods that meet preset speed fluctuation conditions, and generates phase identification content. The maintenance operation triggering module performs a position cross-comparison based on the expansion trend of the grayscale change area of ​​the thermal field image, determines whether the cross-range meets the operation conditions, and outputs the operation command for predictive maintenance of industrial equipment according to the change level of the cross-range area.

[0005] As a further aspect of the present invention, the acoustic signature amplitude trend and thermal field temperature rise trend include amplitude change feature points, amplitude difference between adjacent points, image grayscale distribution features, and grayscale change rate index. The dual-channel comparison results include rise rate difference sequence, time rhythm difference parameter, and inter-channel response difference value. The difference trajectory includes abrupt change point location sequence, response interval category label, and offset amplitude superposition value. The stage identification content includes continuous offset time period identifier, rotation speed fluctuation corresponding time period, and abnormal response stage number. The predictive maintenance operation commands for industrial equipment include location cross-matching results, grayscale change level classification, and maintenance control command type.

[0006] As a further aspect of the present invention, the identification of change points in a continuous time period refers to locating the time position where the change rate exceeds a set threshold by monitoring the abrupt change rate of the dual-channel change values ​​in the time series.

[0007] As a further aspect of the present invention, the time period that meets the preset speed fluctuation condition refers to extracting the time period exceeding the speed fluctuation benchmark value by performing difference judgment on the equipment speed sequence within a continuous offset time period.

[0008] As a further aspect of the present invention, the acoustic-thermal signal acquisition module includes: The acoustic signature amplitude extraction submodule acquires acoustic signature sensing signals from industrial equipment during operation, processes them in segments according to fixed time periods, monitors the amplitude change positions in each segment of the sound pressure value sequence, extracts the sound pressure values ​​at adjacent time points and calculates the difference, and generates a sound pressure amplitude difference sequence. The thermal field grayscale extraction submodule acquires thermal imaging image frames for the corresponding time period, calls the grayscale matrix of the image, locates the boundary of the grayscale change region, extracts the grayscale value of the pixels in the region, calculates the grayscale change rate between adjacent frames, and generates a local grayscale change rate sequence. The acoustic-thermal trend construction submodule calls the time index of the sound pressure amplitude difference sequence and the local grayscale change rate sequence to match the change trend within the same time period, performs joint normalization on the two types of data, establishes a time change curve, and obtains the acoustic amplitude trend and thermal field temperature rise trend.

[0009] As a further aspect of the present invention, the response feature extraction module includes: The rise rate construction submodule extracts the values ​​of three consecutive time points within each time segment based on the acoustic amplitude trend and the thermal field temperature rise trend. It subtracts the differences between the last two points and the first point pairwise to obtain the continuous rise rate difference sequence of each signal segment and obtains the channel rise rate difference matrix. The rhythm difference analysis submodule calls the voiceprint channel and thermal field channel data in the channel rise rate difference matrix, pairs the rise rate differences in the same time period according to the time index, constructs the paired data into a two-dimensional group data set, and performs group aggregation according to the difference range of each group of data to obtain rhythm difference data groups; The dual-channel comparison generation submodule extracts the rhythm change values ​​of the voiceprint and thermal field channel data pairs sequentially according to the time series of each group in the rhythm difference data group, and plots the channel change trend curves in the order of the group sequence. It calculates the inter-group difference value sequence through the deviation of the trend line and establishes the dual-channel comparison result.

[0010] As a further aspect of the present invention, the coupling difference generation module includes: The mutation point identification submodule monitors the differences in channel change values ​​at adjacent time points based on the continuous time series in the dual-channel comparison results, and marks the time positions where the change rate is greater than the set mutation threshold according to the change gradient change rate. It then extracts the time periods between consecutive markers and aggregates the boundary index to obtain the change mutation point sequence. The response interval classification submodule calculates the time response interval value between two points based on the index of each group of adjacent mutation points in the sequence of changes in mutation points, and classifies all mutation groups into the corresponding category interval according to the interval length, counts the number of mutation segments and the average time span in each category, and obtains the response interval classification result set. The difference trajectory construction submodule calls the time index of each mutation group in the response interval classification result set, calculates the difference of the dual-channel change value of the corresponding time period, performs amplitude superposition of the difference between adjacent segments in time order, summarizes the total difference of the classification and the trend line index, and establishes the difference trajectory.

[0011] As a further aspect of the present invention, the abnormal stage identification module includes: The continuous offset detection submodule extracts the amplitude change gradient between adjacent segments based on the continuous offset segments in the difference trajectory, judges the continuity according to the set offset duration threshold, aggregates and marks the index segments that meet the continuity condition, and obtains the continuous offset time period set. The speed fluctuation correlation submodule calls the index of each time period in the continuous offset time period set, and synchronously obtains the equipment speed sequence within the corresponding time period. It calculates the difference between any two adjacent sampling points in each speed sequence and determines whether it exceeds the speed fluctuation benchmark value, and obtains the fluctuation correlation time period sequence. The stage identifier generation submodule filters out the time periods that appear in the difference trajectory and are accompanied by speed fluctuations based on the time period position in the fluctuation-related time period sequence. The filtered time periods are then structured according to the start and end indices to establish stage identifier content.

[0012] As a further aspect of the present invention, the maintenance operation triggering module includes: The grayscale expansion judgment submodule calls the grayscale distribution of the thermal field image of the corresponding time frame according to the time period index in the stage identifier content, extracts the boundary contour of the grayscale change area in each frame, calculates the boundary expansion direction and area increment of the region in adjacent frames, aggregates and judges the region expansion behavior in continuous time periods, and obtains the grayscale expansion trend dataset. The rule cross-filtering submodule performs position coordinate cross-judgment based on the spatial positioning information of the corresponding time index in the gray-scale expansion trend dataset and the stage identifier content, extracts the spatial range of the area cross-section, compares it with the position overlap threshold in the preset operation rules, filters the data segments that meet the conditions, and obtains the rule matching cross-section area set. The maintenance instruction generation submodule calls the rule to match the expansion level value of the cross area in the cross area set, and retrieves the corresponding control instruction from the control parameter set according to the control level mapping relationship corresponding to the level value. It integrates the operation number, action type and execution parameters of the instruction to generate the operation command for predictive maintenance of industrial equipment.

[0013] As a further aspect of the present invention, the position overlap threshold refers to the cross-area standard for determining whether the degree of overlap between the grayscale expansion area and the stage positioning area in spatial coordinates reaches the threshold for triggering maintenance conditions.

[0014] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: By simultaneously acquiring acoustic signature signals and thermal imaging images to construct a joint feature sequence, rhythmic differences and abrupt trajectories in amplitude and grayscale trends are extracted. Combined with response intervals and intensity shifts to generate state change trends, rotational speed linkage analysis is performed on abnormal stages to identify related fault sections. Control criteria are extracted from the intersection of image expansion areas and trajectory shift results to generate command information for maintenance execution. This achieves joint identification of abnormal response rhythm, trend shifts, and spatial diffusion, enhancing the continuity of fault identification and the pertinence of decision-making instructions. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the system of the present invention; Figure 2 This is a flowchart of the acoustic and thermal signal acquisition module in this invention; Figure 3 This is a flowchart of the response feature extraction module in this invention; Figure 4 This is a flowchart of the coupling difference generation module in this invention; Figure 5 This is a flowchart of the abnormal stage identification module in this invention; Figure 6 This is a flowchart of the maintenance operation triggering module in this invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] This invention provides a predictive maintenance system for industrial equipment based on multimodal fusion of voiceprint and thermal imaging, such as... Figure 1 The diagram shown illustrates a predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging. The system includes: The acoustic and thermal signal acquisition module collects acoustic fingerprint sensing signals from industrial equipment in operation and performs segmented processing according to fixed time periods. It identifies the position of amplitude change within each signal segment and extracts the amplitude change of adjacent points. At the same time, it acquires thermal imaging images corresponding to the time period, locates the gray value distribution area in the image, and extracts the gray value change rate between adjacent frames to construct the acoustic fingerprint amplitude trend and the thermal field temperature rise trend. The response feature extraction module calculates the difference in signal rise rate in each time period based on the trend of acoustic amplitude and the trend of thermal field temperature rise. By arranging the differences in time order and performing inter-group comparison, the rhythmic changes of the differentiated sensor signal sources in time response are obtained, and dual-channel comparison results are generated. The coupling difference generation module identifies abrupt change points within a continuous time period based on the dual-channel comparison results, performs classification and summarization on the response interval between each group of abrupt change points, and performs offset amplitude superposition based on the difference in change intensity between adjacent time periods to construct the difference trajectory. The abnormal phase identification module identifies time periods with continuous offsets in the differential trajectory, compares them with the equipment speed change information within the same time period for correlation judgment, filters out time periods that are accompanied by speed fluctuations at the same time as the differential trajectory, and generates phase identification content. The maintenance operation triggering module performs position cross-comparison based on the stage identifier content and the continuous expansion trend of the grayscale change area of ​​the thermal field image, determines whether the cross-range meets the preset operation rules, and obtains control instructions according to the change level of the cross-range area, outputting the operation command for predictive maintenance of industrial equipment.

[0020] The acoustic signature amplitude trend and thermal field temperature rise trend include amplitude change feature points, amplitude difference between adjacent points, image grayscale distribution features, grayscale change rate index, dual-channel comparison results include rise rate difference sequence, time rhythm difference parameter, channel response difference value, difference trajectory includes abrupt change point location sequence, response interval category label, offset amplitude superposition value, stage identification content includes continuous offset time period identifier, speed fluctuation corresponding time period, abnormal response stage number, and predictive maintenance operation commands for industrial equipment include location cross-matching results, grayscale change level classification, and maintenance control command type. The predictive maintenance system for industrial equipment based on the multimodal fusion of acoustic signature and thermal imaging is applicable to industrial operating equipment with rotating parts, heat source distribution and acoustic signature characteristics, especially including but not limited to three-phase asynchronous motors, centrifugal pumps, fans, gearboxes, bearing housings, generator stator windings, cooling pipes and high-temperature compressors.

[0021] Specifically, such as Figure 2 As shown, the acoustic and thermal signal acquisition module includes: The acoustic signature amplitude extraction submodule acquires acoustic signature sensing signals from industrial equipment during operation, processes them in segments according to fixed time periods, monitors the amplitude change positions in each segment of the sound pressure value sequence, extracts the sound pressure values ​​at adjacent time points and calculates the difference, and generates a sound pressure amplitude difference sequence. A broadband industrial piezoelectric sensor array is used to acquire raw analog acoustic signals from industrial equipment under load operation in real time at a sampling frequency of 44.1 kHz. This analog signal is then converted into a 16-bit deep digital acoustic signature signal stream via an analog-to-digital converter. The submodule has a 500-millisecond time window that slides along the time axis of the digital acoustic signature signal stream, dividing the continuous signal stream into a series of discrete sound pressure level (SPL) value segments. Within each 500-millisecond segment, the submodule scans the SPL data point-by-point at millisecond time steps, locating the peaks and troughs where SPL values ​​change abruptly. Subsequently, the submodule executes a difference operation logic between adjacent time points: extracting the SPL value at the current time point tt and the SPL value at the next time point tt plus 1, subtracting the former from the latter, yields the SPL increment within that small time interval. This operation logic iterates through all sampling points within the segment, arranging all calculated increment values ​​in chronological order to construct a sound pressure amplitude difference sequence. For example, when a certain sound pressure value sequence is collected, the sound pressure value at the 10th millisecond is 102 Pascals, the sound pressure value at the 11th millisecond is 105 Pascals, and the sound pressure value at the 12th millisecond is 104 Pascals, the submodule first calculates the difference between 105 Pascals and 102 Pascals to obtain 3 Pascals, then calculates the difference between 104 Pascals and 105 Pascals to obtain -1 Pascals, and so on to form a difference value sequence.

[0022] The thermal field grayscale extraction submodule acquires thermal imaging image frames for the corresponding time period, calls the grayscale matrix of the image, locates the boundary of the grayscale change region, extracts the grayscale value of the pixels in the region, calculates the grayscale change rate between adjacent frames, and generates a local grayscale change rate sequence. Using an uncooled infrared focal plane detector, the thermal radiation distribution on the device surface is captured at a rate of 60 frames per second within a corresponding time period, and converted into thermal imaging image frames with a resolution of 640 x 480 pixels. This submodule reads the raw data of each frame and calls the internal grayscale conversion algorithm to map the color thermal image into a single-channel grayscale matrix, where each element of the matrix corresponds to the grayscale value of a pixel, ranging from 0 to 255. To locate the boundaries of grayscale variation regions, this submodule uses a convolutional neural network to process the grayscale matrix. This network includes an input layer for receiving the 640 x 480 grayscale matrix, followed by a convolutional layer containing 32 3 x 3 kernels to extract edge features from the image. After the convolutional layer, a modified linear unit activation function is connected to remove negative noise, and a max pooling layer is used to reduce the data dimensionality. Finally, the output layer outputs a binary mask that identifies regions with drastic grayscale changes. The submodule extracts the average grayscale value of all pixels within the mask area and calculates the difference between the average grayscale value of the current frame and the average grayscale value of the previous frame. This difference is then divided by the inter-frame time interval to obtain the grayscale change rate. Assuming the average grayscale value of the target area in the current frame is 150, and the previous frame's was 148, with a time interval of 0.016 seconds, the submodule calculates 150 minus 148 to get 2, then divides this by 0.016, resulting in a local grayscale change rate of 125 grayscale values ​​per second.

[0023] The acoustic-thermal trend construction submodule calls the time index of the sound pressure amplitude difference sequence and the local gray level change rate sequence to match the change trend within the same time period, performs joint normalization on the two types of data, establishes a time change curve, and obtains the sound signature amplitude trend and the thermal field temperature rise trend. The module reads the timestamp information from the sound pressure amplitude difference sequence and the local grayscale change rate sequence, aligning the data within the same time window using the time axis as a reference. Since the sound pressure difference is measured in Pascals while the grayscale change rate is measured in grayscale values ​​per second, this submodule performs joint normalization. The specific normalization logic is as follows: obtain the minimum and maximum values ​​of the data within the current time period, subtract the minimum value from the current data value, and then divide the result by the difference between the maximum and minimum values, thus mapping all data to a dimensionless range of 0 to 1. This submodule connects the normalized sound pressure data points in chronological order to form a sound signature amplitude trend curve, and simultaneously connects the normalized grayscale rate data points to form a thermal field temperature rise trend curve. For example, if the maximum sound pressure difference during a certain period is 10 Pascals, the minimum is 0 Pascals, and the current value is 5 Pascals, then the normalization result is 0.5; similarly, if the maximum grayscale change rate during the same period is 200, the minimum is 0, and the current value is 100, then the normalization result is also 0.5. Through this processing, the submodule outputs the synchronous acoustic signature amplitude trend and thermal field temperature rise trend, revealing the coordinated changes of acoustic and thermal physical quantities during equipment operation.

[0024] Specifically, such as Figure 3 As shown, the response feature extraction module includes: The rise rate construction submodule is based on the acoustic amplitude trend and the thermal field temperature rise trend. It extracts the values ​​of three consecutive time points in each time segment, subtracts the difference between the last two points and the first point in pairs, and obtains the continuous rise rate difference sequence of each signal segment, thus obtaining the channel rise rate difference matrix. The module receives normalized voiceprint amplitude trend data and thermal field temperature rise trend data, and extracts data values ​​from three consecutive time points within each set analysis time period, denoted as data point A, data point B, and data point C. This submodule executes a quadratic difference calculation logic: first, it calculates the difference between data point B and data point A to obtain the first-stage change; then, it calculates the difference between data point C and data point B to obtain the second-stage change. Subsequently, the submodule subtracts the first-stage change from the second-stage change to obtain the continuous rise rate difference within that time element. This calculation logic is used to quantify the acceleration characteristics of signal changes. This submodule performs the above operations on both the voiceprint channel data and the thermal field channel data, arranging all calculated differences in chronological order to form a voiceprint channel rise rate difference sequence and a thermal field channel rise rate difference sequence, respectively. These two sequences are then combined to form a channel rise rate difference matrix. For example, in the voiceprint channel, the normalized values ​​at three consecutive time points are 0.2, 0.4 and 0.7 respectively. The submodule first calculates 0.4 minus 0.2 to get 0.2, then calculates 0.7 minus 0.4 to get 0.3, and finally calculates 0.3 minus 0.2 to obtain the difference in the rate of rise at that moment, which is 0.1, indicating that the signal is in an accelerated rising state.

[0025] The rhythm difference analysis submodule calls the voiceprint channel and thermal field channel data in the channel rise rate difference matrix, pairs the rise rate difference of the same time period according to the time index, constructs the paired data into a two-dimensional group data set, and performs group aggregation according to the difference range of each group of data to obtain rhythm difference data groups; The module reads the channel rise rate difference matrix and extracts the voiceprint rise rate difference and thermal field rise rate difference at the same time according to the same time index, forming a two-dimensional data pair. This submodule traverses the entire time period, constructing a dataset containing a massive number of two-dimensional data pairs. Subsequently, the submodule groups and aggregates these data pairs according to a preset difference range standard. The specific difference range standard is set as follows: calculate the algebraic difference between the voiceprint value and the thermal field value in each data group. If the absolute value of the difference is less than 0.05, it is classified into the synchronization group; if the absolute value of the difference is between 0.05 and 0.2, it is classified into the slight difference group; if the absolute value of the difference is greater than 0.2, it is classified into the significant difference group. The submodule counts the number of data pairs falling into each group and their specific values, thereby obtaining the rhythm difference data groups. If at a certain moment the voiceprint difference is 0.1 and the thermal field difference is 0.12, and the absolute value of the difference between the two is 0.02, which is less than 0.05, then the data pair at that moment is classified into the synchronization group; if at another moment the voiceprint difference is 0.1 and the thermal field difference is 0.4, and the absolute value of the difference is 0.3, then it is classified into the significant difference group.

[0026] The dual-channel comparison generation submodule extracts the rhythm change values ​​of the voiceprint and hot field channel data pairs according to the time series of each group in the rhythm difference data group, and plots the channel change trend curves in the order of the group sequence. It calculates the inter-group difference value sequence through the deviation of the trend line and establishes the dual-channel comparison results. For each data group in the rhythm difference data set, the rhythm variation values ​​of the voiceprint channel and the thermal field channel are extracted according to their original time series order. The submodule plots the trend curves of the voiceprint channel and the thermal field channel in a virtual coordinate system, with time as the horizontal axis and the variation value as the vertical axis. Next, the submodule calculates the vertical distance between the two curves at each moment, i.e., the absolute value of the difference between the voiceprint value and the thermal field value, and uses this distance value as the deviation degree at that moment. The submodule sums the deviation values ​​for all moments within the group and divides by the total number of time points to obtain the average difference value for that group. By performing this operation on all groups, a dual-channel comparison result sequence is established. Table 1 shows a dual-channel comparison data segment generated by this submodule during actual processing.

[0027] Table 1. Comparison and Analysis Data of Dual-Channel Rhythm

[0028] As shown in Table 1, the submodule quantifies the specific differences in the rate of change between the two physical quantities, acoustic and thermal, through point-by-point calculations, providing a quantitative basis for subsequent judgment of coupling anomalies. For example, at time T-1003, the deviation reaches 0.40, which is marked as a significant difference, and will serve as an important clue for subsequent modules to identify abrupt changes.

[0029] Specifically, such as Figure 4As shown, the coupling difference generation module includes: The mutation point identification submodule monitors the differences in channel change values ​​between adjacent time points based on the continuous time series in the dual-channel comparison results, and marks the time positions where the change rate is greater than the set mutation threshold according to the change gradient change rate. It then extracts the time periods between consecutive markers and aggregates the boundary index to obtain the change mutation point sequence. The submodule reads continuous time-series data from the dual-channel comparison results, which records the degree of deviation between the acoustic signature and the thermal field. It scans the deviation values ​​of adjacent time points point by point, calculates the difference between the deviation value at the current time point and the deviation value at the previous time point, and divides this difference by the time step to obtain the rate of change of the gradient. The submodule internally sets a mutation threshold, which is determined through statistical analysis of historical normal operation data and is set to three times the standard deviation of the historical data deviation rate of change (e.g., 0.5). The submodule compares the real-time calculated rate of change of the gradient with this mutation threshold; if the result is greater than 0.5, the current time point is marked as a mutation point. After identifying all mutation points that meet the criteria, the submodule extracts the sequence of consecutively marked time points and aggregates the start and end indices of these consecutive points to obtain the sequence of mutation points. For example, if the rate of change of the gradient is 0.6, 0.7, 0.8, 0.7, 0.6, and 0.4 at time points 50 to 55, respectively, all of which are greater than the threshold of 0.5, the submodule will aggregate the time indexes 50 to 55 into a mutation segment and record its boundary.

[0030] The response interval classification submodule calculates the time response interval between two points based on the index of each group of adjacent mutation points in the sequence of changes in mutation points, and classifies all mutation groups into the corresponding category interval according to the interval length. It also counts the number of mutation segments and the average time span in each category to obtain the response interval classification result set. The module iterates through the sequence of mutation points, extracting the index information of each pair of adjacent mutation segments. Specifically, the submodule obtains the end time of the previous mutation segment and the start time of the next mutation segment, calculates the difference between these two time points, and obtains the time response interval value between the two points. Then, the submodule categorizes all mutation groups into corresponding category intervals based on the interval length: if the interval value is less than 1 second, it is classified into the high-frequency response zone; if the interval value is between 1 and 5 seconds, it is classified into the mid-frequency response zone; if the interval value is greater than 5 seconds, it is classified into the low-frequency response zone. The submodule counts the number of mutation segments contained in each category interval and calculates the arithmetic mean of all interval values ​​within that interval, thus obtaining the response interval classification result set. For example, if the calculated interval between two mutation segments is 0.8 seconds, it is classified into the high-frequency zone; if the other interval is 3 seconds, it is classified into the mid-frequency zone. The final statistical result may show that there are 5 mutation segments in the high-frequency zone, with an average interval of 0.7 seconds.

[0031] The differential trajectory construction submodule calls the time index of each mutation group in the response interval classification result set, calculates the difference of the dual-channel change value of the corresponding time period, performs amplitude superposition of the difference between adjacent segments in time order, summarizes the total difference of the classification and the trend line index, and establishes the differential trajectory. The submodule reads the time index corresponding to each mutation group in the response interval classification result set and backtracks to retrieve the original dual-channel change values ​​within that time period. The submodule performs a subtraction operation on the acoustic signature change value and the thermal field change value at each moment to obtain the instantaneous difference value. Next, the submodule performs an amplitude summation operation on the differences between adjacent time periods in chronological order; that is, the cumulative difference at the current moment equals the cumulative difference at the previous moment plus the instantaneous difference at the current moment. The submodule applies this accumulation process to each classification group, summarizing and calculating the total difference for each category, and associates the time index of the cumulative difference curve with a trend line to establish a difference trajectory. Assuming the initial cumulative value is 0, and the instantaneous difference in the first second is 2, then the cumulative value is 2; if the instantaneous difference in the second second is 3, then the cumulative value is 5. Through this summation, the submodule constructs a trajectory curve reflecting the evolution of acoustic and thermal differences over time, which visually demonstrates the evolution of abnormal equipment states.

[0032] Specifically, such as Figure 5 As shown, the anomaly stage identification module includes: The continuous offset detection submodule extracts the amplitude change gradient between adjacent segments based on the continuous offset segments in the differential trajectory, judges the continuity according to the set offset duration threshold, aggregates and marks the index segments that meet the continuity condition, and obtains the continuous offset time period set. The submodule accesses the difference trajectory data, which reflects the cumulative trend of acoustic and thermal signal differences. Following the time index, the submodule extracts the amplitude values ​​of two adjacent time periods on the trajectory and calculates their gradient changes by subtracting the amplitude value of the previous period from the amplitude value of the later period and then dividing by the time difference. The submodule determines continuity based on a set offset duration threshold, which is set to the gradient values ​​having the same direction (i.e., all positive or all negative) for five consecutive sampling periods. The submodule scans the entire gradient sequence; once it finds a segment that satisfies the condition of consistent gradient direction for five consecutive periods, it determines that the continuity condition is met and aggregates and marks the start and end indices of this segment to generate a set of continuous offset time periods. For example, if the gradient values ​​of the difference trajectory from second 100 to second 105 are 0.1, 0.15, 0.12, 0.18, and 0.2, all positive and lasting for five periods, the submodule marks second 100 to second 105 as a continuous offset time period.

[0033] The speed fluctuation correlation submodule calls the index of each time period in the continuous offset time period set, and synchronously obtains the equipment speed sequence within the corresponding time period. It calculates the difference between any two adjacent sampling points in each speed sequence and determines whether it exceeds the speed fluctuation benchmark value, and obtains the fluctuation correlation time period sequence. The submodule reads the index of each time period in the continuous offset time period set and synchronously obtains the speed sequence of the equipment motor within that time period via the industrial fieldbus, in revolutions per minute (rpm). The submodule calculates the difference between any two adjacent sampling points in the speed sequence to obtain the speed fluctuation value. An internal speed fluctuation benchmark value is set within the submodule, which is 2% of the equipment's rated speed. For example, if the rated speed is 1500 rpm, the benchmark value is 30 rpm. The submodule determines whether the calculated speed fluctuation value exceeds this benchmark value. If it does, an abnormal fluctuation is identified at that moment. The submodule filters out all moments within the continuous offset time period that exceed the benchmark value to obtain the fluctuation-related time period sequence. For example, within the aforementioned time period from 100 to 105 seconds, the submodule detects that the speed at the 103rd second suddenly drops from 1500 to 1450, a difference of 50 greater than the benchmark value of 30. Therefore, the 103rd second and the adjacent fluctuation periods are identified as fluctuation-related time periods.

[0034] The stage identifier generation submodule filters out the time periods that appear in the difference trajectory and are accompanied by speed fluctuations based on the time period position in the fluctuation-related time period sequence. It then structures the filtered time periods according to the start and end indices to establish the stage identifier content. The module executes cross-filtering logic, comparing the set of continuous offset time periods with the time sequence associated with fluctuations. The submodule filters out time periods that exist both in the continuous offset records of the differential trajectory and are accompanied by abnormal rotational speed fluctuations. For each selected time period, the submodule extracts its specific start and end timestamps, formats them into structured data, and establishes stage identifier content. For example, after filtering, it is determined that there is both continuous offset of the acoustic-thermal differential trajectory and severe fluctuations in rotational speed from 102 seconds to 104 seconds. The submodule then generates the identifier content: "Stage ID: 001, Start Time: 102s, End Time: 104s, Type: Composite Anomaly". This result will serve as a precise spatiotemporal guide for triggering subsequent maintenance operations.

[0035] Specifically, such as Figure 6 As shown, the maintenance operation triggering module includes: The grayscale expansion judgment submodule calls the grayscale distribution of the thermal field image of the corresponding time frame according to the time period index in the stage identifier content, extracts the boundary contour of the grayscale change area in each frame, calculates the boundary expansion direction and area increment of the region in adjacent frames, aggregates and judges the region expansion behavior in continuous time periods, and obtains the grayscale expansion trend dataset. Based on the abnormal time period index locked in the stage identifier, the submodule retrieves multiple frames of thermal field images corresponding to that time period from the database. The submodule applies the Canny edge detection algorithm to each frame to extract the boundary contours of regions with significantly higher grayscale values ​​than the background. Subsequently, the submodule calculates the geometric attribute changes of this region in adjacent frames: first, it determines the centroid coordinates of the region, calculates the displacement vector of the centroid in the next frame relative to the centroid in the previous frame to determine the direction of boundary expansion; simultaneously, it calculates the total number of pixels contained in the region, subtracting the number of pixels in the previous frame from the number of pixels in the next frame to obtain the area increment. The submodule accumulates the area increments within consecutive time periods and judges the consistency of the expansion direction to obtain a grayscale expansion trend dataset. For example, if in three consecutive frames the area of ​​the high-temperature region increases by 100 pixels, 120 pixels, and 150 pixels respectively, and the centroid moves to the upper right in all three frames, the submodule records that the region is in a "faster expansion to the upper right" state and quantifies its expansion rate.

[0036] The rule cross-filtering submodule performs position coordinate cross-judgment based on the spatial positioning information of the corresponding time index in the gray-scale expansion trend dataset and the stage identifier content, extracts the spatial range of the area cross-section, compares it with the position overlap threshold in the preset operation rules, filters the data segments that meet the conditions, and obtains the rule matching cross-section area set. The submodule reads the spatial positioning information (i.e., the centroid coordinates and boundary range of the expanded area) from the grayscale expansion trend dataset and calls the preset operation rule library. This rule library stores the spatial coordinate ranges and corresponding safety thresholds of key equipment components (such as bearing housings, coil windings, and cooling pipes). The submodule performs position coordinate cross-judgment, calculating the overlap area between the abnormal thermal field area and the preset areas of each key component. If the ratio of the overlap area to the total area of ​​the key component is greater than the preset position overlap threshold (e.g., set to 10%), the data segment is deemed to meet the filtering conditions. The submodule extracts the spatial range data of these cross-regions that meet the conditions, obtaining a rule-matching cross-region set. For example, assuming the coordinate range of the thermal field expansion area is (x:100-200, y:100-200), and the coordinate range of the "main bearing" in the rule library is (x:150-250, y:150-250), and there is an overlap, with the overlap portion accounting for 25% of the main bearing area, exceeding the 10% threshold, then this area is included in the matching set, and the associated component is marked as the "main bearing".

[0037] The maintenance instruction generation submodule calls the rule to match the expansion level value of the cross area in the cross area set, and retrieves the corresponding control instruction from the control parameter set according to the control level mapping relationship corresponding to the level value. It integrates the operation number, action type and execution parameters of the instruction to generate the operation command for predictive maintenance of industrial equipment. The receiving module matches the set of cross-regions and extracts the expansion level value for each cross-region. This expansion level value is a quantitative indicator calculated based on the area increment rate (e.g., level 1 is slow, level 3 is rapid). The submodule retrieves the corresponding operation command from the control parameter set based on the control level mapping relationship corresponding to the level value. The mapping logic is as follows: if the level value is 1, retrieve the warning record command; if the level value is 2, retrieve the speed reduction operation command; if the level value is 3, retrieve the emergency stop command. The submodule integrates the retrieved commands, including the operation number, the specific action type (e.g., "reduce voltage", "cut off power"), and detailed execution parameters (e.g., "reduce to 380 volts", "delay 0 seconds to execute"), and finally generates the operation command for predictive maintenance of industrial equipment.

[0038] Table 2. Maintenance Command Generation Logic Comparison Table

[0039] As shown in Table 2, the submodule maps the actual calculated physical parameters (area increment rate) to specific control logic, achieving an automated closed loop from monitoring to execution. For example, when the calculated area increment rate is 250 pixels / second, the submodule determines it to be Level 3 and immediately generates a command "Operation Number CMD-99, Action Type: Emergency Stop, Execution Parameter: Execute Immediately," effectively preventing further escalation of equipment failure. This logic ensures the timeliness and targeted nature of maintenance operations.

[0040] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the technical solution.

Claims

1. A predictive maintenance system for industrial equipment based on multimodal fusion of voiceprint and thermal imaging, characterized in that, The system includes a piezoelectric acoustic sensor and an infrared thermal imaging device; The acoustic and thermal signal acquisition module collects acoustic fingerprint sensing signals during the operation of industrial equipment, performs segmented processing according to fixed time periods, identifies the position of amplitude change in each signal segment, extracts the amplitude change of adjacent points, acquires thermal imaging images corresponding to the time period, locates the gray value distribution area, extracts the gray value change rate between adjacent frames, and constructs the acoustic fingerprint amplitude trend and thermal field temperature rise trend. Based on the acoustic amplitude trend and the thermal field temperature rise trend, the response feature extraction module calculates the difference in signal rise rate in each time period, arranges the difference in time order and performs inter-group comparison to obtain dual-channel comparison results. Based on the dual-channel comparison results, the coupling difference generation module identifies change points in a continuous time period, calculates and classifies the response intervals between change points, and combines the difference in change intensity between adjacent time periods to perform offset amplitude accumulation and construct a difference trajectory. Based on the difference trajectory, the abnormal phase identification module identifies continuous offset time periods, performs synchronous judgment by comparing the equipment speed change information within the time period, filters time periods that meet preset speed fluctuation conditions, and generates phase identification content. The maintenance operation triggering module performs a position cross-comparison based on the expansion trend of the gray-scale change area of ​​the thermal field image, determines whether the cross-range meets the operation conditions, and outputs the operation command for predictive maintenance of industrial equipment according to the change level of the cross-range area. The maintenance operation triggering module includes: The grayscale expansion judgment submodule calls the grayscale distribution of the thermal field image of the corresponding time frame according to the time period index in the stage identifier content, extracts the boundary contour of the grayscale change area in each frame, calculates the boundary expansion direction and area increment of the region in adjacent frames, aggregates and judges the region expansion behavior in continuous time periods, and obtains the grayscale expansion trend dataset. The rule cross-filtering submodule performs position coordinate cross-judgment based on the spatial positioning information of the corresponding time index in the gray-scale expansion trend dataset and the stage identifier content, extracts the spatial range of the area cross-section, compares it with the position overlap threshold in the preset operation rules, filters the data segments that meet the conditions, and obtains the rule matching cross-section area set. The maintenance instruction generation submodule calls the rule to match the expansion level value of the cross area in the cross area set, and retrieves the corresponding control instruction from the control parameter set according to the control level mapping relationship corresponding to the level value. It integrates the operation number, action type and execution parameters of the instruction to generate the operation command for predictive maintenance of industrial equipment.

2. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that: The acoustic signature amplitude trend and thermal field temperature rise trend include amplitude change feature points, amplitude difference between adjacent points, image grayscale distribution features, and grayscale change rate index. The dual-channel comparison results include rise rate difference sequence, time rhythm difference parameter, and inter-channel response difference value. The difference trajectory includes abrupt change point location sequence, response interval category label, and offset amplitude superposition value. The stage identification content includes continuous offset time period identifier, rotation speed fluctuation corresponding time period, and abnormal response stage number. The predictive maintenance operation commands for industrial equipment include location cross-matching results, grayscale change level classification, and maintenance control command type.

3. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that: The identification of change points in a continuous time period refers to locating the time position where the rate of change exceeds a set threshold by monitoring the abrupt change rate of dual-channel change values ​​in the time series.

4. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that: The time period that meets the preset speed fluctuation conditions refers to the time period that exceeds the speed fluctuation benchmark value by judging the difference between the equipment speed sequence within a continuous offset time period.

5. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that, The acoustic-thermal signal acquisition module includes: The acoustic signature amplitude extraction submodule acquires acoustic signature sensing signals from industrial equipment during operation, processes them in segments according to fixed time periods, monitors the amplitude change positions in each segment of the sound pressure value sequence, extracts the sound pressure values ​​at adjacent time points and calculates the difference, and generates a sound pressure amplitude difference sequence. The thermal field grayscale extraction submodule acquires thermal imaging image frames for the corresponding time period, calls the grayscale matrix of the image, locates the boundary of the grayscale change region, extracts the grayscale value of the pixels in the region, calculates the grayscale change rate between adjacent frames, and generates a local grayscale change rate sequence. The acoustic-thermal trend construction submodule calls the time index of the sound pressure amplitude difference sequence and the local grayscale change rate sequence to match the change trend within the same time period, performs joint normalization on the two types of data, establishes a time change curve, and obtains the acoustic amplitude trend and thermal field temperature rise trend.

6. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that, The response feature extraction module includes: The rise rate construction submodule extracts the values ​​of three consecutive time points within each time segment based on the acoustic amplitude trend and the thermal field temperature rise trend. It subtracts the differences between the last two points and the first point pairwise to obtain the continuous rise rate difference sequence of each signal segment and obtains the channel rise rate difference matrix. The rhythm difference analysis submodule calls the voiceprint channel and thermal field channel data in the channel rise rate difference matrix, pairs the rise rate differences in the same time period according to the time index, constructs the paired data into a two-dimensional group data set, and performs group aggregation according to the difference range of each group of data to obtain rhythm difference data groups; The dual-channel comparison generation submodule extracts the rhythm change values ​​of the voiceprint and thermal field channel data pairs sequentially according to the time series of each group in the rhythm difference data group, and plots the channel change trend curves in the order of the group sequence. It calculates the inter-group difference value sequence through the deviation of the trend line and establishes the dual-channel comparison result.

7. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that, The coupling difference generation module includes: The mutation point identification submodule monitors the differences in channel change values ​​at adjacent time points based on the continuous time series in the dual-channel comparison results, and marks the time positions where the change rate is greater than the set mutation threshold according to the change gradient change rate. It then extracts the time periods between consecutive markers and aggregates the boundary index to obtain the change mutation point sequence. The response interval classification submodule calculates the time response interval value between two points based on the index of each group of adjacent mutation points in the sequence of changes in mutation points, and classifies all mutation groups into the corresponding category interval according to the interval length, counts the number of mutation segments and the average time span in each category, and obtains the response interval classification result set. The difference trajectory construction submodule calls the time index of each mutation group in the response interval classification result set, calculates the difference of the dual-channel change value of the corresponding time period, performs amplitude superposition of the difference between adjacent segments in time order, summarizes the total difference of the classification and the trend line index, and establishes the difference trajectory.

8. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging according to claim 1, characterized in that, The abnormal phase identification module includes: The continuous offset detection submodule extracts the amplitude change gradient between adjacent segments based on the continuous offset segments in the difference trajectory, judges the continuity according to the set offset duration threshold, aggregates and marks the index segments that meet the continuity condition, and obtains the continuous offset time period set. The speed fluctuation correlation submodule calls the index of each time period in the continuous offset time period set, and synchronously obtains the equipment speed sequence within the corresponding time period. It calculates the difference between any two adjacent sampling points in each speed sequence and determines whether it exceeds the speed fluctuation benchmark value, and obtains the fluctuation correlation time period sequence. The stage identifier generation submodule filters out the time periods that appear in the difference trajectory and are accompanied by speed fluctuations based on the time period position in the fluctuation-related time period sequence. The filtered time periods are then structured according to the start and end indices to establish stage identifier content.

9. The predictive maintenance system for industrial equipment based on multimodal fusion of acoustic signature and thermal imaging as described in claim 1, characterized in that: The location overlap threshold refers to the cross-area standard used to determine whether the degree of overlap between the grayscale expansion area and the stage positioning area in spatial coordinates reaches the threshold for triggering maintenance conditions.

Images