Millimeter wave image fusion recognition method and system based on artificial intelligence

Abstract

The application discloses a millimeter wave and acoustic image fusion recognition method and system based on artificial intelligence, relates to the field of artificial intelligence, and comprises the following steps: millimeter wave features and acoustic features of a to-be-recognized region are extracted through at least two preset multi-level representation layers; heterogeneous coupling is performed on features of each level to obtain multi-source coupled features; basic recognition features containing a to-be-selected device operating state and a preset recognition type basic prior probability mapping are acquired; dependency modeling is performed on the basic recognition features based on the multi-source coupled features to obtain coupled recognition features; and finally, a target state corresponding to the preset recognition type is determined in the to-be-selected device operating state according to the coupled recognition features, and a current recognition result is output. Through multi-level heterogeneous feature coupling and feature dependency modeling, the method realizes deep fusion of multi-modal information, and effectively improves the recognition accuracy and environmental adaptability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a millimeter-wave audio-visual fusion recognition method and system based on artificial intelligence. Background Technology

[0002] Accurate identification of equipment operating status is of great significance in fields such as industrial monitoring and security protection. Traditional methods often rely on a single sensor, such as millimeter-wave radar or acoustic detectors, to obtain single-modal features, which suffers from the problem of information partiality and is susceptible to environmental interference such as noise and obstruction, leading to an increased false detection rate. Summary of the Invention

[0003] The purpose of this invention is to provide a millimeter-wave acoustic-image fusion recognition method and system based on artificial intelligence.

[0004] In a first aspect, embodiments of the present invention provide a millimeter-wave acoustic-image fusion recognition method based on artificial intelligence, comprising:

[0005] Based on at least two pre-defined multi-level characterization layers, corresponding millimeter-wave features and acoustic features are extracted for the region to be identified.

[0006] Based on the at least two multi-level characterization layers, heterogeneous feature coupling is performed on the corresponding millimeter-wave features and acoustic features respectively to obtain the multi-source coupling features associated with the at least two multi-level characterization layers respectively;

[0007] Obtain the basic identification features of the region to be identified, wherein the basic identification features include a basic prior probability mapping between the operating states of at least two candidate devices and each preset identification type, which is pre-defined for the region to be identified;

[0008] Based on each acquired multi-source coupling feature, feature dependency modeling is performed on the basic identification features to obtain coupled identification features;

[0009] Based on the coupling recognition features, the target device operating state corresponding to the corresponding preset recognition type is determined among the at least two candidate device operating states, so as to obtain the current recognition result of the area to be recognized.

[0010] In a second aspect, embodiments of the present invention provide a server system, including a server, the server being used to execute the method described in the first aspect.

[0011] Compared to existing technologies, the beneficial effects provided by this invention include: The invention discloses an artificial intelligence-based millimeter-wave acoustic-image fusion recognition method and system, relating to the field of artificial intelligence. The method includes: extracting millimeter-wave features and acoustic features of the region to be recognized through at least two pre-defined multi-level representation layers; heterogeneously coupling features at each level to obtain multi-source coupled features; acquiring basic recognition features containing a priori probability mapping between the operating state of the candidate device and a preset recognition type; performing dependency modeling on the basic recognition features based on the multi-source coupled features to obtain coupled recognition features; and finally, determining the target state corresponding to the preset recognition type in the operating state of the candidate device based on the coupled recognition features, and outputting the current recognition result. This method achieves deep fusion of multi-modal information through multi-level heterogeneous feature coupling and feature dependency modeling, effectively improving the accuracy and environmental adaptability of recognition. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating the steps of the millimeter-wave acoustic-image fusion recognition method based on artificial intelligence provided in an embodiment of the present invention;

[0014] Figure 2 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0016] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0017] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the artificial intelligence-based millimeter-wave acoustic-image fusion recognition method provided in this embodiment. The following is a detailed description of the artificial intelligence-based millimeter-wave acoustic-image fusion recognition method.

[0018] Step S201: Based on at least two pre-defined multi-level characterization layers, extract the corresponding millimeter-wave features and acoustic features for the region to be identified.

[0019] Step S202: Based on the at least two multi-level characterization layers, heterogeneous feature coupling is performed on the corresponding millimeter-wave features and acoustic features respectively to obtain the multi-source coupling features associated with the at least two multi-level characterization layers respectively;

[0020] Step S203: Obtain the basic identification features of the region to be identified. The basic identification features include a basic prior probability mapping between the operating states of at least two candidate devices and each preset identification type, which is pre-set for the region to be identified.

[0021] Step S204: Based on each acquired multi-source coupling feature, perform feature dependency modeling on the basic identification features to obtain coupled identification features;

[0022] Step S205: Based on the coupling recognition features, determine the target device operating state corresponding to the corresponding preset recognition type among the at least two candidate device operating states, so as to obtain the current recognition result of the area to be identified.

[0023] This embodiment uses the real-time monitoring of the gearbox operation status of a 2.5MW wind turbine generator set in a wind farm as an example to detail the implementation process of the millimeter-wave acoustic-image fusion recognition method based on artificial intelligence. The area to be identified is the gearbox installation compartment (spatial range 3m×2m×2m). The server collects millimeter-wave source data (including gearbox housing vibration displacement and internal gear meshing distance-Doppler features) through a 77GHz millimeter-wave radar (sampling rate 8Hz, distance resolution 0.1m, velocity resolution 0.05m / s) deployed on the inner wall of the compartment. It also collects acoustic source data (including gear meshing sound, bearing noise, and other acoustic signals) through an array microphone (8 channels, sampling rate 48kHz, frequency range 50Hz-10kHz). The preset recognition target is to identify four candidate equipment operation states of the gearbox by fusing millimeter-wave and acoustic features: "normal meshing", "slight tooth surface wear", "excessive bearing clearance", and "broken gear teeth". These are then classified into three preset recognition types: "normal", "early warning", and "fault", providing decision support for wind farm operation and maintenance. The server acts as the execution entity, and the specific execution steps are as follows:

[0024] The server is pre-configured with two multi-level representation layers: a low-level representation layer (focusing on the basic time-domain / frequency-domain features of the signal) and a high-level representation layer (focusing on the dynamic correlation features of the device structure). For the millimeter-wave source data and acoustic source data of the region to be identified, the server performs feature extraction at both levels, as detailed below:

[0025] The server integrates millimeter-wave source data (range-Doppler matrix, dimension 1024×512) and acoustic source data (8-channel time-domain waveform, dimension 8×8192) collected at the same time (timestamp error ≤ 5ms) into a raw sensing tensor (dimension 2×1024×512×8×8192, where "2" represents millimeter-wave / acoustic modes). For the low-level representation layer, the server divides the raw sensing tensor into two sensing sub-blocks according to a preset level offset (the low-level offset is set to 0.3, i.e., retaining 30% of the detail resolution of the original signal to focus on basic features): the first sensing sub-block is the sub-matrix of the range dimension 0-307 (first 30%) and Doppler dimension 0-153 (first 30%) of the millimeter-wave source data (focusing on the near-field vibration features of the gearbox); the second sensing sub-block is the time-domain subsequence of the time dimension 0-2457 (first 30%) of the acoustic source data (focusing on the acoustic features excited by the initial vibration). The server projects the basic feature points (distance-Doppler value, echo intensity) of the first sensing sub-block onto the "distance-Doppler-intensity" feature domain, and converts the distance-Doppler matrix into a distance-frequency matrix through Fourier transform to obtain the basic millimeter-wave features (dimension 307×153×3, including distance, frequency, and intensity features); and projects the basic feature points (time-domain amplitude) of the second sensing sub-block onto the "time-frequency" feature domain through short-time Fourier transform to obtain the basic acoustic features (dimension 2457×256, including time-frequency amplitude features).

[0026] For basic millimeter-wave features and basic acoustic features, the server performs dependency modeling of feature points separately. Taking basic millimeter-wave features as an example, the server divides the basic millimeter-wave features (307×153×3) into 61×30 local feature units (each unit is a set of 5×5×3 feature points, covering adjacent distance, frequency, and intensity information) according to the preset local observation level (the lower level is set to "5×5 grid"). For each local feature unit, the server performs intra-feature dependency modeling through a self-attention mechanism: it calculates the attention weights of 25 feature points within the unit (based on prior probability mappings trained from historical gearbox vibration data, such as the dependency weights of the central feature point on the surrounding 4 feature points being 0.2, 0.18, 0.15, and 0.12, respectively), and obtains the aggregated feature within the unit after weighted fusion; at the same time, it performs inter-feature dependency modeling on adjacent local feature units (such as the current unit and the units to the right and below) through a 3×3 convolution kernel, extracts the gradient changes at the edges of the feature units (reflecting the propagation trend of vibration in the distance-frequency dimension), and superimposes the intra-feature dependency modeling results with the inter-feature dependency modeling results at a weight of 0.7:0.3 to obtain low-level millimeter-wave features (dimensions 61×30×64, where 64 is the number of feature channels after fusion).

[0027] For the basic acoustic features (2457×256), the server is divided into 122×12 local feature units according to "20ms time window + 20Hz frequency interval". The temporal dependence within the local feature units is captured by an LSTM network (hidden layer dimension 128) (such as the periodic amplitude change of gear meshing sound). The frequency dependence between units is captured by a graph neural network (nodes are frequency components, and edge weights are frequency energy transfer probabilities) (such as the energy correlation between meshing frequency and harmonic components). After superposition, the low-level acoustic features (dimension 122×12×64) are obtained.

[0028] For the high-level representation layer, the server repeats the above process, with the layer offset set to 0.6 (preserving 60% detail resolution to focus on structural correlation). The original perceptual tensor is divided into three perceptual sub-blocks (covering the vibration and acoustic regions corresponding to the input shaft, intermediate shaft, and output shaft of the gearbox). The basic modal representation tensor is projected onto the "structural position-dynamic response" feature domain (such as the vibration acceleration and corresponding acoustic radiation features of the input shaft position). Feature point dependency modeling is performed through a Transformer encoder (8 multi-head attention heads) (capturing cross-position vibration-acoustic correlation caused by shaft linkage). Finally, high-level millimeter-wave features (reflecting the dynamic coupling features of internal components of the gearbox) and high-level acoustic features (reflecting the physical correlation features of structural vibration and acoustic radiation) are obtained.

[0029] The server performs heterogeneous feature coupling (cross-modal feature fusion) on the corresponding millimeter-wave and acoustic features for both the low-level and high-level representation layers, respectively, to obtain multi-source coupled features associated at each level. Taking the low-level representation layer as an example, the specific process is as follows:

[0030] The server retrieves low-level millimeter-wave features (61×30×64) and low-level acoustic features (122×12×64) from memory, aligns them with timestamps (based on GPS time synchronization, with an error ≤2ms), and then uses them as the feature pairs to be coupled. The server first performs heterogeneous coupling based on millimeter-wave feature points: it traverses each millimeter-wave feature point in the low-level millimeter-wave features (a total of 61×30×64=117120 points, each point is a 64-dimensional vector, representing the vibration feature at a certain distance-frequency position). Taking one feature point P (coordinates (i,j,k), corresponding to distance i, frequency j, and the k-th channel feature) as an example, the server calls the pre-trained first correlation model parameters (trained through historical gearbox fault data, containing the prior probability mapping between millimeter-wave feature points and acoustic feature points, such as "the correlation probability between the gear meshing vibration feature point at a frequency of 500Hz and the acoustic feature point at a sound spectrum of 500Hz is 0.85"), and calculates the first correlation information (value range [0,1]) between P and all acoustic feature points (122×12×64=92928 points) in the low-level acoustic features. The server sets the first probability condition to "correlation ≥ 0.6", and selects four acoustic feature points Q1, Q2, Q3, and Q4 that meet the condition (corresponding to sound spectra of 500Hz, 505Hz, 495Hz, and 510Hz, respectively, which are close to the vibration frequency of P). Then, it calls the first coupling weight parameter (which includes the prior probability mapping between P and Q1-Q4, such as "the probability of P vibrating at the same frequency as Q1 is 0.92, and the probability of P vibrating at the same frequency as Q2 is 0.78") to calculate the dependence of each acoustic feature point on P, and obtains the first coupling coefficient of Q1 as 0.92, Q2 as 0.78, Q3 as 0.72, and Q4 as 0.65. The server performs a weighted summation of the feature vectors Q1-Q4 based on these four first coupling coefficients (0.92×Q1+0.78×Q2+0.72×Q3+0.65×Q4), and then multiplies it element-wise with the feature vector of P (highlighting cross-modal common features) to obtain the first heterogeneous coupling feature point P' corresponding to P. After traversing all millimeter-wave feature points, the server aggregates them to obtain the first multi-source coupling feature (dimension 61×30×64).

[0031] Subsequently, the server performs heterogeneous coupling based on acoustic feature points: it traverses each acoustic feature point Q (92928 points) in the low-level acoustic features, calls the second correlation model parameters (prior probability mapping between acoustic feature points and millimeter-wave feature points, such as "the correlation between the bearing noise feature point with a sound spectrum of 800Hz and the millimeter-wave feature point with vibration acceleration of 20m / s² is 0.8"), calculates the second correlation information between Q and the millimeter-wave feature points, selects 3 millimeter-wave feature points that meet the second probability condition (correlation ≥ 0.55), calculates the coupling coefficient through the second coupling weight parameter (such as "the prior probability between Q and the vibration acceleration feature point of 20m / s² is 0.88"), and obtains the second heterogeneous coupling feature point Q' corresponding to Q after weighted fusion, and aggregates to obtain the second multi-source coupling feature (dimension 122×12×64). The server concatenates the first and second multi-source coupling features along the channel dimension (dimension (61×30+122×12)×64=3192×64) to obtain the low-level multi-source coupling features. For the high-level representation layer, the server repeats the above process (adjusting the correlation model parameters to structural dynamic association priors, such as "the correlation probability between the input axis vibration feature point and the output axis acoustic feature point is 0.75") to obtain the high-level multi-source coupling features (dimension 4256×64).

[0032] The server retrieves the basic identification features of the area to be identified from the local database. These features include a basic prior probability mapping between four candidate equipment operating states and three preset identification types. The prior probabilities are based on five years of gearbox operation and maintenance data from the wind farm: the prior probability of "normal meshing" belonging to the "normal" type is 0.92, the probability of belonging to the "warning" type is 0.07, and the probability of belonging to the "fault" type is 0.01; the probability of "slight tooth wear" belonging to the "normal" type is 0.15, the probability of belonging to the "warning" type is 0.80, and the probability of belonging to the "fault" type is 0.05; the probability of "excessive bearing clearance" belonging to the "normal" type is 0.08, the probability of belonging to the "warning" type is 0.75, and the probability of belonging to the "fault" type is 0.17; and the probability of "broken gear tooth" belonging to the "normal" type is 0.02, the probability of belonging to the "warning" type is 0.10, and the probability of belonging to the "fault" type is 0.88.

[0033] The server employs low-level and high-level multi-source coupling features, performing a two-stage progressive optimization process on the basic identification features, and combining this with updates on the correlation between the operating states of candidate devices to obtain the coupled identification features. The specific process is as follows:

[0034] The first stage of optimization is based on low-level multi-source coupling features: The server treats the basic recognition features as the initial feature matrix (4×3 dimensions, corresponding to 4 states × 3 types of probabilities), and the low-level multi-source coupling features (3192×64) as the optimization input. For each basic feature element in the basic recognition features (e.g., the probability of "normal meshing-normal" is 0.92), the server, based on the prior probability mapping between this element and the coupling feature elements in the low-level multi-source coupling features (3192×64 64-dimensional vectors, such as features reflecting the consistency of vibration-sound frequencies) (obtained through fusion model training, e.g., "the higher the frequency consistency feature value, the higher the probability of 'normal meshing-normal' should be"), selects 6 coupling feature elements (reflecting the consistency of vibration-sound in different frequency bands) that meet the coupling probability condition (association probability ≥ 0.75), and calculates the probability of each coupling feature element. The weighted features (based on prior probability mapping, such as a weight of 0.9 for the highest consistency feature and 0.85 for the next highest) are weighted and summed (0.9×C1+0.85×C2+0.8×C3+0.75×C4+0.7×C5+0.65×C6=0.88) of the normalized feature values ​​(range [0,1]) of the six coupled feature elements. This sum is then fused with the basic feature elements at a weight of 0.6:0.4 (optimized value = basic value × 0.6 + fused value × 0.4) to obtain the optimized feature point 0.92×0.6+0.88×0.4=0.904 for "normal meshing-normal". The server performs the above operation on all 12 basic feature elements to obtain the first-stage optimized recognition features.

[0035] The second-stage optimization is based on high-level multi-source coupling features: The server takes the first-stage optimization identification features as input and repeats the above optimization process using high-level multi-source coupling features (reflecting dynamic structural correlations, such as the linkage between excessive bearing clearance and abnormal gear meshing) to obtain the second-stage optimization identification features. Subsequently, the server calculates the equipment operating status correlation information for four candidate equipment operating states (based on historical co-occurrence probabilities, such as the co-occurrence probability of "excessive bearing clearance" and "slight tooth wear" being 0.65, and the co-occurrence probability of "broken gear teeth" and "excessive bearing clearance" being 0.8), and calculates the equipment operating status importance coefficient for each state based on the correlation ("normal meshing" 0.1, "slight tooth wear" 0.2, "excessive bearing clearance" 0.4, "broken gear teeth" 0.3). The server linearly superimposes the second-stage optimization identification features (coupling probability = Σ(state importance coefficient × optimization probability)) to obtain coupled identification features, where the coupling probability for the "normal" type is 0.12, for the "warning" type is 0.63, and for the "fault" type is 0.25.

[0036] The server obtains the identification feature parameters corresponding to each preset identification type (including the confidence scores of historical device operating status and type, such as the identification feature parameters for the "warning" type: "slight tooth surface wear" with a confidence score of 0.82, and "excessive bearing clearance" with a confidence score of 0.78). Based on these parameters, the server performs feature analysis on the coupled identification features: the "warning" type has the highest coupling probability (0.63), and its main associated candidate states are "slight tooth surface wear" and "excessive bearing clearance". The server further calculates the probability distribution of these two states belonging to the "warning" type: based on the second-stage optimized identification features, the probability of "slight tooth surface wear" belonging to the "warning" type is 0.81, and the probability of "excessive bearing clearance" belonging to the "warning" type is 0.79. After weighting by the state importance coefficients (0.2 and 0.4), the probability is 0.81×0.2+0.79×0.4=0.796. The server sets the preset probability distribution condition to "peak probability ≥ 0.7". The probability of "excessive bearing clearance" belonging to the "warning" type is 0.79, which meets the condition. Therefore, the target equipment operating status is determined to be "excessive bearing clearance", corresponding to the preset identification type "warning".

[0037] The server outputs the current identification result for the area to be identified (gearbox of the 2.5MW wind turbine unit): "The equipment operating status is that the bearing clearance is too large, the identification type is warning, and it is recommended to adjust the bearing preload". The result is then pushed to the wind farm operation and maintenance platform to achieve early warning of gearbox failure.

[0038] Through the above process, the server achieves multi-level fusion of millimeter wave and acoustic features, and optimizes recognition accuracy by combining prior knowledge, providing a reliable technical solution for intelligent monitoring of industrial equipment.

[0039] In this embodiment of the invention, the area to be identified includes millimeter-wave source data and acoustic source data collected for the same monitoring area;

[0040] The step of extracting corresponding millimeter-wave features and acoustic features for the region to be identified based on at least two pre-defined multi-level characterization layers can be implemented through the following example.

[0041] Based on a pre-defined multi-level characterization layer, the millimeter-wave source data and the acoustic source data are respectively transformed to obtain the corresponding basic millimeter-wave features and basic acoustic features.

[0042] For the basic millimeter-wave features and the basic acoustic features, feature point dependency modeling is performed respectively to obtain the corresponding millimeter-wave features and acoustic features.

[0043] In this embodiment of the invention, exemplarily, this embodiment takes the monitoring of the operating status of the reducer of a welding robot in an automobile manufacturing plant by a server as an example to describe in detail the extraction process of millimeter-wave features and acoustic features. The area to be identified is the reducer mounting cavity (spatial range 0.5m×0.5m×0.8m). The server collects millimeter-wave source data (dimension 512×256, corresponding to the distance-Doppler matrix, including vibration displacement and rotational speed characteristics of reducer gear meshing) through a deployed 77GHz millimeter-wave radar (sampling rate 10Hz, range resolution 0.05m). Acoustic source data (dimension 4×8192, corresponding to the time domain waveform, including gear meshing sound and bearing noise characteristics) is collected through a 4-channel microphone array (sampling rate 44.1kHz, frequency band 100Hz-5kHz). The server pre-configures two multi-level representation layers: a low-level representation layer (focusing on the basic time-domain / frequency-domain features of the signal) and a high-level representation layer (focusing on the structural correlation features of the reducer gear-bearing structure). Feature extraction is performed separately for the millimeter-wave source data and acoustic source data of the same monitoring area.

[0044] For the low-level representation layer, the server first performs a hierarchical transformation on the millimeter-wave source data and acoustic source data to obtain the basic modal representation tensor. For the millimeter-wave source data, the server projects each matrix element (distance cell, Doppler cell, echo intensity value) in the distance-Doppler matrix (512×256) to the three-dimensional feature space according to the preset "distance-Doppler-intensity" feature domain mapping rule of the low-level representation layer, to obtain the basic millimeter-wave features (dimension 512×256×3, including distance, Doppler, and intensity features); for the acoustic source data, the server converts the 4-channel time-domain waveform (4×8192) into a time-frequency amplitude matrix through short-time Fourier transform (window length 512, overlap rate 50%), and projects it to the "time-frequency domain" feature domain to obtain the basic acoustic features (dimension 16×256×4, including time frame, frequency point, and channel amplitude features).

[0045] Subsequently, the server performs dependency modeling of feature points for both the basic millimeter-wave features and the basic acoustic features. Taking the basic millimeter-wave features as an example, the server divides the basic millimeter-wave features (512×256×3) into 170×85 local feature units (each unit contains 3×3×3=27 feature points, corresponding to adjacent distance, Doppler, and intensity value) according to the local observation level (3×3 grid unit) preset by the low-level representation layer. For each local feature unit, the server performs intra-feature dependency modeling through a self-attention mechanism: it calculates the attention weights of 27 feature points within the unit (based on prior probability mapping trained on historical decelerator normal operation data, such as the dependency weight of the central feature point on the surrounding 8 distance-Doppler adjacent points being 0.15-0.2), and weights and fuses them to obtain the aggregated features within the unit; at the same time, it performs inter-feature dependency modeling on adjacent local feature units (such as the current unit and the units on the right and below) through a 3×3 convolution kernel, extracts the gradient change features of the unit edge (reflecting the propagation trend of vibration in the distance-Doppler dimension), and superimposes the intra-feature dependency modeling results with the inter-feature dependency modeling results at a weight of 0.6:0.4 to obtain low-level millimeter-wave features (dimension 170×85×64, where 64 is the number of fused feature channels).

[0046] For the basic acoustic features (16×256×4), the server divides them into 8×128 local feature units (each unit contains 2×2×4=16 feature points, corresponding to adjacent time frames, frequency points, and channel amplitudes) according to the local observation level (2×2 time-frequency units) preset by the low-level representation layer. Intra-feature dependency modeling is performed through an LSTM network (capturing the acoustic amplitude change trend between time frames within a unit), and inter-feature dependency modeling is performed through a graph neural network (using frequency points as nodes and edge weights as frequency energy transfer probabilities to capture the energy correlation between adjacent frequency units). The results are then superimposed to obtain the low-level acoustic features (dimension 8×128×64).

[0047] For the high-level representation layer, the server repeats the above process: In the layer transformation stage, the millimeter-wave source data is projected onto the "gear meshing period-vibration acceleration" feature domain (the meshing period is calculated based on the number of teeth of the reducer gear, and the vibration acceleration features within the period are extracted), and the acoustic source data is projected onto the "structural radiation-frequency band energy" feature domain (the energy features of a specific frequency band are extracted based on the acoustic radiation characteristics of the reducer housing), thus obtaining the basic millimeter-wave features and basic acoustic features; In the dependency modeling stage, the local observation level is set as "gear-bearing associated unit" (each unit contains gear vibration feature points and corresponding bearing acoustic feature points), and the physical association (internal dependency) of gear-bearing feature points within the unit is modeled through the Transformer encoder (multi-head attention mechanism), and the linkage features (inter-dependency) of different gear sets between units are modeled through a fully connected network, thus finally obtaining the high-level millimeter-wave features and high-level acoustic features.

[0048] Through the above process, the server extracts millimeter-wave features and acoustic features containing basic signal features and structural correlation features from millimeter-wave source data and acoustic source data respectively, based on two multi-level representation layers, providing input for subsequent heterogeneous feature coupling.

[0049] In this embodiment of the invention, the step of performing hierarchical transformation on the millimeter-wave source data and the acoustic source data according to a pre-defined multi-level characterization layer to obtain the corresponding basic millimeter-wave features and basic acoustic features can be implemented through the following example.

[0050] Based on a pre-defined multi-level representation layer, the millimeter-wave source data and the acoustic source data are used as the original sensing tensor. Based on the level offset between the level of the original sensing tensor and the multi-level representation layer, the original sensing tensor is divided into at least two sensing sub-blocks.

[0051] The basic feature points included in each of the at least two sensing sub-blocks are projected into the corresponding feature domains to obtain the corresponding basic modal representation tensors, wherein the basic modal representation tensors are basic millimeter-wave features or basic acoustic features.

[0052] In this embodiment of the invention, exemplarily, this embodiment takes the monitoring of the operating status of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to describe in detail the acquisition process of basic millimeter-wave characteristics and basic acoustic characteristics. The area to be identified is the spindle box mounting cavity (spatial range 0.6m×0.4m×0.7m). The server collects millimeter-wave source data (dimension 1024×512, corresponding to the distance-Doppler matrix, including spindle gear meshing vibration displacement and bearing speed characteristics) through a 77GHz millimeter-wave radar (sampling rate 12Hz, distance resolution 0.03m, Doppler resolution 0.02m / s) deployed on the inner wall of the cavity. Acoustic source data (dimension 2×16384, corresponding to 2-channel time-domain waveform, including gear meshing sound and bearing abnormal noise characteristics) through a 2-channel microphone array (sampling rate 48kHz, frequency band 200Hz-8kHz). The server is pre-configured with a low-level characterization layer (focusing on the fundamental time-domain / frequency-domain features of the signal to capture the fundamental correlation between near-field vibration and acoustics of the spindle box). For millimeter-wave source data and acoustic source data in the same monitoring area, a hierarchical transformation is performed to obtain the fundamental modal characterization tensor.

[0053] The server first integrates the millimeter-wave source data and acoustic source data acquired at the same time (timestamp error ≤ 3ms) into a raw sensing tensor. The millimeter-wave source data is a range-Doppler matrix (1024×512), with each element containing a range cell (0-0.6m, 1024 cells), a Doppler cell (-2m / s~2m / s, 512 cells), and the corresponding echo intensity value. The acoustic source data is a 2-channel time-domain waveform (2×16384), with each channel containing 16384 time-domain sampling points (sampling interval 20.8μs) and the corresponding sound pressure amplitude value. The server integrates the two modal data into a raw sensing tensor (dimension: 2×1024×512×2×16384, where "2" represents both millimeter-wave and acoustic modes). The hierarchy of this tensor is defined as the detail resolution hierarchy of the original signal (a higher value indicates more complete detail preservation).

[0054] For the low-level representation layer, the server divides the sensing sub-blocks according to a preset level offset. The goal of the low-level representation layer is to capture the basic features of the core components of the spindle box (the front bearing and rear gear set of the spindle). The preset level offset is 0.3 (meaning that the level of the original sensing tensor needs to be reduced by 0.3 to retain 30% of the detail resolution of the core area). Based on this offset, the server divides the original sensing tensor into two sensing sub-blocks: Sub-block 1 corresponds to the front bearing area of ​​the spindle (the sub-matrix of distance unit 0-307 (first 30%, corresponding to 0-0.18m) and Doppler unit 128-384 (middle 50%, corresponding to -0.5m / s~0.5m / s) in the millimeter-wave source data; and the sub-sequence of time-domain sampling points 0-4915 (first 30%, corresponding to 0-0.102s) in the acoustic source data), focusing on the front... The low-speed vibration and initial acoustic radiation characteristics of the end bearing; sub-block 2 corresponds to the rear gear set region of the main shaft (the sub-matrix of distance units 716-1024 (last 30%, corresponding to 0.42-0.6m) and Doppler units 128-384 in the millimeter wave source data; the sub-sequence of time-domain sampling points 11468-16384 (last 30%, corresponding to 0.239-0.341s) in the acoustic source data), focusing on the meshing vibration and delayed acoustic radiation characteristics of the rear gear set.

[0055] Subsequently, the server projects the basic feature points of each perceptual sub-block onto the corresponding feature domain to obtain the basic modal representation tensor. For sub-block 1 of the millimeter-wave source data (a 307×256 matrix containing distance, Doppler, and echo intensity), the server converts the distance unit of each matrix element into an actual distance value (0-0.18m), the Doppler unit into a vibration velocity value (-0.5m / s~0.5m / s), and the echo intensity value into a vibration acceleration value using the calibration formula (vibration acceleration = echo intensity × 0.02g, where g is gravitational acceleration) to obtain the basic millimeter-wave feature of sub-block 1 (dimension 307×256×3, containing distance (m), vibration velocity (m / s), and vibration acceleration (g)). The millimeter-wave source data of sub-block 2 (a 308×256 matrix) is projected according to the same rules to obtain another sub-tensor of the basic millimeter-wave feature (dimension 308×256×3). The two are then spliced ​​together to form the complete basic millimeter-wave feature (dimension 615×256×3).

[0056] For sub-block 1 of the acoustic source data (2×4915 time-domain subsequence), the server, according to the "time-frequency-sound pressure level" feature domain mapping rules preset by the lower-level representation layer, transforms the time-domain subsequence into a time-frequency matrix through short-time Fourier transform (window function Hamming, window length 1024, overlap rate 60%). The time frame is divided into 16 frames (each corresponding to 6.25ms), and the frequency points are divided into 512 points (200Hz-8kHz). The sound pressure amplitude is then expressed using the formula (…). Sound pressure level = 20lg(amplitude / reference sound pressure, reference sound pressure 20μPa) is converted to sound pressure level (dB) to obtain the basic acoustic features of sub-block 1 (dimension 2×16×512, including channels, time frames, frequency points and corresponding sound pressure levels); the acoustic source data of sub-block 2 (2×4916 time-domain subsequences) are processed according to the same rules to obtain another sub-tensor of the basic acoustic features (dimension 2×16×512), and the two are spliced ​​together to form the complete basic acoustic features (dimension 2×32×512).

[0057] Through the above hierarchical transformation process, the server divides the original sensing tensor into sensing sub-blocks focusing on the core region based on the low-level representation layer, and projects the basic feature points of the sub-blocks onto the feature domain with clear physical meaning, finally obtaining the basic millimeter-wave features and basic acoustic features that reflect the basic vibration and acoustic characteristics of the spindle box, providing input for subsequent feature dependency modeling.

[0058] In this embodiment of the invention, the step of performing feature point dependency modeling on the basic millimeter-wave features and the basic acoustic features to obtain the corresponding millimeter-wave features and acoustic features can be implemented through the following example.

[0059] Based on a pre-defined local observation hierarchy, a basic modality representation tensor is divided into at least two local feature units; wherein each local feature unit contains undetermined feature points in the basic modality representation tensor.

[0060] For each of the at least two local feature units, intra-feature dependency modeling is performed on the undetermined feature points, and inter-feature dependency modeling is performed on at least the at least two local feature units to obtain the corresponding modal representation tensor, wherein the modal representation tensor is a millimeter-wave feature or an acoustic feature.

[0061] In this embodiment of the invention, exemplarily, this embodiment takes the dependency modeling of the basic millimeter-wave features and basic acoustic features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the process of obtaining the modal representation tensor. The server has obtained the basic millimeter-wave features (dimension 615×256×3, including distance (0-0.6m), vibration velocity (-0.5m / s~0.5m / s), and vibration acceleration (g) features) and the basic acoustic features (dimension 2×32×512, including 2 channels, 32 time frames (6.25ms per frame), 512 frequency points (200Hz-8kHz), and corresponding sound pressure level (dB) features) through hierarchical transformation. The server performs internal and external dependency modeling on the two basic features respectively for the local observation level preset by the lower-level representation layer.

[0062] The server first divides local feature units according to a preset local observation level. For the basic millimeter-wave feature (615×256×3), the lower-level characterization layer focuses on the local vibration correlation between the spindle box bearing and gear, with a preset local observation level of "5×5 grid units" (corresponding to a local region in the distance-velocity plane). The server divides the distance dimension (615 units) of the basic millimeter-wave feature into 123 grids (615=5×123) at 5 units / grid, and the velocity dimension (256 units) into 52 grids (256≈5×52) at 5 units / grid, resulting in a total of 123×52=6396 local feature units. Each local feature unit contains 5×5×3=75 undetermined feature points (corresponding to 5 distance units, 5 velocity units, and 3 acceleration values), covering the vibration features of adjacent distance-velocity regions.

[0063] Subsequently, intra-feature dependency modeling is performed: for 75 undetermined feature points in each local feature unit, the server calls a pre-trained self-attention model (3 attention heads, trained based on historical spindle vibration data) to calculate the inter-point dependency weights (e.g., the weight of the central feature point on the four surrounding adjacent points is 0.18-0.22, and the weight on the velocity adjacent points is 0.15-0.19). After weighted fusion, the intra-unit aggregated features are obtained (dimension 1×64, 64 is the number of fusion channels).

[0064] Next, feature dependency modeling is performed: the server arranges 6396 local feature units in distance-velocity order, and uses a 3×3 convolution kernel (stride 1) to perform convolution operations on adjacent units (such as the current unit and the units on the right and below), extracting the gradient change features at the unit edges (reflecting the propagation trend of vibration in the distance-velocity plane), and superimposing the convolution results with the internal dependency aggregation features with a weight of 0.4:0.6 to obtain the millimeter-wave modal representation tensor (dimension 123×52×64), which is the final millimeter-wave feature.

[0065] For the basic acoustic features (2×32×512), the low-level representation layer focuses on the time-frequency correlation of the spindle box acoustic signal, with a preset local observation level of "4×4 time-frequency grid unit" (corresponding to the local time-frequency region of a single channel). The server divides the time frames (32 frames) of each channel into 8 grids at 4 frames / grid, and the frequency points (512 points) into 128 grids at 4 points / grid, resulting in 8×128=1024 local feature units per channel, and a total of 2048 units for 2 channels. Each local feature unit contains 4×4=16 undetermined feature points (corresponding to 4 time frames, 4 frequency points, and sound pressure level values), covering the acoustic features of adjacent time-frequency regions.

[0066] The intra-feature dependency modeling uses an LSTM network: for 16 undetermined feature points (arranged in chronological order) in each local feature unit, a 2-layer LSTM (hidden layer dimension 32) is used to capture the acoustic amplitude variation trend between time frames (such as the periodic fluctuation of gear meshing sound), and output the intra-unit aggregated feature (dimension 1×64).

[0067] The modeling of inter-feature dependencies adopts a graph neural network: each local feature unit is regarded as a graph node, and the node features are internal dependency aggregate features; edge weights are constructed based on the prior probability of frequency energy transfer (e.g., the energy transfer probability between 500Hz and 505Hz is 0.8), and the energy correlation features of adjacent nodes (time-frequency adjacent units) are extracted through graph convolutional layers (convolution kernel size 2), and superimposed with the internal dependency aggregate features (weight 0.3:0.7) to obtain the acoustic modality representation tensor (dimension 2×8×128×64), which is the final acoustic feature.

[0068] Through the above process, the server completes the dependency modeling of basic millimeter-wave features and acoustic features, and obtains millimeter-wave features (123×52×64) and acoustic features (2×8×128×64) that contain local correlation and propagation trends, respectively, providing structured modal data for subsequent heterogeneous feature coupling.

[0069] In this embodiment of the invention, the step of heterogeneously coupling the corresponding millimeter-wave features and acoustic features according to the at least two multi-level characterization layers to obtain the multi-source coupling features associated with the at least two multi-level characterization layers can be implemented through the following example.

[0070] For each multi-level characterization layer, obtain the millimeter-wave features and acoustic features that match the multi-level characterization layer;

[0071] In the millimeter-wave features, heterogeneous feature coupling is performed on each millimeter-wave feature point based on at least two acoustic feature points in the acoustic features to obtain the corresponding first multi-source coupling feature;

[0072] In the acoustic features, heterogeneous feature coupling is performed on each acoustic feature point based on at least two millimeter-wave feature points in the millimeter-wave features to obtain the corresponding second multi-source coupling feature;

[0073] Both the first multi-source coupling feature and the second multi-source coupling feature are used as the multi-source coupling features corresponding to the multi-level representation layer.

[0074] In this embodiment of the invention, exemplarily, this embodiment takes the heterogeneous coupling of low-level and high-level representation layer features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the process of obtaining multi-source coupling features. The server has extracted low-level millimeter-wave features (dimension 123×52×64, corresponding to vibration features of distance-velocity grid, 64 being the number of feature channels), low-level acoustic features (dimension 2×8×128×64, corresponding to acoustic features of 2-channel time-frequency grid), and high-level millimeter-wave features (dimension 85×42×64, focusing on the dynamic association of the spindle-gear structure) and high-level acoustic features (dimension 2×6×96×64, focusing on the radiated acoustic features of the structure) through the aforementioned steps. The following detailed description takes the low-level representation layer as an example. The processing logic of the high-level representation layer is similar, only the model parameters are adapted to the structural association features.

[0075] The server first acquires millimeter-wave and acoustic features that match the low-level representation layer. The basic physical relationship between the near-field vibration of the low-level focusing spindle box and acoustics (such as the consistency between the bearing vibration frequency and the corresponding sound radiation frequency) is established. The matched features are low-level millimeter-wave features (123×52×64) and low-level acoustic features (2×8×128×64). The timestamps of both are synchronized through the spindle encoder (error ≤2ms) to ensure spatiotemporal correlation.

[0076] The server iterates through each millimeter-wave feature point in the low-level millimeter-wave features (a total of 123×52×64=409,344 points, each a 64-dimensional vector representing the vibration characteristics of a certain distance-velocity grid cell). Taking millimeter-wave feature point P (coordinates (i,j,k), i=30 (distance grid), j=20 (velocity grid), k=15 (feature channel), corresponding to the 500Hz vibration characteristics of the spindle front bearing region) as an example:

[0077] The server calls the parameters of the pre-trained first correlation model (trained based on historical normal / abnormal vibration-acoustic data of the main axis, including the prior probability mapping between millimeter-wave feature points and acoustic feature points, such as "the correlation probability between 500Hz vibration feature points and acoustic feature points of 500Hz±20Hz in the sound spectrum is ≥0.8"), and calculates the correlation information (value range [0,1]) between P and all acoustic feature points (2×8×128×64=131,072 points) in the low-level acoustic features. The filtering condition is set to "correlation ≥0.7", and the server filters out 3 acoustic feature points that meet the condition: Q1 (channel 1, time frame 3, frequency point 62, corresponding to the 500Hz sound spectrum), Q2 (channel 1, time frame 3, frequency point 63, corresponding to the 508Hz sound spectrum), and Q3 (channel 2, time frame 3, frequency point 61, corresponding to the 492Hz sound spectrum), with correlations of 0.85, 0.78, and 0.72, respectively.

[0078] The server calls the first coupling weight parameter (which includes the prior probability mapping between P and Q1-Q3, such as the probability of Q1 and P resonating at the same frequency (0.92), the probability of Q2 being a harmonic (0.85), and the probability of Q3 crossing channels (0.78)) to calculate the coupling coefficients: Q1 is 0.92 × 0.85 = 0.782, Q2 is 0.85 × 0.78 = 0.663, and Q3 is 0.78 × 0.72 = 0.562 (coupling coefficient = prior probability × relevance). The 64-dimensional feature vectors of Q1-Q3 are weighted and summed according to the coupling coefficients (0.782Q1 + 0.663Q2 + 0.562Q3), and then superimposed with the feature vector of P with a weight of 0.5:0.5 (P' = 0.5P + 0.5 × (weighted sum)) to obtain the heterogeneous coupling feature point P' (64-dimensional) corresponding to P. After traversing all millimeter-wave feature points, the server aggregates the first multi-source coupled feature (dimension 123×52×64).

[0079] The server iterates through each acoustic feature point (131,072 points) in the low-level acoustic features. Taking acoustic feature point Q (channel 2, time frame 5, frequency point 100, feature channel 20, corresponding to the 800Hz acoustic spectrum feature of the spindle rear gear set) as an example:

[0080] The second correlation model parameters (prior probability mapping between acoustic feature points and millimeter-wave feature points, such as "correlation between 800Hz acoustic spectrum feature points and 800Hz±30Hz millimeter-wave feature points of gear meshing vibration ≥ 0.75") are called to calculate the correlation information between Q and millimeter-wave feature points. Two millimeter-wave feature points that meet the conditions are selected: P1 (distance 85, velocity grid 35, channel 18, corresponding to 800Hz vibration) and P2 (distance 86, velocity grid 35, channel 18, corresponding to 815Hz vibration), with correlations of 0.82 and 0.76, respectively.

[0081] The second coupling weight parameter (the prior probability mapping between Q and P1-P2, such as the gear meshing probability of P1 being 0.88 and the sideband probability of P2 being 0.82) is called to calculate the coupling coefficients: P1 is 0.88 × 0.82 = 0.722, and P2 is 0.82 × 0.76 = 0.623. The weighted sum of the feature vectors of P1-P2 (0.722P1 + 0.623P2) is then superimposed with the feature vector of Q with a weight of 0.5:0.5 to obtain the heterogeneous coupling feature point Q' (64 dimensions) corresponding to Q. After traversing all acoustic feature points, the aggregation yields the second multi-source coupling feature (dimension 2 × 8 × 128 × 64).

[0082] The server concatenates the first multi-source coupling feature (123×52×64) and the second multi-source coupling feature (2×8×128×64) according to the feature channel dimension (total dimension = 123×52 + 2×8×128 = 6396 + 2048 = 8444, number of channels 64) to obtain the multi-source coupling feature (dimension 8444×64) corresponding to the lower-level representation layer.

[0083] For the higher-level representation layer, the server employs the same process, adapting only the correlation model parameters to the structural association (such as the "structural transmission probability of the main shaft vibration feature point and the gearbox housing acoustic radiation feature point"), ultimately obtaining higher-level multi-source coupled features. Through this process, the server achieves deep coupling between millimeter waves and acoustic features at multiple levels, providing multi-source fusion data for subsequent feature optimization.

[0084] In this embodiment of the invention, in the millimeter-wave features, heterogeneous feature coupling is performed on each millimeter-wave feature point according to at least two acoustic feature points in the acoustic features to obtain the corresponding first multi-source coupling feature. This can be implemented through the following example.

[0085] For each millimeter-wave feature point in the millimeter-wave features, based on the prior probability mapping between the millimeter-wave feature point and each acoustic feature point in the acoustic features, at least two acoustic feature points that meet the first probability condition and their corresponding first coupling coefficients are obtained; wherein, each first coupling coefficient contains the prior probability mapping between the corresponding acoustic feature point and the millimeter-wave feature point.

[0086] Based on the obtained at least two first coupling coefficients, feature coupling is performed on the at least two acoustic feature points and the millimeter wave feature points to obtain the first heterogeneous coupling feature points corresponding to the millimeter wave feature points;

[0087] Based on the first heterogeneous coupling feature points associated with each millimeter-wave feature point, the corresponding first multi-source coupling feature is obtained.

[0088] In this embodiment of the invention, exemplarily, this embodiment takes the heterogeneous coupling of low-level millimeter-wave features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the acquisition process of the first multi-source coupling feature. The server has extracted low-level millimeter-wave features (dimension 123×52×64, corresponding to vibration features of distance-velocity grid, where 123 is the number of distance grids, 52 is the number of velocity grids, and 64 is the number of feature channels) and low-level acoustic features (dimension 2×8×128×64, corresponding to acoustic features of 2-channel time-frequency grid, where 8 is the number of time frames and 128 is the number of frequency points) through the aforementioned steps. The low-level focus is on the vibration of the core component (front bearing) of the spindle box and its acoustic basis. The following describes the coupling process of millimeter-wave feature points as an example. The server obtains the first multi-source coupling feature by traversing all millimeter-wave feature points and aggregating the results.

[0089] The server iterates through each millimeter-wave feature point in the low-level millimeter-wave features (a total of 123×52×64=409,344 points). Each point is a 64-dimensional vector, representing the vibration characteristics (such as amplitude, phase, energy, etc.) of a certain distance-velocity grid cell. Taking a typical millimeter-wave feature point P as an example: the coordinates of P are (i=45, j=25, k=12), where i=45 corresponds to the distance grid (the front bearing area of ​​the spindle, distance 0.22m), j=25 corresponds to the velocity grid (vibration velocity 0.5m / s), and k=12 corresponds to the feature channel (500Hz vibration frequency feature). Its physical meaning is "the 500Hz vibration characteristic of the front bearing of the spindle at a distance of 0.22m and a vibration velocity of 0.5m / s".

[0090] For a millimeter-wave feature point P, the server invokes a pre-trained prior probability mapping model (trained based on vibration-acoustic data of historical spindle bearings under normal / wear conditions, stored in the server's local model library, containing the frequency correlation probability between millimeter-wave and acoustic feature points). This model defines: "There is a strong correlation between vibration feature points at 500Hz±20Hz and acoustic feature points at 500Hz±20Hz, with a prior probability mapping range of 0.6-0.9."

[0091] Based on this prior probability mapping, the server calculates the association probability between P and all acoustic feature points in the low-level acoustic features (a total of 2 channels × 8 time frames × 128 frequency points × 64 feature channels = 131,072 points). The coordinates of the acoustic feature points are defined as (c, t, f, k), where c is the channel (1 / 2), t is the time frame (1-8, 6.25ms per frame), f is the frequency point (1-128, corresponding to 200Hz-8kHz, with a frequency interval of approximately 60Hz), and k is the feature channel (1-64). The server sets the first probability condition to "association probability ≥ 0.7" and selects 3 acoustic feature points that meet the condition, as follows:

[0092] Q1: (c=1, t=4, f=6, k=12), corresponding to channel 1, time frame 4 (25ms), frequency point 6 (frequency point 6 corresponds to 200+(6-1)×60=500Hz, frequency interval 60Hz), association probability 0.85;

[0093] Q2: (c=1, t=4, f=7, k=12), corresponding to channel 1, time frame 4, frequency point 7 (560Hz, 500+60), with an association probability of 0.78 (second harmonic of 500Hz vibration).

[0094] Q3: (c=2, t=4, f=5, k=12), corresponding to channel 2, time frame 4, frequency point 5 (440Hz, 500-60), with an association probability of 0.72 (fundamental frequency sidelobe of 500Hz vibration).

[0095] The server calculates the first coupling coefficient of each acoustic feature point to the millimeter-wave feature point P based on the "feature point dependency weight" parameter in the prior probability mapping model. This coefficient comprehensively considers the frequency correlation (prior probability) between the acoustic feature point and P and the signal strength consistency (intensity correlation coefficient obtained through historical data statistics). The calculation formula is: First coupling coefficient = Prior probability × Intensity correlation coefficient.

[0096] Given that the prior probabilities of Q1, Q2, Q3 and P are 0.85, 0.78, and 0.72 respectively, and the intensity correlation coefficients (the linear correlation between acoustic amplitude and vibration acceleration) are 0.92, 0.85, and 0.78 respectively (obtained from training with historical data), then: the first coupling coefficient of Q1 = 0.85 × 0.92 = 0.782; the first coupling coefficient of Q2 = 0.78 × 0.85 = 0.663; the first coupling coefficient of Q3 = 0.72 × 0.78 = 0.562.

[0097] The server performs feature coupling on the three selected acoustic feature points (Q1, Q2, Q3) and the millimeter-wave feature point P. Each feature point is a 64-dimensional vector, and the server operates according to the following steps:

[0098] 1) Weighted summation of acoustic feature points: Multiply the 64-dimensional vectors of Q1, Q2, and Q3 by their first coupling coefficients, and then sum them to obtain the coupled acoustic vector V. acoustic =0.782×Q1+0.663×Q2+0.562×Q3 (each dimension of the vector is calculated separately);

[0099] 2) Superimposed with millimeter-wave feature points: The coupled acoustic vector V acoustic The 64-dimensional vector of the millimeter-wave feature point P is superimposed with the vector of the millimeter-wave feature point P by a weight of 0.4:0.6 (with a higher weight due to the vibration feature being the core), resulting in the first heterogeneous coupling feature point P' corresponding to P: P' = 0.6 × P + 0.4 × V acoustic (Each dimension is stacked separately).

[0100] P' is a 64-dimensional vector that retains the core vibrational features of P while incorporating details of associated acoustic features (such as the sound radiation energy distribution corresponding to a 500Hz vibration).

[0101] The server repeats the above operation on all 409,344 millimeter-wave feature points in the low-level millimeter-wave features, generating a corresponding first heterogeneous coupling feature point (64-dimensional vector) for each feature point. The server aggregates these heterogeneous coupling feature points according to the spatial distribution of the original millimeter-wave features (123×52 grid) to obtain the first multi-source coupling feature, whose dimension is consistent with the original millimeter-wave features (123×52×64), but each feature point is a heterogeneous feature that integrates vibration and acoustic correlation.

[0102] Through the above process, the server completes heterogeneous coupling based on millimeter-wave feature points and obtains the first multi-source coupling feature, which provides a foundation for subsequent merging with the second multi-source coupling feature and feature optimization.

[0103] In this embodiment of the invention, the step of obtaining at least two acoustic feature points that meet the first probability condition and their corresponding first coupling coefficients based on the prior probability mapping between the millimeter-wave feature points and each acoustic feature point in the acoustic features can be implemented through the following example.

[0104] Based on the first correlation model parameters, the first correlation information between each acoustic feature point and the millimeter-wave feature point is obtained; wherein, the first correlation model parameters include the prior probability mapping between the millimeter-wave feature point and each acoustic feature point in the acoustic features;

[0105] In each obtained first relevance information, at least two first relevance information that meet the first probability condition are selected, and at least two corresponding acoustic feature points are obtained.

[0106] Based on the first coupling weight parameter, the dependence influence of each acoustic feature point on the millimeter-wave feature point is calculated to obtain the first coupling coefficient associated with the at least two acoustic feature points respectively; wherein, the first coupling weight parameter contains the prior probability mapping between the millimeter-wave feature point and the at least two acoustic feature points.

[0107] In this embodiment of the invention, exemplarily, this embodiment takes the heterogeneous coupling of low-level millimeter-wave feature points of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the process of screening acoustic feature points that meet the first probability condition and calculating the first coupling coefficient. The server has extracted low-level millimeter-wave features (123×52×64, distance-velocity grid vibration features) and low-level acoustic features (2×8×128×64, 2-channel time-frequency grid acoustic features), focusing on the correlation between the 500Hz vibration of the front bearing of the spindle (distance range of 0.2-0.3m) and the corresponding acoustic radiation.

[0108] The server selects a millimeter-wave feature point P with coordinates (i=48, j=22, k=15). i=48 corresponds to the distance grid (0.24m, the bearing area at the front end of the spindle), j=22 corresponds to the velocity grid (0.45m / s vibration velocity), and k=15 corresponds to the 500Hz vibration frequency feature channel. The server calls the parameters of the first correlation model (trained based on historical vibration-acoustic data, including a prior probability mapping of 500Hz±25Hz vibration and sound spectrum, conforming to a Gaussian distribution: center frequency 500Hz, standard deviation 20Hz, the smaller the deviation, the higher the correlation).

[0109] The server calculates the first correlation information between P and all acoustic feature points (2 channels × 8 time frames × 128 frequency points × 64 channels = 131072) in the low-level acoustic features. The coordinates of the acoustic feature points are (c, t, f, k), where c = 1 / 2 (channel), t = 1-8 (time frames, each frame is 6.25ms), f = 1-128 (frequency points, 200Hz-8kHz, interval 60.9Hz), and k = 1-64 (feature channels).

[0110] The server sets the first probability condition as "first relevance ≥ 0.7", and selects 3 acoustic feature points:

[0111] Q1: (c=1, t=5, f=6, k=15) → f=6 corresponds to 504.5Hz (200+(6-1)×60.9) → deviation 4.5Hz → correlation 0.88;

[0112] Q2: (c=1, t=5, f=5, k=15) → f=5 ​​corresponds to 443.6Hz → deviation 56.4Hz (sideband) → correlation 0.75;

[0113] Q3: (c=2, t=5, f=7, k=15) → f=7 corresponds to 565.4Hz → deviation 65.4Hz (second harmonic) → correlation 0.72.

[0114] The server calls the first coupling weight parameter (including the prior probability mapping of P and Q1-Q3: frequency association probability × strength association probability):

[0115] Q1: Frequency correlation probability 0.9 (deviation 4.5Hz) × Intensity correlation probability 0.92 (vibration-sound pressure linear correlation) → Coupling coefficient 0.828;

[0116] Q2: Frequency correlation probability 0.8 (sideband) × strength correlation probability 0.85 → coupling coefficient 0.680;

[0117] Q3: Frequency correlation probability 0.75 (harmonic) × intensity correlation probability 0.8 → coupling coefficient 0.600.

[0118] The server stores the coordinates and coupling coefficients of Q1-Q3 in a cache for subsequent heterogeneous coupling of millimeter-wave feature points P.

[0119] Through the above process, the server completes the acoustic feature point screening and coupling coefficient calculation for a single millimeter-wave feature point, and after traversing all millimeter-wave feature points, it aggregates them to obtain the first multi-source coupling feature.

[0120] In this embodiment of the invention, the acoustic feature is heterogeneously coupled to each acoustic feature point based on at least two millimeter-wave feature points in the millimeter-wave feature to obtain a corresponding second multi-source coupling feature. This can be implemented through the following example.

[0121] For each acoustic feature point in the acoustic features, based on the prior probability mapping between the acoustic feature point and each millimeter-wave feature point in the millimeter-wave features, at least two millimeter-wave feature points that meet the second probability condition and their corresponding second coupling coefficients are obtained; wherein, each second coupling coefficient contains the prior probability mapping between the corresponding millimeter-wave feature point and the acoustic feature point;

[0122] Based on the obtained at least two second coupling coefficients, the at least two millimeter-wave feature points and the acoustic feature points are fused to obtain the second heterogeneous coupling feature points corresponding to the acoustic feature points;

[0123] Based on the second heterogeneous coupling feature points associated with each acoustic feature point, the corresponding second multi-source coupling feature is obtained.

[0124] In this embodiment of the invention, exemplarily, this embodiment uses the heterogeneous coupling of low-level acoustic features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to describe in detail the process of obtaining the second multi-source coupling feature. The server has extracted low-level acoustic features (2×8×128×64, 2-channel time-frequency grid acoustic features) and low-level millimeter-wave features (123×52×64, distance-velocity grid vibration features), focusing on the correlation between the 800Hz meshing sound and the corresponding vibration of the rear gear set of the spindle (distance range of 0.4-0.5m). Taking the processing of a typical acoustic feature point Q as an example, the server obtains the second heterogeneous coupling feature point through this process, and after traversing all acoustic feature points, aggregates them into the second multi-source coupling feature.

[0125] The server iterates through each acoustic feature point in the low-level acoustic features (a total of 2×8×128×64=131,072 points, each a 64-dimensional vector representing the acoustic features of a certain time-frequency region). Acoustic feature point Q is selected: coordinates (c=2, t=6, f=10, k=20), where c=2 (second channel, acoustic radiation region of the main shaft rear gear set), t=6 (time frame 6, corresponding to 31.25ms, each frame 6.25ms, t=1-8 corresponding to 0-50ms), f=10 (frequency point 10, corresponding to 200+(10-1)×60.9≈748.1Hz, close to the gear meshing fundamental frequency of 800Hz), and k=20 (feature channel, corresponding to sound pressure level energy characteristics). Its physical meaning is "sound pressure level energy characteristics of the second channel of the main shaft rear gear set at time 31.25ms and frequency 748.1Hz".

[0126] The server calls the pre-trained second correlation model parameters (trained based on historical acoustic-vibration data of gear meshing, stored in the local model library, containing a prior probability mapping between acoustic feature points and millimeter-wave feature points: "frequency association probability between 750Hz±30Hz acoustic feature points and 750Hz±30Hz vibration millimeter-wave feature points"). This parameter is modeled using a Lorentz distribution, with a center frequency of 750Hz and a half-width at half-maximum of 40Hz. That is, the smaller the frequency deviation, the higher the correlation probability (e.g., a probability of 0.85 when the deviation is 0Hz, and a probability of 0.7 when the deviation is 30Hz).

[0127] Based on this parameter, the server calculates the second correlation information (value [0,1]) between Q and all millimeter-wave feature points (123×52×64=409,344) in the low-level millimeter-wave features. The coordinates of the millimeter-wave feature points are (i,j,k), i=1-123 (distance grid, 0-0.6m), j=1-52 (velocity grid, -2-2m / s), k=1-64 (feature channel). The second probability condition is set to "correlation ≥ 0.7", and two millimeter-wave feature points that meet the condition are selected:

[0128] P1: (i=85, j=30, k=20), i=85 corresponds to the distance grid (0.42m, rear gear set area), j=30 corresponds to the velocity grid (vibration velocity 0.8m / s), k=20 corresponds to the feature channel (750Hz vibration frequency feature), correlation 0.82;

[0129] P2: (i=86, j=30, k=20), i=86 (0.43m), j=30 (0.8m / s), k=20 (780Hz vibration frequency characteristics, fundamental frequency + 30Hz sideband), correlation 0.75.

[0130] The server calls the second coupling weight parameter (which includes the prior probability mapping between Q and P1-P2, containing "frequency association probability" and "strength association probability"):

[0131] Frequency correlation probability: P1 (750Hz) deviates from Q (748.1Hz) by 1.9Hz, with a probability of 0.85; P2 (780Hz) deviates from Q by 31.9Hz, with a probability of 0.72 (sideband correlation).

[0132] Intensity correlation probability: P1 vibration acceleration and Q sound pressure level linear correlation is 0.88 (high consistency between vibration and sound intensity during gear meshing); P2 is 0.8 (weak sideband energy).

[0133] Calculated using the formula "Second Coupling Coefficient = Frequency Correlation Probability × Strength Correlation Probability": P1: 0.85 × 0.88 = 0.748; P2: 0.72 × 0.8 = 0.576.

[0134] The server performs a weighted summation of the 64-dimensional eigenvectors of P1 and P2: 0.748×P1 + 0.576×P2, to obtain the coupled vibration vector V. vib (64 dimensions). V vib The 64-dimensional vector of the acoustic feature point Q is superimposed with a weight of 0.4:0.6 (primarily acoustic features, preserving sound radiation details): Q' = 0.6 × Q + 0.4 × V vib We obtain the second heterogeneous coupling feature point Q' (64-dimensional) corresponding to Q, which integrates the correlation features of gear vibration and sound radiation.

[0135] The server iterates through all 131,072 acoustic feature points in the low-level acoustic features, repeating the above filtering and coupling process, generating a corresponding second heterogeneous coupled feature point for each point. These feature points are aggregated according to the spatial distribution of the original acoustic features (2×8×128×64) to obtain the second multi-source coupled feature (dimension 2×8×128×64), where each feature point incorporates associated vibration features, providing multi-source data support for subsequent optimization of basic identification features.

[0136] Through the above process, the server completes heterogeneous coupling based on acoustic feature points, obtains the second multi-source coupling feature, and together with the first multi-source coupling feature, constitutes a hierarchical fusion feature.

[0137] In this embodiment of the invention, the step of obtaining at least two millimeter-wave feature points and their corresponding second coupling coefficients that meet the second probability condition based on the prior probability mapping between the acoustic feature points and each millimeter-wave feature point in the millimeter-wave features can be implemented through the following example.

[0138] Based on the second correlation model parameters, the second correlation information between each millimeter-wave feature point and the acoustic feature point is obtained; wherein, the second correlation model parameters include the prior probability mapping between the acoustic feature point and each millimeter-wave feature point in the millimeter-wave features;

[0139] In each obtained second correlation information, at least two second correlation information that meet the second probability condition are selected, and at least two corresponding millimeter-wave feature points are obtained.

[0140] Based on the second coupling weight parameter, the dependence influence of each millimeter-wave feature point on the acoustic feature point is calculated respectively to obtain the second coupling coefficients associated with the at least two millimeter-wave feature points respectively; wherein, the second coupling weight parameter contains the prior probability mapping between the acoustic feature point and the at least two millimeter-wave feature points.

[0141] In this embodiment of the invention, exemplarily, this embodiment takes the heterogeneous coupling of low-level acoustic features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the process of screening millimeter-wave feature points that meet the second probability condition and calculating the second coupling coefficient. The server has extracted low-level acoustic features (2×8×128×64, 2-channel time-frequency grid acoustic features) and low-level millimeter-wave features (123×52×64, distance-velocity grid vibration features), focusing on the correlation between the 800Hz meshing sound and the corresponding vibration of the rear gear set of the spindle (distance range of 0.4-0.5m). The following uses the processing of a typical acoustic feature point Q as an example. The server obtains the second coupling coefficient through this process, providing a basis for subsequent feature fusion.

[0142] The server iterates through low-level acoustic feature points and selects acoustic feature point Q: coordinates (c=2, t=6, f=10, k=20). Here, c=2 (second channel, corresponding to the acoustic radiation area of ​​the rear gear set of the main shaft; the microphone array is installed on the rear cover of the gearbox); t=6 (time frame 6, each frame is 6.25ms, corresponding to 6×6.25=37.5ms, capturing the acoustic features of the stable gear meshing phase); f=10 (frequency point 10, frequency calculation: 200Hz+(10-1)×60.9Hz=200+548.1=748.1Hz, close to the gear meshing fundamental frequency of 800Hz, with a 51.9Hz deviation due to gear processing error); k=20 (feature channel, corresponding to the sound pressure level energy characteristics, reflecting the acoustic radiation intensity at this frequency). Its physical meaning is "sound pressure level energy characteristics of the second channel of the rear gear set of the main shaft, at 37.5ms, and at a frequency of 748.1Hz".

[0143] The server calls the pre-trained second correlation model parameters (trained based on historical acoustic-vibration data of gear meshing, stored in the local model library, containing a prior probability mapping between acoustic feature points and millimeter-wave feature points: "frequency correlation probability between 750Hz±50Hz acoustic feature points and 750Hz±50Hz vibration millimeter-wave feature points"). These parameters are modeled using a Lorentz distribution: center frequency 750Hz, half-width at half-maximum 40Hz (i.e., when the frequency deviation Δf satisfies |Δf|≤20Hz, the correlation probability ≥0.8; when |Δf|=50Hz, the probability ≥0.65).

[0144] Based on this parameter, the server calculates the second correlation information (value [0,1]) between Q (748.1Hz) and all millimeter-wave feature points (123×52×64=409,344 points) in the low-level millimeter-wave features. The coordinates of the millimeter-wave feature points are (i,j,k), i=1-123 (distance grid, 0-0.6m), j=1-52 (velocity grid, -2-2m / s); k=1-64 (feature channel, including vibration frequency and amplitude features).

[0145] The server sets the second probability condition to "second relevance information ≥ 0.7" and filters millimeter-wave feature points that meet the condition. Calculations show that two points satisfy the condition:

[0146] P1: Coordinates (i=85, j=30, k=20). i=85 (distance from the grid, 0.42m, corresponding to the vibration region of the rear gear set); j=30 (velocity grid, vibration velocity 0.8m / s, gear meshing velocity); k=20 (characteristic channel, 752Hz vibration frequency characteristic, deviation from Q's 748.1Hz by 3.9Hz). Correlation calculation: According to the Lorentz distribution, Δf=3.9Hz≤20Hz, correlation=0.85.

[0147] P2: Coordinates (i=86, j=30, k=20). i=86 (0.43m); j=30 (0.8m / s); k=20 (781Hz vibration frequency characteristic, deviation 32.9Hz, gear meshing sideband). Correlation = 0.72 (Δf=32.9Hz is close to the full width at half maximum, which meets the probability condition).

[0148] The server extracts the coordinates and relevance of P1 and P2 and stores them in a temporary cache.

[0149] The server calls the second coupling weight parameter (containing the prior probability mapping between Q and P1 / P2, based on historical data statistics: composed of the product of "frequency association probability" and "strength association probability"):

[0150] Frequency correlation probability: P1 and Q have a Δf of 3.9 Hz, indicating a high frequency matching degree, with a probability of 0.9; P2 and Q have a Δf of 32.9 Hz (sideband), with a probability of 0.75.

[0151] Intensity correlation probability: The linear correlation between P1 vibration acceleration and Q sound pressure level is 0.88 (high consistency between vibration and sound intensity when gears are meshing normally); P2 has a correlation of 0.75 due to its weaker sideband energy.

[0152] The following calculations are performed using the formula "Second Coupling Coefficient = Frequency Correlation Probability × Strength Correlation Probability": P1 Coupling Coefficient = 0.9 × 0.88 = 0.792; P2 Coupling Coefficient = 0.75 × 0.75 = 0.562.

[0153] The server stores the coordinates and coupling coefficients (0.792, 0.562) of P1 and P2 in the feature coupling buffer for subsequent generation of the second heterogeneous coupling feature point of Q (formula: Q'=0.6×Q+0.4×(0.792×P1+0.562×P2)). After traversing all acoustic feature points, the second multi-source coupling feature is obtained by aggregation.

[0154] Through the above process, the server achieves accurate screening of acoustic feature points for millimeter-wave feature points and calculation of coupling weights, providing a quantitative basis for cross-modal feature fusion.

[0155] In this embodiment of the invention, the step of performing feature dependency modeling on the basic identification features based on each acquired multi-source coupling feature to obtain coupled identification features can be implemented through the following example.

[0156] Each acquired multi-source coupling feature is used to sequentially perform multi-stage progressive optimization processing on the basic identification feature to obtain the corresponding optimized identification feature; wherein, in one optimization stage, feature correlation information between a multi-source coupling feature and the basic identification feature is acquired, and the basic identification feature is optimized based on the optimization gain of the basic identification feature according to the feature correlation information.

[0157] Based on the correlation information of device operating states between at least two candidate device operating states indicated by the optimized identification features, the optimized identification features are updated to obtain coupled identification features.

[0158] In this embodiment of the invention, exemplarily, this embodiment takes the optimization of basic identification features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the process of obtaining coupled identification features. The server has obtained low-level multi-source coupling features (8444×64, focusing on basic vibration-acoustic correlation) and high-level multi-source coupling features (7250×64, focusing on structural dynamic correlation) through the aforementioned steps. The basic identification features include the prior probability mapping of four candidate equipment operating states ("normal meshing" S1, "slight tooth wear" S2, "excessive bearing clearance" S3, "broken gear tooth" S4) and three preset identification types ("normal" T1, "warning" T2, "fault" T3) (based on five years of historical operation and maintenance data statistics, such as S1-T1 probability 0.92, S2-T2 probability 0.80, etc.). The server finally obtains the coupled identification features through multi-stage optimization and state correlation updates.

[0159] The first stage of optimization (based on low-level multi-source coupling features): The server uses the basic identification features (4×3 probability matrix) as initial input. The low-level multi-source coupling features (8444×64) focus on the basic physical relationship between the near-field vibration of the spindle box and acoustics (such as the consistency between the 500Hz vibration sideband of S2 and the corresponding acoustic spectrum). The server calls the feature correlation model (including the prior mapping between coupling features and state probabilities: "the stronger the sideband feature, the higher the probability of S2-T2 should be") to calculate the feature correlation information between the low-level coupling features and the basic identification features (values ​​[0,1], such as the correlation between S2 and the coupling features being 0.82). Based on the correlation, the optimization gain is calculated (gain = correlation × 0.3, 0.3 is the adjustment coefficient), and the probability (0.80) of S2-T2 in the basic identification features is optimized as follows: 0.80 + 0.82 × 0.3 × (1 - 0.80) = 0.85 (to avoid probability overflow). Similarly, by adjusting the probabilities of other states, we obtain the first-stage optimized recognition features (S1-T1: 0.90, S2-T2: 0.85, S3-T2: 0.78, S4-T3: 0.88, etc.).

[0160] The second stage of optimization (based on high-level multi-source coupling features): The server takes the features identified in the first stage of optimization as input, and the high-level coupling features focus on structural correlations (such as the linkage between bearing vibration in S3 and tooth surface wear in S2). The correlation information between the high-level coupling features and the optimization features is calculated (e.g., the correlation between S3 and the coupling features is 0.75). After optimizing and adjusting the gain, the second stage of optimization identification features are obtained (S1-T1: 0.89, S2-T2: 0.87, S3-T2: 0.82, S4-T3: 0.86, etc.).

[0161] The server calls the state relevance model parameters (based on historical fault data to statistically determine the co-occurrence probability of candidate states: S2 and S3 co-occurrence probability 0.65, S3 and S4 co-occurrence probability 0.80), and calculates the equipment operating state importance coefficient for each state (importance = Σ (co-occurrence probability × state probability), normalized to a sum of 1): S1: 0.12, S2: 0.23, S3: 0.40, S4: 0.25.

[0162] The server performs a linear superposition update of the second-stage optimized identification features based on the importance coefficient: Coupling probability = Σ (State importance coefficient × Optimization probability). For example, the coupling probability of type T2 is 0.23 × 0.87 + 0.40 × 0.82 = 0.20 + 0.33 = 0.53, ultimately resulting in coupled identification features (T1: 0.15, T2: 0.53, T3: 0.32), completing the update from basic identification features to coupled identification features.

[0163] Through the above process, the server integrates multi-source coupling features with state correlation to achieve dynamic optimization of recognition features, providing accurate basis for subsequent target state decisions.

[0164] In this embodiment of the invention, the step of obtaining feature correlation information between a multi-source coupling feature and the basic identification feature, and optimizing the basic identification feature based on the feature correlation information and the optimization gain of the basic identification feature, can be implemented through the following example.

[0165] For each basic feature element in the basic identification features, based on the prior probability mapping between the basic feature element and each coupled feature element in a multi-source coupled feature, at least two coupled feature elements that meet the coupling probability condition and their corresponding fusion association weights are obtained; wherein, each fusion association weight contains the prior probability mapping between the corresponding coupled feature element and the basic feature element.

[0166] Based on the obtained at least two fusion association weights, the at least two coupled feature elements and the basic feature elements are fused to obtain the optimized feature points corresponding to the basic feature elements.

[0167] In this embodiment of the invention, exemplarily, this embodiment takes the optimization of basic identification features of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to explain in detail the optimization process of basic feature elements. The server has obtained basic identification features (a 4×3 probability matrix, containing the prior probabilities of 4 candidate equipment operating states S1-S4 and 3 preset identification types T1-T3, such as the prior probability of "slight tooth surface wear" S2 and "early warning" T2 being 0.80) and low-level multi-source coupling features (8444×64, focusing on the near-field vibration-acoustic basis correlation of the spindle box, such as the consistency feature of the 500Hz vibration sideband of S2 and the corresponding acoustic spectrum). Taking the basic feature element "prior probability of S2-T2 0.80" as an example, the server obtains optimized feature points through coupling feature element screening, weight calculation and fusion processing, and completes the first stage of optimization after traversing all basic feature elements.

[0168] The basic feature elements in the basic identification features are the probability values ​​of the candidate state and the identification type. The prior probability of S2-T2 is selected as 0.80 (S2: "Slight tooth surface wear", T2: "Warning"). Its physical meaning is: "In historical data, when the spindle box exhibits slight tooth surface wear, the probability of it being classified as 'Warning' is 80%." This element needs to be optimized in conjunction with the current low-level multi-source coupling features (reflecting real-time vibration-acoustic correlation) to improve adaptability to the current operating state.

[0169] The server invokes a pre-trained prior probability mapping model (trained based on historical tooth surface wear data, stored in a local model library, containing the association probabilities of basic feature elements and coupled feature elements: "the coupling probability of the 500Hz±30Hz vibration sideband features of the S2 state and the corresponding acoustic spectrum features"). This model is modeled using a Gaussian mixture distribution and contains two components: the principal component (500Hz, standard deviation 20Hz, weight 0.7) captures the fundamental frequency association, and the secondary component (550Hz, standard deviation 30Hz, weight 0.3) captures the second harmonic association. The total probability is the weighted sum of the components.

[0170] The coupling feature elements in the low-level multi-source coupling features (8444×64) are 64-dimensional vectors, each element corresponding to a certain vibration-acoustic correlation feature (such as sideband energy difference, phase consistency, etc.). The server iterates through all coupling feature elements (8444), calculates the correlation probability with the basic feature element S2-T2 (value [0,1]), sets the coupling probability condition to "correlation probability ≥ 0.7", and selects 3 coupling feature elements that meet the condition:

[0171] C1: Coordinates (index 2156, feature channel 12), corresponding to the energy difference feature between the 502Hz vibration sideband and the sound spectrum, with a correlation probability of 0.85 (principal component contribution 0.7×0.92=0.644, secondary component contribution 0.3×0.68=0.204, total 0.848≈0.85).

[0172] C2: Coordinates (index 3420, feature channel 12), corresponding to the phase consistency feature of the 485Hz vibration sideband and the sound spectrum, with an association probability of 0.78 (principal component 0.7×0.88=0.616, secondary component 0.3×0.54=0.162, total 0.778≈0.78).

[0173] C3: Coordinates (index 5689, feature channel 12), corresponding to the amplitude ratio feature of the second harmonic vibration and the sound spectrum at 555Hz, with a correlation probability of 0.72 (principal component 0.7×0.62=0.434, secondary component 0.3×0.96=0.288, total 0.722≈0.72).

[0174] The server calls the fusion weight parameters (including the prior probability mapping between basic feature elements and coupled feature elements: "association probability × strength normalization coefficient", where the strength normalization coefficient is the energy normalization value of the coupled feature element, ranging from [0,1]). Calculations show: C1 has a strength normalization value of 0.90 (strongest energy), fusion association weight = 0.85 × 0.90 = 0.765; C2 has a strength normalization value of 0.85, fusion association weight = 0.78 × 0.85 = 0.663; C3 has a strength normalization value of 0.75, fusion association weight = 0.72 × 0.75 = 0.540. Weight normalization processing (total 0.765 + 0.663 + 0.540 = 1.968, normalized C1: 0.389, C2: 0.337, C3: 0.274).

[0175] The server performs a fusion process on the three coupled feature elements:

[0176] Weighted summation of coupling feature elements: The 64-dimensional feature vectors of C1, C2, and C3 (normalized to [0,1]) are summed according to normalized weights to obtain the fused coupling vector V = 0.389×C1 + 0.337×C2 + 0.274×C3 (the value range of each dimension of the vector is [0,1], such as the energy difference dimension value of 0.62).

[0177] Fusion with basic feature elements: The formula “Optimized feature point = basic feature element + mean of fusion coupling vector × optimization coefficient” is adopted, where the mean of fusion coupling vector is 0.62 (mean of each dimension) and the optimization coefficient is 0.2 (to avoid probability overflow). Then: Optimized feature point = 0.80 + 0.62 × 0.2 × (1 - 0.80) = 0.80 + 0.025 = 0.825 (rounded to 0.83).

[0178] The server optimizes the basic feature element of S2-T2 from 0.80 to 0.83, completing the optimization of this element. It then iterates through all 12 basic feature elements (4 states × 3 types) in the basic recognition features, repeating the above process to obtain the first-stage optimized recognition features.

[0179] Through the above process, the server achieves dynamic fusion of basic feature elements and multi-source coupled features, improving the adaptability of the recognition features to the current device operating state and laying the foundation for the subsequent acquisition of coupled recognition features.

[0180] In this embodiment of the invention, the step of updating the optimized identification feature based on the device operating state correlation information between at least two candidate device operating states indicated by the optimized identification feature to obtain the coupled identification feature can be implemented through the following example.

[0181] For the operating states of at least two candidate devices indicated by the optimized identification features, the importance coefficients of the operating states associated with the at least two candidate devices are obtained based on the prior probability mapping of the operating states of each candidate device and the operating states of the at least two candidate devices.

[0182] Based on the importance coefficients of at least two device operating states, the optimized identification features are linearly superimposed to obtain coupled identification features.

[0183] In this embodiment of the invention, for example, this embodiment takes the optimization and identification feature update of a precision CNC machine tool spindle box (model HTM-80H) by the server as an example to explain in detail the process of obtaining the coupled identification features. The server has obtained the second-stage optimized identification features (a 4×3 probability matrix, containing the optimized probabilities of 4 candidate device operating states S1-S4 and 3 preset identification types T1-T3: S1 "Normal meshing", S2 "Slight tooth wear", S3 "Excessive bearing clearance", S4 "Broken gear teeth"; T1 "Normal", T2 "Warning", T3 "Fault") through multi-stage optimization. The specific optimized probabilities are as follows: S1-T1: 0.89, S1-T2: 0.10, S1-T3: 0.01; S2-T1: 0.12, S2-T2: 0.87, S2-T3: 0.01; S3-T1: 0.08, S3-T2: 0.82, S3-T3: 0.10; S4-T1: 0.02, S4-T2: 0.12, S4-T3: 0.86. The server calculates the importance coefficients of device operating states and linearly superimposes them, ultimately updating the optimized identification features into coupled identification features to fuse the correlation between states.

[0184] The server first calls the prior probability mapping model of the equipment operating status (based on the statistics of historical fault data of the spindle box, stored in the local model library, containing the co-occurrence probability between 4 candidate equipment operating states, reflecting the correlation between states: such as "excessive bearing clearance (S3) easily leads to slight wear on the tooth surface (S2), and the co-occurrence probability of the two is high"). The model stores co-occurrence probabilities in the form of a 4×4 matrix. The main diagonal represents the co-occurrence probability of a state itself (1.0), and the off-diagonal represents the co-occurrence probability between states. The specific values ​​are as follows: S1 with other states: S1-S2: 0.15 (low probability of slight wear during normal meshing), S1-S3: 0.10, S1-S4: 0.05; S2 with other states: S2-S1: 0.15, S2-S3: 0.65 (tooth surface wear is often accompanied by excessive bearing clearance), S2-S4: 0.30 (severe wear may lead to tooth breakage); S3 with other states: S3-S1: 0.10, S3-S2: 0.65, S3-S4: 0.80 (excessive bearing clearance easily leads to gear tooth breakage); S4 with other states: S4-S1: 0.05, S4-S2: 0.30, S4-S3: 0.80.

[0185] The server calculates the importance coefficient of each candidate device's operating status based on the mapping matrix. The importance coefficient reflects the weight of a certain status's influence on the overall identification result. The calculation logic is: "Status Importance = Σ (Co-occurrence probability of this status with other statuses × Total probability of other statuses)", where "Total probability of other statuses" is the sum of the probabilities of this status under all types in the optimized identification features (normalized).

[0186] First, calculate the total probability of each candidate state (the sum of the probabilities of each row in the optimized recognition features): Total probability of S1: 0.89(T1) + 0.10(T2) + 0.01(T3) = 0.90; Total probability of S2: 0.12(T1) + 0.87(T2) + 0.01(T3) = 1.00; Total probability of S3: 0.08(T1) + 0.82(T2) + 0.10(T3) = 1.00; Total probability of S4: 0.02(T1) + 0.12(T2) + 0.86(T3) = 1.00.

[0187] Normalizing the total probability (total 0.90 + 1.00 + 1.00 + 1.00 = 3.90), we obtain the state probability vectors: S1: 0.90 / 3.90 ≈ 0.23, S2: 1.00 / 3.90 ≈ 0.26, S3: 1.00 / 3.90 ≈ 0.26, S4: 1.00 / 3.90 ≈ 0.26.

[0188] Then, the importance coefficient is calculated based on the co-occurrence probability matrix and the state probability vector:

[0189] Importance of S1: (S1 - S1 × S1 probability) + (S1 - S2 × S2 probability) + (S1 - S3 × S3 probability) + (S1 - S4 × S4 probability) = 1.0 × 0.23 + 0.15 × 0.26 + 0.10 × 0.26 + 0.05 × 0.26 ≈ 0.23 + 0.04 + 0.03 + 0.01 = 0.31;

[0190] Importance of S2: (S2 - S1 × S1 probability) + (S2 - S2 × S2 probability) + (S2 - S3 × S3 probability) + (S2 - S4 × S4 probability) = 0.15 × 0.23 + 1.0 × 0.26 + 0.65 × 0.26 + 0.30 × 0.26 ≈ 0.03 + 0.26 + 0.17 + 0.08 = 0.54;

[0191] Importance of S3: (S3 - S1 × S1 probability) + (S3 - S2 × S2 probability) + (S3 - S3 × S3 probability) + (S3 - S4 × S4 probability) = 0.10 × 0.23 + 0.65 × 0.26 + 1.0 × 0.26 + 0.80 × 0.26 ≈ 0.02 + 0.17 + 0.26 + 0.21 = 0.66;

[0192] Importance of S4: (S4 - S1 × S1 probability) + (S4 - S2 × S2 probability) + (S4 - S3 × S3 probability) + (S4 - S4 × S4 probability) = 0.05 × 0.23 + 0.30 × 0.26 + 0.80 × 0.26 + 1.0 × 0.26 ≈ 0.01 + 0.08 + 0.21 + 0.26 = 0.56.

[0193] After normalizing the importance coefficients (total 0.31 + 0.54 + 0.66 + 0.56 = 2.07), the final importance coefficients are: S1: 0.31 / 2.07 ≈ 0.15, S2: 0.54 / 2.07 ≈ 0.26, S3: 0.66 / 2.07 ≈ 0.32, S4: 0.56 / 2.07 ≈ 0.27 (total 1.00).

[0194] Based on the obtained device operating status importance coefficients (S1: 0.15, S2: 0.26, S3: 0.32, S4: 0.27), the server linearly superimposes the second-stage optimized identification features and calculates the coupling probability of each preset identification type (T1 / T2 / T3). The formula is: Coupling probability = Σ (state importance coefficient × optimization probability of the state under the type).

[0195] The specific calculations are as follows:

[0196] T1 (normal) coupling probability: S1-T1×0.15+S2-T1×0.26+S3-T1×0.32+S4-T1×0.27=0.89×0.15+0.12×0.26+0.08×0.32+0.02×0.27≈0.13+0.03+0.03+0.01=0.19;

[0197] T2 (early warning) coupling probability: S1-T2×0.15+S2-T2×0.26+S3-T2×0.32+S4-T2×0.27=0.10×0.15+0.87×0.26+0.82×0.32+0.12×0.27≈0.02+0.23+0.26+0.03=0.54;

[0198] T3 (fault) coupling probability: S1-T3×0.15+S2-T3×0.26+S3-T3×0.32+S4-T3×0.27=0.01×0.15+0.01×0.26+0.10×0.32+0.86×0.27≈0.00+0.00+0.03+0.23=0.26.

[0199] The sum of the above coupling probabilities is 0.19 + 0.54 + 0.26 = 0.99 (due to slight deviations from rounding, the sum after normalization is 1.00). The server uses this as the coupling identification feature, i.e., T1: 0.19, T2: 0.54, T3: 0.27.

[0200] The coupling identification feature comprehensively considers the correlation between the operating states of the candidate equipment (e.g., the high co-occurrence probability of S3 with S2 and S4 increases the importance coefficient of S3), which is more in line with the actual operating rules of the equipment compared with the optimized identification feature. For example, the coupling probability of T2 (early warning) is 0.54, which is significantly higher than other types, reflecting that the spindle box is more likely to be in a state that requires an early warning (e.g., S2 or S3), providing a quantitative basis for subsequent decision-making on the operating state of the target equipment (e.g., determining S3 as the target state).

[0201] Through the above process, the server completes the dynamic update of optimized identification features, obtains coupled identification features that can reflect state correlation, and improves the robustness and accuracy of device operating status identification.

[0202] In this embodiment of the invention, the step of determining the target device operating state corresponding to the corresponding preset identification type among the at least two candidate device operating states based on the coupling identification features can be implemented through the following example.

[0203] Obtain the identification feature parameters corresponding to each preset identification type; wherein, each identification feature parameter includes the correlation confidence between each historical device operating state and the corresponding preset identification type;

[0204] Based on each acquired identification feature parameter, feature analysis processing is performed on the coupled identification feature to obtain the probability distribution of the operating status of the at least two candidate devices being classified into each preset identification type;

[0205] For the at least two candidate device operating states, if the peak probability associated with the candidate device operating state conforms to a preset probability distribution, the candidate device operating state is determined to be the target device operating state corresponding to the preset identification type of the peak probability.

[0206] In this embodiment of the invention, exemplarily, this embodiment takes the coupling recognition feature analysis of a precision CNC machine tool spindle box (model HTM-80H) by a server as an example to describe in detail the process of determining the operating status of the target equipment. The server has obtained the coupling recognition features through the aforementioned steps (preset recognition type probability distribution: T1 "Normal" 0.19, T2 "Warning" 0.54, T3 "Fault" 0.27), and the operating status of the candidate equipment is S1 "Normal meshing", S2 "Slight tooth wear", S3 "Excessive bearing clearance", and S4 "Broken gear tooth". By acquiring the recognition feature parameters, analyzing the probability distribution and peak value judgment, the server finally determines the target status corresponding to the "Warning" type.

[0207] The server retrieves the recognition feature parameters corresponding to each preset recognition type from the local model library. These parameters are trained based on historical spindle box fault diagnosis data and include the "correlation confidence between historical equipment operating status and preset recognition type" (value [0,1], reflecting the representativeness of a certain historical status to the type). Specific parameters are as follows:

[0208] T1 "Normal": Primarily associated with S1 (normal meshing), with a confidence level of 0.95 (95% of "normal" cases in historical data are caused by S1); S2-S4 confidence levels are all ≤0.1 (abnormal states are rarely classified as normal).

[0209] T2 "Warning": mainly associated with S2 (slight tooth surface wear) and S3 (excessive bearing clearance), with confidence levels of 0.85 (85% of warnings are caused by S2) and 0.82 (82% are caused by S3), respectively; S1 confidence level is 0.12, and S4 confidence level is 0.20 (may provide warning in the early stage of tooth breakage);

[0210] T3 "Fault": Primarily associated with S4 (broken gear teeth), with a confidence level of 0.90 (90% of faults are caused by S4); S3 confidence level of 0.35 (severe bearing failure may be classified as fault), and S1-S2 confidence levels ≤ 0.05.

[0211] The server performs feature analysis on the coupled recognition features (T1: 0.19, T2: 0.54, T3: 0.27) based on the recognition feature parameters, and calculates the probability distribution of the four candidate states to be classified into each preset type. The analysis logic is: "State-Type Probability = Type Probability in Coupled Recognition Features × Confidence of the State to the Type" (normalization process to ensure that the sum of the probabilities of each state is 1).

[0212] The specific calculations are as follows:

[0213] S1 "Normal meshing": Probability of classification into T1 = 0.19 × 0.95 = 0.1805, T2 = 0.54 × 0.12 = 0.0648, T3 = 0.27 × 0.05 = 0.0135; Normalized distribution: T1: 0.70 (0.1805 / 0.2588), T2: 0.25 (0.0648 / 0.2588), T3: 0.05;

[0214] S2 "Slight wear on tooth surface": Probability of being classified as T2 = 0.54 × 0.85 = 0.459, T1 = 0.19 × 0.10 = 0.019, T3 = 0.27 × 0.05 = 0.0135; Normalized distribution: T2: 0.93 (0.459 / 0.4915), T1: 0.04, T3: 0.03;

[0215] S3 "Excessive bearing clearance": Probability of classification into T2 = 0.54 × 0.82 = 0.4428, T3 = 0.27 × 0.35 = 0.0945, T1 = 0.19 × 0.08 = 0.0152; Normalized distribution: T2: 0.80 (0.4428 / 0.5525), T3: 0.17, T1: 0.03;

[0216] S4 "Broken Gear Tooth": The probability of being classified as T3 is 0.27 × 0.90 = 0.243, T2 = 0.54 × 0.20 = 0.108, and T1 = 0.19 × 0.02 = 0.0038; the normalized distribution is: T3: 0.69 (0.243 / 0.3548), T2: 0.30, and T1: 0.01.

[0217] The server extracts the peak probabilities (the highest probability type in each state distribution) and their corresponding types from the probability distributions of the four candidate states: S1 peak probability 0.70 (T1), S2 peak probability 0.93 (T2), S3 peak probability 0.80 (T2), and S4 peak probability 0.69 (T3). The server sets the preset probability distribution condition as "peak probability ≥ 0.7, and the corresponding type is consistent with the peak type of the coupled identification feature". The peak type of the coupled identification feature is T2 (0.54), so only the peak probabilities of S2 and S3 need to be checked: S2 peak probability 0.93 ≥ 0.7, corresponding to T2; S3 peak probability 0.80 ≥ 0.7, corresponding to T2.

[0218] Further comparison of the peak probabilities of S2 and S3 with the T2 probability in the coupled identification features revealed that the T2 probability of S3 (0.4428) was closer to the T2 probability of the coupled identification features (0.54), and the co-occurrence frequency of S3 and T2 in historical data (0.82) was higher than that of S2 (0.85, but the difference was slight). The server ultimately determined that S3 "excessive bearing clearance" corresponded to the target equipment operating status of the T2 "warning" type.

[0219] Through the above process, the server integrates historical confidence levels with real-time coupling characteristics to accurately locate the specific equipment operating status that caused the "early warning," providing a clear basis for subsequent operation and maintenance decisions (such as adjusting bearing preload).

[0220] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned artificial intelligence-based millimeter-wave acoustic-image fusion recognition method. Figure 2 As shown, Figure 2This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0221] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A millimeter-wave acoustic-image fusion recognition method based on artificial intelligence, characterized in that, include: Based on at least two pre-defined multi-level representation layers, corresponding millimeter-wave features and acoustic features are extracted for the region to be identified. The at least two multi-level representation layers include a low-level representation layer and a high-level representation layer. The extraction of the corresponding millimeter-wave features and acoustic features includes: integrating millimeter-wave source data and acoustic source data collected for the same monitoring region into an original sensing tensor based on one multi-level representation layer; dividing the original sensing tensor into at least two sensing sub-blocks based on the level offset between the original sensing tensor and the multi-level representation layers; and projecting the basic feature points in each sensing sub-block onto the corresponding feature domain to obtain... Basic millimeter-wave features and basic acoustic features are derived. Dependency modeling of feature points is performed on the basic millimeter-wave features and basic acoustic features respectively to obtain the corresponding millimeter-wave features and acoustic features. Specifically, when performing dependency modeling on the basic features of the lower-level representation layer, a self-attention mechanism is used for intra-feature dependency modeling of millimeter-wave features, combined with a convolutional neural network for inter-feature dependency modeling. For acoustic features, an LSTM network is used for temporal dependency modeling, combined with a graph neural network for frequency domain dependency modeling. When performing dependency modeling on the basic features of the higher-level representation layer, a Transformer encoder is used for feature point dependency modeling. Based on the at least two multi-level representation layers, heterogeneous feature coupling is performed on the corresponding millimeter-wave features and acoustic features respectively to obtain multi-source coupled features associated with the at least two multi-level representation layers respectively; wherein, the heterogeneous feature coupling includes: for each millimeter-wave feature point in the millimeter-wave features, calculating the first correlation information between it and each acoustic feature point according to the first correlation model parameters containing the prior probability mapping between the millimeter-wave feature point and the acoustic feature point, screening acoustic feature points that meet the first probability conditions, calculating the first coupling coefficient of each screened acoustic feature point according to the first coupling weight parameter, performing a weighted summation of the feature vectors of the screened acoustic feature points based on the first coupling coefficient, and fusing it with the feature vectors of the millimeter-wave feature points to obtain the first heterogeneous coupling. Feature points are aggregated to obtain the first multi-source coupling feature; for each acoustic feature point in the acoustic features, according to the second correlation model parameters containing the prior probability mapping between acoustic feature points and millimeter-wave feature points, the second correlation information between it and each millimeter-wave feature point is calculated, millimeter-wave feature points that meet the second probability conditions are selected, and the second coupling coefficient of each selected millimeter-wave feature point is calculated according to the second coupling weight parameter. Based on the second coupling coefficient, the feature vectors of the selected millimeter-wave feature points are weighted and summed, and fused with the feature vectors of the acoustic feature points to obtain the second heterogeneous coupling feature points, which are aggregated to obtain the second multi-source coupling feature; the first multi-source coupling feature and the second multi-source coupling feature are concatenated as the multi-source coupling feature corresponding to the multi-level representation layer; Obtain the basic identification features of the region to be identified, wherein the basic identification features include a basic prior probability mapping between the operating states of at least two candidate devices and each preset identification type, which is pre-defined for the region to be identified; Based on each acquired multi-source coupling feature, feature dependency modeling is performed on the basic identification features to obtain coupled identification features. The process of obtaining coupled identification features includes: using each acquired multi-source coupling feature, sequentially performing multi-stage progressive optimization processing on the basic identification features to obtain optimized identification features; in each optimization stage, for each basic feature element in the basic identification features, based on its prior probability mapping with each coupled feature element in a multi-source coupling feature, at least two coupled feature elements that meet the coupling probability conditions are selected and their corresponding fusion association weights are calculated; the selected coupled feature elements are weighted and summed based on the fusion association weights, and then fused with the basic feature elements to obtain optimized feature points; after obtaining optimized identification features, based on the device operation state correlation information between the at least two candidate device operation states, the device operation state importance coefficient of each candidate device operation state is calculated; the optimized identification features are linearly superimposed based on the device operation state importance coefficients to obtain coupled identification features. Based on the coupling recognition features, the target device operating state corresponding to the corresponding preset recognition type is determined among the at least two candidate device operating states, so as to obtain the current recognition result of the area to be recognized.

2. The method according to claim 1, characterized in that, The step of determining the target device operating state corresponding to the corresponding preset identification type among the at least two candidate device operating states based on the coupling identification features includes: Obtain the identification feature parameters corresponding to each preset identification type; wherein, each identification feature parameter includes the correlation confidence between each historical device operating state and the corresponding preset identification type; Based on each acquired identification feature parameter, feature analysis processing is performed on the coupled identification feature to obtain the probability distribution of the operating status of the at least two candidate devices being classified into each preset identification type; For the at least two candidate device operating states, if the peak probability associated with the candidate device operating state conforms to a preset probability distribution, the candidate device operating state is determined to be the target device operating state corresponding to the preset identification type of the peak probability.

3. A server system, characterized in that, Includes a server, the server being used to perform the method according to any one of claims 1-2.

Images