A special equipment inspection system and method based on a multi-modal large model

Abstract

This invention proposes a special equipment inspection system and method based on a multimodal large model, comprising: collecting and preprocessing multimodal data of special equipment; constructing a spatiotemporal fusion attention network for feature extraction to obtain fused feature representations; using a pre-trained multimodal large model for inference analysis, and distinguishing between normal and abnormal states through comparative learning; constructing a dynamic Bayesian decision network by combining historical data and an expert knowledge base to generate fault warnings and maintenance suggestions; optimizing inspection strategies based on equipment importance and warning levels; displaying inspection results through augmented reality and marking abnormal areas with digital twin technology to provide maintenance guidance. This invention can improve fault prediction accuracy, reduce false alarms and missed detections, optimize inspection paths, significantly reduce maintenance costs, extend equipment lifespan, and improve production safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, specifically to a special equipment inspection system and method based on a multimodal large model. Background Technology

[0002] In the field of special equipment health management, with the accelerated advancement of digitalization, the demand for full lifecycle status perception of complex systems is growing exponentially. Traditional inspection systems rely on periodic manual verification mechanisms, and their inspection efficiency is limited by manpower and the complexity of operating conditions, resulting in systemic defects such as difficulty in eliminating blind spots and insufficient data traceability. Against this backdrop, intelligent inspection methods integrating IoT sensing and machine learning algorithms have been developed. However, most existing intelligent inspection methods are based on single sensors or single-modal data, making it difficult to comprehensively and accurately reflect the operating status of equipment. Summary of the Invention

[0003] Based on the above-mentioned problems, this invention proposes a special equipment inspection system and method based on a multimodal large model. Through the solution of this invention, the accuracy of fault prediction can be improved, false alarms and missed detections can be reduced, inspection paths can be optimized, maintenance costs can be significantly reduced, equipment service life can be extended, and production safety can be improved.

[0004] In view of this, one aspect of the present invention proposes a special equipment inspection method based on a multimodal large model, comprising:

[0005] Collect multimodal data from special equipment;

[0006] The multimodal data is preprocessed to obtain the first multimodal data;

[0007] A spatiotemporal fusion attention network is constructed, and spatiotemporal domain features are extracted from the first multimodal data through the spatiotemporal fusion attention network to obtain a fusion feature representation;

[0008] The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning.

[0009] Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions.

[0010] Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization.

[0011] The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

[0012] Optionally, the multimodal large model employs a hierarchical attention mechanism for intermodal information fusion, including:

[0013] Extract the internal features of each modality to obtain the intramodal representation;

[0014] Construct cross-modal attention graphs to establish connections between data from different modalities;

[0015] By capturing long-range dependencies through a self-attention mechanism, effective fusion of information between modalities can be achieved.

[0016] Optionally, the step of collecting multimodal data from special equipment includes:

[0017] Visual data is acquired using a high-resolution camera array, which includes RGB cameras and depth cameras. The optimal acquisition position is determined according to the following formula:

[0018] P=argmin{∑(ω1·d(p,v1,ξ)+ω2·c(p,η)+ω3·o(p,e,μ)+ω4·ρ(p,ε))};

[0019] Wherein, d(p,v1,ξ) represents the coverage metric function of position p to device viewpoint v1, ξ is the field of view parameter, considering the impact of focal length changes on coverage; c(p,η) represents the cost function of position p, η is the environmental complexity coefficient, reflecting the deployment difficulty in different environments; o(p,e,μ) represents the occlusion metric function of position p to device e, μ is the occlusion perspective coefficient, considering the information acquisition capability under partial occlusion; ρ(p,ε) represents the stability evaluation function, measuring the degree of environmental interference to position p, ε is the environmental vibration threshold; ω1, ω2, ω3, ω4 represent weighting coefficients, dynamically adjusted to balance the influence of various factors;

[0020] The device's operating parameters are collected using a distributed intelligent sensor network, and the sensor sampling frequency fs is adaptively adjusted according to the device's dynamic characteristics.

[0021] fs=max(2fmax·(1+θ·CT),κ·SNR -0.5 ·σ 2 •λenvironment);

[0022] Where fs represents the adaptive sampling frequency; fmax represents the maximum vibration frequency of the key components of the equipment; θ represents the temperature compensation coefficient, considering the influence of temperature on the vibration characteristics of the equipment; CT represents the temperature difference between the equipment surface and the environment; κ represents the adjustment coefficient, which changes dynamically according to the equipment status; SNR represents the signal-to-noise ratio; σ 2 λ represents the signal variance, ensuring sufficient sampling accuracy under various operating conditions; λenvironment represents the environmental impact factor.

[0023] Optionally, the spatiotemporal fusion attention network achieves the fusion of temporal and spatial features through the following methods:

[0024] F=σa(Wτ·[H1,H2,...,Hτmax]·Qτ+bτ)⊙σa(Ws·[S1,S2,...,Ssmax]·Qs+bs)+ζ·I(Hτ,Ss);

[0025] Where F represents the fused feature representation, H1 to Hτmax represent the feature vectors at τmax time points, S1 to Ssmax represent the feature vectors at smax spatial locations, Wτ and Ws are the temporal and spatial attention weight matrices, respectively; Qτ represents the temporal feature projection matrix, used for dimensionality reduction and feature enhancement; Qs represents the spatial feature projection matrix, used for dimensionality reduction and feature enhancement; bτ represents the temporal attention bias term, bs represents the spatial attention bias term, σa is the Swish activation function, defined as x3·sigmoid(x3), providing nonlinear transformation capability; ⊙ is the Hadamard product; ζ is the mutual information weight coefficient, controlling the degree of influence of spatiotemporal information complementarity; I(Hτ,Ss): the mutual information measurement function between temporal and spatial features, capturing spatiotemporal correlation.

[0026] Optionally, the multimodal large model distinguishes between the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning. The multimodal large model employs a cross-modal contrastive learning framework, and the loss function is defined as:

[0027] L=-log[exp(sim(z1,z2) / τtemp) / ∑exp(sim(z1,z) / τtemp)]+Ω·∑||z i || 2 +γ·D(z1,z2,zneg);

[0028] Where L is the contrastive learning loss function, z1 and z2 are the feature representations of different modalities under the same device state; sim is the cosine similarity function, which measures the similarity of feature vectors; τtemp is a temperature parameter that controls the concentration of feature distribution, ranging from [0.1, 1.0]; Ω is the regularization coefficient, which controls the L2 norm of the feature vector to prevent feature collapse; ||z i || 2 γ is the L2 norm of the feature vectors, promoting a uniform distribution in the feature space; γ is the triplet loss weight, enhancing inter-class discrimination; D is the triplet distance function, defined as [d1(z1,zneg)-d1(z1,z2)+margin]. + , where d1 is the Euclidean distance, margin is the boundary parameter; zneg is the negative sample feature representation;

[0029] The model achieves semantic alignment of information from different modalities by minimizing this loss function, and calculates the anomaly score AS as follows:

[0030] AS=1-max{sim(z,zj)|zj∈Z}·exp(-υ·var({sim(z,zj)|zj∈Z}));

[0031] Where z is the feature representation of the current sample, Z is the feature set of normal samples; υ is the variance penalty coefficient, which increases the anomaly score when the similarity fluctuates greatly; var is the variance of the similarity set, which reflects the consistency of feature matching. The higher the anomaly score, the more likely the sample is to be in an abnormal state.

[0032] Optionally, the dynamic Bayesian decision network update method is as follows:

[0033] P(S|E,χ)=P(E|S,χ)·P(S|χ) / ∑P(E|S',χ)·P(S'|χ);

[0034] Where S represents the device state variable, containing multiple discrete states; E represents the observation evidence variable, reflecting sensor readings; χ represents the environmental condition variable, considering external factors such as temperature and humidity; P(S|E,χ) is the posterior probability that the device is in state S given the observation evidence E and environmental condition χ; P(E|S,χ) is the likelihood probability, representing the probability that the device will produce observation E under state S and environmental condition χ; P(S|χ) is the prior probability considering environmental factors; S' is a summation variable, representing each possible state in the state space; ∑P(E|S',χ)·P(S'|χ) represents the summation over all possible states S', and this summation constitutes the normalization constant (also called marginal likelihood or evidence) in Bayes' theorem; the relationships between nodes in the network are represented by the conditional probability table CPT. t express;

[0035] CPTt The update method is as follows:

[0036] CPT t =γ t ·CPT t-1 +(1-γ t )·CPTnew·f(Δt);

[0037] γ t =γbase·exp(-φ·R t );

[0038] Among them, CPT t Here is the conditional probability table at time t; CPT t-1 CPTnew is the conditional probability table at time t-1; CPTnew is the conditional probability table calculated based on the new observation data, γ t γ is the smoothing coefficient, ranging from [0,1], controlling the weight ratio of historical and new information; γbase is the basic smoothing coefficient, ranging from [0.7,0.95]; φ is the anomaly coefficient, controlling the degree of influence of abnormal events on model updates; R t Let f(Δt) be the anomaly rate at time t, which measures the degree of difference between the current observation and the historical model; f(Δt) is the time decay function, defined as exp(-λ·Δt), where Δt is the time interval and λ is the decay rate.

[0039] Optionally, the adaptive inspection strategy optimization is based on a multi-objective optimization model:

[0040] min J=[J1(x,y),J2(x,y),...,J m [(x,y)];

[0041] stg(x,y)≤0,h(x,y)=0,x∈X,y∈Y;

[0042] Where x is the inspection decision variable vector, including decision variables such as inspection frequency, resource allocation, and route selection; y is the environmental variable vector, including temperature, humidity, and vibration; J1 to J m Let m be the optimization objective functions, including inspection cost, fault detection rate, equipment availability, etc.; g(x,y) is the inequality constraint function, representing resource constraints; h(x,y) is the equality constraint function, representing the equilibrium conditions that must be satisfied; X is the feasible region of the decision variables, representing physical and operational constraints; Y is the range of values ​​for environmental variables.

[0043] The formula for calculating the inspection frequency f is:

[0044] f=fbase·(1+δ·RPN)·Φ(t,ω);

[0045] Where fbase is the basic inspection frequency, a standard frequency determined according to the equipment type; δ is the risk adjustment coefficient, ranging from [0.1, 2.0]; RPN is the risk priority number, which comprehensively considers multiple risk factors; Φ(t, ω) is the time modulation function, where t is the equipment running time and ω is the seasonal parameter;

[0046] RPN is calculated using the following formula:

[0047] RPN=S1^αs·O^αo·D1^αd·In^αi·C^αc;

[0048] Wherein, S1 is the fault severity factor, reflecting the degree of impact of the fault on production; O is the fault occurrence probability factor, reflecting the possibility of the fault occurring; D1 is the fault detectability factor, reflecting the ease with which the fault can be detected; In is the equipment importance factor, reflecting the criticality of the equipment in production; C is the environmental complexity factor, reflecting the complexity of the inspection environment; αs, αo, αd, αi, αc are the weight indices of the corresponding factors, satisfying αs+αo+αd+αi+αc=1.

[0049] Optionally, presenting the inspection results to the inspection personnel in an augmented reality manner includes:

[0050] A spatial registration algorithm is used to achieve precise alignment between virtual information and real devices. The registration error E1 is calculated as follows:

[0051] E1=√(∑w i ·||T v (p i ,d i )-p' i || 2 / ∑w i );

[0052] Where, p i Let d be the three-dimensional coordinates of the i-th feature point in the real world; i Provides additional matching information for the descriptor vectors of feature points; p i ' represents the corresponding image coordinates; T v Given the view transformation matrix, consider perspective distortion compensation; w i The feature point weights are dynamically allocated based on reliability.

[0053] Adaptive rendering technology is used to adjust the display effect of virtual content according to ambient light conditions. The brightness adjustment function B is defined as:

[0054] B(I,χe)=I max ·(I / I amh )^(1 / γv)·(1-xξ·cos(θ v ));

[0055] Where I is the original brightness value; χe is the environmental characteristic vector, which includes parameters such as color temperature and contrast; I amh This refers to the measured ambient light level; I max γv is the maximum display brightness; γv is the viewing angle adaptive gamma correction parameter, which adjusts according to changes in the user's line of sight; xξ is the viewing angle attenuation coefficient, which controls the degree to which the viewing angle affects the brightness; θ v The viewing angle is the angle between the user's line of sight and the display plane.

[0056] Optionally, before the step of collecting multimodal data of special equipment, a digital twin modeling step of the equipment is also included:

[0057] A high-precision geometric model is constructed based on the equipment CAD model and laser scanning point cloud data;

[0058] A physical behavior model of the equipment is established through finite element analysis, and the state equation is:

[0059]

[0060] Where M is the mass matrix, representing the mass distribution of each part of the system; It is the acceleration vector; This is the temperature-dependent damping matrix, considering the effect of temperature T on the damping characteristics; is the velocity vector; K(x1,T) is the temperature-dependent stiffness matrix, considering the influence of temperature T on material stiffness; x1 is the displacement vector; F(t) is the known external excitation force vector, such as the driving force; G(t,ν) is the random disturbance force vector, where ν is the disturbance parameter;

[0061] A digital twin model is trained using deep reinforcement learning to predict device states. The state transition function Sf is defined as follows:

[0062]

[0063] Where Sf(t) is the system state at time t, which includes physical and operational parameters; A(t) is the control action vector, representing operational intervention; W(t) is the random disturbance vector, simulating environmental noise; and Π(t) is the operational parameter vector, reflecting the impact of human intervention. is a nonlinear state transition mapping function, modeled by a deep neural network; ∈(t) is a time-varying residual coefficient, which controls the influence of the residual term; R(Sf(t),t) is a state-dependent residual function, which captures the dynamic characteristics that the model fails to express.

[0064] Another aspect of the present invention provides a special equipment inspection system based on a multimodal large model, for executing a special equipment inspection method based on a multimodal large model, comprising: an intelligent data acquisition device and a server;

[0065] The intelligent data acquisition device is configured to: acquire multimodal data from special equipment;

[0066] The server is configured as follows:

[0067] The multimodal data is preprocessed to obtain the first multimodal data;

[0068] A spatiotemporal fusion attention network is constructed, and spatiotemporal domain features are extracted from the first multimodal data through the spatiotemporal fusion attention network to obtain a fusion feature representation;

[0069] The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning.

[0070] Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions.

[0071] Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization.

[0072] The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

[0073] The technical solution of this invention provides a special equipment inspection method based on a multimodal large model, comprising: collecting multimodal data of the special equipment; preprocessing the multimodal data to obtain first multimodal data; constructing a spatiotemporal fusion attention network, and extracting spatiotemporal features from the first multimodal data through the spatiotemporal fusion attention network to obtain a fused feature representation; using a pre-trained multimodal large model to perform inference analysis on the fused feature representation, wherein the multimodal large model maps and distinguishes the normal and abnormal states of the special equipment in a high-dimensional feature space through contrastive learning; constructing a dynamic Bayesian decision network based on historical equipment inspection data and an expert knowledge base, and combining the inference results of the multimodal large model with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions; dynamically adjusting the inspection path and frequency according to the preset equipment importance level and fault warning level to achieve adaptive inspection strategy optimization; presenting the inspection results to the inspection personnel in an augmented reality manner, and simultaneously comparing the equipment digital twin model with the actual inspection data in real time, marking abnormal areas and providing maintenance guidance suggestions. This solution significantly improves the coverage and accuracy of anomaly detection; maximizes the extraction of effective information from various types of data and reduces noise interference; enhances the ability to identify gradual faults, enables effective detection of unseen fault types, reduces false alarm rates, improves inspection efficiency, and effectively enhances maintenance efficiency and accuracy; the overall solution significantly reduces maintenance costs, extends equipment lifespan, and improves production safety by improving fault prediction accuracy, reducing false alarms and missed detections, and optimizing inspection paths. Attached Figure Description

[0074] Figure 1 This is a flowchart of a special equipment inspection method based on a multimodal large model provided in an embodiment of the present invention;

[0075] Figure 2 This is a schematic block diagram of a special equipment inspection system based on a multimodal large model provided in an embodiment of the present invention. Detailed Implementation

[0076] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0077] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0078] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0079] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0080] The following reference Figures 1 to 2 This invention describes a special equipment inspection system and method based on a multimodal large model, provided by some embodiments of the present invention.

[0081] like Figure 1 As shown, one embodiment of the present invention provides a special equipment inspection method based on a multimodal large model, comprising:

[0082] Collect multimodal data from special equipment;

[0083] In this embodiment, the multimodal data includes visual data, acoustic data, thermal imaging data, vibration data, and equipment operation data;

[0084] The multimodal data is preprocessed to obtain the first multimodal data, including: denoising and enhancing the visual data, performing spectral analysis on the acoustic data, performing temperature gradient analysis on the thermal imaging data, performing Fourier transform on the vibration data, and standardizing the equipment operation data.

[0085] A spatiotemporal fusion attention network is constructed (the spatiotemporal fusion attention network includes a temporal attention module for processing data at different time scales and a spatial attention module for processing data from different parts of the device). The spatiotemporal domain features of the first multimodal data are extracted through the spatiotemporal fusion attention network to obtain a fusion feature representation.

[0086] The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning.

[0087] Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions.

[0088] Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization.

[0089] The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

[0090] The technical solution adopted in this embodiment achieves comprehensive perception of the status of special equipment by collecting multimodal data such as visual, acoustic, thermal imaging, vibration, and operating parameters, significantly improving the coverage and accuracy of anomaly detection. Targeted preprocessing of the multimodal data (such as visual enhancement, spectrum analysis, and temperature gradient analysis) maximizes the extraction of effective information from various data types and reduces noise interference. Through a spatiotemporal fusion attention network, the correlation of operating states at different points in time and different parts of the equipment is effectively captured, identifying complex spatiotemporal anomaly patterns and improving the ability to identify gradual faults. High-dimensional feature mapping and comparative learning using a pre-trained multimodal large model can distinguish between normal and abnormal states, achieving effective detection of unseen fault types. Combining a dynamic Bayesian decision network with historical data... The system leverages data and expert knowledge bases to provide highly reliable fault warnings and targeted maintenance suggestions, reducing false alarm rates. It dynamically adjusts inspection paths and frequencies based on equipment importance and fault risk, optimizing inspection resource allocation and improving efficiency. Augmented reality technology visually presents abnormal areas to inspection personnel, while digital twin technology provides maintenance guidance, effectively enhancing maintenance efficiency and accuracy. The overall solution significantly reduces maintenance costs, extends equipment lifespan, and improves production safety by improving fault prediction accuracy, reducing false alarms and missed detections, and optimizing inspection paths. This multimodal, large-scale model-based special equipment inspection method overcomes the limitations of traditional single-sensor methods, transforming the inspection mode from passive response to proactive prediction, and providing a new technological path for special equipment safety management.

[0091] In some possible embodiments of the present invention, the multimodal large model employs a hierarchical attention mechanism for intermodal information fusion, including:

[0092] Extract the internal features of each modality to obtain the intramodal representation;

[0093] Construct cross-modal attention graphs to establish connections between data from different modalities;

[0094] By capturing long-range dependencies through a self-attention mechanism, effective fusion of information between modalities can be achieved.

[0095] In some possible embodiments of the present invention, the dynamic Bayesian decision network is updated in the following manner:

[0096] Compare the current inspection results with historical data to calculate the state transition probability matrix;

[0097] The network parameters are updated based on new samples, and an incremental learning approach is used to maintain the model's adaptability.

[0098] Regularly optimize the network structure and dynamically adjust the causal relationships between nodes.

[0099] In some possible embodiments of the present invention, the adaptive inspection strategy optimization includes:

[0100] Allocate inspection resources based on anomaly warning levels;

[0101] A reinforcement learning framework is adopted, with inspection efficiency and fault detection rate as reward functions, to optimize the inspection path;

[0102] A personalized inspection task allocation plan is generated by taking into account the professional background of the inspection personnel and the location of the equipment.

[0103] In some possible embodiments of the present invention, the special equipment inspection method based on a multimodal large model further includes:

[0104] The inspection data is sent back to the cloud knowledge base to update the equipment lifecycle management model.

[0105] Based on the federated learning framework, while protecting data privacy, the model is jointly trained using inspection data from multiple similar devices to improve the model's generalization ability.

[0106] Quantum enhancement algorithms accelerate the model update process and improve the efficiency of large-scale data processing.

[0107] In some possible embodiments of the present invention, the step of collecting multimodal data from special equipment includes:

[0108] Visual data is acquired using a high-resolution camera array, which includes RGB cameras and depth cameras. The optimal acquisition position is determined according to the following formula:

[0109] P=argmin{∑(ω1·d(p,v1,ξ)+ω2·c(p,η)+ω3·o(p,e,μ)+ω4·ρ(p,ε))};

[0110] Wherein, d(p,v1,ξ) represents the coverage metric function of position p to device viewpoint v1, ξ is the field of view parameter, considering the impact of focal length changes on coverage; c(p,η) represents the cost function of position p, η is the environmental complexity coefficient, reflecting the deployment difficulty in different environments; o(p,e,μ) represents the occlusion metric function of position p to device e, μ is the occlusion perspective coefficient, considering the information acquisition capability under partial occlusion; ρ(p,ε) represents the stability evaluation function, measuring the degree of environmental interference to position p, ε is the environmental vibration threshold; ω1, ω2, ω3, ω4 represent weighting coefficients, dynamically adjusted to balance the influence of various factors;

[0111] The device's operating parameters are collected using a distributed intelligent sensor network, and the sensor sampling frequency fs is adaptively adjusted according to the device's dynamic characteristics.

[0112] fs=max(2fmax·(1+θ·CT),κ·SNR -0.5 ·σ 2 •λenvironment);

[0113] Where fs represents the adaptive sampling frequency; fmax represents the maximum vibration frequency of the key components of the equipment; θ represents the temperature compensation coefficient, considering the influence of temperature on the vibration characteristics of the equipment; CT represents the temperature difference between the equipment surface and the environment; κ represents the adjustment coefficient, which changes dynamically according to the equipment status; SNR represents the signal-to-noise ratio; σ 2 λ represents the signal variance, ensuring sufficient sampling accuracy under various operating conditions; λenvironment represents the environmental impact factor, taking into account environmental factors such as humidity and electromagnetic interference.

[0114] In some possible embodiments of the present invention, the spatiotemporal fusion attention network achieves the fusion of temporal and spatial features through the following methods:

[0115] F=σa(Wτ·[H1,H2,...,Hτmax]·Qτ+bτ)⊙σa(Ws·[S1,S2,...,Ssmax]·Qs+bs)+ζ·I(Hτ,Ss);

[0116] Where F represents the fused feature representation, H1 to Hτmax represent the feature vectors at τmax time points, S1 to Ssmax represent the feature vectors at smax spatial locations, Wτ and Ws are the temporal and spatial attention weight matrices, respectively; Qτ represents the temporal feature projection matrix, used for dimensionality reduction and feature enhancement; Qs represents the spatial feature projection matrix, used for dimensionality reduction and feature enhancement; bτ represents the temporal attention bias term, bs represents the spatial attention bias term, σa is the Swish activation function, defined as x3·sigmoid(x3), providing nonlinear transformation capability; ⊙ is the Hadamard product (element-wise multiplication); ζ is the mutual information weight coefficient, controlling the degree of influence of spatiotemporal information complementarity; I(Hτ,Ss): the mutual information measurement function between temporal and spatial features, capturing spatiotemporal correlation. The fusion method in this embodiment can adaptively capture the spatiotemporal change pattern of device operating status, improving the accuracy of anomaly detection.

[0117] In some possible embodiments of the present invention, the multimodal large model distinguishes between the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning. The multimodal large model employs a cross-modal contrastive learning framework, and the loss function is defined as:

[0118] L=-log[exp(sim(z1,z2) / τtemp) / ∑exp(sim(z1,z) / τtemp)]+Ω·∑||z i || 2 +γ·D(z1,z2,zneg);

[0119] Where L is the contrastive learning loss function, z1 and z2 are the feature representations of different modalities under the same device state; sim is the cosine similarity function, which measures the similarity of feature vectors; τtemp is a temperature parameter that controls the concentration of feature distribution, ranging from [0.1, 1.0]; Ω is the regularization coefficient, which controls the L2 norm of the feature vector to prevent feature collapse; ||z i ||2 represents the L2 norm of the feature vectors, promoting a uniform distribution in the feature space; γ represents the triplet loss weights, enhancing inter-class discrimination; D is the triplet distance function, defined as [d1(z1,zneg)-d1(z1,z2)+margin] + , where d1 is the Euclidean distance, margin is the boundary parameter; zneg is the negative sample feature representation;

[0120] The model achieves semantic alignment of information from different modalities by minimizing this loss function, and calculates the anomaly score AS as follows:

[0121] AS=1-max{sim(z,zj)|zj∈Z}·exp(-υ·var({sim(z,zj)|zj∈Z}));

[0122] Where z is the feature representation of the current sample, Z is the feature set of normal samples; υ is the variance penalty coefficient, which increases the anomaly score when the similarity fluctuates greatly; var is the variance of the similarity set, which reflects the consistency of feature matching. The higher the anomaly score, the more likely the sample is to be in an abnormal state.

[0123] In some possible embodiments of the present invention, the dynamic Bayesian decision network update method is as follows:

[0124] P(S|E,χ)=P(E|S,χ)·P(S|χ) / ∑P(E|S',χ)·P(S'|χ);

[0125] Where S represents the device state variable, containing multiple discrete states; E represents the observation evidence variable, reflecting sensor readings; χ represents the environmental condition variable, considering external factors such as temperature and humidity; P(S|E,χ) is the posterior probability that the device is in state S given the observation evidence E and environmental condition χ; P(E|S,χ) is the likelihood probability, representing the probability that the device will produce observation E under state S and environmental condition χ; P(S|χ) is the prior probability considering environmental factors; S' is a summation variable, representing each possible state in the state space; ∑P(E|S',χ)·P(S'|χ) represents the summation over all possible states S', and this summation constitutes the normalization constant (also called marginal likelihood or evidence) in Bayes' theorem; the relationships between nodes in the network are represented by the conditional probability table CPT. t express;

[0126] CPT t The update method is as follows:

[0127] CPT t =γ t ·CPT t-1 +(1-γ t )·CPTnew·f(Δt);

[0128] γ t =γbase·exp(-φ·R t );

[0129] Among them, CPT t Here is the conditional probability table at time t; CPT t-1 CPTnew is the conditional probability table at time t-1; CPTnew is the conditional probability table calculated based on the new observation data, γ tγ is the smoothing coefficient, ranging from [0,1], controlling the weight ratio of historical and new information; γbase is the basic smoothing coefficient, ranging from [0.7,0.95]; φ is the anomaly coefficient, controlling the degree of influence of abnormal events on model updates; R t Let f(Δt) be the anomaly rate at time t, which measures the degree of difference between the current observation and the historical model; f(Δt) is the time decay function, defined as exp(-λ·Δt), where Δt is the time interval and λ is the decay rate.

[0130] In some possible embodiments of the present invention, the adaptive inspection strategy optimization is based on a multi-objective optimization model:

[0131] min J=[J1(x,y),J2(x,y),...,J m [(x,y)];

[0132] stg(x,y)≤0,h(x,y)=0,x∈X,y∈Y;

[0133] Where x is the inspection decision variable vector, including decision variables such as inspection frequency, resource allocation, and route selection; y is the environmental variable vector, including temperature, humidity, and vibration; J1 to J m Let m be the optimization objective functions, including inspection cost, fault detection rate, equipment availability, etc.; g(x,y) is the inequality constraint function, representing resource constraints; h(x,y) is the equality constraint function, representing the equilibrium conditions that must be satisfied; X is the feasible region of the decision variables, representing physical and operational constraints; Y is the range of values ​​for environmental variables.

[0134] The formula for calculating the inspection frequency f is:

[0135] f=fbase·(1+δ·RPN)·Φ(t,ω);

[0136] Where fbase is the basic inspection frequency, a standard frequency determined according to the equipment type; δ is the risk adjustment coefficient, ranging from [0.1, 2.0]; RPN is the risk priority number, which comprehensively considers multiple risk factors; Φ(t, ω) is the time modulation function, where t is the equipment running time and ω is the seasonal parameter;

[0137] RPN is calculated using the following formula:

[0138] RPN=S1^αs·O^αo·D1^αd·In^αi·C^αc;

[0139] Wherein, S1 is the fault severity factor, reflecting the degree of impact of the fault on production; O is the fault occurrence probability factor, reflecting the possibility of the fault occurring; D1 is the fault detectability factor, reflecting the ease with which the fault can be detected; In is the equipment importance factor, reflecting the criticality of the equipment in production; C is the environmental complexity factor, reflecting the complexity of the inspection environment; αs, αo, αd, αi, αc are the weight indices of the corresponding factors, satisfying αs+αo+αd+αi+αc=1.

[0140] In some possible embodiments of the present invention, presenting the inspection results to the inspection personnel in an augmented reality manner includes:

[0141] A spatial registration algorithm is used to achieve precise alignment between virtual information and real devices. The registration error E1 is calculated as follows:

[0142] E1=√(∑w i ·||T v (p i ,d i )-p' i || 2 / ∑w i );

[0143] Where, p i Let d be the three-dimensional coordinates of the i-th feature point in the real world; i Provides additional matching information for the descriptor vectors of feature points; p i ' represents the corresponding image coordinates; T v Given the view transformation matrix, consider perspective distortion compensation; w i The feature point weights are dynamically allocated based on reliability.

[0144] Adaptive rendering technology is used to adjust the display effect of virtual content according to ambient light conditions. The brightness adjustment function B is defined as:

[0145] B(I,χe)=I max ·(I / I amh )^(1 / γv)·(1-xξ·cos(θ v ));

[0146] Where I is the original brightness value; χe is the environmental characteristic vector, which includes parameters such as color temperature and contrast; I amh This refers to the measured ambient light level; I max γv is the maximum display brightness; γv is the viewing angle adaptive gamma correction parameter, which adjusts according to changes in the user's line of sight; xξ is the viewing angle attenuation coefficient, which controls the degree to which the viewing angle affects the brightness; θ v The viewing angle is the angle between the user's line of sight and the display plane.

[0147] In some possible embodiments of the present invention, a digital twin modeling step for the equipment is included before the step of acquiring multimodal data of the special equipment:

[0148] A high-precision geometric model is constructed based on the equipment CAD model and laser scanning point cloud data;

[0149] A physical behavior model of the equipment is established through finite element analysis, and the state equation is:

[0150]

[0151] Where M is the mass matrix, representing the mass distribution of each part of the system; It is the acceleration vector; This is the temperature-dependent damping matrix, considering the effect of temperature T on the damping characteristics; is the velocity vector; K(x1,T) is the temperature-dependent stiffness matrix, considering the influence of temperature T on material stiffness; x1 is the displacement vector; F(t) is the known external excitation force vector, such as the driving force; G(t,ν) is the random disturbance force vector, where ν is the disturbance parameter;

[0152] A digital twin model is trained using deep reinforcement learning to predict device states. The state transition function Sf is defined as follows:

[0153]

[0154] Where Sf(t) is the system state at time t, which includes physical and operational parameters; A(t) is the control action vector, representing operational intervention; W(t) is the random disturbance vector, simulating environmental noise; and Π(t) is the operational parameter vector, reflecting the impact of human intervention. is a nonlinear state transition mapping function, modeled by a deep neural network; ∈(t) is a time-varying residual coefficient, which controls the influence of the residual term; R(Sf(t),t) is a state-dependent residual function, which captures the dynamic characteristics that the model fails to express.

[0155] In some possible embodiments of the present invention, the multimodal large model adopts a hierarchical structure:

[0156] The bottom layer is a modality-specific feature extraction network, and the feature extraction function for each modality m is:

[0157]

[0158] Where x2 is the input data, W k W and Wv are the weight matrices for query, key, and value, respectively; MultiHeadAttention is a multi-head attention mechanism to improve feature extraction capabilities; M mModality-specific attention masks highlight key information regions; MLP stands for Multilayer Perceptron, employing nonlinear transformations; D r The dropout rate controls the degree of network regularization; LayerNorm normalizes layers to stabilize the training process.

[0159] The middle layer is a cross-modal fusion network, and the fusion function is:

[0160] g([f1,...,f n ])=∑(α i ·U i ·f i )+β·CrossAttention([f1,...,f n ]);

[0161] Among them, f i U represents the feature representation of the i-th mode; i The feature projection matrix is ​​used to align the feature space; α i β is the attention weight, which controls the importance of each modality; β is the cross-modal attention weight coefficient; CrossAttention is the cross-modal attention function, which captures the inter-modal relationships.

[0162] α i Calculated using the softmax function:

[0163] α i =exp(e i ) / ∑exp(e j );

[0164] e i =v T ·tanh(W e ·f i +b e )+r T ·f i ;

[0165] Among them, e i The attention score for the i-th modality is given by v; the attention parameter vector is given by W. e b is the attention weight matrix; e is the attention bias vector; r is the residual connection parameter vector, which enhances gradient flow capability;

[0166] The top layer is a decision network, composed of a Transformer structure, with a multi-head attention mechanism. The attention of each head h is calculated as follows:

[0167] head_h=Attention(Q·W_h^Q,K·W_h^K,V·W_h^V);

[0168] Where Q, K, and V are the query, key, and value matrices, respectively, and W_h^Q, W_h^K, and W_h^V are the corresponding weight matrices.

[0169] In some possible embodiments of the present invention, the preprocessing of the multimodal data includes:

[0170] A method combining adaptive wavelet transform and convolutional neural network is used for denoising and enhancing visual data:

[0171]

[0172] Where tI represents the original image, WT represents the wavelet transform, and WT -1 The inverse wavelet transform (FT) is represented by FT, and the wavelet domain threshold function (FT) is represented by FT. The parameter controls the soft / hard thresholding characteristics; tG is a global image enhancement parameter that controls contrast and sharpness.

[0173] λoptimal represents the adaptive threshold parameter, which is dynamically adjusted based on the local signal-to-noise ratio of the image.

[0174] λoptimal=β1·log(σ 2 noise / σ 2 signal)·√(2log(N))·(1+ψ·entropy(tI));

[0175] Where β1 is the adjustment coefficient, ranging from [0.5, 2]; σ 2 noise is the estimated noise variance; σ 2 signal is the estimated signal variance, N is the number of pixels in the image; entropy(tI) is the image entropy, which quantifies the information richness of the image; CNN represents a convolutional neural network for image enhancement, with the training objective being to maximize the structural similarity index SSIM.

[0176] In some possible embodiments of the present invention, the preprocessing of the multimodal data includes processing the vibration data using a combination of wavelet packet transform and empirical mode decomposition, specifically:

[0177] Wavelet packet transform is used to extract the time-frequency features of vibration signals:

[0178] WPT(f,j,k,ψt1)=∫f(t)·ψ j,k,t 1(t)dt;

[0179] Where f(t) is the vibration signal; ψ j,k,tFor parameterized wavelet basis functions, t1 is the basis function type index; j is the decomposition scale, controlling the frequency resolution; k is the translation parameter, controlling the time positioning.

[0180] Empirical Mode Decomposition (EMD) is used for adaptive separation of the Intrinsic Mode Function (IMF) of vibration signals.

[0181] f(t)=∑c i (t)·Π i (t)+r(t)·Ψ(t);

[0182] Among them, c i (t) represents the i-th intrinsic mode function (IMF) component; Π i Ψ(t) is the IMF weighting function, which is dynamically adjusted according to the signal energy distribution; r(t) is the residual term; Ψ(t) is the residual weighting function, which controls the contribution of the residual term.

[0183] Calculate the Hilbert spectrum for each IMF:

[0184] H(Jω,t)=∑a i (t)·exp(j·∫Jω i (t)dt)·Γ i (Jω,t);

[0185] Where H(ω,t) is the Hilbert spectrum, representing the time-frequency distribution of the signal; Jω is the angular frequency; a i (t) represents the instantaneous amplitude of the i-th IMF, Jω i (t) is the instantaneous frequency, calculated using the Hilbert transform:

[0186]

[0187] in, For c i Hilbert transform of (t); Γ i (Jω,t) is the spectral correction function, which enhances spectral resolution; η1 is the amplitude correction coefficient, which controls the smoothness of amplitude calculation; a i '(t) is the first derivative of the amplitude, reflecting the trend of amplitude change; yξ i d represents the frequency correction coefficient, used to compensate for second-order frequency variations. 2 / dt 2 The second-order time derivative is used to capture the acceleration change at the capture frequency. The processing method in this embodiment can effectively extract the nonlinear and non-stationary characteristics in equipment vibration signals, improving the accuracy of fault diagnosis.

[0188] Please see Figure 2Another embodiment of the present invention provides a special equipment inspection system based on a multimodal large model, for executing a special equipment inspection method based on a multimodal large model, including: intelligent acquisition device and server;

[0189] The intelligent data acquisition device is configured to: acquire multimodal data from special equipment;

[0190] The server is configured as follows:

[0191] The multimodal data is preprocessed to obtain the first multimodal data;

[0192] A spatiotemporal fusion attention network is constructed, and spatiotemporal domain features are extracted from the first multimodal data through the spatiotemporal fusion attention network to obtain a fusion feature representation;

[0193] The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning.

[0194] Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions.

[0195] Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization.

[0196] The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

[0197] It should be known that, Figure 2 The block diagram of the special equipment inspection system based on a multimodal large model shown is for illustrative purposes only, and the number of modules shown does not limit the scope of protection of this invention. The special equipment inspection system based on a multimodal large model provided in this embodiment can be used to execute various embodiments of the corresponding special equipment inspection method based on a multimodal large model. For specific implementation details, please refer to the descriptions of the respective method embodiments, which will not be repeated here.

[0198] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0199] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0201] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0203] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0204] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0205] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0206] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of the present invention, and various modifications and alterations can be made, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of the present invention.

Claims

1. A method for inspecting special equipment based on a multimodal large model, characterized in that, include: Collect multimodal data from special equipment; The multimodal data is preprocessed to obtain the first multimodal data; A spatiotemporal fusion attention network is constructed, and spatiotemporal domain features are extracted from the first multimodal data through the spatiotemporal fusion attention network to obtain a fusion feature representation; The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning. Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions. Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization. The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

2. The special equipment inspection method based on a multimodal large model according to claim 1, characterized in that, The multimodal large model employs a hierarchical attention mechanism for intermodal information fusion, including: Extract the internal features of each modality to obtain the intramodal representation; Construct cross-modal attention graphs to establish connections between data from different modalities; By capturing long-range dependencies through a self-attention mechanism, effective fusion of information between modalities can be achieved.

3. The special equipment inspection method based on a multimodal large model according to claim 2, characterized in that, The steps for collecting multimodal data from special equipment include: Visual data is acquired using a high-resolution camera array, which includes RGB cameras and depth cameras. The optimal acquisition position is determined according to the following formula: P = argmin{∑(ω1·d(p,v1,ξ) + ω2·c(p,η) + ω3·o(p,e,μ) + ω4·ρ(p,ε))}; Wherein, d(p,v1,ξ) represents the coverage metric function of position p to device viewpoint v1, ξ is the field of view parameter, considering the impact of focal length changes on coverage; c(p,η) represents the cost function of position p, η is the environmental complexity coefficient, reflecting the deployment difficulty in different environments; o(p,e,μ) represents the occlusion metric function of position p to device e, μ is the occlusion perspective coefficient, considering the information acquisition capability under partial occlusion; ρ(p,ε) represents the stability evaluation function, measuring the degree of environmental interference to position p, ε is the environmental vibration threshold; ω1, ω2, ω3, ω4 represent weighting coefficients, dynamically adjusted to balance the influence of various factors; The device's operating parameters are collected using a distributed intelligent sensor network, and the sensor sampling frequency fs is adaptively adjusted according to the device's dynamic characteristics. fs = max(2fmax·(1+θ·CT), κ·SNR -0 · 5 ·σ²·λenvironment); Where fs represents the adaptive sampling frequency; fmax represents the maximum vibration frequency of the key components of the equipment; θ represents the temperature compensation coefficient, which takes into account the influence of temperature on the vibration characteristics of the equipment; CT represents the temperature difference between the equipment surface and the environment; κ represents the adjustment coefficient, which changes dynamically according to the equipment status; SNR represents the signal-to-noise ratio; σ² represents the signal variance, which ensures sufficient sampling accuracy under various operating conditions; and λenvironment represents the environmental impact factor.

4. The special equipment inspection method based on a multimodal large model according to claim 3, characterized in that, The spatiotemporal fusion attention network achieves the fusion of temporal and spatial features through the following methods: F = σa(Wτ·[H1, H2, ..., Hτmax]·Qτ + bτ)⊙σa(Ws·[S1, S2, ..., Ssmax]·Qs+ bs) + ζ·I(Hτ, Ss); Where F represents the fused feature representation, H1 to Hτmax represent the feature vectors at τmax time points, S1 to Ssmax represent the feature vectors at smax spatial locations, Wτ and Ws are the temporal and spatial attention weight matrices, respectively; Qτ represents the temporal feature projection matrix, used for dimensionality reduction and feature enhancement; Qs represents the spatial feature projection matrix, used for dimensionality reduction and feature enhancement; bτ represents the temporal attention bias term, bs represents the spatial attention bias term, σa is the Swish activation function, defined as x3·sigmoid(x3), providing nonlinear transformation capability; ⊙ is the Hadamard product; ζ is the mutual information weight coefficient, controlling the degree of influence of spatiotemporal information complementarity; I(Hτ, Ss): the mutual information measurement function between temporal and spatial features, capturing spatiotemporal correlation.

5. The special equipment inspection method based on a multimodal large model according to claim 4, characterized in that, The multimodal large model uses contrastive learning to map and distinguish the normal and abnormal states of special equipment in a high-dimensional feature space. The multimodal large model employs a cross-modal contrastive learning framework, and the loss function is defined as: L=-log[exp(sim(z1,z2) / τtemp) / ∑exp(sim(z1,z) / τtemp)]+Ω·∑‖z i ‖²+γ·D(z1,z2,zneg); Where L is the contrastive learning loss function, z1 and z2 are the feature representations of different modalities under the same device state; sim is the cosine similarity function, which measures the similarity of feature vectors; τtemp is a temperature parameter that controls the concentration of feature distribution, ranging from [0.1, 1.0]; Ω is the regularization coefficient, which controls the L2 norm of feature vectors to prevent feature collapse; ||z|| i ||² represents the L2 norm of the feature vectors, promoting a uniform distribution in the feature space; γ represents the triple loss weights, enhancing inter-class discrimination; D is the triple distance function, defined as [d1(z1,zneg)-d1(z1,z2)+margin]. + , where d1 is the Euclidean distance, margin is the boundary parameter; zneg is the negative sample feature representation; The model achieves semantic alignment of information from different modalities by minimizing this loss function, and calculates the anomaly score AS as follows: AS = 1-max{sim(z,z j )|from j ∈Z}·exp(-υ·var({sim(z,z j )|from j ∈Z})); Where z is the feature representation of the current sample, Z is the feature set of normal samples; υ is the variance penalty coefficient, which increases the anomaly score when the similarity fluctuates greatly; var is the variance of the similarity set, reflecting the consistency of feature matching; the higher the anomaly score, the more likely the sample is to be in an abnormal state.

6. The special equipment inspection method based on a multimodal large model according to claim 5, characterized in that, The dynamic Bayesian decision network update method is as follows: P(S|E,χ) = P(E|S,χ)·P(S|χ) / ∑P(E|S',χ)·P(S'|χ); Where S represents the device state variable, containing multiple discrete states; E represents the observation evidence variable, reflecting sensor readings; χ represents the environmental condition variable, including temperature and humidity; P(S|E,χ) is the posterior probability that the device is in state S given the observation evidence E and environmental condition χ; P(E|S,χ) is the likelihood probability, representing the probability that the device will produce observation E under state S and environmental condition χ; P(S|χ) is the prior probability considering environmental factors; S' is a summation variable, representing each possible state in the state space; ∑P(E|S',χ)·P(S'|χ) represents the summation over all possible states S', and this summation constitutes the normalization constant in Bayes' formula; the relationships between nodes in the network are represented by the conditional probability table CPT. t express; CPT t The update method is as follows: CPT t = c t ·CPT t-1 + (1-c t )·CPTnew·f(Δt); c t = γbase·exp(- ·R t ); Among them, CPT t Here is the conditional probability table at time t; CPT t-1 Here is the conditional probability table for time t-1; CPTnew is the conditional probability table calculated based on the new observation data, γ t γbase is the smoothing coefficient, with a value range of [0,1], which controls the weight ratio of historical information to new information; γbase is the basic smoothing coefficient, with a value range of [0.7,0.95]. R is the anomaly coefficient, which controls the degree of impact of abnormal events on model updates. t Let f(Δt) be the anomaly rate at time t, which measures the degree of difference between the current observation and the historical model; f(Δt) is the time decay function, defined as exp(-λ·Δt), where Δt is the time interval and λ is the decay rate.

7. The special equipment inspection method based on a multimodal large model according to claim 6, characterized in that, The adaptive inspection strategy optimization is based on a multi-objective optimization model: min J = [J1(x,y), J2(x,y), ..., J m (x,y)]; st g(x,y) ≤ 0, h(x,y) = 0, x∈X, y∈Y; Where x is the inspection decision variable vector, including inspection frequency, resource allocation, and route selection; y is the environmental variable vector, including temperature, humidity, and vibration; J1 to J m Let m be the optimization objective functions, including inspection cost, fault detection rate, and equipment availability; g(x,y) is the inequality constraint function, representing resource constraints; h(x,y) is the equality constraint function, representing the equilibrium conditions that must be satisfied; X is the feasible region of the decision variables, representing physical and operational constraints; Y is the range of values ​​for environmental variables. The formula for calculating the inspection frequency f is: f = fbase·(1+δ·RPN)·Φ(t,ω); Where fbase is the basic inspection frequency, a standard frequency determined according to the equipment type; δ is the risk adjustment coefficient, ranging from [0.1, 2.0]; RPN is the risk priority number, which comprehensively considers multiple risk factors; Φ(t, ω) is the time modulation function, where t is the equipment running time and ω is the seasonal parameter; RPN is calculated using the following formula: RPN = S1^αs·O^αo·D1^αd·In^αi·C^αc; Wherein, S1 is the fault severity factor, reflecting the degree of impact of the fault on production; O is the fault occurrence probability factor, reflecting the possibility of the fault occurring; D1 is the fault detectability factor, reflecting the ease with which the fault can be detected; In is the equipment importance factor, reflecting the criticality of the equipment in production; C is the environmental complexity factor, reflecting the complexity of the inspection environment; αs, αo, αd, αi, αc are the weight indices of the corresponding factors, satisfying αs+αo+αd+αi+αc=1.

8. The special equipment inspection method based on a multimodal large model according to claim 7, characterized in that, Presenting inspection results to inspection personnel in an augmented reality manner includes: A spatial registration algorithm is used to achieve precise alignment between virtual information and real devices. The registration error E1 is calculated as follows: E1 = √(∑w i ·‖T v (p i ,d i ) - p' i ‖² / ∑w i ); Where, p i Let d be the three-dimensional coordinates of the i-th feature point in the real world; i Provides additional matching information for the descriptor vectors of feature points; p i ' represents the corresponding image coordinates; T v Given the view transformation matrix, consider perspective distortion compensation; w i The feature point weights are dynamically allocated based on reliability. Adaptive rendering technology is used to adjust the display effect of virtual content according to ambient light conditions. The brightness adjustment function B is defined as: B(I,xe) = I max ·(I / I amh )^(1 / γv)·(1-xξ·cos(θ v )); Where I is the original brightness value; χe is the environmental characteristic vector, which includes parameters such as color temperature and contrast; I amh This refers to the measured ambient light level; I max γv is the maximum display brightness; γv is the viewing angle adaptive gamma correction parameter, which adjusts according to changes in the user's line of sight; xξ is the viewing angle attenuation coefficient, which controls the degree to which the viewing angle affects the brightness; θ v The viewing angle is the angle between the user's line of sight and the display plane.

9. The special equipment inspection method based on a multimodal large model according to claim 8, characterized in that, Prior to the step of collecting multimodal data from special equipment, a digital twin modeling step for the equipment is also included: A high-precision geometric model is constructed based on the equipment CAD model and laser scanning point cloud data; A physical behavior model of the equipment is established through finite element analysis, and the state equation is: M + C( ,T)· + K(x1,T)·x1 = F(t) + G(t,ν); Where M is the mass matrix, representing the mass distribution of each part of the system; C is the acceleration vector; (T) is the temperature-dependent damping matrix, considering the effect of temperature T on damping characteristics; is the velocity vector; K(x1,T) is the temperature-dependent stiffness matrix, considering the influence of temperature T on material stiffness; x1 is the displacement vector; F(t) is the known external excitation force vector, including the driving force; G(t,ν) is the random disturbance force vector, where ν is the disturbance parameter; A digital twin model is trained using deep reinforcement learning to predict device states. The state transition function Sf is defined as follows: Sf(t+1) = fφ(Sf(t), A(t), W(t), Π(t)) + (t)·R(Sf(t),t); Where Sf(t) is the system state at time t, which includes physical and operational parameters; A(t) is the control action vector, representing operational intervention; W(t) is the random disturbance vector, simulating environmental noise; Π(t) is the operational parameter vector, reflecting the impact of human intervention; and fφ is the nonlinear state transition mapping function, modeled by a deep neural network. (t) is the time-varying residual coefficient, which controls the degree of influence of the residual term; R(Sf(t),t) is the state-dependent residual function, which captures the dynamic characteristics that the model fails to express.

10. A special equipment inspection system based on a multimodal large model, used to execute the special equipment inspection method based on a multimodal large model as described in any one of claims 1 to 9, characterized in that, include: Intelligent data acquisition devices and servers; The intelligent data acquisition device is configured to: acquire multimodal data from special equipment; The server is configured as follows: The multimodal data is preprocessed to obtain the first multimodal data; A spatiotemporal fusion attention network is constructed, and spatiotemporal domain features are extracted from the first multimodal data through the spatiotemporal fusion attention network to obtain a fusion feature representation; The fused feature representation is inferred and analyzed using a pre-trained multimodal large model. The multimodal large model maps and distinguishes the normal and abnormal states of special equipment in a high-dimensional feature space through contrastive learning. Based on historical equipment inspection data and expert knowledge base, a dynamic Bayesian decision network is constructed. The reasoning results of the multimodal large model are combined with the prior knowledge of the dynamic Bayesian decision network to generate fault warning levels and predictive maintenance suggestions. Based on the preset equipment importance level and fault warning level, the inspection path and frequency are dynamically adjusted to achieve adaptive inspection strategy optimization. The inspection results are presented to the inspection personnel in an augmented reality manner, and the digital twin model of the equipment is compared with the actual inspection data in real time to mark abnormal areas and provide maintenance guidance and suggestions.

Images