A method, device and system based on acoustic-thermal multimodal fusion imaging

Abstract

The present invention discloses a method, device and system based on acoustic-thermal multimodal fusion imaging, which relates to the field of artificial intelligence technology. The method includes: First, an acoustic imager is used to collect ultrasonic and thermal imaging data of a target industrial device, and acoustic and thermal imaging features are respectively obtained. After feature integration processing, a first acoustic-thermal joint feature is obtained to determine a first modal integration unit corresponding to the acoustic feature, and a first feature descriptor is obtained through cross-modal fusion. Similarly, a second acoustic-thermal joint feature, a second modal integration unit and a second feature descriptor are obtained. The fault evaluation result is determined by combining the fault type label. Then, the image data is calibrated to construct a three-dimensional acoustic-thermal fusion base image and the result is marked to obtain a diagnostic fusion image, so as to realize the accurate diagnosis of the target industrial device.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly, to a method, device, and system for acoustic-thermal multimodal fusion imaging. Background Art

[0002] In the field of industrial equipment detection, traditional single detection methods, such as relying solely on ultrasonic detection or thermal imaging detection, have limitations. Although ultrasonic detection can reflect the internal structure information of the equipment, it is difficult to detect potential faults caused by temperature changes; thermal imaging detection can detect temperature abnormal areas, but it cannot deeply understand the details of the internal structure of the equipment. Summary of the Invention

[0003] The purpose of the present invention is to provide a method, device, and system for acoustic-thermal multimodal fusion imaging.

[0004] In a first aspect, an embodiment of the present invention provides a method for acoustic-thermal multimodal fusion imaging, including:

[0005] Using an acoustic imager to collect ultrasonic images and thermal images of a target industrial equipment, obtaining ultrasonic image data and thermal image data;

[0006] Based on the ultrasonic image data, obtaining acoustic imaging features corresponding to the target industrial equipment, and based on the thermal image data, obtaining thermal imaging features corresponding to the target industrial equipment;

[0007] Performing feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and based on the first acoustic-thermal joint feature, determining a first modal integration unit corresponding to the acoustic imaging features among a plurality of modal integration units; the plurality of modal integration units are used to encode acoustic and thermal imaging features with different modal weight coefficients;

[0008] In the first modal integration unit, performing cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor;

[0009] Performing feature integration processing on the thermal imaging features and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and based on the second acoustic-thermal joint feature, determining a second modal integration unit corresponding to the thermal imaging features among the plurality of modal integration units;

[0010] In the second modal integration unit, performing cross-modal fusion on the thermal imaging features according to the first feature descriptor to obtain a second feature descriptor;

[0011] Obtain the fault type label, and determine the fault evaluation result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor;

[0012] Calibrate the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain the diagnostic fusion image of the target industrial device.

[0013] In a second aspect, an embodiment of the present invention provides a device based on acoustic-thermal multimodal fusion imaging, including:

[0014] An acquisition module, configured to use an acoustic imager to perform ultrasonic image acquisition and thermal imaging acquisition on a target industrial device to obtain ultrasonic image data and thermal imaging data;

[0015] A fusion module, configured to obtain an acoustic imaging feature corresponding to a target industrial device based on the ultrasonic image data, and obtain a thermal imaging feature corresponding to the target industrial device based on the thermal imaging data; perform feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modal integration unit corresponding to the acoustic imaging feature in multiple modal integration units according to the first acoustic-thermal joint feature; the multiple modal integration units are configured to encode acoustic and thermal imaging features using different modal weight coefficients; in the first modal integration unit, perform cross-modal fusion on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor; perform feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determine a second modal integration unit corresponding to the thermal imaging feature in the multiple modal integration units according to the second acoustic-thermal joint feature; in the second modal integration unit, perform cross-modal fusion on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; obtain a fault type label, and determine the fault evaluation result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor;

[0016] An imaging module, configured to calibrate the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain the diagnostic fusion image of the target industrial device.

[0017] In a third aspect, the present invention provides a system based on acoustic-thermal multimodal fusion imaging, including a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the method based on acoustic-thermal multimodal fusion imaging described above is executed.

[0018] Compared with the prior art, the beneficial effects provided by the present invention include: By using the method, device and system based on acoustic-thermal multimodal fusion imaging disclosed in the present invention, ultrasonic and thermal imaging data of a target industrial device are collected by an ultrasonic imager, acoustic and thermal imaging features are respectively obtained, a first acoustic-thermal joint feature is obtained through feature integration processing to determine a first modal integration unit corresponding to the acoustic feature, cross-modal fusion is completed to obtain a first feature descriptor. Similarly, a second acoustic-thermal joint feature, a second modal integration unit and a second feature descriptor are obtained, a fault evaluation result is determined in combination with a fault type label, and then the image data is calibrated to construct a three-dimensional acoustic-thermal fusion base image and mark the result, obtaining a diagnostic fusion image, realizing accurate diagnosis of the target industrial device. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flowchart of the steps of the method based on acoustic-thermal multimodal fusion imaging provided by the embodiment of the present invention;

[0021] Figure 2 It is a schematic block diagram of the structure of the device based on acoustic-thermal multimodal fusion imaging provided by the embodiment of the present invention;

[0022] Figure 3 It is a schematic block diagram of the structure of the system based on acoustic-thermal multimodal fusion imaging provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] The following will describe in detail the specific embodiments of the present invention with reference to the drawings.

[0025] To solve the technical problems in the foregoing background art, Figure 1 FIG. 2 is a schematic flowchart of a method based on acoustic-thermal multimodal fusion imaging provided by an embodiment of the present disclosure. The method based on acoustic-thermal multimodal fusion imaging will be introduced in detail below.

[0026] Step S201: Use an ultrasonic imager to collect ultrasonic images and thermal images of the target industrial equipment, and obtain ultrasonic image data and thermal image data;

[0027] Step S202: Obtain acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and obtain thermal imaging features corresponding to the target industrial equipment based on the thermal image data;

[0028] Step S203: Perform feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain first acoustic-thermal joint features, and determine a first modal integration unit corresponding to the acoustic imaging features among multiple modal integration units; the multiple modal integration units are used to encode acoustic and thermal imaging features with different modal weight coefficients;

[0029] Step S204: In the first modal integration unit, perform cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor;

[0030] Step S205: Perform feature integration processing on the thermal imaging features and the first feature descriptor in the same feature alignment domain to obtain second acoustic-thermal joint features, and determine a second modal integration unit corresponding to the thermal imaging features among the multiple modal integration units;

[0031] Step S206: In the second modal integration unit, perform cross-modal fusion on the thermal imaging features according to the first feature descriptor to obtain a second feature descriptor;

[0032] Step S207: Obtain a fault type label, and determine a fault evaluation result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor;

[0033] Step S208: Calibrate the ultrasonic image data and the thermal image data based on a spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

[0034] In an embodiment of the present invention, by way of example, assume that the server is responsible for monitoring large stamping equipment in an automobile manufacturing factory. The server controls the audio-video device installed around the stamping equipment to perform ultrasonic image acquisition and thermal imaging acquisition on the equipment. The ultrasonic part of the audio-video device emits ultrasonic waves towards the stamping equipment. The ultrasonic waves propagate inside the equipment and will be reflected back when encountering different materials or structural changes. The audio-video device receives these reflected waves and converts them into ultrasonic image data. These data can present the structural details of the internal parts of the equipment, such as the integrity of the mold and the tightness of the connecting components. At the same time, the thermal imaging part of the audio-video device detects the thermal radiation on the surface of the equipment and generates thermal imaging data, reflecting the temperature distribution of each area on the surface of the equipment, such as the temperature conditions of parts like the stamping head and transmission components. The server acquires and stores these ultrasonic image data and thermal imaging data. The server divides the acquired ultrasonic imaging data into multiple local ultrasonic images according to the structural areas of the equipment, such as separately dividing out the mold area, the transmission rod area, etc. Then, feature extraction is performed on these local ultrasonic images to extract feature descriptors of each local image, such as edge features, texture features, etc. Then, by calculating the importance of the features of each local image, a dynamic weight is assigned to it to obtain a feature significance coefficient. Finally, the feature descriptor and the significance coefficient are linearly superimposed to obtain an acoustic imaging feature reflecting the acoustic characteristics of the stamping equipment. The server performs thermal radiation feature segmentation on the thermal imaging data to obtain multi-scale thermodynamic feature maps reflecting the thermal state of the equipment at different scales. By constructing a thermal imaging feature pyramid network and using its spatial attention layer to enhance key thermal features, enhanced thermodynamic features are obtained. Then, it is aligned with real-time thermodynamic parameters (such as ambient temperature, normal operating temperature range of the equipment) to generate multi-dimensional thermal field fusion features. Finally, through spatio-temporal correlation coding by a thermodynamic feature encoder, the thermal imaging feature of the stamping equipment is obtained. The server inputs the acoustic imaging feature and the thermal imaging feature into an adaptive feature selection unit. Through a non-linear transformation network, the acoustic imaging feature is respectively transformed into a structural anomaly feature, and the thermal imaging feature is transformed into a thermodynamic state feature, so that the two are in the same feature alignment domain. Operations such as energy intensity value acquisition, normalization, enhancement, and aggregation are respectively performed on the structural anomaly feature and the thermodynamic state feature to obtain a first steady-state feature component and a second steady-state feature component. After the two are added, high-dimensional feature embedding is performed through an affine transformation network to obtain a first acoustic-thermal joint feature. Then, the server calculates the modal priority scores of the first acoustic-thermal joint feature and multiple modal integration units through a dynamic feature routing network, and determines the modal integration unit with the highest score as the first modal integration unit corresponding to the acoustic imaging feature. Assume that the first modal integration unit includes a cross-modal collaborative adapter.The server performs multimodal tensor stacking on the acoustic imaging features and thermal imaging features to obtain a first cross-modal joint tensor, and then transforms it into different vectors through a cross-modal collaborative adapter. After a series of vector operations, an acoustic-thermal correlation transition vector is constructed, and finally a first cross-modal correlation weight spectrum is obtained. The dominant correlation features in it are determined as the first feature descriptor. The server performs a feature integration process similar to the third step on the thermal imaging features and the first feature descriptor within the same feature alignment domain to obtain a second acoustic-thermal joint feature. Similarly, a second-modal integration unit corresponding to the thermal imaging features is determined through a dynamic feature routing network. In the second-modal integration unit, in a manner similar to the fourth step, the thermal imaging features are cross-modally fused according to the first feature descriptor to obtain a second feature descriptor. The server obtains fault type labels related to stamping equipment from the equipment failure database, such as "die crack", "wear of transmission components", etc. The first feature descriptor and the second feature descriptor are subjected to multimodal tensor stacking, and after configuring the fault type labels, a fused feature descriptor is obtained. The weights are calculated through a multimodal collaborative attention network to obtain a physical field coupling feature, and after high-dimensional feature embedding, the fault assessment result of the stamping equipment is determined, such as judging whether the equipment has faults and the specific fault type, severity, etc. The server first needs to clarify the coordinate systems of the ultrasonic image data and the thermal imaging data respectively. For ultrasonic images, their coordinates may be set based on the three-dimensional space of the internal structure of the equipment. Taking a certain fixed point of the equipment as the origin, the coordinate axes in the length, width, and height directions are determined to locate the positions of internal parts. The coordinates of the thermal imaging data are usually based on the two-dimensional plane of the equipment surface. Taking a corner point on the equipment surface as the origin, the horizontal and vertical coordinate axes are determined to represent the temperature distribution positions. The server establishes a spatial coordinate mapping relationship between the two. For example, through precise modeling of the equipment, knowing the coordinate position of a certain part inside the equipment in the ultrasonic image and also knowing the projection position of this part on the equipment surface, the mapping relationship between the coordinates of this part in the ultrasonic image and the corresponding projection point coordinates on the thermal imaging diagram is determined. Or using the known marking points on the equipment, these marking points can be recognized in both the ultrasonic image and the thermal imaging diagram, and the overall coordinate mapping relationship is constructed through the coordinates of the marking points. Based on the calibrated spatial coordinate mapping relationship, the server begins to construct a three-dimensional acoustic-thermal fusion base image. It presents the internal structure information of the equipment contained in the ultrasonic image data in the form of a three-dimensional model. For example, the three-dimensional shape of the die and the connection method of the transmission components can be seen. At the same time, the temperature information in the thermal imaging data is fused into this three-dimensional model in the form of colors or isosurfaces, etc. For example, areas with higher temperatures are represented in red on the three-dimensional model, and areas with lower temperatures are represented in blue, so that operators can intuitively see the temperature distribution of the equipment surface in three-dimensional space and its correspondence with the internal structure. The server marks the previously obtained fault assessment results on the three-dimensional acoustic-thermal fusion base image.If the fault assessment result shows that there are cracks in the mold, the server will mark the location and approximate trend of the cracks with a specific identifier (such as a red flashing line) at the corresponding position of the mold in the three-dimensional acoustic-thermal fusion base image, and attach a text description of the fault type as "mold crack" and the severity level (such as mild, moderate, severe). If wear of the transmission components is also detected, similar markings will be made at the corresponding positions of the transmission components, such as circling the worn parts with a yellow circle and marking information such as "wear of transmission components, severity level: moderate". The final diagnostic fusion image of the target industrial equipment (stamping equipment) enables the technical and maintenance personnel in the factory to comprehensively and intuitively understand the operating conditions of the equipment, including the internal structure, surface temperature, and possible faults, so as to quickly formulate maintenance and repair strategies.

[0035] In the embodiment of the present invention, the obtaining of the acoustic imaging features corresponding to the target industrial equipment includes:

[0036] Obtaining ultrasonic imaging data corresponding to the target industrial equipment, and dividing the ultrasonic imaging data into a plurality of local ultrasonic images;

[0037] Performing feature extraction processing on the plurality of local ultrasonic images to obtain image feature descriptors corresponding to the plurality of local ultrasonic images;

[0038] Performing dynamic weight assignment calculation on the image feature descriptors corresponding to the plurality of local ultrasonic images to obtain feature significance coefficients corresponding to the plurality of local ultrasonic images;

[0039] Linearly superimposing the image feature descriptors corresponding to the plurality of local ultrasonic images and the feature significance coefficients corresponding to the plurality of local ultrasonic images to obtain the acoustic imaging features corresponding to the ultrasonic imaging data.

[0040] In an embodiment of the present invention, exemplarily, the server is responsible for monitoring the steam turbine, which is a target industrial device in a thermal power plant. The server acquires the ultrasonic imaging data corresponding to the steam turbine. These data are collected by ultrasonic detection devices installed around the steam turbine and reflect the internal structure information of the steam turbine. The server divides the ultrasonic imaging data into multiple local ultrasonic images according to different structural parts of the steam turbine. For example, the steam inlet and outlet parts responsible for steam inlet and outlet are divided into one local image, the area where the blades are located is divided into one local image, and the rotor part is also separately divided into one local image, etc. Then, the server performs feature extraction processing on the multiple local ultrasonic images. For the local ultrasonic images of the steam inlet and outlet, acoustic reflection features such as port shape and edge are extracted through a specific algorithm to obtain corresponding image feature descriptors. These descriptors can characterize acoustic properties such as whether there are deformations and cracks in the ports. For the local blade images, acoustic-related features such as blade thickness change and surface texture are extracted as image feature descriptors to determine whether the blades are worn. For the local rotor images, internal density distribution, acoustic features of the connection parts, etc. are extracted as image feature descriptors to reflect whether the rotor structure is stable. Then, the server performs dynamic weight assignment calculation on the image feature descriptors corresponding to the multiple local ultrasonic images. Since the state of the blades has a significant impact on the overall performance of the steam turbine, the image feature descriptors corresponding to the local blade images are given a higher weight, and a relatively large feature significance coefficient is calculated. If the current operating states of the steam inlet and outlet are relatively stable, their feature significance coefficients are relatively small. The rotor part will also calculate an appropriate feature significance coefficient according to its key degree during the operation of the equipment. Finally, the server linearly superimposes the image feature descriptors and feature significance coefficients corresponding to the multiple local ultrasonic images. The feature descriptors corresponding to each local ultrasonic image such as the steam inlet and outlet, blades, and rotor are weighted and summed according to their respective feature significance coefficients, so as to obtain the acoustic imaging features corresponding to the ultrasonic imaging data. This feature comprehensively and with emphasis reflects the acoustic characteristics of the steam turbine based on ultrasonic detection, providing an important basis for subsequent analysis of the steam turbine state.

[0041] In an embodiment of the present invention, to obtain the thermal imaging features corresponding to the target industrial device, the following example can be executed.

[0042] Acquire the thermal imaging data of the target industrial device, and perform thermal radiation feature segmentation on the thermal imaging data to obtain a multi-scale thermodynamic feature map;

[0043] Construct a thermal imaging feature pyramid network, and perform feature enhancement on the multi-scale thermodynamic feature map through the spatial attention layer of the pyramid network to obtain enhanced thermodynamic features;

[0044] Perform cross-modal alignment of the enhanced thermodynamic features and real-time thermodynamic parameters to generate multi-dimensional thermal field fusion features;

[0045] Perform spatio-temporal correlation encoding on the multi-dimensional thermal field fusion features through a thermodynamic feature encoder to obtain the thermal imaging features corresponding to the target industrial equipment.

[0046] In an embodiment of the present invention, exemplarily, the server continuously monitors the blast furnace of a large steel plant, which is the target industrial equipment this time. The server first obtains the thermal imaging data of the blast furnace, which is collected by thermal imaging devices installed around the blast furnace and presents the temperature distribution on the surface of the blast furnace. The server performs thermal radiation feature segmentation on the thermal imaging data, which is divided according to the differences in thermal radiation characteristics of different regions of the blast furnace. For example, different parts such as the furnace top, furnace body, and furnace bottom are segmented into regions of different scales according to the temperature change gradient and distribution characteristics, resulting in a multi-scale thermodynamic feature map. The furnace top area dissipates heat relatively quickly, with a relatively low temperature and a relatively gentle change, and is segmented as a region of one scale during segmentation; while the furnace body part is close to the internal high-temperature reaction area, with a high temperature and possibly local high-temperature points, and is segmented at a finer scale. In this way, the feature maps of different scales can comprehensively capture the thermal radiation characteristics of each part of the blast furnace. Next, the server constructs a thermal imaging feature pyramid network. In this network, the multi-scale thermodynamic feature map is feature-enhanced through the spatial attention layer of the pyramid network. For example, for the multi-scale thermodynamic feature map of the furnace body part, the spatial attention layer will focus on the areas where local high temperatures may occur, highlighting the thermal features of these areas, making the key thermal information closely related to the operating state of the blast furnace more prominent, and thus obtaining enhanced thermodynamic features. This is like adding a "magnifying glass" to these thermal features to make the important thermal information more clearly displayed. After that, the server performs cross-modal alignment of the enhanced thermodynamic features and the real-time thermodynamic parameters. The real-time thermodynamic parameters include the ambient temperature, the temperature range of each part during normal operation of the blast furnace in theory, etc. For example, at a certain moment, the ambient temperature is 25°C, and the normal operating temperature of the furnace body should be between 800 - 1000°C. The server compares and calibrates the furnace body temperature feature in the enhanced thermodynamic features with these real-time parameters to generate a multi-dimensional thermal field fusion feature. This not only considers the temperature situation reflected by the thermal imaging of the blast furnace itself but also combines the real-time environmental and theoretical operating parameters, making the thermal feature information more accurate and comprehensive. Finally, the server performs spatio-temporal correlation encoding on the multi-dimensional thermal field fusion feature through a thermodynamic feature encoder. Considering the temperature change over time and the spatial distribution difference during the operation of the blast furnace, the encoder processes these features. For example, analyze the temperature change trend of a high-temperature point on the furnace body in the past few hours and the thermal conduction relationship between this high-temperature point and the surrounding area in space, so as to obtain the thermal imaging feature corresponding to the target industrial equipment (blast furnace). This feature comprehensively reflects the thermal characteristics of the blast furnace in the time and space dimensions, providing strong support for accurately evaluating the operating state of the blast furnace.

[0047] In an embodiment of the present invention, the feature integration process of the acoustic imaging feature and the thermal imaging feature within the same feature alignment domain to obtain the first acoustic-thermal joint feature can be implemented through the following examples.

[0048] Input the acoustic imaging features and the thermal imaging features into an adaptive feature selection unit, and transform the acoustic imaging features through a first non-linear transformation network in the adaptive feature selection unit to obtain structural anomaly features;

[0049] Transform the thermal imaging features through a second non-linear transformation network in the adaptive feature selection unit to obtain thermodynamic state features; the structural anomaly features and the thermodynamic state features have the same feature alignment domain;

[0050] Perform feature integration processing on the structural anomaly features and the thermodynamic state features to obtain first acoustic-thermal combined features.

[0051] In an embodiment of the present invention, by way of example, assume that the server is monitoring a large reactor in a chemical plant. After the server obtains the acoustic imaging features and thermal imaging features of the reactor, it inputs them into the adaptive feature selection unit. For the acoustic imaging features, the first non-linear transformation network in the adaptive feature selection unit starts to work. For example, the acoustic imaging features reflect the acoustic information of the internal structure of the reactor, including the thickness of the reactor wall, the acoustic characteristics of the connection part between the agitator and the reactor body, etc. The first non-linear transformation network transforms this information into features that can more intuitively reflect structural abnormalities. For example, if under normal circumstances, the acoustic reflection signal at a specific position on the reactor wall is within a certain range, when it exceeds this range, after being processed by the first non-linear transformation network, the feature of deviating from the normal range will be highlighted, obtaining structural abnormality features, which can clearly show whether there are structural abnormalities such as thinning of the reactor wall or loosening of the connection part. At the same time, the second non-linear transformation network in the adaptive feature selection unit processes the thermal imaging features. The thermal imaging features reflect the surface temperature distribution of the reactor, such as the heat dissipation or heat generation conditions in different regions of the reactor. The second non-linear transformation network transforms these thermal information into thermodynamic state features. For example, by analyzing the temperature distribution, it is judged whether the reaction in the reactor is in a normal thermodynamic state. If the temperature in a certain area is too high or too low, after transformation, features representing abnormal thermodynamic states will be formed, thereby obtaining thermodynamic state features, which reflect thermodynamic state information such as whether the reaction process in the reactor is normal and whether the heat transfer is reasonable. Moreover, through specific algorithms and settings, the structural abnormality features and the thermodynamic state features are in the same feature alignment domain, facilitating subsequent processing. Next, feature integration processing is performed to obtain the first acoustic-thermal combined feature. The server comprehensively considers the relationship between the structural abnormality features and the thermodynamic state features. For example, if a structural abnormality is detected in a certain area, and at the same time the thermodynamic state in this area also shows an abnormality, such as an abnormal increase in temperature, then during the integration process, these two abnormal features will be correlated and strengthened. The server will fuse the abnormal information about this area in the structural abnormality features with the corresponding temperature abnormality information in the thermodynamic state features. Through specific algorithms, the feature values of the two are weighted or integrated in other forms, so that the finally obtained first acoustic-thermal combined feature can not only reflect the structural abnormalities but also the related thermodynamic state abnormalities, providing comprehensive feature information for more accurately analyzing the overall operating condition of the reactor in the subsequent stage.

[0052] In an embodiment of the present invention, the step of performing feature integration processing on the structural abnormality features and the thermodynamic state features to obtain the first acoustic-thermal combined feature can be implemented through the following example.

[0053] Obtain the first energy intensity value corresponding to the structural anomaly feature, and perform dynamic range normalization on the structural anomaly feature according to the first energy intensity value to obtain a structural anomaly calibration feature;

[0054] Perform non-linear enhancement processing on the structural anomaly calibration feature to obtain a structural anomaly enhancement feature, and perform time-domain smoothing aggregation on the structural anomaly enhancement feature to obtain a first steady-state feature component;

[0055] Obtain the second energy intensity value corresponding to the thermodynamic state feature, and perform thermodynamic dynamic range calibration on the thermodynamic state feature according to the second energy intensity value to obtain a thermodynamic state calibration feature;

[0056] Perform threshold response processing on the thermodynamic state calibration feature to obtain a thermodynamic state enhancement feature, and perform spatial consistency integration on the thermodynamic state enhancement feature to obtain a second steady-state feature component;

[0057] Add the first steady-state feature component and the second steady-state feature component to obtain a multi-physical field coupling feature, and perform high-dimensional feature embedding on the multi-physical field coupling feature according to the multi-physical field coupling weight corresponding to the affine transformation network in the adaptive feature selection unit to obtain the first acoustic-thermal joint feature.

[0058] In an embodiment of the present invention, by way of example, taking a server monitoring a large distillation column in a refinery as an example, after the server obtains the structural anomaly features and thermodynamic state features of the distillation column, it starts the integration process to obtain the first acoustic-thermal coupling feature. First, for the structural anomaly features, the server obtains the corresponding first energy intensity value. This value reflects the strength of the information carried by the structural anomaly features. For example, if a welding point inside the distillation column becomes loose, the intensity of the corresponding acoustic anomaly signal is reflected in the first energy intensity value. Based on this value, the server performs dynamic range normalization on the structural anomaly features. Just like adjusting the volume, the intensity range of the structural anomaly features is adjusted to a standard interval to obtain the structural anomaly calibration features, so that structural anomaly features at different positions and of different types have a unified and comparable basis. Then, the server performs non-linear enhancement processing on the structural anomaly calibration features. Features that may imply serious structural problems are highlighted to a greater extent. For example, if the structural anomaly in a certain part may lead to serious safety hazards, its related features will be significantly enhanced to obtain the structural anomaly enhancement features. Then, the structural anomaly enhancement features are subjected to time-domain smoothing aggregation. Since the structural state of the distillation column fluctuates to a certain extent over time during operation, through this processing, some instantaneous and possibly interfering fluctuations are removed to obtain the first steady-state feature component, which more stably reflects the key information of the distillation column structure. At the same time, for the thermodynamic state features, the server obtains the corresponding second energy intensity value, which reflects the energy strength of the thermodynamic state features. For example, the severity of the abnormal temperature change in a certain area of the distillation column is reflected in this value. Based on the second energy intensity value, the server performs thermodynamic dynamic range calibration on the thermodynamic state features to make different regions and different types of thermodynamic state features have a unified measurement standard for energy intensity, obtaining the thermodynamic state calibration features. After that, the server performs threshold response processing on the thermodynamic state calibration features. A threshold related to the normal thermodynamic state is set to highlight those abnormal features that exceed the threshold, such as areas with too high or too low temperatures, to obtain the thermodynamic state enhancement features. Then, these enhancement features are subjected to spatial consistency integration to make the thermodynamic anomaly features at different spatial positions correlated with each other and more coordinated, obtaining the second steady-state feature component, which stably reflects the key information of the distillation column thermodynamic state. Finally, the server adds the first steady-state feature component and the second steady-state feature component to obtain the multi-physical field coupling feature, which integrates the key information of both the structure and thermodynamics of the distillation column. Then, according to the multi-physical field coupling weights corresponding to the affine transformation network in the adaptive feature selection unit, high-dimensional feature embedding is performed on the multi-physical field coupling feature. The affine transformation network determines the relative importance weights of the structural and thermodynamic information in the overall evaluation based on the characteristics of the distillation column and past data experience.Through high-dimensional feature embedding, the multi-physical field coupling features are extended to a higher dimension, containing more levels of information, and finally the first acoustic-thermal joint feature is obtained, providing a strong basis for comprehensively and accurately evaluating the operating status of the distillation column.

[0059] In the embodiment of the present invention, determining the first modal integration unit corresponding to the acoustic imaging feature among multiple modal integration units according to the first acoustic-thermal joint feature can be implemented through the following examples.

[0060] The multi-modal fitness of the first acoustic-thermal joint feature is calculated through the dynamic feature routing network in the adaptive feature selection unit, and the modal priority scores corresponding to multiple modal integration units are obtained;

[0061] The modal integration unit corresponding to the largest modal priority score is determined as the first modal integration unit corresponding to the acoustic imaging feature.

[0062] In an embodiment of the present invention, by way of example, it is assumed that the server is responsible for monitoring a lithography machine in a semiconductor manufacturing factory, which is an extremely precise and critical target industrial device. After obtaining the first acoustic-thermal joint feature of the lithography machine, the server determines the first modal integration unit corresponding to the acoustic imaging feature according to the following steps. The server uses the dynamic feature routing network in the adaptive feature selection unit to calculate the multimodal fitness of the first acoustic-thermal joint feature. During the semiconductor manufacturing process, the acoustic imaging feature of the lithography machine can reflect the operating conditions of internal precision mechanical components, such as the sound characteristics of shutter opening and closing, the smoothness of lens movement, etc., while the thermal imaging feature can reflect the heat generation of optical components and circuit modules. These factors are crucial for the imaging accuracy and stability of the lithography machine. The dynamic feature routing network will consider the fitness between the first acoustic-thermal joint feature and multiple modal integration units. For example, one of the modal integration units focuses on enhancing the high-frequency vibration information in the acoustic imaging feature to highlight the mechanical vibration problems that may cause imaging blur; another modal integration unit pays more attention to the spatial position correlation between the thermal imaging feature and the acoustic imaging feature to analyze whether local overheating will cause performance changes in nearby mechanical components. The dynamic feature routing network evaluates the fitness between the first acoustic-thermal joint feature and each modal integration unit from multiple dimensions, such as the similarity, complementarity of the features, and the impact on the key performance indicators of the device, and then calculates the modal priority scores corresponding to each of the multiple modal integration units. For example, if there is a slight jitter problem in the current imaging of the lithography machine, and the high-frequency vibration information in the acoustic imaging feature is highly relevant to solving this problem, then the modal integration unit that focuses on enhancing this part of the acoustic imaging feature will have a relatively high modal priority score. After the calculation is completed, the server determines the modal integration unit corresponding to the maximum modal priority score as the first modal integration unit corresponding to the acoustic imaging feature. This means that under the current operating conditions of the lithography machine, the selected modal integration unit is most suitable for subsequent processing of the acoustic imaging feature, can maximize the extraction of useful information in the acoustic imaging feature, and effectively fuse it with the thermal imaging feature to more accurately analyze the possible problems of the lithography machine, helping technicians take corresponding measures in a timely manner to ensure the high-precision operation of the lithography machine and the smooth progress of the semiconductor manufacturing process.

[0063] In an embodiment of the present invention, the first modal integration unit includes a cross-modal collaborative adapter;

[0064] In the first modal integration unit, cross-modal fusion of the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor can be implemented through the following example.

[0065] Perform multimodal tensor stacking on the acoustic imaging feature and the thermal imaging feature to obtain a first cross-modal joint tensor;

[0066] The automatic dynamic weight assignment calculation is performed on the first cross-modal joint tensor by the cross-modal collaborative adapter to obtain a first cross-modal correlation weight spectrum;

[0067] The dominant correlation features in the first cross-modal correlation weight spectrum are determined as the first feature descriptors.

[0068] In an embodiment of the present invention, by way of example, it is assumed that the server is responsible for monitoring the engine assembly line of an automobile engine manufacturing plant, and the target industrial device is a detection device for detecting the quality of engine blocks. After the server obtains the acoustic imaging features and thermal imaging features of the detection device, cross-modal fusion is performed in the first modal integration unit (including the cross-modal collaborative adapter) to obtain the first feature descriptor. First, the server stacks the acoustic imaging features and thermal imaging features in a multi-modal tensor. The acoustic imaging features reflect the acoustic characteristics of the internal structure of the engine block, such as the thickness of the block wall, the acoustic reflection of internal holes, etc.; the thermal imaging features reflect the surface temperature distribution of the block, such as local hot spots generated by friction or pressure during the assembly process. The server stacks these two different types of features in a specific dimension like stacking building blocks to obtain the first cross-modal joint tensor. This tensor integrates information from different modalities and provides a basis for subsequent processing. Next, the cross-modal collaborative adapter performs an automatic dynamic weight assignment calculation on the first cross-modal joint tensor. The cross-modal collaborative adapter analyzes the degree of association between each part of the acoustic imaging features and thermal imaging features in the tensor based on the working principle of the detection device and the experience accumulated from a large amount of past detection data. For example, if a region with a relatively thin block wall in the acoustic imaging features shows an abnormal temperature at the corresponding position in the thermal imaging features, the cross-modal collaborative adapter will assign higher weights to these two related features because they are likely to jointly reflect potential quality problems of the block. By comprehensively analyzing all feature associations in the entire first cross-modal joint tensor, the weights of each association relationship are calculated to obtain the first cross-modal correlation weight spectrum. This weight spectrum details the importance of different feature associations in the entire cross-modal information. Finally, the server determines the dominant correlation features from the first cross-modal correlation weight spectrum and determines them as the first feature descriptors. For example, in the analyzed weight spectrum, it is found that the association weight between the abnormal acoustic reflection at a specific part of the block and the abnormal temperature at the corresponding position is the largest, indicating that this association is the most critical for judging the quality of the block. Then this association feature is determined as the first feature descriptor. This first feature descriptor integrates the key information of both the acoustic and thermal imaging modalities and can provide an important basis for accurately evaluating the operating state of the engine block detection device and the quality of the detected engine blocks in the subsequent process.

[0069] In an embodiment of the present invention, the automatic dynamic weight assignment calculation of the first cross-modal joint tensor by the cross-modal collaborative adapter to obtain the first cross-modal correlation weight spectrum can be implemented through the following examples.

[0070] According to the multi-modal projection vectors corresponding to the cross-modal collaborative adapter, transform the first cross-modal joint tensor into a first acoustic imaging feature mapping vector, a first thermodynamic feature mapping vector, and a first acoustic-thermal coupling basis vector;

[0071] Construct a first acoustic-thermal correlation transition vector according to the multiplication operation result between the first acoustic imaging feature mapping vector and the acoustic imaging feature, and the multiplication operation result between the first acoustic imaging feature mapping vector and the thermal imaging feature;

[0072] Construct a second acoustic-thermal correlation transition vector according to the conjugate feature mapping of the multiplication operation result between the first thermodynamic feature mapping vector and the acoustic imaging feature, and the conjugate feature mapping of the multiplication operation result between the first thermodynamic feature mapping vector and the thermal imaging feature;

[0073] Construct a third acoustic-thermal correlation transition vector according to the multiplication operation result between the first acoustic-thermal coupling basis vector and the acoustic imaging feature, and the multiplication operation result between the first acoustic-thermal coupling basis vector and the thermal imaging feature;

[0074] Determine the multiplication operation result between the first acoustic-thermal correlation transition vector and the second acoustic-thermal correlation transition vector as the fourth acoustic-thermal correlation transition vector, and obtain the acoustic modal dimension of the acoustic imaging feature;

[0075] Normalize the division operation result between the fourth acoustic-thermal correlation transition vector and the scaling factor of the acoustic modal dimension to obtain a self-feature significance coefficient vector, and determine the multiplication operation result between the self-feature significance coefficient vector and the third acoustic-thermal correlation transition vector as the first cross-modal correlation weight spectrum.

[0076] In an embodiment of the present invention, by way of example, assume that the server is monitoring a turbine blade detection device in an aero-engine manufacturing factory. The server has stacked the acoustic imaging features and thermal imaging features of the turbine blade detection device in a multi-modal tensor to obtain a first cross-modal joint tensor. Next, the cross-modal collaborative adapter performs an automatic weight assignment calculation on it to obtain a first cross-modal correlation weight spectrum. First, according to the multi-modal projection vectors corresponding to the cross-modal collaborative adapter, the first cross-modal joint tensor is transformed. The multi-modal projection vectors are like a special "filter" that decomposes the first cross-modal joint tensor into different vectors. The acoustic imaging features can reflect the subtle defects in the internal structure of the blade, such as abnormal acoustic reflections caused by cracks; the thermal imaging features can show the temperature changes on the blade surface due to friction or stress concentration. After the transformation, a first acoustic imaging feature mapping vector, a first thermodynamic feature mapping vector, and a first acoustic-thermal coupling basis vector are obtained. These vectors re-express the original acoustic and thermal feature information from different angles, preparing for the subsequent construction of the correlation transition vector. Then, a first acoustic-thermal correlation transition vector is constructed. The first acoustic imaging feature mapping vector is multiplied by the acoustic imaging features and thermal imaging features respectively. For example, the first acoustic imaging feature mapping vector may highlight the acoustic reflection features of a specific part of the blade. Multiplying it by the acoustic imaging features can further emphasize the acoustic characteristics of this part; multiplying it by the thermal imaging features can explore the potential connection between this acoustic characteristic and the thermal distribution. The first acoustic-thermal correlation transition vector is constructed through the results of these two multiplication operations, initially establishing the correlation between the acoustic and thermal features. Then, a second acoustic-thermal correlation transition vector is constructed. The conjugate feature mapping is taken for the results of the multiplication operations of the first thermodynamic feature mapping vector with the acoustic imaging features and thermal imaging features. The conjugate feature mapping can uncover the hidden symmetric or complementary relationships between the features. For example, if the first thermodynamic feature mapping vector emphasizes the temperature gradient in a certain area of the blade, after multiplying it by the acoustic imaging features and taking the conjugate feature mapping, the deep connection between the temperature gradient and the acoustic anomaly can be discovered. Similarly, the conjugate feature mapping is taken for the multiplication result with the thermal imaging features. Based on these two conjugate feature mappings, a second acoustic-thermal correlation transition vector is constructed to further enrich the correlation information between the acoustic and thermal features. After that, a third acoustic-thermal correlation transition vector is constructed. The first acoustic-thermal coupling basis vector is multiplied by the acoustic imaging features and thermal imaging features respectively. The first acoustic-thermal coupling basis vector captures the basic features of the interaction between the acoustic and thermal modalities. The multiplication operation can deeply reveal the coupling effect of the acoustic and thermal in blade detection, thereby constructing a third acoustic-thermal correlation transition vector to strengthen the coupling connection of the acoustic and thermal features. Subsequently, the first acoustic-thermal correlation transition vector and the second acoustic-thermal correlation transition vector are multiplied to obtain a fourth acoustic-thermal correlation transition vector. This step further fuses the acoustic-thermal correlation information contained in the first two transition vectors, making it more rich and comprehensive. At the same time, the acoustic modal dimension of the acoustic imaging features is obtained, which reflects the dimensional characteristics such as the space or frequency of the acoustic features.Finally, divide the fourth thermal correlation transition vector by the scaling factor of the acoustic modal dimension and perform normalization to obtain the self-feature significance coefficient vector. The scaling factor is used to adjust the adaptability between the fourth thermal correlation transition vector and the acoustic modal dimension, and the normalization process makes the coefficient vector more comparable under a unified scale. Multiply the self-feature significance coefficient vector by the third thermal correlation transition vector to obtain the first cross-modal correlation weight spectrum. This weight spectrum comprehensively and meticulously reflects the correlation weights between various parts of the acoustic imaging features and thermal imaging features in the turbine blade detection device, providing a key basis for accurately evaluating the blade state. For example, it can be used to judge the tightness between internal defects and surface temperature anomalies of the blade, helping engineers identify potential quality problems.

[0077] In an embodiment of the present invention, the first modal integration unit includes a cross-modal collaborative adapter;

[0078] In the first modal integration unit, cross-modal fusion of the acoustic imaging features based on the thermal imaging features to obtain a first feature descriptor can be implemented through the following examples.

[0079] Through the cross-modal collaborative adapter, transform the acoustic imaging features into a second acoustic imaging feature mapping vector, and transform the thermal imaging features into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector;

[0080] Perform a vector multiplication calculation on the conjugate feature mapping of the multiplication operation result between the second acoustic imaging feature mapping vector and the acoustic imaging features, and the multiplication operation result between the second thermodynamic feature mapping vector and the thermal imaging features to obtain a fifth thermal correlation transition vector, and obtain the acoustic modal dimension of the acoustic imaging features;

[0081] Normalize the division operation result between the fifth thermal correlation transition vector and the scaling factor of the acoustic modal dimension to obtain an acoustic-thermal feature significance coefficient vector, and perform a vector multiplication calculation on the acoustic-thermal feature significance coefficient vector and the multiplication operation result between the second acoustic-thermal coupling basis vector and the thermal imaging features to obtain an acoustic-thermal energy aggregation feature;

[0082] Determine the dominant correlation feature in the acoustic-thermal energy aggregation feature as the first feature descriptor.

[0083] In an embodiment of the present invention, illustratively, taking the server monitoring the power bogie detection equipment of the high-speed train as an example, after the server obtains the acoustic imaging features and thermal imaging features of the power bogie detection equipment, a cross-modal fusion operation is performed in the first modal integration unit including the cross-modal collaborative adapter. First, the cross-modal collaborative adapter starts working. The acoustic imaging features reflect the operating sound characteristics of the internal components of the power bogie, such as gears, bearings, etc., and can detect whether there is abnormal wear or noise caused by looseness; the thermal imaging features show the temperature distribution on the surface of the bogie, and can find local overheating areas caused by frictional heat. The cross-modal collaborative adapter transforms the acoustic imaging features into a second acoustic imaging feature mapping vector, which re-characterizes the acoustic information from a specific angle, such as highlighting the sound features of certain key frequency bands; at the same time, the thermal imaging features are transformed into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector, the former highlighting the features of a specific temperature change area in thermal imaging, and the latter capturing the potential coupling relationship features between acoustics and heat. Next, vector operations are performed to construct the fifth acoustic-thermal correlation transition vector. The second acoustic imaging feature mapping vector is multiplied with the acoustic imaging feature to highlight the part of the original acoustic imaging feature related to the transformed feature; the conjugate feature mapping is taken for the result of the multiplication of the second thermodynamic feature mapping vector and the thermal imaging feature to mine the symmetric or complementary information related to the acoustic feature hidden in the thermal imaging feature, and then the two results are vector-multiplied. For example, if an abnormal sound is detected at a certain gear position in the acoustic imaging, and the temperature near the position is slightly increased in the thermal imaging, these two seemingly independent information can be more closely linked after these operations, thereby obtaining the fifth acoustic-thermal correlation transition vector. At the same time, the acoustic modal dimension of the acoustic imaging feature is obtained. This dimensional information reflects the characteristics of the acoustic feature in terms of frequency, spatial distribution, etc. After that, the fifth acoustic-thermal correlation transition vector is processed to obtain the acoustic-thermal feature significance coefficient vector. The fifth acoustic-thermal correlation transition vector is divided by the scale factor of the acoustic modal dimension. The scale factor is used to adjust the degree of adaptation between the vector and the acoustic modal dimension to make the calculation result more reasonable. The division result is then standardized so that the obtained vector can more clearly reflect the relative importance of each part of the features at a unified scale, thereby obtaining the acoustic-thermal feature significance coefficient vector. Next, the vector is multiplied with the result of multiplying the second acoustic-thermal coupling basis vector and the thermal imaging feature to obtain the acoustic-thermal energy aggregation feature. This step integrates the coupling relationship between acoustics and heat and the thermal imaging feature information, and weights it according to the significance coefficient obtained previously, so that the final feature can better reflect the key information after the acoustic-thermal fusion. For example, if the acoustic-thermal coupling feature of a certain area is more important in the overall evaluation, after this series of operations, the corresponding features of the area will be strengthened accordingly in the acoustic-thermal energy aggregation feature. Finally, the dominant correlation feature is determined from the acoustic-thermal energy aggregation feature and used as the first feature descriptor.For example, in the acoustic-thermal energy aggregation feature, it is found that the acoustic-thermal correlation feature at a certain key bearing position of the power bogie is the most prominent, and the information it contains is crucial for judging whether the bogie is operating normally. Then this dominant correlation feature is determined as the first feature descriptor. This descriptor synthesizes the key information of the power bogie detection equipment in both acoustic and thermal modalities, providing an important basis for accurately evaluating the equipment status and judging whether there are faults in the power bogie in the follow-up.

[0084] In an embodiment of the present invention, the first modality integration unit includes a target cross-modal collaborative adapter;

[0085] In the first modality integration unit, performing cross-modal fusion on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor can be implemented through the following example.

[0086] Obtain an acoustic-thermal coupling feature basis, and perform multi-modal tensor stacking on the acoustic-thermal coupling feature basis and the thermal imaging feature to obtain a second cross-modal joint tensor; the acoustic-thermal coupling feature basis is used to indicate the physical field correlation between the acoustic and thermal imaging features;

[0087] Through the first multi-modal correlation control network in the target cross-modal collaborative adapter, perform automatic dynamic weight assignment calculation on the second cross-modal joint tensor to obtain a second cross-modal correlation weight spectrum, and determine the dominant correlation feature in the second cross-modal correlation weight spectrum as a candidate feature descriptor;

[0088] Perform multi-modal tensor stacking on the acoustic imaging feature and the candidate feature descriptor to obtain a third cross-modal joint tensor, and perform automatic dynamic weight assignment calculation on the third cross-modal joint tensor through the second multi-modal correlation control network in the target cross-modal collaborative adapter to obtain a third cross-modal correlation weight spectrum;

[0089] Determine the dominant correlation feature in the third cross-modal correlation weight spectrum as the first feature descriptor.

[0090] In an embodiment of the present invention, by way of example, it is assumed that the server is responsible for monitoring the crusher equipment in a large-scale mine. First, the server obtains the acoustic-thermal coupling feature basis. During the operation of the crusher, the acoustic features reflect the sound characteristics generated by internal components such as the crushing hammer and the liner during operation, and can reveal whether there are conditions such as looseness and wear of the components; the thermal imaging features show the temperature distribution on the surface of the equipment, and can detect local overheating areas caused by reasons such as friction and overload. The acoustic-thermal coupling feature basis is obtained based on the analysis of a large amount of historical operation data of the crusher, and is used to indicate the physical field correlation between the acoustic and thermal imaging features. For example, an abnormal sound at a specific location is often accompanied by a corresponding temperature change pattern, and the acoustic-thermal coupling feature basis records this correlation. The server stacks the acoustic-thermal coupling feature basis and the thermal imaging features in a multi-modal tensor, just like stacking relevant information cards according to specific rules, to obtain the second cross-modal joint tensor. Then, through the first multi-modal correlation control network in the target cross-modal collaborative adapter, an automatic weight distribution calculation is performed on the second cross-modal joint tensor. This network analyzes the importance of each part of the information in the second cross-modal joint tensor based on the physical field correlation contained in the acoustic-thermal coupling feature basis. For example, if the acoustic-thermal coupling feature basis indicates that the temperature increase in a certain area of the crusher's crushing chamber is closely related to an abnormal sound at a specific frequency, then the thermal imaging feature information in this area will be given a higher weight in the calculation. By comprehensively analyzing the entire tensor, a second cross-modal correlation weight spectrum is obtained, which details the correlation weights between each part of the thermal imaging features and the acoustic-thermal coupling feature basis. Then, the dominant correlation features in the second cross-modal correlation weight spectrum are determined as candidate feature descriptors. For example, the strong correlation part between the abnormal temperature increase at a key position in the crushing chamber and the corresponding acoustic features in the acoustic-thermal coupling feature basis is selected as a candidate feature descriptor, which preliminarily integrates the key information related to thermal imaging and acoustic-thermal coupling. After that, the server stacks the acoustic imaging features and the candidate feature descriptors in a multi-modal tensor, integrating different types of important information together again to obtain the third cross-modal joint tensor. Then, through the second multi-modal correlation control network in the target cross-modal collaborative adapter, an automatic weight distribution calculation is performed on the third cross-modal joint tensor. This network comprehensively considers the relationship between the acoustic imaging features and the candidate feature descriptors. For example, if the candidate feature descriptor points to an overheated area in the crushing chamber, and there are abnormal sound reflection features near this area in the acoustic imaging features, then this part of the acoustic features will be given a higher weight in the calculation. After calculation, a third cross-modal correlation weight spectrum is obtained, which comprehensively reflects the correlation weights between the acoustic imaging features and the candidate feature descriptors. Finally, the server determines the dominant correlation features in the third cross-modal correlation weight spectrum as the first feature descriptor. For example, in the weight spectrum, it is found that the strong correlation feature between the acoustic abnormality and the thermal abnormality at a specific location in the crusher's crushing chamber is the most prominent, and this dominant correlation feature is determined as the first feature descriptor.This descriptor highly integrates the acoustic and thermal imaging feature information of the crusher, providing a key basis for accurately judging the operating state of the crusher and detecting potential faults, helping mine workers take maintenance measures in a timely manner to ensure the stable operation of the crusher.

[0091] In an embodiment of the present invention, the first modality integration unit includes a thermodynamic fusion device;

[0092] In the first modality integration unit, cross-modal fusion of the acoustic imaging features is performed according to the thermal imaging features to obtain a first feature descriptor, which can be implemented through the following examples.

[0093] Obtain the structural anomaly calibration features corresponding to the acoustic imaging features and the thermodynamic state calibration features corresponding to the thermal imaging features;

[0094] Through the first dynamic coding network in the thermodynamic fusion device, feature enhancement transmission is performed on the acoustic imaging features to obtain first dynamic coding features;

[0095] Through the second dynamic coding network in the thermodynamic fusion device, feature enhancement transmission is performed on the thermal imaging features to obtain second dynamic coding features;

[0096] Perform energy equalization aggregation on the second dynamic coding features to obtain a third steady-state feature component, and add the first dynamic coding features and the third steady-state feature component to obtain a first feature descriptor.

[0097] In an embodiment of the present invention, by way of example, assume that the server is monitoring a steam turbine in a thermal power plant.

[0098] When monitoring a steam turbine, the server first obtains the structural anomaly calibration features corresponding to the acoustic imaging features and the thermodynamic state calibration features corresponding to the thermal imaging features. For the acoustic imaging features, features reflecting the structural information of the steam turbine have been obtained through processing of ultrasonic image data before, and after dynamic range normalization, the structural anomaly calibration features are obtained. For example, the normalized features that can clearly show whether there are structural anomalies in key components such as blades and shafts. The thermal imaging features are also calibrated through the thermodynamic dynamic range to obtain the thermodynamic state calibration features, which can accurately present whether the thermodynamic states of various parts of the steam turbine, such as bearings and seals, are normal. Then, the server uses the first dynamic encoding network in the thermodynamic fusion device to perform feature enhancement transfer on the acoustic imaging features. The first dynamic encoding network will highlight the parts in the acoustic imaging features that are closely related to potential structural problems according to the operating characteristics of the steam turbine and past fault data. For example, if historical data shows that there are specific manifestations of small cracks in the blades of the steam turbine in the acoustic features, the first dynamic encoding network will enhance this part of the acoustic features to obtain the first dynamic encoding features, making the structural anomaly information more prominent. At the same time, the second dynamic encoding network in the thermodynamic fusion device performs feature enhancement transfer on the thermal imaging features. It will focus on the key thermal information in the thermal imaging features that can reflect abnormal equipment operation. For example, for the situation where the temperature of some parts of the steam turbine rises abnormally due to friction or uneven load, the second dynamic encoding network will enhance these abnormal thermal features to obtain the second dynamic encoding features, highlighting the parts in the thermal imaging features related to potential faults. After that, the server performs energy balance aggregation on the second dynamic encoding features. Since the energy distribution of the thermal imaging features may vary in different regions and times, this step will make the energy distribution of the thermal features more uniform and reasonable, eliminating the interference of local energy being too high or too low on the overall analysis, and obtaining the third steady-state feature component. For example, in different operating stages of the steam turbine, the temperature change ranges of some components are different. Through energy balance aggregation, the energy levels of these thermal features can be unified, facilitating better fusion with the acoustic features. Finally, the server adds the first dynamic encoding features and the third steady-state feature component. By combining the enhanced acoustic structure anomaly features with the energy-balanced thermal features, the first feature descriptor is obtained. This first feature descriptor comprehensively combines the key acoustic and thermal information of the steam turbine, fully reflecting the operating state of the equipment. For example, it can simultaneously reflect the structural anomalies of the blades and the possible related local overheating problems, providing an important basis for accurately evaluating whether there are faults in the steam turbine and the types of faults, helping the power plant maintenance personnel to detect and solve potential problems in time, and ensuring the stable operation of the steam turbine.

[0099] In the embodiment of the present invention, the first mode integration unit includes an acoustic dominant selector;

[0100] In the first modal integration unit, the acoustic imaging features are cross-modally fused according to the thermal imaging features to obtain a first feature descriptor, which can be implemented through the following examples.

[0101] Delete the thermal imaging features and use the acoustic imaging features as the first feature descriptor output by the acoustic dominant selector.

[0102] In an embodiment of the present invention, by way of example, it is assumed that the server is responsible for monitoring a large cement mill in a cement plant. When monitoring the cement mill, the server uses the first modal integration unit including the acoustic dominant selector to cross-modally fuse the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor. First, the server obtains the acoustic imaging features and thermal imaging features of the cement mill. The acoustic imaging features are obtained through ultrasonic detection and can reflect the internal structure conditions of the cement mill, such as the wear condition of the grinding media and the fitting degree of the lining plates, which are presented in the form of specific acoustic signal features. The thermal imaging features show the temperature distribution on the surface of the cement mill and can be used to judge the heat generation conditions of various parts during the operation of the equipment. For example, the temperature change near the motor due to the energy loss during the conversion of electrical energy into mechanical energy. However, in certain specific cases, based on the long-term accumulated operation experience and data analysis of the cement plant, it is found that for the cement mill, the acoustic imaging features have a higher priority and key role in judging the core faults of the equipment. For example, when the grinding media inside the mill are broken, unique acoustic abnormal signals will be generated, while the thermal imaging features may not show obvious abnormal changes at this time. Or when the equipment is operating normally, the temperature changes reflected by the thermal imaging features are mostly within the normal fluctuation range, and their indication effect on judging equipment faults is relatively weak. Based on this situation, in the first modal integration unit, the server performs corresponding operations through the acoustic dominant selector. The acoustic dominant selector directly deletes the thermal imaging features, only retains the acoustic imaging features, and uses them as the output first feature descriptor. This is because in the current monitoring scenario, the acoustic imaging features can more effectively reflect whether there are faults in the cement mill and the types and locations of the faults. For example, when the lining plates inside the cement mill are loose, the acoustic imaging features will clearly capture the abnormal acoustic reflection signals caused by the looseness. By analyzing these signal features, the location and degree of the lining plate looseness can be determined. In this case, the thermal imaging features may not provide valuable fault information because the looseness of the lining plates has not caused obvious temperature changes. Therefore, using the acoustic imaging features as the first feature descriptor can provide direct and key equipment status information for the maintenance personnel of the cement plant, helping them quickly and accurately judge the operation status of the cement mill, take timely maintenance measures, ensure the stable operation of the cement mill, and guarantee the production efficiency of the cement plant.

[0103] In an embodiment of the present invention, determining the fault assessment result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor can be implemented through the following examples.

[0104] Perform multi-modal tensor stacking on the first feature descriptor and the second feature descriptor to obtain a fourth cross-modal joint tensor, and configure the fault type label at the reference feature position of the fourth cross-modal joint tensor to obtain a fused feature descriptor;

[0105] Perform dynamic weight allocation calculation on the fused feature descriptor through a multi-modal collaborative attention network to obtain a physical field coupling feature, and perform high-dimensional feature embedding on the physical field coupling feature to obtain the fault assessment result corresponding to the target industrial device.

[0106] In an embodiment of the present invention, by way of example, assume that the server is monitoring a welding robot in an automobile manufacturing factory as the target industrial device. The server has obtained the first feature descriptor and the second feature descriptor of the welding robot. The first feature descriptor is obtained by cross-modal fusion of the thermal imaging feature and the acoustic imaging feature. For example, it may fuse the sound features generated during welding and the temperature distribution features of the welding head, reflecting potential problems such as insufficient welding or overheating of the welding head during the welding process. The second feature descriptor is obtained by cross-modal fusion of the thermal imaging feature based on the first feature descriptor, further integrating two types of modal information, such as information on the influence of the temperature field distribution and sound features in the welding area on the welding quality. First, the server performs multi-modal tensor stacking on the first feature descriptor and the second feature descriptor. This is like merging two tables containing different but related information together according to specific rules to obtain the fourth cross-modal joint tensor. Then, fault type labels are configured at the reference feature positions of the fourth cross-modal joint tensor. For example, common fault types of the welding robot include "loose welding", "welding head blockage", "circuit overheating", etc. The server places these fault type labels at specific positions in the joint tensor to obtain the fused feature descriptor. This fused feature descriptor integrates the features obtained from different modal analyses of the welding robot and the information on possible fault types. Next, the server calculates the dynamic weight distribution for the fused feature descriptor through a multi-modal collaborative attention network. The multi-modal collaborative attention network analyzes the correlation degree between each part of the fused feature descriptor and different fault types. For example, for the "loose welding" fault, the network will focus on the features related to unstable welding sound in the first feature descriptor and the features of uneven temperature in the welding area in the second feature descriptor, and assign higher weights to these features closely related to "loose welding". Through a comprehensive analysis of all fault types and related features, the physical field coupling feature is obtained, which highlights the distribution of the correlation weights between different physical field (acoustic and thermal) features and fault types. Finally, the server performs high-dimensional feature embedding on the physical field coupling feature. This step expands the physical field coupling feature to a higher dimension to make it contain more levels of information. For example, through a specific algorithm, the relationship between the feature weights related to different fault types is further refined and enriched, considering non-linear relationships between features, etc., so as to obtain the fault evaluation result corresponding to the target industrial device (welding robot). This result may be to determine whether there is a fault in the welding robot currently. If so, what specific fault it is and the severity of the fault, etc. For example, the evaluation result may show that the welding robot has a "loose welding" fault with a medium severity, which will help the technicians in the automobile manufacturing factory to maintain and adjust the welding robot in a timely manner, ensure the welding quality, and guarantee the smooth progress of automobile production.

[0107] Please refer to Figure 2, Figure 2 The device 110 for acoustic-thermal multimodal fusion imaging provided by an embodiment of the present invention includes:

[0108] An acquisition module 1101, configured to use an ultrasonic imager to perform ultrasonic image acquisition and thermal imaging acquisition on a target industrial device, so as to obtain ultrasonic image data and thermal imaging data;

[0109] A fusion module 1102, configured to obtain an acoustic imaging feature corresponding to the target industrial device based on the ultrasonic image data, and obtain a thermal imaging feature corresponding to the target industrial device based on the thermal imaging data; perform feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modal integration unit corresponding to the acoustic imaging feature in a plurality of modal integration units according to the first acoustic-thermal joint feature; the plurality of modal integration units are configured to encode acoustic and thermal imaging features with different modal weight coefficients; in the first modal integration unit, perform cross-modal fusion on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor; perform feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determine a second modal integration unit corresponding to the thermal imaging feature in the plurality of modal integration units according to the second acoustic-thermal joint feature; in the second modal integration unit, perform cross-modal fusion on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; obtain a fault type label, and determine a fault evaluation result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor;

[0110] An imaging module 1103, configured to calibrate the ultrasonic image data and the thermal imaging data based on a spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial device.

[0111] It should be noted that the implementation principle of the foregoing device 110 for acoustic-thermal multimodal fusion imaging may refer to the implementation principle of the foregoing method for acoustic-thermal multimodal fusion imaging, which will not be elaborated here.

[0112] An embodiment of the present invention provides a system for acoustic-thermal multimodal fusion imaging, including a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the foregoing method for acoustic-thermal multimodal fusion imaging is executed. As Figure 3 shown, Figure 3The system provided by the embodiment of the present invention based on acoustic-thermal multimodal fusion imaging includes an acoustic-thermal multimodal fusion imaging device 110, a memory 111, a processor 112, and a communication unit 113. To achieve data transmission or interaction, the elements of the memory 111, the processor 112, and the communication unit 113 are electrically connected to each other directly or indirectly.

[0113] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best illustrate the principles of the disclosure and its practical application, to thereby enable those skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular application contemplated.

Claims

1. A method based on acoustic-thermal multimodal fusion imaging, characterized in that, Including: Using an acoustic imager to perform ultrasonic image acquisition and thermal imaging acquisition on a target industrial device to obtain ultrasonic image data and thermal imaging data; Based on the ultrasonic image data, obtaining the acoustic imaging features corresponding to the target industrial device, and based on the thermal imaging data, obtaining the thermal imaging features corresponding to the target industrial device; Performing feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal combined feature, and based on the first acoustic-thermal combined feature, determining a first modal integration unit corresponding to the acoustic imaging features among multiple modal integration units; the multiple modal integration units are used to encode acoustic and thermal imaging features with different modal weight coefficients; In the first modal integration unit, performing cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor; Performing feature integration processing on the thermal imaging features and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal combined feature, and based on the second acoustic-thermal combined feature, determining a second modal integration unit corresponding to the thermal imaging features among the multiple modal integration units; In the second modal integration unit, performing cross-modal fusion on the thermal imaging features according to the first feature descriptor to obtain a second feature descriptor; Obtaining a fault type label, and based on the fault type label, the first feature descriptor, and the second feature descriptor, determining a fault evaluation result corresponding to the target industrial device; Calibrating the ultrasonic image data and the thermal imaging data based on a spatial coordinate mapping relationship, constructing a three-dimensional acoustic-thermal fusion base image, and marking the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial device.

2. The method according to claim 1, wherein The obtaining of the acoustic imaging features corresponding to the target industrial device includes: Obtaining ultrasonic imaging data corresponding to the target industrial device, and dividing the ultrasonic imaging data into multiple local ultrasonic images; Performing feature extraction processing on the multiple local ultrasonic images to obtain image feature descriptors corresponding to the multiple local ultrasonic images; Performing dynamic weight distribution calculation on the image feature descriptors corresponding to the multiple local ultrasonic images to obtain feature significance coefficients corresponding to the multiple local ultrasonic images; Linearly superimposing the image feature descriptors corresponding to the multiple local ultrasonic images and the feature significance coefficients corresponding to the multiple local ultrasonic images to obtain the acoustic imaging features corresponding to the ultrasonic imaging data.

3. The method according to claim 1, wherein The obtaining of the thermal imaging features corresponding to the target industrial device includes: Obtaining the thermal imaging data of the target industrial device, and performing thermal radiation feature segmentation on the thermal imaging data to obtain a multi-scale thermodynamic feature map; Constructing a thermal imaging feature pyramid network, and enhancing the features of the multi-scale thermodynamic feature map through the spatial attention layer of the pyramid network to obtain enhanced thermodynamic features; Performing cross-modal alignment on the enhanced thermodynamic features and real-time thermodynamic parameters to generate multi-dimensional thermal field fusion features; Perform spatio-temporal correlation encoding on the multi-dimensional thermal field fusion features through a thermodynamic feature encoder to obtain the thermal imaging features corresponding to the target industrial equipment.

4. The method according to claim 1, characterized in that, Performing feature integration processing on the acoustic imaging features and the thermal imaging features within the same feature alignment domain to obtain a first acoustic-thermal joint feature, including: Input the acoustic imaging features and the thermal imaging features into an adaptive feature selection unit, and perform feature transformation on the acoustic imaging features through the first non-linear transformation network in the adaptive feature selection unit to obtain structure anomaly features; Perform feature transformation on the thermal imaging features through the second non-linear transformation network in the adaptive feature selection unit to obtain thermodynamic state features; the structure anomaly features and the thermodynamic state features have the same feature alignment domain; Obtain the first energy intensity value corresponding to the structure anomaly features, and perform dynamic range normalization on the structure anomaly features according to the first energy intensity value to obtain structure anomaly calibration features; Perform non-linear enhancement processing on the structure anomaly calibration features to obtain structure anomaly enhancement features, and perform time-domain smoothing aggregation on the structure anomaly enhancement features to obtain a first steady-state feature component; Obtain the second energy intensity value corresponding to the thermodynamic state features, and perform thermodynamic dynamic range calibration on the thermodynamic state features according to the second energy intensity value to obtain thermodynamic state calibration features; Perform thresholding response processing on the thermodynamic state calibration features to obtain thermodynamic state enhancement features, and perform spatial consistency integration on the thermodynamic state enhancement features to obtain a second steady-state feature component; Add the first steady-state feature component and the second steady-state feature component to obtain a multi-physical field coupling feature, and perform high-dimensional feature embedding on the multi-physical field coupling feature according to the multi-physical field coupling weights corresponding to the affine transformation network in the adaptive feature selection unit to obtain the first acoustic-thermal joint feature.

5. The method according to claim 4, wherein Determining the first modal integration unit corresponding to the acoustic imaging features in multiple modal integration units according to the first acoustic-thermal joint feature, including: Perform multi-modal fitness calculation on the first acoustic-thermal joint feature through the dynamic feature routing network in the adaptive feature selection unit to obtain modal priority scores corresponding to multiple modal integration units; Determine the modal integration unit corresponding to the maximum modal priority score as the first modal integration unit corresponding to the acoustic imaging features.

6. The method according to claim 1, characterized in that The first modal integration unit includes a cross-modal collaborative adapter; In the first modal integration unit, perform cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor, including: Perform multi-modal tensor stacking on the acoustic imaging features and the thermal imaging features to obtain a first cross-modal joint tensor; According to the multi-modal projection vectors corresponding to the cross-modal collaborative adapter, transform the first cross-modal joint tensor into a first acoustic imaging feature mapping vector, a first thermodynamic feature mapping vector, and a first acoustic-thermal coupling basis vector; Construct a first acoustic-thermal correlation transition vector based on the multiplication result between the first acoustic imaging feature mapping vector and the acoustic imaging feature, and the multiplication result between the first acoustic imaging feature mapping vector and the thermal imaging feature; Construct a second acoustic-thermal correlation transition vector based on the conjugate feature mapping of the multiplication result between the first thermodynamic feature mapping vector and the acoustic imaging feature, and the conjugate feature mapping of the multiplication result between the first thermodynamic feature mapping vector and the thermal imaging feature; Construct a third acoustic-thermal correlation transition vector based on the multiplication result between the first acoustic-thermal coupling basis vector and the acoustic imaging feature, and the multiplication result between the first acoustic-thermal coupling basis vector and the thermal imaging feature; Determine the multiplication result between the first acoustic-thermal correlation transition vector and the second acoustic-thermal correlation transition vector as the fourth acoustic-thermal correlation transition vector, and obtain the acoustic modal dimension of the acoustic imaging feature; Perform normalization processing on the division result between the fourth acoustic-thermal correlation transition vector and the scaling factor of the acoustic modal dimension to obtain a self-feature significance coefficient vector, and determine the multiplication result between the self-feature significance coefficient vector and the third acoustic-thermal correlation transition vector as the first cross-modal correlation weight spectrum; Determine the dominant correlation feature in the first cross-modal correlation weight spectrum as the first feature descriptor; or, In the first modal integration unit, cross-modal fusion of the acoustic imaging feature based on the thermal imaging feature to obtain the first feature descriptor includes: Transform the acoustic imaging feature into a second acoustic imaging feature mapping vector through the cross-modal collaborative adapter, and transform the thermal imaging feature into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector; Perform vector multiplication calculation on the multiplication result between the second acoustic imaging feature mapping vector and the acoustic imaging feature, and the conjugate feature mapping of the multiplication result between the second thermodynamic feature mapping vector and the thermal imaging feature to obtain a fifth acoustic-thermal correlation transition vector, and obtain the acoustic modal dimension of the acoustic imaging feature; Perform normalization processing on the division result between the fifth acoustic-thermal correlation transition vector and the scaling factor of the acoustic modal dimension to obtain an acoustic-thermal feature significance coefficient vector, and perform vector multiplication calculation on the acoustic-thermal feature significance coefficient vector and the multiplication result between the second acoustic-thermal coupling basis vector and the thermal imaging feature to obtain an acoustic-thermal energy aggregation feature; Determine the dominant correlation feature in the acoustic-thermal energy aggregation feature as the first feature descriptor.

7. The method according to claim 1, characterized in that The first modal integration unit includes a thermodynamic fusion device; In the first modal integration unit, cross-modal fusion of the acoustic imaging feature based on the thermal imaging feature to obtain the first feature descriptor includes: Obtain the structural anomaly calibration feature corresponding to the acoustic imaging feature, and the thermodynamic state calibration feature corresponding to the thermal imaging feature; Through the first dynamic encoding network in the thermodynamic fusion device, perform feature enhancement transfer on the acoustic imaging features to obtain first dynamic encoding features; Through the second dynamic encoding network in the thermodynamic fusion device, perform feature enhancement transfer on the thermal imaging features to obtain second dynamic encoding features; Perform energy equilibrium aggregation on the second dynamic encoding features to obtain a third steady-state feature component, and accumulate the first dynamic encoding features and the third steady-state feature component to obtain a first feature descriptor; The first modality integration unit further includes an acoustic dominance selector; In the first modality integration unit, performing cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor further includes: Delete the thermal imaging features, and use the acoustic imaging features as the first feature descriptor output by the acoustic dominance selector.

8. The method according to claim 1, characterized in that Determining the fault assessment result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor includes: Perform multi-modal tensor stacking on the first feature descriptor and the second feature descriptor to obtain a fourth cross-modal joint tensor, and configure the fault type label at the reference feature position of the fourth cross-modal joint tensor to obtain a fused feature descriptor; Perform dynamic weight assignment calculation on the fused feature descriptor through a multi-modal collaborative attention network to obtain a physical field coupling feature, and perform high-dimensional feature embedding on the physical field coupling feature to obtain the fault assessment result corresponding to the target industrial device.

9. An apparatus based on acoustic-thermal multimodal fusion imaging, characterized in that, Including: An acquisition module, configured to use a sound imager to perform ultrasonic image acquisition and thermal imaging acquisition on a target industrial device to obtain ultrasonic image data and thermal imaging data; A fusion module, configured to obtain acoustic imaging features corresponding to the target industrial device based on the ultrasonic image data, and obtain thermal imaging features corresponding to the target industrial device based on the thermal imaging data; perform feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modality integration unit corresponding to the acoustic imaging features in a plurality of modality integration units according to the first acoustic-thermal joint feature; the plurality of modality integration units are configured to encode acoustic and thermal imaging features with different modality weight coefficients; in the first modality integration unit, perform cross-modal fusion on the acoustic imaging features according to the thermal imaging features to obtain a first feature descriptor; perform feature integration processing on the thermal imaging features and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determine a second modality integration unit corresponding to the thermal imaging features in the plurality of modality integration units according to the second acoustic-thermal joint feature; in the second modality integration unit, perform cross-modal fusion on the thermal imaging features according to the first feature descriptor to obtain a second feature descriptor; obtain a fault type label, and determine the fault assessment result corresponding to the target industrial device according to the fault type label, the first feature descriptor, and the second feature descriptor; An imaging module for calibrating the ultrasonic image data and the thermal imaging data based on a spatial coordinate mapping relationship, constructing a three-dimensional acoustic-thermal fusion base image, and marking the fault evaluation result on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial device.

10. A system based on acoustic-thermal multimodal fusion imaging, characterized in that, It includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the method described in any one of claims 1-8 is executed.

Images