Method, device and system based on acoustic-thermal multi-modal fusion imaging

Abstract

The invention discloses a method, device and system based on acoustic-thermal multi-mode fusion imaging, and relates to the technical field of artificial intelligence, and the method comprises the steps: firstly, collecting ultrasonic and thermal imaging data of target industrial equipment through an acoustic imager, and obtaining acoustic and thermal imaging features respectively; performing feature integration processing to obtain a first acoustic-thermal joint feature so as to determine a first modal integration unit corresponding to the acoustic feature, completing cross-modal fusion to obtain a first feature descriptor, similarly obtaining a second acoustic-thermal joint feature, a second modal integration unit and a second feature descriptor, and determining a fault evaluation result in combination with a fault type label. And then calibrating image data to construct a three-dimensional acoustic-thermal fusion substrate image and marking a result to obtain a diagnosis fusion image so as to realize accurate diagnosis of the target industrial equipment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and system based on acoustic-thermal multimodal fusion imaging. Background Art

[0002] In the field of industrial equipment inspection, traditional single inspection methods, such as relying solely on ultrasonic inspection or thermal imaging inspection, have limitations. Although ultrasonic inspection can reflect the internal structure information of the equipment, it is difficult to detect potential failures caused by temperature changes; thermal imaging inspection can detect abnormal temperature areas, but it cannot deeply understand the details of the internal structure of the equipment. Summary of the invention

[0003] The object of the present invention is to provide a method, device and system based on acoustic-thermal multimodal fusion imaging.

[0004] In a first aspect, the present invention provides a method based on acoustic-thermal multimodal fusion imaging, comprising: Use an ultrasonic imager to collect ultrasonic images and thermal images of target industrial equipment to obtain ultrasonic image data and thermal imaging data; Acquire acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and acquire thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; Performing feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determining a first modal integration unit corresponding to the acoustic imaging feature from a plurality of modal integration units according to the first acoustic-thermal joint feature; the plurality of modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor; Performing feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determining a second modal integration unit corresponding to the thermal imaging feature from among the multiple modal integration units according to the second acoustic-thermal joint feature; In the second modality integration unit, cross-modal fusion is performed on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; Acquire a fault type label, and determine a fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor; The ultrasonic image data and the thermal imaging data are calibrated based on the spatial coordinate mapping relationship to construct a three-dimensional acoustic-thermal fusion base image, and the fault assessment result is marked on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

[0005] In a second aspect, an embodiment of the present invention provides a device based on acoustic-thermal multimodal fusion imaging, comprising: An acquisition module is used to acquire ultrasonic images and thermal images of target industrial equipment using an acoustic imager to obtain ultrasonic image data and thermal imaging data; A fusion module is used to obtain the acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and to obtain the thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; perform feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modal integration unit corresponding to the acoustic imaging feature in multiple modal integration units according to the first acoustic-thermal joint feature; the multiple modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; in the first modal integration unit, the acoustic imaging feature is integrated according to the thermal imaging feature. Perform cross-modal fusion on the thermal imaging feature to obtain a first feature descriptor; perform feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determine a second modal integration unit corresponding to the thermal imaging feature in the multiple modal integration units according to the second acoustic-thermal joint feature; in the second modal integration unit, perform cross-modal fusion on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; obtain a fault type label, and determine a fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor; An imaging module is used to calibrate the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault assessment result to the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

[0006] In a third aspect, the present invention provides a system based on acoustic and thermal multimodal fusion imaging, comprising a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the aforementioned method based on acoustic and thermal multimodal fusion imaging is executed.

[0007] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a method, device and system based on acoustic and thermal multimodal fusion imaging disclosed in the present invention, by using an acoustic imager to collect ultrasonic and thermal imaging data of the target industrial equipment, respectively obtaining acoustic and thermal imaging features, obtaining a first acoustic and thermal joint feature through feature integration processing to determine the first modal integration unit corresponding to the acoustic feature, completing cross-modal fusion to obtain a first feature descriptor, and similarly obtaining a second acoustic and thermal joint feature, a second modal integration unit and a second feature descriptor, determining a fault assessment result in combination with a fault type label, then calibrating the image data to construct a three-dimensional acoustic and thermal fusion base image and marking the result, obtaining a diagnostic fusion image, and realizing accurate diagnosis of the target industrial equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0009] Figure 1 A schematic diagram of the steps of a method for acoustic-thermal multimodal fusion imaging provided by an embodiment of the present invention; Figure 2 A schematic block diagram of the structure of a device based on acoustic-thermal multimodal fusion imaging provided by an embodiment of the present invention; Figure 3 A schematic block diagram of the structure of a system based on acoustic-thermal multimodal fusion imaging provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0011] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0012] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flow chart of a method based on acoustic-thermal multimodal fusion imaging provided in an embodiment of the present disclosure is provided. The method based on acoustic-thermal multimodal fusion imaging is introduced in detail below.

[0013] Step S201, using an ultrasonic imager to collect ultrasonic images and thermal images of target industrial equipment to obtain ultrasonic image data and thermal imaging data; Step S202, acquiring acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and acquiring thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; Step S203, performing feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determining a first modal integration unit corresponding to the acoustic imaging feature from a plurality of modal integration units according to the first acoustic-thermal joint feature; the plurality of modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; Step S204, in the first modality integration unit, cross-modal fusion is performed on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor; Step S205, performing feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determining a second modal integration unit corresponding to the thermal imaging feature from among the multiple modal integration units according to the second acoustic-thermal joint feature; Step S206, in the second modality integration unit, cross-modal fusion is performed on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; Step S207, obtaining a fault type label, and determining a fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor; Step S208, calibrating the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, constructing a three-dimensional acoustic-thermal fusion base image, and marking the fault assessment result to the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

[0014] In the embodiment of the present invention, for example, it is assumed that the server is responsible for monitoring a large-scale stamping equipment in an automobile manufacturing plant. The server controls the sound imager installed around the stamping equipment to collect ultrasonic images and thermal images of the equipment. The ultrasonic part of the sound imager transmits ultrasonic waves to the stamping equipment. The ultrasonic waves propagate inside the equipment and are reflected back when encountering different materials or structural changes. The sound imager receives these reflected waves and converts them into ultrasonic image data. These data can present the structural details of the parts inside the equipment, such as the integrity of the mold, the tightness of the connecting parts, etc. At the same time, the thermal imaging part of the sound imager detects the thermal radiation on the surface of the equipment and generates thermal imaging data to reflect the temperature distribution of various areas on the surface of the equipment, such as the temperature of the stamping head, transmission parts, etc. The server obtains and stores these ultrasonic image data and thermal imaging data. The server divides the acquired ultrasonic imaging data into multiple local ultrasonic images according to the structural area of ​​the equipment, such as dividing the mold area, the transmission rod area, etc. separately. Then, these local ultrasonic images are feature extracted to extract the feature descriptors of each local image, such as edge features, texture features, etc. Then, by calculating the importance of each local image feature, a dynamic weight is assigned to it to obtain the feature significance coefficient. Finally, the feature descriptor and the significance coefficient are linearly superimposed to obtain the acoustic imaging features reflecting the acoustic characteristics of the stamping equipment. The server performs thermal radiation feature segmentation on the thermal imaging data to obtain a multi-scale thermodynamic feature map reflecting the thermal state of the equipment at different scales. By constructing a thermal imaging feature pyramid network and using its spatial attention layer to enhance key thermal features, enhanced thermodynamic features are obtained. Then it is aligned with real-time thermodynamic parameters (such as ambient temperature and normal operating temperature range of the equipment) to generate multi-dimensional thermal field fusion features. Finally, the thermodynamic feature encoder performs spatiotemporal correlation encoding to obtain the thermal imaging features of the stamping equipment. The server inputs the acoustic imaging features and thermal imaging features into the adaptive feature selection unit. The acoustic imaging features are converted into structural anomaly features and the thermal imaging features are converted into thermodynamic state features through the nonlinear transformation network, so that the two are in the same feature alignment domain. The structural anomaly features and thermodynamic state features are respectively subjected to energy intensity value acquisition, standardization, enhancement and aggregation operations to obtain the first steady-state feature component and the second steady-state feature component. After adding the two, the high-dimensional feature embedding is performed through the affine transformation network to obtain the first acoustic-thermal joint feature. Afterwards, the server calculates the modal priority scores of the first acoustic-thermal joint feature and multiple modal integration units through a dynamic feature routing network, and determines the modal integration unit with the highest score as the first modal integration unit corresponding to the acoustic imaging feature. It is assumed that the first modal integration unit includes a cross-modal collaborative adapter.The server stacks the acoustic imaging features and thermal imaging features as multi-modal tensors to obtain the first cross-modal joint tensor, and then transforms it into different vectors through a cross-modal collaborative adapter. After a series of vector operations, the acoustic-thermal correlation transition vector is constructed, and finally the first cross-modal correlation weight spectrum is obtained, and the dominant correlation feature is determined as the first feature descriptor. In the same feature alignment domain, the server performs feature integration processing similar to the third step on the thermal imaging features and the first feature descriptor to obtain the second acoustic-thermal joint feature. Similarly, the second modal integration unit corresponding to the thermal imaging feature is determined by the dynamic feature routing network. In the second modal integration unit, the thermal imaging features are cross-modally fused according to the first feature descriptor in a manner similar to the fourth step to obtain the second feature descriptor. The server obtains the fault type labels related to the stamping equipment from the equipment fault database, such as "die crack" and "transmission component wear". The first feature descriptor and the second feature descriptor are stacked as multi-modal tensors, and the fused feature descriptor is obtained after configuring the fault type label. The weights are calculated through the multimodal collaborative attention network to obtain the physical field coupling characteristics. After high-dimensional feature embedding, the fault assessment results of the stamping equipment are determined, such as whether the equipment has a fault and the specific fault type and severity. The server must first clarify the coordinate system of the ultrasonic image data and the thermal imaging data. For ultrasonic images, their coordinates may be based on the three-dimensional space setting of the internal structure of the equipment. With a fixed point of the equipment as the origin, the coordinate axes in the length, width, and height directions are determined to locate the position of the internal parts. The coordinates of thermal imaging data are usually based on the two-dimensional plane of the equipment surface. With a corner point on the equipment surface as the origin, the horizontal and vertical coordinate axes are determined to represent the temperature distribution position. The server establishes a spatial coordinate mapping relationship between the two. For example, by accurately modeling the equipment, the coordinate position of a part inside the equipment in the ultrasonic image is known, and the projection position of the part on the surface of the equipment is also known, so as to determine the mapping relationship between the coordinates of the part in the ultrasonic image and the coordinates of the corresponding projection point on the thermal image. Alternatively, the known marking points on the equipment can be used to identify these marking points in both the ultrasonic image and the thermal image, and the overall coordinate mapping relationship can be constructed through the coordinates of the marking points. Based on the calibrated spatial coordinate mapping relationship, the server begins to construct a three-dimensional acoustic-thermal fusion base image. It presents the internal structure information of the device contained in the ultrasound image data in the form of a three-dimensional model, such as the three-dimensional shape of the mold, the connection method of the transmission parts, etc. At the same time, the temperature information in the thermal imaging data is fused to this three-dimensional model in the form of color or isosurface. For example, areas with higher temperatures are represented in red on the three-dimensional model, and areas with lower temperatures are represented in blue, so that operators can intuitively see the distribution of the surface temperature of the equipment in three-dimensional space, as well as the corresponding relationship with the internal structure. The server marks the previously obtained fault assessment results on the three-dimensional acoustic-thermal fusion base image.If the fault assessment results show that there are cracks in the mold, the server will mark the location and general direction of the cracks with a specific mark (such as a red flashing line) at the corresponding position of the mold in the 3D acoustic-thermal fusion substrate image, and attach a text description of the fault type as "mold crack" and the severity (such as mild, moderate, severe). If transmission component wear is also detected, similar markings will be made at the corresponding transmission component position, such as circling the worn area with a yellow circle and marking information such as "transmission component wear, severity: moderate". The final diagnostic fusion image of the target industrial equipment (stamping equipment) allows the factory's technicians and maintenance personnel to fully and intuitively understand the operating status of the equipment, including the internal structure, surface temperature, and possible faults, so that maintenance and repair strategies can be quickly formulated.

[0015] In an embodiment of the present invention, the step of acquiring acoustic imaging features corresponding to the target industrial equipment includes: Acquire ultrasonic imaging data corresponding to the target industrial equipment, and divide the ultrasonic imaging data into a plurality of local ultrasonic images; Performing feature extraction processing on the multiple local ultrasound images to obtain image feature descriptors corresponding to the multiple local ultrasound images; Performing dynamic weight allocation calculation on image feature descriptors corresponding to the multiple local ultrasound images to obtain feature significance coefficients corresponding to the multiple local ultrasound images; The image feature descriptors corresponding to the multiple local ultrasonic images and the feature significance coefficients corresponding to the multiple local ultrasonic images are linearly superimposed to obtain the acoustic imaging features corresponding to the ultrasonic imaging data.

[0016] In an embodiment of the present invention, illustratively, the server is responsible for monitoring the target industrial equipment, the steam turbine of a thermal power plant. The server obtains ultrasonic imaging data corresponding to the steam turbine, which is collected by ultrasonic detection equipment installed around the steam turbine and reflects the internal structural information of the steam turbine. The server divides the ultrasonic imaging data into multiple local ultrasonic images according to different structural parts of the steam turbine. For example, the steam inlet and exhaust port parts responsible for steam in and out are divided into a local image, the area where the blades are located is divided into a local image, and the rotor part is also divided into a local image separately. Then, the server performs feature extraction processing on multiple local ultrasonic images. For the local ultrasonic images of the steam inlet and exhaust ports, acoustic reflection features such as port shape and edge are extracted through a specific algorithm to obtain corresponding image feature descriptors, which can characterize whether the port has acoustic characteristics such as deformation and cracks. For the local image of the blade, acoustic related features such as blade thickness change and surface texture are extracted as image feature descriptors to determine whether the blade is worn. For the local image of the rotor, its internal density distribution, acoustic features of the connection parts, etc. are extracted as image feature descriptors to reflect whether the rotor structure is stable. Then, the server dynamically calculates the weight distribution of the image feature descriptors corresponding to the multiple local ultrasonic images. Since the state of the blade has a significant impact on the overall performance of the turbine, the image feature descriptors corresponding to the local images of the blade will be assigned a higher weight, and a relatively large feature significance coefficient will be calculated. If the current operating state of the steam inlet and exhaust port is relatively stable, their feature significance coefficients are relatively small. The rotor part will also calculate the corresponding appropriate feature significance coefficient according to its criticality in the operation of the equipment. Finally, the server linearly superimposes the image feature descriptors and feature significance coefficients corresponding to the multiple local ultrasonic images. The feature descriptors corresponding to the local ultrasonic images of the steam inlet and exhaust port, blades, rotors, etc. are weighted and summed according to their respective feature significance coefficients, thereby obtaining the acoustic imaging features corresponding to the ultrasonic imaging data. This feature comprehensively and emphatically reflects the acoustic characteristics of the turbine based on ultrasonic detection, and provides an important basis for the subsequent analysis of the turbine state.

[0017] In the embodiment of the present invention, the step of acquiring the thermal imaging features corresponding to the target industrial equipment may be implemented through the following examples.

[0018] Acquire thermal imaging data of the target industrial equipment, perform thermal radiation feature segmentation on the thermal imaging data, and obtain a multi-scale thermodynamic feature map; Constructing a thermal imaging feature pyramid network, and enhancing the multi-scale thermodynamic feature map through a spatial attention layer of the pyramid network to obtain enhanced thermodynamic features; Cross-modally aligning the enhanced thermodynamic features with real-time thermodynamic parameters to generate a multi-dimensional thermal field fusion feature; The multi-dimensional thermal field fusion features are temporally and spatially encoded using a thermodynamic feature encoder to obtain thermal imaging features corresponding to the target industrial equipment.

[0019] In an embodiment of the present invention, illustratively, the server continuously monitors the blast furnace of a large steel plant, which is the target industrial equipment of this time. The server first obtains the thermal imaging data of the blast furnace, which is collected by the thermal imaging equipment installed around the blast furnace, presenting the temperature distribution of the blast furnace surface. The server performs thermal radiation feature segmentation on the thermal imaging data, and divides it according to the differences in thermal radiation characteristics of different regions of the blast furnace. For example, different parts such as the furnace top, furnace body, and furnace bottom are divided into regions of different scales according to the temperature change gradient and distribution characteristics, and a multi-scale thermodynamic feature map is obtained. For example, the furnace top area dissipates heat faster, the temperature is relatively low and the change is relatively slow, and it is used as a scale area during segmentation; while the furnace body part is close to the internal high-temperature reaction zone, the temperature is high and there may be local high-temperature points, which are segmented at a finer scale, so that feature maps of different scales can fully capture the thermal radiation characteristics of various parts of the blast furnace. Then, the server constructs a thermal imaging feature pyramid network. In this network, the multi-scale thermodynamic feature map is feature enhanced through the spatial attention layer of the pyramid network. For example, for the multi-scale thermodynamic feature map of the furnace body, the spatial attention layer will focus on the areas where local high temperature may occur, highlight the thermal features of these areas, and make the key thermal information closely related to the operation status of the blast furnace more prominent, thereby obtaining enhanced thermodynamic features. This is like adding a "magnifying glass" to these thermal features, allowing important thermal information to be displayed more clearly. After that, the server cross-modally aligns the enhanced thermodynamic features with the real-time thermodynamic parameters. Real-time thermodynamic parameters include ambient temperature, theoretical temperature range of various parts during normal operation of the blast furnace, etc. For example, at a certain moment, the ambient temperature is 25°C, and the normal operating temperature of the furnace body should be between 800-1000°C. The server compares and calibrates the furnace body temperature features in the enhanced thermodynamic features with these real-time parameters to generate multi-dimensional thermal field fusion features. This not only takes into account the temperature conditions reflected by the thermal imaging of the blast furnace itself, but also combines the real-time environment and theoretical operation parameters to make the thermal feature information more accurate and comprehensive. Finally, the server performs spatiotemporal correlation encoding of the multi-dimensional thermal field fusion features through the thermodynamic feature encoder. Considering the temperature changes over time and the differences in spatial distribution during the operation of the blast furnace, the encoder processes these features. For example, the temperature change trend of a certain high-temperature point in the furnace body in the past few hours and the heat conduction relationship between the high-temperature point and the surrounding area in space are analyzed to obtain the thermal imaging features corresponding to the target industrial equipment (blast furnace). This feature fully reflects the thermal characteristics of the blast furnace in time and space dimensions, providing strong support for accurately evaluating the operating status of the blast furnace.

[0020] In the embodiment of the present invention, the feature integration processing is performed on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, which can be implemented through the following examples.

[0021] Inputting the acoustic imaging feature and the thermal imaging feature into an adaptive feature selection unit, performing feature transformation on the acoustic imaging feature through a first nonlinear transformation network in the adaptive feature selection unit to obtain a structural abnormality feature; The thermal imaging feature is transformed by a second nonlinear transformation network in the adaptive feature selection unit to obtain a thermodynamic state feature; the structural anomaly feature and the thermodynamic state feature have the same feature alignment domain; The structural anomaly feature and the thermodynamic state feature are integrated to obtain a first acoustic-thermal joint feature.

[0022] In an embodiment of the present invention, for example, it is assumed that the server is monitoring a large reactor in a chemical plant. After the server obtains the acoustic imaging features and thermal imaging features of the reactor, it inputs them into the adaptive feature selection unit. For the acoustic imaging features, the first nonlinear transformation network in the adaptive feature selection unit starts to work. For example, the acoustic imaging features reflect the acoustic information of the internal structure of the reactor, including the thickness of the reactor wall, the acoustic characteristics of the connection between the agitator and the reactor body, etc. The first nonlinear transformation network transforms this information and converts it into features that can more intuitively reflect structural abnormalities. For example, if the acoustic reflection signal at a specific position of the reactor wall is within a certain range under normal circumstances, when it exceeds this range, after being processed by the first nonlinear transformation network, this feature that deviates from the normal range will be highlighted, and structural abnormality features will be obtained. These features can clearly show whether the reactor wall is thinning, whether the connection part is loose, and other structural abnormalities. At the same time, the second nonlinear transformation network in the adaptive feature selection unit processes the thermal imaging features. The thermal imaging features reflect the surface temperature distribution of the reactor, such as the heat dissipation or heating of different areas of the reactor. The second nonlinear transformation network converts these thermal information into thermodynamic state features. For example, by analyzing the temperature distribution, it is possible to determine whether the reaction in the reactor is in a normal thermodynamic state. If the temperature in a certain area is too high or too low, after transformation, features representing abnormal thermodynamic states will be formed, thereby obtaining thermodynamic state features. These features reflect thermodynamic state information such as whether the reaction process in the reactor is normal and whether the heat transfer is reasonable. Moreover, through specific algorithms and settings, the structural abnormality features and thermodynamic state features are in the same feature alignment domain, which is convenient for subsequent processing. Next, feature integration processing is performed to obtain the first acoustic-thermal joint feature. The server will comprehensively consider the relationship between the structural abnormality features and the thermodynamic state features. For example, if a structural abnormality is detected in a certain area, and the thermodynamic state of the area is also abnormal, such as abnormal temperature increase, then during the integration process, the two abnormal features will be mutually correlated and strengthened. The server will fuse the abnormal information about the area in the structural abnormality features with the corresponding temperature abnormality information in the thermodynamic state features. Through a specific algorithm, the characteristic values ​​of the two are weighted calculated or integrated in other forms, so that the final first acoustic and thermal joint feature can reflect both structural anomalies and related thermodynamic state anomalies, providing comprehensive characteristic information for subsequent more accurate analysis of the overall operating status of the reactor.

[0023] In the embodiment of the present invention, the structural anomaly feature and the thermodynamic state feature are subjected to feature integration processing to obtain a first acoustic-thermal joint feature, which can be implemented through the following example.

[0024] Acquiring a first energy intensity value corresponding to the structural abnormality feature, and performing dynamic range normalization on the structural abnormality feature according to the first energy intensity value to obtain a structural abnormality calibration feature; Performing nonlinear enhancement processing on the structural abnormality calibration feature to obtain a structural abnormality enhancement feature, and performing time domain smoothing aggregation on the structural abnormality enhancement feature to obtain a first steady-state feature component; Acquiring a second energy intensity value corresponding to the thermodynamic state characteristic, and performing thermodynamic dynamic range calibration on the thermodynamic state characteristic according to the second energy intensity value to obtain a thermodynamic state calibration characteristic; Performing threshold response processing on the thermodynamic state calibration feature to obtain a thermodynamic state enhancement feature, and performing spatial consistency integration on the thermodynamic state enhancement feature to obtain a second steady-state feature component; The first steady-state characteristic component and the second steady-state characteristic component are added to obtain a multi-physical field coupling feature, and high-dimensional feature embedding is performed on the multi-physical field coupling feature according to the multi-physical field coupling weight corresponding to the affine transformation network in the adaptive feature selection unit to obtain the first acoustic-thermal joint feature.

[0025] In the embodiment of the present invention, for example, taking the server monitoring a large distillation tower of an oil refinery as an example, after the server obtains the structural anomaly characteristics and thermodynamic state characteristics of the distillation tower, it starts to integrate and process to obtain the first acoustic and thermal joint characteristics. First, for the structural anomaly characteristics, the server obtains the corresponding first energy intensity value. This value reflects the strength of the information carried by the structural anomaly characteristics. For example, if a welding point inside the distillation tower is loose, the corresponding acoustic abnormality signal intensity is reflected in the first energy intensity value. According to this value, the server performs dynamic range standardization on the structural anomaly characteristics. Just like adjusting the volume, the intensity range of the structural anomaly characteristics is adjusted to a standard interval to obtain the structural anomaly calibration characteristics, so that the structural anomaly characteristics of different locations and types have a unified and comparable basis. Next, the server performs nonlinear enhancement processing on the structural anomaly calibration characteristics. Features that may indicate serious structural problems are given a greater degree of prominence. For example, if a structural anomaly in a certain part may cause serious safety hazards, its related features will be significantly enhanced to obtain the structural anomaly enhancement characteristics. Then, the structural anomaly enhancement features are smoothed and aggregated in the time domain. Because the structural state of the distillation tower will fluctuate over time during operation, this processing removes some instantaneous fluctuations that may be interference, and obtains the first steady-state characteristic component, which more stably reflects the key information of the distillation tower structure. At the same time, for the thermodynamic state feature, the server obtains the corresponding second energy intensity value, which reflects the energy strength of the thermodynamic state feature. For example, the severity of the abnormal temperature change in a certain area of ​​the distillation tower is reflected in this value. According to the second energy intensity value, the server calibrates the thermodynamic dynamic range of the thermodynamic state feature so that the thermodynamic state features of different regions and types have a unified measurement standard in energy intensity, and obtains the thermodynamic state calibration feature. After that, the server performs threshold response processing on the thermodynamic state calibration feature. A threshold value related to the normal thermodynamic state is set to highlight those abnormal features that exceed the threshold, such as areas with too high or too low temperature, and obtain the thermodynamic state enhancement feature. Then these enhanced features are spatially consistent and integrated, so that the thermodynamic abnormal features at different spatial locations are interrelated and more coordinated, and the second steady-state characteristic component is obtained, which stably reflects the key information of the thermodynamic state of the distillation tower. Finally, the server adds the first steady-state feature component and the second steady-state feature component to obtain the multi-physics coupling feature, which integrates the key information of the distillation tower structure and thermodynamics. Then, the multi-physics coupling feature is embedded in high dimensions according to the multi-physics coupling weights corresponding to the affine transformation network in the adaptive feature selection unit. The affine transformation network determines the relative importance weights of the structure and thermodynamic information in the overall evaluation based on the characteristics of the distillation tower and past data experience.Through high-dimensional feature embedding, the multi-physical field coupling characteristics are expanded to higher dimensions, including more levels of information, and finally the first acoustic and thermal joint characteristics are obtained, providing a strong basis for comprehensive and accurate evaluation of the operating status of the distillation tower.

[0026] In the embodiment of the present invention, determining the first modal integration unit corresponding to the acoustic imaging feature among multiple modal integration units according to the first acoustic-thermal joint feature can be implemented through the following examples.

[0027] Performing multi-modal fitness calculation on the first acoustic-thermal joint feature through a dynamic feature routing network in the adaptive feature selection unit to obtain modal priority scores corresponding to multiple modal integration units; The modal integration unit corresponding to the maximum modal priority score is determined as the first modal integration unit corresponding to the acoustic imaging feature.

[0028] In the embodiment of the present invention, for example, it is assumed that the server is responsible for monitoring the lithography machine in a semiconductor manufacturing plant, which is an extremely precise and critical target industrial equipment. After obtaining the first acoustic and thermal joint feature of the lithography machine, the server determines the first modal integration unit corresponding to the acoustic imaging feature according to the following steps. The server uses the dynamic feature routing network in the adaptive feature selection unit to perform multi-modal fitness calculation on the first acoustic and thermal joint feature. In the semiconductor manufacturing process, the acoustic imaging feature of the lithography machine can reflect the operating conditions of the internal precision mechanical parts, such as the sound characteristics of the shutter opening and closing, the stability of the lens movement, etc., while the thermal imaging feature can reflect the heating conditions of the optical components and circuit modules. These factors are crucial to the imaging accuracy and stability of the lithography machine. The dynamic feature routing network considers the degree of adaptation between the first acoustic and thermal joint feature and multiple modal integration units. For example, one of the modal integration units focuses on strengthening the high-frequency vibration information in the acoustic imaging feature to highlight the mechanical vibration problem that may cause imaging blur; the other modal integration unit pays more attention to the spatial relationship between the thermal imaging feature and the acoustic imaging feature, which is used to analyze whether local overheating will cause performance changes of nearby mechanical parts. The dynamic feature routing network evaluates the fit of the first acoustic-thermal joint feature with each modal integration unit from multiple dimensions, such as the similarity and complementarity of the features and the degree of influence on the key performance indicators of the equipment, and then calculates the modal priority scores corresponding to each of the multiple modal integration units. For example, if the current lithography machine has a slight jitter problem in its imaging, and the high-frequency vibration information in the acoustic imaging feature is highly relevant to solving the problem, then the modal integration unit that focuses on strengthening this part of the acoustic imaging feature will have a relatively high modal priority score. After the calculation is completed, the server determines the modal integration unit corresponding to the largest modal priority score as the first modal integration unit corresponding to the acoustic imaging feature. This means that under the current operating conditions of the lithography machine, the selected modal integration unit is most suitable for subsequent processing of the acoustic imaging feature, and can maximize the mining of useful information in the acoustic imaging feature, and effectively integrate it with the thermal imaging feature, so as to more accurately analyze the possible problems of the lithography machine, help technicians take corresponding measures in a timely manner, ensure the high-precision operation of the lithography machine, and ensure the smooth progress of the semiconductor manufacturing process.

[0029] In an embodiment of the present invention, the first modality integration unit includes a cross-modality collaboration adapter; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor, which can be implemented through the following example.

[0030] Performing multi-modal tensor stacking on the acoustic imaging feature and the thermal imaging feature to obtain a first cross-modal joint tensor; Performing automatic dynamic weight allocation calculation on the first cross-modal joint tensor by the cross-modal collaborative adapter to obtain a first cross-modal association weight spectrum; A dominant correlation feature in the first cross-modal correlation weight spectrum is determined as a first feature descriptor.

[0031] In an embodiment of the present invention, illustratively, it is assumed that the server is responsible for monitoring the engine assembly production line of an automobile engine manufacturing plant, and the target industrial equipment is a detection device for detecting the quality of the engine cylinder. After the server obtains the acoustic imaging features and thermal imaging features of the detection equipment, it performs cross-modal fusion in the first modal integration unit (including the cross-modal collaborative adapter) to obtain the first feature descriptor. First, the server stacks the acoustic imaging features and thermal imaging features into multi-modal tensors. The acoustic imaging features reflect the acoustic characteristics of the internal structure of the engine cylinder, such as the thickness of the cylinder wall, the acoustic reflection of the internal holes, and other information; the thermal imaging features reflect the surface temperature distribution of the cylinder, such as local hot spots caused by friction or pressure during the assembly process. The server stacks these two different types of features in a specific dimension like stacking building blocks to obtain a first cross-modal joint tensor. This tensor integrates information from different modes and provides a basis for subsequent processing. Then, the cross-modal collaborative adapter performs self-dynamic weight allocation calculation on the first cross-modal joint tensor. The cross-modal collaborative adapter analyzes the degree of association between the acoustic imaging features and the various parts of the thermal imaging features in the tensor based on the working principle of the detection equipment and the experience accumulated from a large amount of past detection data. For example, if there is a thinner cylinder wall area in the acoustic imaging feature and a temperature anomaly appears at the corresponding position in the thermal imaging feature, the cross-modal collaborative adapter will give these two related features a higher weight because they are likely to jointly reflect the potential quality problem of the cylinder. Through a comprehensive analysis of all feature associations in the entire first cross-modal joint tensor, the weights of each association relationship are calculated to obtain the first cross-modal association weight spectrum. This weight spectrum records in detail the importance of different feature associations in the entire cross-modal information. Finally, the server determines the dominant association feature from the first cross-modal association weight spectrum and determines it as the first feature descriptor. For example, in the weight spectrum obtained by analysis, it is found that the association weight between the acoustic reflection anomaly at a specific part of the cylinder and the temperature anomaly at the corresponding position is the largest, which indicates that this association is most critical to judging the quality of the cylinder, and then this association feature is determined as the first feature descriptor. This first feature descriptor integrates the key information of both acoustic and thermal imaging modes, and can provide an important basis for the subsequent accurate evaluation of the operating status of the engine cylinder inspection equipment and the quality status of the inspected cylinder.

[0032] In the embodiment of the present invention, the cross-modal collaborative adapter performs automatic dynamic weight allocation calculation on the first cross-modal joint tensor to obtain a first cross-modal association weight spectrum, which can be implemented through the following examples.

[0033] According to the multimodal projection vector corresponding to the cross-modal collaborative adapter, transforming the first cross-modal joint tensor into a first acoustic imaging feature mapping vector, a first thermodynamic feature mapping vector and a first acoustic-thermal coupling basis vector; Constructing a first acoustic-thermal correlation transition vector according to a multiplication result between the first acoustic imaging feature mapping vector and the acoustic imaging feature, and a multiplication result between the first acoustic imaging feature mapping vector and the thermal imaging feature; Constructing a second acoustic-thermal correlation transition vector according to a conjugate feature map of a multiplication result between the first thermodynamic feature map vector and the acoustic imaging feature, and a conjugate feature map of a multiplication result between the first thermodynamic feature map vector and the thermal imaging feature; constructing a third acoustic-thermal correlation transition vector according to a multiplication result between the first acoustic-thermal coupling basis vector and the acoustic imaging feature, and a multiplication result between the first acoustic-thermal coupling basis vector and the thermal imaging feature; Determining a result of a multiplication operation between the first acoustic-thermal correlation transition vector and the second acoustic-thermal correlation transition vector as a fourth acoustic-thermal correlation transition vector, and acquiring an acoustic modal dimension of the acoustic imaging feature; The result of the division operation between the fourth acoustic-thermal association transition vector and the scaling factor of the acoustic modal dimension is standardized to obtain a self-feature significance coefficient vector, and the result of the multiplication operation between the self-feature significance coefficient vector and the third acoustic-thermal association transition vector is determined as the first cross-modal association weight spectrum.

[0034] In an embodiment of the present invention, for example, it is assumed that the server is monitoring the turbine blade detection equipment of an aircraft engine manufacturing plant. The server has stacked the acoustic imaging features and thermal imaging features of the turbine blade detection equipment into multi-modal tensors to obtain a first cross-modal joint tensor. Next, the cross-modal collaborative adapter performs an automatic dynamic weight allocation calculation on it to obtain a first cross-modal association weight spectrum. First, the first cross-modal joint tensor is transformed according to the multi-modal projection vector corresponding to the cross-modal collaborative adapter. The multi-modal projection vector is like a special "filter" that decomposes the first cross-modal joint tensor into different vectors. The acoustic imaging feature can reflect the subtle defects of the internal structure of the blade, such as abnormal acoustic reflections caused by cracks; the thermal imaging feature can show the temperature changes on the blade surface caused by friction or stress concentration. After transformation, the first acoustic imaging feature mapping vector, the first thermodynamic feature mapping vector and the first acoustic-thermal coupling basis vector are obtained. These vectors re-express the original acoustic and thermal feature information from different angles, preparing for the subsequent construction of the association transition vector. Then, the first acoustic-thermal association transition vector is constructed. The first acoustic imaging feature mapping vector is multiplied with the acoustic imaging feature and the thermal imaging feature respectively. For example, the first acoustic imaging feature mapping vector may highlight the acoustic reflection characteristics of a specific part of the blade. Multiplying it with the acoustic imaging feature can further emphasize the acoustic characteristics of the part; multiplying it with the thermal imaging feature can explore the potential connection between the acoustic characteristics and the thermal distribution. The first acoustic thermal correlation transition vector is constructed through the results of these two multiplication operations, and the association between the acoustic thermal features is preliminarily established. Then, the second acoustic thermal correlation transition vector is constructed. The conjugate feature mapping is taken for the multiplication result of the first thermodynamic feature mapping vector with the acoustic imaging feature and the thermal imaging feature. The conjugate feature mapping can explore the hidden symmetry or complementary relationship between the features. For example, if the first thermodynamic feature mapping vector emphasizes the temperature gradient of a certain area of ​​the blade, after multiplying it with the acoustic imaging feature and taking the conjugate feature mapping, the deep connection between the temperature gradient and the acoustic anomaly can be found. Similarly, the conjugate feature mapping is taken for the multiplication result of it with the thermal imaging feature. The second acoustic thermal correlation transition vector is constructed based on these two conjugate feature mappings to further enrich the association information between the acoustic thermal features. Afterwards, the third acoustic-thermal correlation transition vector is constructed. The first acoustic-thermal coupling basis vector is used to multiply the acoustic imaging feature and the thermal imaging feature respectively. The first acoustic-thermal coupling basis vector captures the basic characteristics of the interaction between the acoustic and thermal modes. The multiplication operation can deeply reveal the coupling effect of acoustic and thermal in blade detection, thereby constructing the third acoustic-thermal correlation transition vector and strengthening the coupling connection of the acoustic-thermal features. Subsequently, the first acoustic-thermal correlation transition vector is multiplied by the second acoustic-thermal correlation transition vector to obtain the fourth acoustic-thermal correlation transition vector. This step further integrates the acoustic-thermal correlation information contained in the first two transition vectors, making it richer and more comprehensive. At the same time, the acoustic modal dimension of the acoustic imaging feature is obtained, which reflects the dimensional characteristics of the acoustic feature such as space or frequency.Finally, the fourth acoustic-thermal correlation transition vector is divided by the scale factor of the acoustic modal dimension and normalized to obtain the self-feature significance coefficient vector. The scale factor is used to adjust the adaptability of the fourth acoustic-thermal correlation transition vector to the acoustic modal dimension, and the normalization process makes the coefficient vector more comparable under a unified scale. The self-feature significance coefficient vector is multiplied by the third acoustic-thermal correlation transition vector to obtain the first cross-modal correlation weight spectrum. This weight spectrum comprehensively and meticulously reflects the correlation weights between the acoustic imaging features and the thermal imaging features in the turbine blade inspection equipment, providing a key basis for accurately evaluating the blade status. For example, it can be used to judge the closeness between the internal defects of the blade and the surface temperature anomalies, helping engineers identify potential quality problems.

[0035] In an embodiment of the present invention, the first modality integration unit includes a cross-modality collaboration adapter; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor, which can be implemented through the following example.

[0036] The acoustic imaging feature is transformed into a second acoustic imaging feature mapping vector, and the thermal imaging feature is transformed into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector by the cross-modal collaborative adapter; Performing vector multiplication calculation on the conjugate feature mapping of the multiplication operation result between the second acoustic imaging feature mapping vector and the acoustic imaging feature and the multiplication operation result between the second thermodynamic feature mapping vector and the thermal imaging feature to obtain a fifth acoustic-thermal correlation transition vector, and acquiring the acoustic modal dimension of the acoustic imaging feature; The result of the division operation between the fifth acoustic-thermal correlation transition vector and the scale factor of the acoustic modal dimension is normalized to obtain an acoustic-thermal feature significance coefficient vector, and a vector multiplication calculation is performed on the acoustic-thermal feature significance coefficient vector and a result of the multiplication operation between the second acoustic-thermal coupling basis vector and the thermal imaging feature to obtain an acoustic-thermal energy aggregation feature; The dominant correlation feature in the acoustic thermal energy aggregation feature is determined as the first feature descriptor.

[0037] In an embodiment of the present invention, illustratively, taking the server monitoring the power bogie detection equipment of the high-speed train as an example, after the server obtains the acoustic imaging features and thermal imaging features of the power bogie detection equipment, a cross-modal fusion operation is performed in the first modal integration unit including the cross-modal collaborative adapter. First, the cross-modal collaborative adapter starts working. The acoustic imaging features reflect the operating sound characteristics of the internal components of the power bogie, such as gears, bearings, etc., and can detect whether there is abnormal wear or noise caused by looseness; the thermal imaging features show the temperature distribution on the surface of the bogie, and can find local overheating areas caused by frictional heat. The cross-modal collaborative adapter transforms the acoustic imaging features into a second acoustic imaging feature mapping vector, which re-characterizes the acoustic information from a specific angle, such as highlighting the sound features of certain key frequency bands; at the same time, the thermal imaging features are transformed into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector, the former highlighting the features of a specific temperature change area in thermal imaging, and the latter capturing the potential coupling relationship features between acoustics and heat. Next, vector operations are performed to construct the fifth acoustic-thermal correlation transition vector. The second acoustic imaging feature mapping vector is multiplied with the acoustic imaging feature to highlight the part of the original acoustic imaging feature related to the transformed feature; the conjugate feature mapping is taken for the result of the multiplication of the second thermodynamic feature mapping vector and the thermal imaging feature to mine the symmetric or complementary information related to the acoustic feature hidden in the thermal imaging feature, and then the two results are vector-multiplied. For example, if an abnormal sound is detected at a certain gear position in the acoustic imaging, and the temperature near the position is slightly increased in the thermal imaging, these two seemingly independent information can be more closely linked after these operations, thereby obtaining the fifth acoustic-thermal correlation transition vector. At the same time, the acoustic modal dimension of the acoustic imaging feature is obtained. This dimensional information reflects the characteristics of the acoustic feature in terms of frequency, spatial distribution, etc. After that, the fifth acoustic-thermal correlation transition vector is processed to obtain the acoustic-thermal feature significance coefficient vector. The fifth acoustic-thermal correlation transition vector is divided by the scale factor of the acoustic modal dimension. The scale factor is used to adjust the degree of adaptation between the vector and the acoustic modal dimension to make the calculation result more reasonable. The division result is then standardized so that the obtained vector can more clearly reflect the relative importance of each part of the features at a unified scale, thereby obtaining the acoustic-thermal feature significance coefficient vector. Next, the vector is multiplied with the result of multiplying the second acoustic-thermal coupling basis vector and the thermal imaging feature to obtain the acoustic-thermal energy aggregation feature. This step integrates the coupling relationship between acoustics and heat and the thermal imaging feature information, and weights it according to the significance coefficient obtained previously, so that the final feature can better reflect the key information after the acoustic-thermal fusion. For example, if the acoustic-thermal coupling feature of a certain area is more important in the overall evaluation, after this series of operations, the corresponding features of the area will be strengthened accordingly in the acoustic-thermal energy aggregation feature. Finally, the dominant correlation feature is determined from the acoustic-thermal energy aggregation feature and used as the first feature descriptor.For example, in the acoustic-thermal energy aggregation features, it is found that the acoustic-thermal correlation feature of a key bearing position of the power bogie is the most prominent, and the information contained in it is crucial to determine whether the bogie is operating normally, so this dominant correlation feature is determined as the first feature descriptor. This descriptor combines the key information about the power bogie detection equipment in both acoustic and thermal modes, providing an important basis for subsequent accurate evaluation of the equipment status and judgment of whether the power bogie is faulty.

[0038] In an embodiment of the present invention, the first modality integration unit includes a target cross-modality collaborative adapter; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor, which can be implemented through the following example.

[0039] Acquiring an acoustic-thermal coupling feature basis, and stacking the acoustic-thermal coupling feature basis and the thermal imaging feature into a multi-modal tensor to obtain a second cross-modal joint tensor; the acoustic-thermal coupling feature basis is used to indicate the physical field correlation between the acoustic and thermal imaging features; By using the first multimodal association control network in the target cross-modal collaborative adapter, the second cross-modal joint tensor is automatically and dynamically weighted to obtain a second cross-modal association weight spectrum, and the dominant association feature in the second cross-modal association weight spectrum is determined as a candidate feature descriptor; The acoustic imaging feature and the candidate feature descriptor are stacked as multimodal tensors to obtain a third cross-modal joint tensor, and the third cross-modal joint tensor is automatically and dynamically weighted by the second multimodal association control network in the target cross-modal collaborative adapter to obtain a third cross-modal association weight spectrum; The dominant correlation feature in the third cross-modal correlation weight spectrum is determined as a first feature descriptor.

[0040] In an embodiment of the present invention, for example, it is assumed that the server is responsible for monitoring the crusher equipment in a large mine. First, the server obtains the acoustic-thermal coupling feature base. During the operation of the crusher, the acoustic feature reflects the sound characteristics of the internal components of the equipment, such as the crushing hammer, liner, etc., when they are working, and can reveal whether the components are loose, worn, etc.; the thermal imaging feature shows the temperature distribution on the surface of the equipment, and can detect local overheating areas caused by friction, overload, etc. The acoustic-thermal coupling feature base is based on the analysis of a large amount of historical operation data of the crusher, and is used to indicate the physical field correlation between acoustic and thermal imaging features. For example, abnormal sounds in specific parts are often accompanied by corresponding temperature change patterns, and the acoustic-thermal coupling feature base records this correlation. The server stacks the acoustic-thermal coupling feature base and the thermal imaging feature as multi-modal tensors, just like stacking related information cards according to specific rules, to obtain a second cross-modal joint tensor. Then, the second cross-modal joint tensor is automatically dynamic weighted and calculated through the first multi-modal association control network in the target cross-modal collaborative adapter. The network analyzes the importance of each part of the information in the second cross-modal joint tensor based on the physical field correlation contained in the acoustic-thermal coupling feature base. For example, if the acoustic-thermal coupling feature base indicates that the temperature increase in a certain area of ​​the crusher's crushing chamber is closely related to the abnormal sound of a specific frequency, then the thermal imaging feature information in this area will be given a higher weight in the calculation. By comprehensively analyzing the entire tensor, the second cross-modal correlation weight spectrum is obtained. This weight spectrum records in detail the correlation weights between each part of the thermal imaging feature and the acoustic-thermal coupling feature base. Then, the dominant correlation features in the second cross-modal correlation weight spectrum are determined as candidate feature descriptors. For example, the part of the abnormal temperature increase in a key part of the crushing chamber that is strongly correlated with the corresponding acoustic feature in the acoustic-thermal coupling feature base is selected as a candidate feature descriptor, which preliminarily integrates the key information related to thermal imaging and acoustic-thermal coupling. After that, the server stacks the acoustic imaging features and candidate feature descriptors into multi-modal tensors, and integrates different types of important information together again to obtain the third cross-modal joint tensor. Then, the third cross-modal joint tensor is automatically and dynamically weighted by the second multimodal association control network in the target cross-modal collaborative adapter. This network will comprehensively consider the relationship between the acoustic imaging feature and the candidate feature descriptor. For example, if the candidate feature descriptor points to an overheated area in the crushing chamber, and there are abnormal sound reflection features near this area in the acoustic imaging feature, then this part of the acoustic feature will be given a higher weight in the calculation. The third cross-modal association weight spectrum is calculated, which fully reflects the association weights between the acoustic imaging feature and the candidate feature descriptor. Finally, the server determines the dominant association feature in the third cross-modal association weight spectrum as the first feature descriptor. For example, in the weight spectrum, it is found that the strong correlation feature between the acoustic anomaly and the thermal anomaly at a specific position in the crushing chamber of the crusher is the most prominent, and this dominant association feature is determined as the first feature descriptor.This descriptor highly integrates the acoustic and thermal imaging feature information of the crusher, providing a key basis for accurately judging the operating status of the crusher and discovering potential faults, helping mine workers to take maintenance measures in a timely manner to ensure the stable operation of the crusher.

[0041] In an embodiment of the present invention, the first modality integration unit includes a thermodynamic fuser; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor, which can be implemented through the following example.

[0042] Acquiring a structural anomaly calibration feature corresponding to the acoustic imaging feature, and a thermodynamic state calibration feature corresponding to the thermal imaging feature; The acoustic imaging feature is subjected to feature enhancement transmission through a first dynamic coding network in the thermodynamic fusion device to obtain a first dynamic coding feature; The thermal imaging feature is subjected to feature enhancement transmission through a second dynamic coding network in the thermodynamic fusion device to obtain a second dynamic coding feature; The second dynamic coding feature is subjected to energy-balanced aggregation to obtain a third steady-state feature component, and the first dynamic coding feature and the third steady-state feature component are accumulated to obtain a first feature descriptor.

[0043] In the embodiment of the present invention, for example, it is assumed that the server is monitoring a steam turbine of a thermal power plant.

[0044] When monitoring a steam turbine, the server first obtains the structural anomaly calibration features corresponding to the acoustic imaging features and the thermodynamic state calibration features corresponding to the thermal imaging features. For the acoustic imaging features, the features reflecting the structural information of the steam turbine have been obtained by processing the ultrasonic image data. After the dynamic range is normalized, the structural anomaly calibration features are obtained, such as standardized features that can clearly show whether there are structural anomalies in key components such as blades and shafts. The thermal imaging features are also calibrated by the thermodynamic dynamic range to obtain the thermodynamic state calibration features, which can accurately show whether the thermodynamic states of various parts of the steam turbine, such as bearings and seals, are normal. Then, the server uses the first dynamic coding network in the thermodynamic fusion device to perform feature enhancement transmission on the acoustic imaging features. The first dynamic coding network will highlight the parts of the acoustic imaging features that are closely related to potential structural problems based on the operating characteristics of the steam turbine and past fault data. For example, if historical data shows that the tiny cracks in the steam turbine blades have specific manifestations in the acoustic features, the first dynamic coding network will strengthen this part of the acoustic features to obtain the first dynamic coding features, making the structural anomaly information more significant. At the same time, the second dynamic coding network in the thermodynamic fusion device performs feature enhancement transmission on the thermal imaging features. It will focus on the key thermal information in the thermal imaging features that can reflect the abnormal operation of the equipment. For example, for the abnormal temperature increase caused by friction or uneven load in some parts of the steam turbine, the second dynamic coding network will enhance these abnormal thermal features to obtain the second dynamic coding features, highlighting the parts of the thermal imaging features related to potential faults. After that, the server performs energy balance aggregation on the second dynamic coding features. Since the energy distribution of thermal imaging features in different regions and times may be different, this step will make the energy distribution of thermal features more uniform and reasonable, eliminate the interference of local excessive or low energy on the overall analysis, and obtain the third steady-state feature component. For example, in different operating stages of the steam turbine, the temperature change amplitude of some components is different. Through energy balance aggregation, the energy level of these thermal features can be unified, which is convenient for better fusion with acoustic features. Finally, the server accumulates the first dynamic coding features and the third steady-state feature components. The enhanced acoustic structural abnormality features are combined with the thermal features after energy balance to obtain the first feature descriptor. This first feature descriptor integrates the key acoustic and thermal information of the steam turbine and comprehensively reflects the operating status of the equipment. For example, it can simultaneously reflect the structural abnormalities of the blades and the local overheating problems that may be related to them. It provides an important basis for accurately assessing whether there is a fault in the steam turbine and the type of fault, helping power plant maintenance personnel to promptly discover and resolve potential problems and ensure the stable operation of the steam turbine.

[0045] In an embodiment of the present invention, the first modal integration unit includes an acoustic dominant selector; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor, which can be implemented through the following example.

[0046] The thermal imaging feature is deleted, and the acoustic imaging feature is used as the first feature descriptor output by the acoustic dominant selector.

[0047] In an embodiment of the present invention, for example, it is assumed that the server is responsible for monitoring a large cement mill in a cement plant. When monitoring the cement mill, the server uses a first modal integration unit including an acoustic dominant selector to realize cross-modal fusion of acoustic imaging features according to thermal imaging features to obtain a first feature descriptor. First, the server obtains the acoustic imaging features and thermal imaging features of the cement mill. The acoustic imaging features are obtained through ultrasonic detection and can reflect the internal structural conditions of the cement mill, such as the wear of the grinding body, the degree of fit of the liner, etc., and these information are presented in specific acoustic signal features. The thermal imaging features show the temperature distribution on the surface of the cement mill, which can be used to determine the heating conditions of various parts of the equipment during operation, such as the temperature changes caused by energy loss in the process of converting electrical energy into mechanical energy near the motor. However, in some specific cases, according to the long-term operating experience and data analysis accumulated by the cement plant, it is found that for cement mills, acoustic imaging features have a higher priority and a key role in determining the core faults of the equipment. For example, when the grinding body inside the mill is broken, a unique acoustic abnormal signal will be generated, while the thermal imaging features may not have obvious abnormal changes. Or when the equipment is operating normally, the temperature changes reflected by the thermal imaging features are mostly within the normal fluctuation range, and the indicative effect on judging equipment failure is relatively weak. Based on this situation, in the first modal integration unit, the server performs corresponding operations through the acoustic dominant selector. The acoustic dominant selector directly deletes the thermal imaging features, retains only the acoustic imaging features, and uses them as the first feature descriptor of the output. This is done because in the current monitoring scenario, the acoustic imaging features can more effectively reflect whether there is a fault in the cement mill and the type and location of the fault. For example, when the lining inside the cement mill is loose, the acoustic imaging features will clearly capture the abnormal acoustic reflection signals caused by the looseness. By analyzing these signal features, the loose position and degree of the lining can be determined. In this case, the thermal imaging features may not provide valuable fault information because the loose lining has not caused obvious temperature changes. Therefore, using the acoustic imaging features as the first feature descriptor can provide direct and critical equipment status information for the maintenance personnel of the cement plant, helping them to quickly and accurately judge the operating status of the cement mill, take maintenance measures in time, ensure the stable operation of the cement mill, and ensure the production efficiency of the cement plant.

[0048] In the embodiment of the present invention, determining the fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor can be implemented through the following example.

[0049] Performing multi-modal tensor stacking on the first feature descriptor and the second feature descriptor to obtain a fourth cross-modal joint tensor, configuring the fault type label at the reference feature bit of the fourth cross-modal joint tensor to obtain a fused feature descriptor; The fusion feature descriptor is dynamically weighted and calculated through a multimodal collaborative attention network to obtain physical field coupling features, and high-dimensional feature embedding is performed on the physical field coupling features to obtain a fault assessment result corresponding to the target industrial equipment.

[0050] In an embodiment of the present invention, for example, it is assumed that the server is monitoring a welding robot in an automobile manufacturing plant as a target industrial equipment. The server has obtained the first feature descriptor and the second feature descriptor of the welding robot. The first feature descriptor is obtained by cross-modal fusion of acoustic imaging features through thermal imaging features. For example, it may fuse the sound features generated during welding and the temperature distribution features of the welding head, reflecting whether there are potential problems such as cold welding and overheating of the welding head during welding. The second feature descriptor is obtained by cross-modal fusion of thermal imaging features based on the first feature descriptor, and further integrates the two modal information, such as the influence of the temperature field distribution of the welding area and the sound features on the welding quality. First, the server stacks the first feature descriptor and the second feature descriptor into multi-modal tensors. This is like merging two tables containing different but related information according to specific rules to obtain a fourth cross-modal joint tensor. Then, the fault type label is configured in the reference feature position of the fourth cross-modal joint tensor. For example, common fault types of welding robots include "unsound welding", "blocked welding head", "overheated line", etc. The server places these fault type labels in specific positions of the joint tensor to obtain a fused feature descriptor. This fusion feature descriptor integrates the features obtained from different modal analyses of the welding robot and the information on possible fault types. Next, the server dynamically calculates the weight assignment of the fusion feature descriptor through a multimodal collaborative attention network. The multimodal collaborative attention network analyzes the degree of association between each part of the fusion feature descriptor and different fault types. For example, for the "weak welding" fault, the network will focus on the features related to the unstable welding sound in the first feature descriptor and the features of the uneven temperature in the welding area in the second feature descriptor, and assign higher weights to these features closely related to "weak welding". Through a comprehensive analysis of all fault types and related features, the physical field coupling feature is obtained, which highlights the distribution of association weights between different physical field (acoustic and thermal) features and fault types. Finally, the server performs high-dimensional feature embedding on the physical field coupling feature. This step is to expand the physical field coupling feature to a higher dimension so that it contains more information at more levels. For example, through a specific algorithm, the feature weight relationship related to different fault types is further refined and enriched, and the nonlinear relationship between features is considered, so as to obtain the fault assessment result corresponding to the target industrial equipment (welding robot). This result may be to determine whether the welding robot currently has a fault, and if so, what kind of fault it is and how severe the fault is. For example, the evaluation result may show that the welding robot has a "weak welding" fault with a moderate severity, which will help technicians in the automobile manufacturing plant to maintain and adjust the welding robot in a timely manner, ensure welding quality, and ensure the smooth progress of automobile production.

[0051] Please refer to Figure 2, Figure 2 The device 110 based on acoustic-thermal multimodal fusion imaging provided in the embodiment of the present invention includes: The acquisition module 1101 is used to acquire ultrasonic images and thermal images of target industrial equipment using an acoustic imager to obtain ultrasonic image data and thermal imaging data; The fusion module 1102 is used to obtain the acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and to obtain the thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; perform feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modal integration unit corresponding to the acoustic imaging feature in multiple modal integration units according to the first acoustic-thermal joint feature; the multiple modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; in the first modal integration unit, the acoustic imaging features are encoded according to the thermal imaging features. The thermal imaging feature is cross-modally fused to obtain a first feature descriptor; the thermal imaging feature and the first feature descriptor are feature integrated in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and a second modal integration unit corresponding to the thermal imaging feature is determined from the multiple modal integration units according to the second acoustic-thermal joint feature; in the second modal integration unit, the thermal imaging feature is cross-modally fused according to the first feature descriptor to obtain a second feature descriptor; a fault type label is obtained, and a fault assessment result corresponding to the target industrial equipment is determined according to the fault type label, the first feature descriptor and the second feature descriptor; The imaging module 1103 is used to calibrate the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault assessment result to the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

[0052] It should be noted that the implementation principle of the aforementioned device 110 based on acoustic and thermal multimodal fusion imaging can refer to the implementation principle of the aforementioned method based on acoustic and thermal multimodal fusion imaging, which will not be repeated here.

[0053] The embodiment of the present invention provides a system based on acoustic and thermal multimodal fusion imaging, including a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the aforementioned method based on acoustic and thermal multimodal fusion imaging is executed. Figure 3 As shown, Figure 3The system based on acoustic-thermal multimodal fusion imaging provided by the embodiment of the present invention includes an apparatus 110 based on acoustic-thermal multimodal fusion imaging, a memory 111, a processor 112, and a communication unit 113. To achieve data transmission or interaction, the memory 111, the processor 112, and the communication unit 113 are electrically connected to each other directly or indirectly.

[0054] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A method based on acoustic and thermal multimodal fusion imaging, characterized in that: include: Use an ultrasonic imager to collect ultrasonic images and thermal images of target industrial equipment to obtain ultrasonic image data and thermal imaging data; Acquire acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and acquire thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; Performing feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determining a first modal integration unit corresponding to the acoustic imaging feature from a plurality of modal integration units according to the first acoustic-thermal joint feature; the plurality of modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; In the first modality integration unit, the acoustic imaging feature is cross-modally fused according to the thermal imaging feature to obtain a first feature descriptor; Performing feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determining a second modal integration unit corresponding to the thermal imaging feature from among the multiple modal integration units according to the second acoustic-thermal joint feature; In the second modality integration unit, cross-modal fusion is performed on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; Acquire a fault type label, and determine a fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor; The ultrasonic image data and the thermal imaging data are calibrated based on the spatial coordinate mapping relationship to construct a three-dimensional acoustic-thermal fusion base image, and the fault assessment result is marked on the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

2. The method according to claim 1, characterized in that The step of obtaining the acoustic imaging features corresponding to the target industrial equipment includes: Acquire ultrasonic imaging data corresponding to the target industrial equipment, and divide the ultrasonic imaging data into a plurality of local ultrasonic images; Performing feature extraction processing on the multiple local ultrasonic images to obtain image feature descriptors corresponding to the multiple local ultrasonic images; Performing dynamic weight allocation calculation on image feature descriptors corresponding to the multiple local ultrasound images to obtain feature significance coefficients corresponding to the multiple local ultrasound images; The image feature descriptors corresponding to the multiple local ultrasonic images and the feature significance coefficients corresponding to the multiple local ultrasonic images are linearly superimposed to obtain the acoustic imaging features corresponding to the ultrasonic imaging data.

3. The method according to claim 1, characterized in that: The step of obtaining the thermal imaging features corresponding to the target industrial equipment includes: Acquire thermal imaging data of the target industrial equipment, perform thermal radiation feature segmentation on the thermal imaging data, and obtain a multi-scale thermodynamic feature map; Constructing a thermal imaging feature pyramid network, and enhancing the multi-scale thermodynamic feature map through a spatial attention layer of the pyramid network to obtain enhanced thermodynamic features; Cross-modally aligning the enhanced thermodynamic features with real-time thermodynamic parameters to generate a multi-dimensional thermal field fusion feature; The multi-dimensional thermal field fusion features are temporally and spatially encoded using a thermodynamic feature encoder to obtain thermal imaging features corresponding to the target industrial equipment.

4. The method according to claim 1, characterized in that The step of performing feature integration processing on the acoustic imaging feature and the thermal imaging feature in the same feature alignment domain to obtain a first acoustic-thermal joint feature includes: Inputting the acoustic imaging feature and the thermal imaging feature into an adaptive feature selection unit, performing feature transformation on the acoustic imaging feature through a first nonlinear transformation network in the adaptive feature selection unit to obtain a structural abnormality feature; The thermal imaging feature is transformed by a second nonlinear transformation network in the adaptive feature selection unit to obtain a thermodynamic state feature; the structural anomaly feature and the thermodynamic state feature have the same feature alignment domain; Acquiring a first energy intensity value corresponding to the structural abnormality feature, and performing dynamic range normalization on the structural abnormality feature according to the first energy intensity value to obtain a structural abnormality calibration feature; Performing nonlinear enhancement processing on the structural abnormality calibration feature to obtain a structural abnormality enhancement feature, and performing time domain smoothing aggregation on the structural abnormality enhancement feature to obtain a first steady-state feature component; Acquiring a second energy intensity value corresponding to the thermodynamic state characteristic, and performing thermodynamic dynamic range calibration on the thermodynamic state characteristic according to the second energy intensity value to obtain a thermodynamic state calibration characteristic; Performing threshold response processing on the thermodynamic state calibration feature to obtain a thermodynamic state enhancement feature, and performing spatial consistency integration on the thermodynamic state enhancement feature to obtain a second steady-state feature component; The first steady-state characteristic component and the second steady-state characteristic component are added to obtain a multi-physical field coupling feature, and high-dimensional feature embedding is performed on the multi-physical field coupling feature according to the multi-physical field coupling weight corresponding to the affine transformation network in the adaptive feature selection unit to obtain the first acoustic-thermal joint feature.

5. The method according to claim 4, characterized in that The determining, according to the first acoustic-thermal joint feature, a first modal integration unit corresponding to the acoustic imaging feature from a plurality of modal integration units comprises: Performing multi-modal fitness calculation on the first acoustic-thermal joint feature through a dynamic feature routing network in the adaptive feature selection unit to obtain modal priority scores corresponding to multiple modal integration units; The modal integration unit corresponding to the maximum modal priority score is determined as the first modal integration unit corresponding to the acoustic imaging feature.

6. The method according to claim 1, characterized in that The first modality integration unit includes a cross-modality collaboration adapter; In the first modality integration unit, cross-modal fusion is performed on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor, including: Performing multi-modal tensor stacking on the acoustic imaging feature and the thermal imaging feature to obtain a first cross-modal joint tensor; According to the multimodal projection vector corresponding to the cross-modal collaborative adapter, transforming the first cross-modal joint tensor into a first acoustic imaging feature mapping vector, a first thermodynamic feature mapping vector and a first acoustic-thermal coupling basis vector; Constructing a first acoustic-thermal correlation transition vector according to a multiplication result between the first acoustic imaging feature mapping vector and the acoustic imaging feature, and a multiplication result between the first acoustic imaging feature mapping vector and the thermal imaging feature; Constructing a second acoustic-thermal correlation transition vector according to a conjugate feature map of a multiplication result between the first thermodynamic feature map vector and the acoustic imaging feature, and a conjugate feature map of a multiplication result between the first thermodynamic feature map vector and the thermal imaging feature; constructing a third acoustic-thermal correlation transition vector according to a multiplication result between the first acoustic-thermal coupling basis vector and the acoustic imaging feature, and a multiplication result between the first acoustic-thermal coupling basis vector and the thermal imaging feature; Determining a result of a multiplication operation between the first acoustic-thermal correlation transition vector and the second acoustic-thermal correlation transition vector as a fourth acoustic-thermal correlation transition vector, and acquiring an acoustic modal dimension of the acoustic imaging feature; Standardizing a division operation result between the fourth acoustic-thermal association transition vector and the scaling factor of the acoustic modal dimension to obtain a self-feature significance coefficient vector, and determining a multiplication operation result between the self-feature significance coefficient vector and the third acoustic-thermal association transition vector as the first cross-modal association weight spectrum; determining the dominant correlation feature in the first cross-modal correlation weight spectrum as a first feature descriptor; or, In the first modality integration unit, cross-modal fusion is performed on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor, including: The acoustic imaging feature is transformed into a second acoustic imaging feature mapping vector, and the thermal imaging feature is transformed into a second thermodynamic feature mapping vector and a second acoustic-thermal coupling basis vector by the cross-modal collaborative adapter; Performing vector multiplication calculation on the conjugate feature mapping of the multiplication operation result between the second acoustic imaging feature mapping vector and the acoustic imaging feature and the multiplication operation result between the second thermodynamic feature mapping vector and the thermal imaging feature to obtain a fifth acoustic-thermal correlation transition vector, and acquiring the acoustic modal dimension of the acoustic imaging feature; The result of the division operation between the fifth acoustic-thermal correlation transition vector and the scale factor of the acoustic modal dimension is normalized to obtain an acoustic-thermal feature significance coefficient vector, and a vector multiplication calculation is performed on the acoustic-thermal feature significance coefficient vector and a result of the multiplication operation between the second acoustic-thermal coupling basis vector and the thermal imaging feature to obtain an acoustic-thermal energy aggregation feature; The dominant correlation feature in the acoustic thermal energy aggregation feature is determined as the first feature descriptor.

7. The method according to claim 1, characterized in that The first modality integration unit includes a thermodynamic fuser; In the first modality integration unit, cross-modal fusion is performed on the acoustic imaging feature according to the thermal imaging feature to obtain a first feature descriptor, including: Acquiring a structural anomaly calibration feature corresponding to the acoustic imaging feature, and a thermodynamic state calibration feature corresponding to the thermal imaging feature; The acoustic imaging feature is subjected to feature enhancement transmission through a first dynamic coding network in the thermodynamic fusion device to obtain a first dynamic coding feature; The thermal imaging feature is subjected to feature enhancement transmission through a second dynamic coding network in the thermodynamic fusion device to obtain a second dynamic coding feature; Performing energy-balanced aggregation on the second dynamic coding feature to obtain a third steady-state feature component, and accumulating the first dynamic coding feature and the third steady-state feature component to obtain a first feature descriptor; The first modal integration unit further includes an acoustic dominant selector; The method further comprises: performing cross-modal fusion of the acoustic imaging feature according to the thermal imaging feature in the first modality integration unit to obtain a first feature descriptor; The thermal imaging feature is deleted, and the acoustic imaging feature is used as the first feature descriptor output by the acoustic dominant selector.

8. The method according to claim 1, characterized in that The determining, according to the fault type label, the first feature descriptor, and the second feature descriptor, a fault assessment result corresponding to the target industrial equipment includes: Performing multi-modal tensor stacking on the first feature descriptor and the second feature descriptor to obtain a fourth cross-modal joint tensor, configuring the fault type label at the reference feature bit of the fourth cross-modal joint tensor to obtain a fused feature descriptor; The fusion feature descriptor is dynamically weighted and calculated through a multimodal collaborative attention network to obtain physical field coupling features, and high-dimensional feature embedding is performed on the physical field coupling features to obtain a fault assessment result corresponding to the target industrial equipment.

9. A device based on acoustic and thermal multimodal fusion imaging, characterized in that: include: An acquisition module is used to acquire ultrasonic images and thermal images of target industrial equipment using an acoustic imager to obtain ultrasonic image data and thermal imaging data; A fusion module is used to obtain the acoustic imaging features corresponding to the target industrial equipment based on the ultrasonic image data, and to obtain the thermal imaging features corresponding to the target industrial equipment based on the thermal imaging data; perform feature integration processing on the acoustic imaging features and the thermal imaging features in the same feature alignment domain to obtain a first acoustic-thermal joint feature, and determine a first modal integration unit corresponding to the acoustic imaging feature in multiple modal integration units according to the first acoustic-thermal joint feature; the multiple modal integration units are used to encode the acoustic and thermal imaging features using different modal weight coefficients; in the first modal integration unit, the acoustic imaging feature is integrated according to the thermal imaging feature. Perform cross-modal fusion on the thermal imaging feature to obtain a first feature descriptor; perform feature integration processing on the thermal imaging feature and the first feature descriptor in the same feature alignment domain to obtain a second acoustic-thermal joint feature, and determine a second modal integration unit corresponding to the thermal imaging feature in the multiple modal integration units according to the second acoustic-thermal joint feature; in the second modal integration unit, perform cross-modal fusion on the thermal imaging feature according to the first feature descriptor to obtain a second feature descriptor; obtain a fault type label, and determine a fault assessment result corresponding to the target industrial equipment according to the fault type label, the first feature descriptor, and the second feature descriptor; An imaging module is used to calibrate the ultrasonic image data and the thermal imaging data based on the spatial coordinate mapping relationship, construct a three-dimensional acoustic-thermal fusion base image, and mark the fault assessment result to the three-dimensional acoustic-thermal fusion base image to obtain a diagnostic fusion image of the target industrial equipment.

10. A system based on acoustic and thermal multimodal fusion imaging, characterized in that: The method comprises a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the method described in any one of claims 1 to 8 is executed.

Images