Real-time sound field imaging method and system based on FPGA and ARM heterogeneous computing

Abstract

This invention relates to the technical field of acoustic field imaging, and discloses a real-time acoustic field imaging method and system based on heterogeneous computing of FPGA and ARM. The method involves using an FPGA module to acquire acoustic signal data from an acoustic sensor array, and simultaneously acquiring image signal data and infrared signal data through a DVP interface and a USB interface. An ARM computing unit performs data fusion on the acoustic signal data, image signal data, and infrared signal data acquired by the FPGA module to obtain acoustic fusion imaging. Based on the fault detection information fed back by the acoustic fusion imaging, processing logic is allocated between the FPGA module and the ARM computing unit to update the acoustic fusion imaging, resulting in several acoustic optimization images. Fault analysis is performed on the detected target based on each acoustic optimization image to guide the combination of the various acoustic optimization images into fault diagnosis acoustic imaging.

Description

Technical Field

[0001] This invention relates to the technical field of sound field imaging, and in particular to a real-time sound field imaging method and system based on heterogeneous computing of FPGA and ARM. Background Technology

[0002] In electrical equipment inspection, traditional detection methods often rely on data from a single type of sensor, such as using acoustic sensors to detect sound signals, optical sensors to acquire images, or infrared sensors to detect thermal radiation. The information provided by a single source is limited and it is difficult to comprehensively and accurately reflect the state of the target being inspected. Traditional methods have certain bottlenecks in detection accuracy and efficiency, and are prone to missed detections and misjudgments. Moreover, the detection speed is slow and cannot meet the needs of rapid and accurate detection. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time sound field imaging method and system based on FPGA and ARM heterogeneous computing, which aims to solve the problem that existing technologies cannot quickly and accurately inspect electrical equipment.

[0004] The present invention is implemented as follows: Firstly, the present invention provides a real-time sound field imaging method based on heterogeneous computing of FPGA and ARM, comprising: Acoustic signal data acquired by the acoustic sensor array is collected using an FPGA module, and image signal data and infrared signal data are simultaneously acquired through the DVP interface and USB interface. The acoustic signal data, image signal data and infrared signal data acquired by the FPGA module are fused using the ARM computing unit to obtain acoustic fusion imaging; Based on the fault detection information fed back by the acoustic fusion imaging, the FPGA module and ARM computing unit are allocated processing logic to update the acoustic fusion imaging and obtain several acoustic optimization images. Based on the aforementioned acoustic optimization imaging, fault analysis is performed on the detected target to guide the combination of the aforementioned acoustic optimization imaging into fault diagnosis acoustic imaging.

[0005] Secondly, the present invention provides a real-time sound field imaging system based on FPGA and ARM heterogeneous computing, used to implement the real-time sound field imaging method based on FPGA and ARM heterogeneous computing as described in any one of the first aspects, comprising: The signal acquisition module is used to acquire acoustic signal data obtained by the acoustic sensor array based on the FPGA module, and simultaneously acquire image signal data and infrared signal data through the DVP interface and USB interface; The data fusion module is used to fuse acoustic signal data, image signal data and infrared signal data acquired by the FPGA module based on the ARM computing unit to obtain acoustic fusion imaging; An imaging optimization module is used to allocate processing logic to the FPGA module and ARM computing unit based on the fault detection information fed back by the acoustic fusion imaging, so as to update the acoustic fusion imaging and obtain several acoustic optimization images. An imaging integration module is used to perform fault analysis on the detected target based on each of the acoustic optimization imaging methods, so as to guide the integration of each of the acoustic optimization imaging methods into fault diagnosis acoustic imaging.

[0006] This invention provides a real-time sound field imaging method based on heterogeneous computing of FPGA and ARM, which has the following beneficial effects: This invention rapidly acquires acoustic, image, and infrared multi-source signal data using an FPGA, ensuring timely and comprehensive data acquisition. Data fusion is then performed using an ARM processor to obtain acoustic fusion imaging, which comprehensively reflects the target's condition based on multiple types of information. The image is updated according to the fault information allocated in the fusion imaging, optimizing the imaging effect. Finally, fault analysis is performed based on the optimized imaging to obtain diagnostic imaging, accurately guiding the fault diagnosis of the detected target, improving the efficiency and accuracy of fault diagnosis, and providing reliable support for related fields. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the steps of a real-time sound field imaging method based on heterogeneous computing of FPGA and ARM provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a real-time sound field imaging system based on heterogeneous computing of FPGA and ARM provided in an embodiment of the present invention. Detailed Implementation

[0008] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0009] The implementation of the present invention will be described in detail below with reference to specific embodiments.

[0010] Reference Figure 1 , Figure 2 The diagram shows a preferred embodiment of the present invention.

[0011] In a first aspect, the present invention provides a real-time sound field imaging method based on heterogeneous computing of FPGA and ARM, comprising: S1: Acoustic signal data acquired by the acoustic sensor array is collected based on the FPGA module, and image signal data and infrared signal data are acquired simultaneously through the DVP interface and USB interface; S2: Based on the ARM computing unit, acoustic signal data, image signal data and infrared signal data acquired by the FPGA module are fused to obtain acoustic fusion imaging; S3: Based on the fault detection information fed back by the acoustic fusion imaging, allocate processing logic to the FPGA module and ARM computing unit to update the acoustic fusion imaging and obtain several acoustic optimization images; S4: Perform fault analysis on the detected target based on the acoustic optimization imaging described above, so as to guide the combination of the acoustic optimization imaging described above into fault diagnosis acoustic imaging.

[0012] Specifically, in step S1 of the embodiment provided by the present invention, the FPGA module sends initial detection commands to the acoustic sensor array, optical sensor and infrared sensor. The FPGA module sends the start detection command signal to each sensor through the pre-configured control signal output port in accordance with a specific communication protocol and timing.

[0013] More specifically, after receiving the initial detection command from the FPGA module, each sensor begins to acquire data from the target. The acoustic sensor array converts the acoustic signal emitted by the target into an electrical signal, the optical sensor converts the optical image of the target into an image signal, and the infrared sensor converts the infrared radiation of the target into an infrared signal. The acoustic signal data acquired by the acoustic sensor array is received through 128 sampling channels. The FPGA module utilizes its rich I / O interface resources to allocate an independent sampling channel for each acoustic sensor, sample and quantize the electrical signal output by the acoustic sensor, convert it into a digital signal, and store it.

[0014] More specifically, the FPGA module receives image signal data from the optical sensor and infrared signal data from the infrared sensor via the DVP interface and USB interface. The DVP interface is a common image data transmission interface, suitable for high-speed, low-cost image data transmission; the USB interface has the characteristics of strong versatility and high transmission rate, and can be used for infrared signal data transmission. The FPGA module receives, parses and stores the image signal and infrared signal according to the communication protocol of the corresponding interface.

[0015] More specifically, FPGAs have abundant logic resources and high-speed parallel processing capabilities, enabling them to handle data acquisition tasks from multiple sensors simultaneously. This ensures the synchronous acquisition of acoustic, image, and infrared signal data, avoiding information mismatch caused by time differences in data acquisition. FPGAs can be flexibly configured and programmed according to different application requirements, facilitating the adjustment of parameters such as sensor control signals, sampling frequency, and data transmission format to adapt to different types of detection targets and scenarios.

[0016] More specifically, the acoustic sensor array contains 128 acoustic sensors, each corresponding to a sampling channel, which can provide more comprehensive and detailed acoustic information. By increasing the number of sampling channels, the sampling resolution of acoustic signals can be improved, thereby capturing the acoustic features of the target more accurately. The parallel processing capability of the FPGA allows the 128 sampling channels to work simultaneously, greatly improving the acquisition efficiency of acoustic signal data and meeting the requirements of real-time sound field imaging for data acquisition speed.

[0017] More specifically, acoustic signals, image signals, and infrared signals reflect the characteristics of the detected target from different perspectives. Fusing these three types of data can obtain more comprehensive and accurate target information. Synchronous acquisition can ensure the consistency of the three types of data in time and space, improving the accuracy and effectiveness of subsequent data fusion. Different types of data have different sensitivities to the fault characteristics of the detected target. Through multimodal data fusion, the advantages of various data can be comprehensively utilized to more accurately identify and locate the faults of the detected target, improving the reliability and accuracy of fault diagnosis.

[0018] Specifically, in step S2 of the embodiment provided by the present invention, the image signal data is interpreted based on the image parsing algorithm configured by the ARM computing unit to identify the target classification information of the detected target. The ARM computing unit calls a pre-trained image classification model (such as a convolutional neural network model), inputs the image signal data into the model for processing, and the model extracts and analyzes the features in the image, and finally outputs the classification information of the detected target, such as whether the target is a mechanical part, electronic device or other type. Identifying the classification information of the detected target can help determine the characteristics of different types of targets and possible fault modes, providing a basis for subsequent adjustment of heterogeneous data fusion algorithm parameters, making the fusion process more targeted. Different types of targets have large differences in acoustic, image and infrared characteristics. Accurate classification can better utilize the advantages of various data and improve the accuracy and reliability of fused imaging.

[0019] More specifically, the parameters of the heterogeneous data fusion algorithm configured in the ARM computing unit are adjusted based on the identified target classification information. Different types of detected targets have different acoustic, image, and infrared features. Therefore, the parameters of the fusion algorithm need to be adjusted according to the target classification information to improve the fusion effect. For example, if the detected target is a mechanical part, the weight of the acoustic signal in the fusion needs to be increased; if it is an electronic device, more attention needs to be paid to the infrared signal. Different types of detected targets have different sensitivities to acoustic, image, and infrared signals. By adjusting the parameters of the fusion algorithm according to the target classification information, the fusion result can be made more consistent with the actual situation of the target, thereby improving the fusion effect. Reasonably adjusting the weight of each data can highlight the importance of different data, enabling the fused imaging to more accurately reflect the fault information of the detected target.

[0020] More specifically, an original digital model of the target is established based on image signal data, and several key identification regions are divided on the original digital model. Image processing techniques (such as edge detection, feature extraction, etc.) can be used to analyze the image to construct a three-dimensional model or a two-dimensional contour model of the target. Then, based on the structural and functional characteristics of the target, key identification regions that may have faults are divided, such as the connection parts of mechanical components and the heating areas of electronic devices. The original digital model provides a spatial framework for the fusion of acoustic and infrared signals, enabling different types of data to be correlated and fused in the same coordinate system. Dividing key identification regions can focus attention on areas that may have faults, reduce unnecessary data processing, and improve the efficiency and accuracy of fault detection.

[0021] More specifically, by combining acoustic signal data and infrared signal data to match key identification areas on the original digital model, the data mapping relationship between acoustic signal data and infrared signal data and the original digital model is obtained. Through feature matching, coordinate transformation and other methods, acoustic and infrared signals can be associated with key identification areas on the original digital model. For example, the intensity distribution of acoustic signals can be mapped to specific areas on the model, and the temperature distribution of infrared signals can be mapped to heat-generating areas on the model. By matching acoustic and infrared signals with key identification areas on the original digital model, a correspondence between different data can be established, providing a foundation for subsequent information mapping. Accurate data mapping relationships can enable acoustic and infrared information to be more accurately superimposed on the model, enhancing the information integrity and accuracy of fused imaging.

[0022] More specifically, based on the data mapping relationship, acoustic signal data and infrared signal data are decomposed into data units and mapped to generate acoustic feedback units and infrared feedback units at various locations on the original digital model, resulting in acoustic fusion imaging. Each unit value of the acoustic and infrared signals is associated with the corresponding position on the original digital model, and the acoustic and infrared information is superimposed on the model to form acoustic fusion imaging. Data decomposition and information mapping can accurately superimpose acoustic and infrared information onto the original digital model, enabling the fusion imaging to reflect the characteristics and fault information of the detected target in more detail. The generated acoustic fusion imaging can intuitively display the acoustic and infrared characteristics of the detected target, providing a clear visual basis for subsequent fault analysis.

[0023] Specifically, in step S3 of the embodiment provided by the present invention, fault detection is performed on the acoustic fusion imaging. Preset fault feature template matching, machine learning classification algorithms (such as support vector machines, decision trees, etc.) or deep learning models (such as convolutional neural networks) can be used to identify possible faults in the acoustic fusion imaging. Fault detection results are labeled on the acoustic fusion imaging, for example, using different colored markers, symbols or text to describe the type, location and other information of the fault. Fault detection and labeling can intuitively display the location and type of the fault on the acoustic fusion imaging, providing a clear basis for subsequent analysis and processing. The labeled fault detection information is the basis for evaluating the completeness of feedback, which helps to determine whether the current detection results are comprehensive and accurate enough. The evaluation of the completeness of the fault detection information feedback can be measured by calculating indicators such as the coverage and accuracy of the fault detection information. For example, the proportion of the detected fault area to the entire detection target area and the accuracy of the fault type judgment can be calculated.

[0024] More specifically, based on the evaluation results, a value analysis of changes in the acoustic signal sensing mode of the acoustic sensor array is performed. The potential value of different sensing modes (such as changing the sampling frequency, adjusting the direction or position of the sensors, etc.) in obtaining more comprehensive fault information is considered. Based on the analysis results, several optimized sensing strategies are generated, such as increasing the sampling frequency of certain key areas and adjusting the orientation of some sensors. Feedback integrity evaluation can identify the deficiencies in the current detection process. By generating optimized sensing strategies, the working mode of the acoustic sensor array can be adjusted to obtain more comprehensive and accurate acoustic signal data, thereby improving the effect of fault detection. Different detection targets and fault conditions require different sensing modes. Dynamically generating optimized sensing strategies based on the evaluation results can enable the system to have adaptive capabilities and better adapt to various complex situations.

[0025] More specifically, based on each optimized sensing strategy, corresponding processing logic is allocated to the FPGA module and the ARM computing unit in sequence. For the FPGA module, its control signals need to be reconfigured to change the working mode of the acoustic sensor array; for the ARM computing unit, the data processing algorithm or parameters need to be adjusted.

[0026] More specifically, the FPGA module sends working instructions to the acoustic sensor array. These instructions contain specific acoustic signal sensing mode information, such as sampling frequency and sensor orientation. The FPGA module and the ARM computing unit perform different tasks in the system. By properly allocating the processing logic, the two can work together to achieve effective control and data processing of the acoustic sensor array.

[0027] More specifically, by issuing working instructions to the acoustic sensor array, the optimized perception strategy can be transformed into actual operation, enabling the acoustic sensor array to operate in a specified mode and acquire the required acoustic signal data. The acoustic sensor array acquires the acoustic signal data of the detection target according to the working instructions and in the specified acoustic signal perception mode. The ARM computing unit deploys the corresponding heterogeneous data fusion algorithm for the newly acquired acoustic signal data, and updates the acoustic fusion imaging by combining it with the previously acquired image signal data and infrared signal data, resulting in optimized acoustic imaging. Updating the acoustic fusion imaging with the newly acquired acoustic signal data can integrate more accurate and comprehensive information into the imaging, improve the quality of acoustic fusion imaging, and provide a more reliable basis for fault analysis. By updating the acoustic fusion imaging multiple times, the fault detection results can be continuously optimized, and the system's ability to diagnose faults in the detected target can be gradually improved.

[0028] Specifically, in step S4 of the embodiment provided by the present invention, multiple imaging superposition processes are performed on each acoustic optimization imaging. A simple superposition method can be adopted here, that is, the corresponding positions of each acoustic optimization imaging are accumulated. If each acoustic optimization imaging is a two-dimensional image and the data represents acoustic-related features (such as sound pressure intensity), then the data of the same position can be added together. Alternatively, a weighted superposition method can be adopted, assigning different weights according to the credibility or importance of different acoustic optimization imaging, and then performing weighted summation. The imaging superposition process yields a primary superposition image. Superposition of multiple acoustic optimization imaging can enhance the characteristics of acoustic signals and make fault information more obvious, because different acoustic optimization imaging obtains fault-related information from different angles or under different conditions. After superposition, this information can be integrated to improve the identification of fault features. Noise may be introduced during the imaging process. Superposition can, to a certain extent, average out random noise, making the image smoother, which is beneficial for subsequent fault analysis.

[0029] More specifically, based on the primary superimposed images, the fault points and fault conditions of the detected target can be located and identified using image processing and machine learning methods. For example, threshold segmentation can be used to extract abnormal regions from the image, which correspond to fault points. For fault condition identification, a pre-trained classification model can be used, and the extracted fault region features can be input into the model to obtain the fault type (such as cracks, loosening, etc.). Accurate fault point location is the key to fault diagnosis. By processing the primary superimposed images, the specific location of the fault on the detected target can be determined, providing accurate information for subsequent maintenance or treatment. Identifying the fault condition can help understand the nature and severity of the fault, which is helpful in developing targeted solutions. For example, different treatment measures are needed for crack faults and loosening faults.

[0030] More specifically, interactive information verification is performed on the fault location and identification information of each primary overlay imaging. This can be achieved by comparing the location and fault type of the same fault point in different primary overlay imaging to determine whether they are consistent. If discrepancies exist, corrections can be made using methods such as majority voting or confidence-based methods. At the same time, the information correction form of each fault location and identification information is recorded, such as location correction or type correction. Different primary overlay imaging may have errors in fault location and type judgment due to various factors (such as sensor errors and environmental interference). Interactive information verification and correction can improve the accuracy and reliability of fault information. Recording the information correction form can provide a basis for subsequent analysis and improvement, making the entire fault diagnosis process more stable and repeatable.

[0031] More specifically, based on the information correction and overlay processing, each acoustic optimization image of the primary overlay imaging generates imaging content feedback labels. For example, if a certain acoustic optimization image plays a key role in fault location, a location-key label can be added to it; if it makes an important contribution to fault type judgment, a type-key label can be added. The imaging content feedback labels of each acoustic optimization image are comprehensively analyzed, and the number and weight of different labels of each acoustic optimization image are statistically analyzed to obtain the content feedback characteristics of each acoustic optimization image. For example, if the location-key label of a certain acoustic optimization image has a high weight, it indicates that it is more valuable in fault location. Different primary overlay imagings have errors in fault location and type judgment due to various factors (such as sensor error, environmental interference, etc.). Interactive information verification and correction can improve the accuracy and reliability of fault information. Recording the information correction form can provide a basis for subsequent analysis and improvement, making the entire fault diagnosis process more stable and repeatable.

[0032] More specifically, based on the content feedback characteristics, effective feedback content of each acoustic optimization image is screened and combined. According to the value of each acoustic optimization image in fault location and fault type judgment, the most representative and contributing parts are selected for combination. Image stitching, fusion and other methods can be used to combine the screened acoustic optimization images into fault diagnosis acoustic images. Screening and combining acoustic optimization images can integrate the effective feedback content of each image together, avoid the interference of redundant information, and generate a comprehensive, complete and accurate fault diagnosis acoustic image that reflects the fault status of the detected target. The final fault diagnosis acoustic image can provide maintenance personnel or relevant decision-makers with clear and intuitive fault information, which can facilitate them to make quick diagnosis and decisions.

[0033] This invention provides a real-time sound field imaging method based on heterogeneous computing of FPGA and ARM, which has the following beneficial effects: This invention rapidly acquires acoustic, image, and infrared multi-source signal data using an FPGA, ensuring timely and comprehensive data acquisition. Data fusion is then performed using an ARM processor to obtain acoustic fusion imaging, which comprehensively reflects the target's condition based on multiple types of information. The image is updated according to the fault information allocated in the fusion imaging, optimizing the imaging effect. Finally, fault analysis is performed based on the optimized imaging to obtain diagnostic imaging, accurately guiding the fault diagnosis of the detected target, improving the efficiency and accuracy of fault diagnosis, and providing reliable support for related fields.

[0034] Preferably, the steps of acquiring acoustic signal data from the acoustic sensor array based on the FPGA module, and simultaneously acquiring image signal data and infrared signal data through the DVP interface and USB interface include: S11: The FPGA module sends initial detection commands to the acoustic sensor array, optical sensor and infrared sensor to drive each sensor to collect data from the target. S12: Receives acoustic signal data acquired by the acoustic sensor array through 128 sampling channels, and simultaneously receives image signal data acquired by the optical sensor and infrared signal data acquired by the infrared sensor through the DVP interface and USB interface.

[0035] Preferably, the acoustic sensor array includes 128 acoustic sensors, each corresponding to one of the 128 sampling channels.

[0036] Specifically, the FPGA module is connected to the acoustic sensor array, optical sensor, and infrared sensor via specific control signal lines. When the system starts detection, the FPGA sends initial detection commands to each sensor according to pre-set logic and timing. These commands can be digital signals, such as high-level or low-level signals. Different signal combinations represent different command meanings, informing the sensors to begin data acquisition of the detection target. For the acoustic sensor array, the command triggers its internal acoustic signal conversion circuit to start working; for the optical sensor, it enables its image capture function; and for the infrared sensor, it initiates infrared signal detection. In practical applications, the FPGA outputs a specific sequence of control signals sequentially within one clock cycle, first sending a start signal to the acoustic sensor array, then sending corresponding start signals to the optical and infrared sensors, ensuring that each sensor starts data acquisition synchronously.

[0037] More specifically, to ensure that the three different types of data—acoustic, image, and infrared—accurately reflect the state of the target at the same moment, all sensors need to start data acquisition synchronously. The FPGA module, as the control core of the entire data acquisition system, can ensure that the acoustic sensor array, optical sensor, and infrared sensor start up simultaneously by issuing initial detection commands. This avoids data mismatch caused by asynchronous acquisition times, which would affect subsequent data analysis and processing. By issuing commands through the FPGA, the working status of the sensors can be easily controlled and managed. In different detection scenarios, the content and timing of the commands can be flexibly adjusted as needed to achieve personalized control of the sensors and improve the adaptability and reliability of the system.

[0038] More specifically, the 128 acoustic sensors in the acoustic sensor array convert the detected acoustic signals into electrical signals, which are then transmitted to the FPGA module through their respective 128 sampling channels. The sampling circuit inside the FPGA module samples and quantizes these electrical signals at a certain sampling frequency, converting them into digital signals. The selection of the sampling frequency is usually determined based on the frequency range of the acoustic signal and the detection accuracy requirements to ensure accurate capture of the acoustic signal characteristics. The image signal data acquired by the optical sensor is transmitted to the FPGA module through the DVP interface. The DVP interface has a specific communication protocol, and the FPGA module receives and parses the image data according to this protocol. During the reception process, the FPGA buffers and processes the data to ensure the integrity and accuracy of the image data.

[0039] More specifically, the infrared signal data acquired by the infrared sensor is transmitted to the FPGA module via a USB interface. The USB interface is a universal high-speed data transmission interface, and the FPGA module follows the USB interface communication protocol to receive and process the infrared signal data. Similarly, data verification and buffering are performed during reception to ensure data quality. When receiving acoustic signal data, the FPGA samples the signal of each sampling channel point-by-point according to the sampling clock, converting the sampled analog signals into digital values ​​and storing them in internal registers. For image and infrared signal data, the FPGA performs frame synchronization and error detection simultaneously upon receiving the data to ensure that the received data frames are complete and valid.

[0040] More specifically, using 128 sampling channels corresponding to 128 acoustic sensors enables high-resolution acquisition of acoustic signals. Each acoustic sensor is responsible for detecting acoustic signals at different positions or directions of the target. Data transmission through independent sampling channels avoids interference between signals, improving the accuracy and completeness of acoustic signal acquisition. This allows for a more comprehensive capture of the acoustic characteristics of the target, providing richer information for subsequent fault diagnosis. The DVP interface and USB interface were chosen to receive image signal data and infrared signal data respectively because these two interfaces have different characteristics and advantages, meeting the transmission needs of different types of data. The DVP interface is suitable for high-speed, low-cost image data transmission, ensuring fast and accurate transmission of image data; the USB interface has strong versatility and high transmission rate, making it suitable for infrared signal data transmission and facilitating connection and communication with various types of infrared sensors. By appropriately selecting the interface, the efficiency and stability of data transmission can be improved.

[0041] Preferably, the step of fusing acoustic signal data, image signal data, and infrared signal data acquired by the FPGA module based on the ARM computing unit to obtain acoustic fusion imaging includes: S21: The image analysis algorithm configured by the ARM computing unit interprets the content of the image signal data to identify the target classification information of the detected target; S22: Adjust the parameters of the heterogeneous data fusion algorithm configured in the ARM computing unit according to the target classification information to perform data fusion of acoustic signal data, image signal data and infrared signal data to obtain acoustic fusion imaging.

[0042] Specifically, the ARM computing unit first performs preprocessing operations on the image signal data acquired by the FPGA module. This includes converting the image to grayscale to reduce data volume and processing complexity; performing image filtering, such as using Gaussian filtering to remove noise and improve image clarity; and performing image enhancement operations, such as histogram equalization to enhance image contrast and make image features more prominent. Then, it employs feature extraction algorithms to extract representative features from the preprocessed image. Common feature extraction methods include SIFT (Scale Invariant Feature Transform) and SURF (Speed ​​Robust Feature Extraction). These algorithms can extract key points and feature descriptors from the image for subsequent target classification. The extracted features are then classified using pre-trained image classification models. These models can be machine learning-based classifiers, such as Support Vector Machines (SVM) and decision trees, or deep learning-based convolutional neural networks (CNNs), such as AlexNet and ResNet. The extracted features are input into a classification model, which outputs the category information of the detected target, such as whether it is a mechanical part, electronic device, or other type of object. In practical applications, the ARM computing unit first converts the acquired color image into a grayscale image, and then smooths the image using a Gaussian filter. Next, the SIFT algorithm is used to extract feature points and feature descriptors from the image. Finally, these features are input into a pre-trained CNN model, which classifies the target based on the features and outputs the category of the detected target.

[0043] More specifically, different types of detection targets possess different characteristics in acoustics, imaging, and infrared imaging. By identifying the classification information of the detection targets, we can understand their characteristics and provide a basis for targeted parameter adjustments to subsequent heterogeneous data fusion algorithms. For example, mechanical parts generate acoustic signals of specific frequencies during operation, while electronic devices generate significant infrared thermal radiation. Adjusting the parameters of the fusion algorithm based on the target classification information can make the fusion results more consistent with the actual situation of the targets, thereby improving the accuracy and reliability of the fusion.

[0044] More specifically, accurate target classification helps to more accurately identify the types of faults that may exist in the target. Different types of targets may have different fault modes. By classifying the target, we can combine the characteristics of the target with common fault modes to make more accurate fault diagnosis. For example, for mechanical parts, we pay more attention to abnormal vibration frequencies in acoustic signals; for electronic equipment, we pay more attention to overheated areas in infrared signals.

[0045] More specifically, based on the target classification information identified in S21, the parameters of the heterogeneous data fusion algorithm configured in the ARM computing unit are adjusted. Different types of detected targets have different acoustic, image, and infrared features, so the parameters of the fusion algorithm need to be adjusted according to the target classification information to improve the fusion effect. For example, if the detected target is a mechanical part, the weight of the acoustic signal in the fusion needs to be increased; if it is an electronic device, more attention needs to be paid to the infrared signal.

[0046] More specifically, a heterogeneous data fusion algorithm with adjusted parameters is used to fuse acoustic signal data, image signal data, and infrared signal data. Common data fusion methods include weighted average method, Kalman filter method, and neural network-based fusion method. During the fusion process, different types of data are associated and integrated. For example, the intensity distribution of acoustic signals is matched with the target area in the image, and the temperature distribution of infrared signals is matched with the heat-generating area in the image, ultimately obtaining acoustic fusion imaging. Assuming the target is a mechanical part, the ARM computing unit sets the weight of acoustic signal data to 0.6, the weight of image signal data to 0.2, and the weight of infrared signal data to 0.2. Then, the weighted average method is used to fuse the three types of data, and the acoustic signal, image signal, and infrared signal are weighted and summed according to their respective weights to obtain acoustic fusion imaging.

[0047] More specifically, acoustic signals, image signals, and infrared signals reflect the characteristics of the detected target from different perspectives. A single type of data cannot comprehensively and accurately describe the target's state. Data fusion combines the advantages of these three types of data, achieving complementary multimodal information. For example, image signals can provide information about the target's appearance and structure, acoustic signals can reflect the target's vibration and acoustic characteristics, and infrared signals can display the target's temperature distribution. Integrating this information provides a more comprehensive understanding of the detected target's state, improving fault detection and diagnosis capabilities. Acoustic fusion imaging combines different types of data into a unified whole, providing a more intuitive display of the target's comprehensive information. Compared to single data representations, acoustic fusion imaging offers richer and more accurate information, enabling operators to more clearly observe and analyze the target's state, thereby facilitating more effective fault diagnosis and decision-making.

[0048] Preferably, the heterogeneous data fusion algorithm includes the following data fusion steps for acoustic signal data, image signal data, and infrared signal data: S221: Based on the image signal data, establish an original digital model of the target to be detected, and divide several key recognition regions on the original digital model; S222: Combine the acoustic signal data and the infrared signal data to perform matching processing on each key identification region on the original digital model, and obtain the data mapping relationship between the acoustic signal data and the infrared signal data relative to the original digital model; S223: Based on the data mapping relationship, the acoustic signal data and the infrared signal data are decomposed into data units and mapped to generate acoustic feedback units and infrared feedback units at various locations on the original digital model, thereby obtaining acoustic fusion imaging.

[0049] Specifically, the ARM computing unit uses image processing technology to construct the original digital model of the detected target based on image signal data. For two-dimensional images, edge detection, contour extraction and other methods can be used to obtain the outline of the target and then construct a two-dimensional digital model. If the image has certain depth information or uses multi-view images, a three-dimensional digital model can also be constructed. For example, the Canny edge detection algorithm is used to extract the edge of the target in the image, and then the contour tracking algorithm is used to determine the outline of the target, thereby constructing a two-dimensional shape model.

[0050] More specifically, based on the structural characteristics, functional features, and potential fault locations of the target object, several key recognition regions are divided on the original digital model. For example, for mechanical parts, key recognition regions are connection points and easily worn areas; for electronic devices, they are heating elements and interface areas. Rules can be set manually based on experience, or automatic division can be performed using machine learning algorithms. For instance, clustering algorithms can be used to group regions with similar features in an image into the same category, and then key recognition regions are determined based on the category. Taking a mechanical gear as an example, the ARM computing unit first performs Canny edge detection on the acquired gear image to obtain the gear's edge contour, constructs a two-dimensional digital model, and then, based on the gear's structure, divides the tooth surface, tooth root, and other parts into key recognition regions.

[0051] More specifically, the original digital model provides a unified spatial framework for the fusion of acoustic signal data and infrared signal data. By establishing a digital model, different types of data can be correlated and processed in the same coordinate system, avoiding the problem of mismatch in the spatial location of data. Dividing key identification areas can focus attention on the areas of the detection target most likely to have faults or require special attention. In this way, unnecessary data processing can be reduced in the subsequent data fusion and analysis process, and the efficiency and accuracy of fault detection can be improved.

[0052] More specifically, features related to the key identification area are extracted from acoustic signal data and infrared signal data respectively. For acoustic signals, features such as frequency, amplitude, and phase can be extracted; for infrared signals, features such as temperature distribution and hotspot locations can be extracted. The extracted acoustic and infrared features are then matched with the key identification area on the original digital model. Feature matching algorithms, such as template matching and feature point matching, can be used. For example, the frequency features of the acoustic signal are compared with the expected frequency range of the corresponding area on the digital model, and the hotspot locations of the infrared signal are matched with areas on the digital model that may generate heat.

[0053] More specifically, by matching the results, the data mapping relationship between acoustic signal data and infrared signal data relative to the original digital model is determined, that is, which features in the acoustic signal and infrared signal correspond to which positions on the digital model. For the aforementioned mechanical gear, vibration features of specific frequencies are extracted from the acoustic signal, and temperature distribution features of the tooth surface are extracted from the infrared signal. These features are matched with key identification areas such as the tooth surface and tooth root on the digital model to determine the correspondence between the acoustic signal and infrared signal and each area of ​​the digital model.

[0054] More specifically, through matching processing, acoustic signal data and infrared signal data can be correlated with key identification areas on the original digital model. This data mapping relationship enables different types of data to be spatially correlated, providing a basis for subsequent information integration and analysis. Accurate data mapping relationship can ensure that acoustic and infrared information can be accurately reflected in the corresponding positions on the digital model, avoiding information misalignment and confusion, thereby improving the accuracy and reliability of acoustic fusion imaging.

[0055] More specifically, based on the data mapping relationship, the acoustic signal data and infrared signal data are decomposed, and the acoustic signal and infrared signal are divided according to the digital model, so that each basic unit corresponds to certain acoustic and infrared information. The decomposed acoustic and infrared data units are mapped to the corresponding positions in the original digital model. For example, the intensity value of the acoustic signal is mapped to the position of the corresponding area on the digital model, and the temperature value of the infrared signal is also mapped to the corresponding position.

[0056] More specifically, acoustic feedback units and infrared feedback units are generated at various locations on the original digital model. The acoustic feedback units can use visual elements such as color and brightness to represent information such as the intensity and frequency of the acoustic signal; the infrared feedback units can use different colors to represent the temperature. Finally, the acoustic and infrared feedback units are integrated into the original digital model to obtain acoustic fusion imaging. On the digital model of the mechanical gear, the intensity value of the acoustic signal is converted into the brightness of the color, and the temperature value of the infrared signal is converted into the hue of the color. The corresponding acoustic and infrared information is displayed at each location of the digital model to form acoustic fusion imaging.

[0057] More specifically, data decomposition and information mapping can accurately overlay acoustic and infrared information onto the original digital model, achieving more refined information fusion. This allows for a more detailed display of the acoustic and infrared characteristics of the target being detected, providing richer information for fault diagnosis. The generated acoustic fusion imaging presents acoustic and infrared information in a visual manner, enabling operators to intuitively observe the state of the target. Through the display of acoustic and infrared feedback units, the acoustic characteristics and temperature distribution of the target can be understood more clearly, facilitating the rapid detection of potential faults.

[0058] Preferably, in the process of matching key identification regions on the original digital model by combining the acoustic signal data and the infrared signal data, the method further includes reproducing the digital shape of the detected target by combining the acoustic signal data and the infrared signal data, so as to correct the structural details of the original digital model.

[0059] Specifically, acoustic signal data is analyzed in both the time and frequency domains. In the time domain, characteristics such as amplitude, energy, and duration of the acoustic signal are calculated. In the frequency domain, the signal spectrum is obtained through methods such as Fourier transform, and features such as main frequency components and frequency bandwidth are extracted. For example, for acoustic signals generated by mechanical vibration, the dominant frequency of the vibration can be determined through spectrum analysis, thereby inferring possible vibration sources. Multi-resolution analysis methods such as wavelet transform can also be used to extract acoustic signal features at different scales to capture the local characteristics of the signal. Temperature distribution features, such as average temperature, maximum temperature, minimum temperature, and temperature gradient, can be extracted from infrared signal data. By analyzing the temperature distribution, the heating status of the target can be understood, hot spots and cold spots in the infrared image can be identified, and their location, size, and shape can be recorded. Hot spots correspond to fault points or high-energy-consuming areas inside the target.

[0060] More specifically, the original digital model has inaccuracies in structural details due to modeling errors and actual changes in the target being detected. By digitally reproducing the shape of acoustic and infrared signals and correcting the original digital model, the actual state of the target being detected can be reflected more accurately. For example, some internal micro-cracks are difficult to detect in the original image, but can be detected through the abnormal propagation of acoustic signals and local hot spots in infrared signals. Integrating this information into the digital model can improve the ability to detect faults such as cracks.

[0061] More specifically, the extracted acoustic and infrared features are matched with features in a pre-established standard model or database. The standard model can be the acoustic and infrared features of the target under normal operating conditions, and the database can store feature patterns corresponding to different fault types. Based on the matching results, the digital morphology of the target in acoustic and infrared aspects is reconstructed. For example, if the spectral features of the acoustic signal have a high degree of matching with the spectral features of a certain fault mode, the acoustic morphology is reconstructed according to the acoustic propagation model of that fault mode. For infrared signals, the heat conduction model of the target is reconstructed based on the location and temperature distribution of hotspot areas, thereby obtaining the infrared digital morphology.

[0062] More specifically, by employing a fusion modeling approach, acoustic and infrared features are combined for digital shape reconstruction. Machine learning algorithms, such as neural networks, can be used to train a model that uses acoustic and infrared features as input to predict the three-dimensional digital shape of the target. During the reconstruction process, the relationship between acoustic and infrared signals is considered. For example, high temperatures in certain areas can cause changes in acoustic signals. By establishing a coupling model between the two, the digital shape can be more accurately reconstructed.

[0063] More specifically, acoustic and infrared signals reflect the characteristics of the detected target from different perspectives. By digitally reproducing their shapes and correcting the original digital model, deep fusion of multi-source data at the model level is achieved. This fusion can more comprehensively integrate acoustic and infrared information, making the final acoustic fusion imaging more accurate and reliable, and providing richer and more accurate evidence for subsequent fault analysis.

[0064] More specifically, the reproduced digital morphology is compared and analyzed with the original digital model to check the differences between the two in terms of structural shape, size, feature distribution, etc. For example, observe whether there are abnormal acoustic propagation paths in the reproduced acoustic morphology, and whether there are additional hot spots in the infrared morphology, which are not reflected in the original digital model.

[0065] More specifically, based on the results of the comparative analysis, the structural details of the original digital model are corrected. If the size of a certain part in the reproduced digital morphology is found to be different from that of the original digital model, the size of that part is adjusted. If a new feature region is found, the feature region is added to the corresponding position in the original digital model. The correction process can use a combination of manual and automatic adjustment. For some obvious differences, manual adjustments can be made precisely. For some minor differences, algorithms can be used to automatically correct them, thereby improving the efficiency and accuracy of the correction.

[0066] More specifically, the state of the target object changes during operation, such as temperature rise and mechanical wear. By performing digital morphology reproduction and model correction on acoustic and infrared signals in real time, the digital model can reflect the dynamic changes of the target object in a timely manner. In this way, the accurate description of the target object can be maintained under different working conditions, improving the adaptability and stability of the fault diagnosis system.

[0067] Preferably, the steps of allocating processing logic to the FPGA module and ARM computing unit based on the fault detection information fed back by the acoustic fusion imaging to update the acoustic fusion imaging and obtain several acoustically optimized images include: S31: Perform fault detection on the acoustic fusion imaging and annotate the fault detection results on the acoustic fusion imaging to generate fault detection information; S32: Evaluate the completeness of the feedback of the fault detection information, and perform value analysis on various optimization methods of the acoustic signal sensing mode of the acoustic sensor array based on the evaluation results, so as to determine several feasible optimization methods, and optimize the acoustic signal sensing mode according to the optimization methods to obtain an optimized sensing strategy. S33: Based on the optimized perception strategies described above, the corresponding processing logic is allocated to the FPGA module and the ARM computing unit in sequence to send working instructions to the acoustic sensor array. The working instructions drive the acoustic sensor array to acquire the acoustic signal data of the detection target in the specified acoustic signal perception mode. S34: The ARM computing unit deploys a corresponding heterogeneous data fusion algorithm for the newly acquired acoustic signal data to update the acoustic fusion imaging based on the acoustic signal data, thereby obtaining acoustically optimized imaging.

[0068] Specifically, acoustic fusion imaging is analyzed using preset fault feature templates, machine learning classification algorithms, or deep learning models. For example, convolutional neural networks (CNNs) are used to extract and classify features in acoustic fusion imaging to determine whether a fault exists and its type. Threshold detection methods can also be used to compare certain parameters in acoustic fusion imaging (such as acoustic intensity, temperature, etc.) with preset thresholds. If the threshold is exceeded, a fault is determined to exist. Fault detection results are annotated on the acoustic fusion imaging. Different colored markers, symbols, or text can be used to describe the location, type, and other information of the fault. For example, a red circle is used to mark the fault location, and the fault type is labeled next to it, such as crack or looseness. For acoustic fusion imaging of a mechanical part, a trained CNN model is used to classify it. The model output determines that the part has a looseness fault, and then a yellow triangle is used to mark the loose part on the image, with "loose" labeled next to it.

[0069] More specifically, the completeness of feedback is evaluated by calculating indicators such as the coverage and accuracy of fault detection information. For example, by calculating the proportion of the detected fault area to the entire detection target area, and the accuracy of fault type identification, a comprehensive evaluation index can be set, such as feedback completeness = 0.6 × coverage + 0.4 × accuracy.

[0070] More specifically, based on the evaluation results, a value analysis is conducted on various optimization methods for the acoustic signal sensing mode of the acoustic sensor array. Considering multiple forms of optimization based on the existing acoustic signal sensing mode, several optimization methods are obtained. Each optimization method can yield different optimized sensing modes (such as changing the sampling frequency, adjusting the direction or position of the sensors, etc.). Through these optimization methods, the potential value of obtaining more comprehensive fault information can be realized. For example, if the feedback completeness is low, it is analyzed whether increasing the sampling frequency of certain key areas or adjusting the orientation of some sensors can obtain more fault information.

[0071] More specifically, based on the results of the optimization value analysis, several optimization perception strategies are generated. For example, if it is found that the fault information detection in a certain area is incomplete, a strategy can be formulated to increase the sampling frequency of the acoustic sensor corresponding to that area; if it is suspected that the acoustic signals in certain directions are not fully captured, a strategy can be formulated to adjust the direction of some sensors. The calculated feedback completeness is 0.4. The analysis found that the fault information detection in a certain key area is insufficient. The generated optimization perception strategy is to double the sampling frequency of the acoustic sensor corresponding to that area.

[0072] More specifically, feedback integrity assessment can identify deficiencies in the current detection process. By generating optimized perception strategies, the operating mode of the acoustic sensor array can be adjusted to obtain more comprehensive and accurate acoustic signal data, thereby improving the fault detection effect. Different detection targets and fault conditions require different perception modes. Dynamically generating optimized perception strategies based on the assessment results can enable the system to have adaptive capabilities and better adapt to various complex situations.

[0073] More specifically, based on various optimized sensing strategies, corresponding processing logic is allocated to the FPGA module and the ARM computing unit. For the FPGA module, its control signals need to be reconfigured to change the working mode of the acoustic sensor array, such as adjusting the sampling frequency and controlling the direction of the sensors. For the ARM computing unit, the data processing algorithm or parameters need to be adjusted to adapt to the new acoustic signal data. The FPGA module sends working instructions to the acoustic sensor array, which contain specific acoustic signal sensing mode information, such as sampling frequency and sensor direction. The FPGA controls the acoustic sensor array to work in the specified mode according to the instructions. According to the optimized sensing strategy, the FPGA module configures the acoustic sensor sampling frequency of the corresponding key area to twice the original value and sends working instructions containing the new sampling frequency information to the acoustic sensor array.

[0074] More specifically, the FPGA module and the ARM computing unit undertake different tasks in the system. Reasonable allocation of processing logic can enable the two to work together to achieve effective control and data processing of the acoustic sensor array. By issuing working instructions to the acoustic sensor array, the optimized sensing strategy can be transformed into actual operation, so that the acoustic sensor array works in the specified mode and acquires the required acoustic signal data.

[0075] More specifically, the acoustic sensor array acquires acoustic signal data of the target in a specified acoustic signal sensing mode according to the working instructions. The ARM computing unit deploys a corresponding heterogeneous data fusion algorithm for the newly acquired acoustic signal data, and fuses the new acoustic signal data with image signal data and infrared signal data based on the previous target classification information and the adjusted parameters.

[0076] More specifically, the acoustic fusion imaging is updated based on the fusion results to obtain acoustically optimized imaging. This can be achieved by using the same method as the previous data fusion, superimposing new data information onto the original acoustic fusion imaging, or by generating a new acoustic fusion imaging. The acoustic sensor array acquires acoustic signal data at a new sampling frequency, and the ARM computing unit uses an adjusted heterogeneous data fusion algorithm to fuse the new acoustic signals with the image and infrared signals, updating the original acoustic fusion imaging to obtain acoustically optimized imaging.

[0077] More specifically, updating acoustic fusion imaging with newly acquired acoustic signal data can incorporate more accurate and comprehensive information into the imaging, improve the quality of acoustic fusion imaging, provide a more reliable basis for fault analysis, and continuously optimize fault detection results by updating acoustic fusion imaging multiple times, thereby gradually improving the system's ability to diagnose faults in the detected target.

[0078] Preferably, the step of performing fault analysis on the detected target based on each of the acoustic optimization images to guide the combination of the acoustic optimization images into fault diagnosis acoustic imaging includes: S41: Based on each of the acoustic optimization imaging methods, the detection target is subjected to multiple imaging superposition processes to obtain a primary superposition imaging corresponding to several acoustic optimization imaging combinations. S42: Based on each of the primary superimposed imaging methods, the fault points of the detected target are located and the fault conditions are identified to obtain several fault location and identification information of the detected target. S43: Perform interactive information verification on the fault location identification information of each of the primary superimposed imaging to correct each fault location identification information, and record the information correction form of each fault location identification information. S44: Based on the information correction form of the fault location identification information, analyze the imaging content feedback tendency of each acoustic optimization image that makes up the primary superimposed imaging, so as to generate the corresponding content feedback label of each acoustic optimization image, and comprehensively analyze the imaging content feedback labels of each acoustic optimization image to obtain the content feedback characteristics of each acoustic optimization image. S45: Based on the content feedback features, the effective feedback content of each acoustic optimization image is screened and combined to generate a fault diagnosis acoustic image.

[0079] Specifically, multiple acoustic optimization images are directly superimposed. Superimposing multiple acoustic optimization images can enhance the characteristics of acoustic signals and make fault information more obvious. Different acoustic optimization images obtain fault-related information from different angles or under different conditions. After superposition, this information can be integrated to improve the identification of fault features. Noise is introduced during the imaging process. Superposition can average out random noise to a certain extent, making the image smoother, which is beneficial for subsequent fault analysis.

[0080] More specifically, image processing and pattern recognition methods are used to analyze the primary overlay imaging. For example, threshold segmentation is used to extract abnormal regions in the image, which correspond to fault points. Edge detection and morphological operations can also be used to find the boundaries of the fault regions, thereby determining the location of the fault points. A pre-trained classification model is used to classify the fault regions and identify the type of fault, such as cracks, loosening, or wear. The classification model can be a machine learning-based algorithm, such as support vector machines or decision trees, or a deep learning model, such as a convolutional neural network. The image features of the fault regions are input into the classification model to obtain the fault type. Threshold segmentation is performed on the primary overlay imaging, and regions with fault feedback values ​​greater than a certain threshold are extracted as suspected fault regions. Then, the image features of these fault regions are input into the trained convolutional neural network to determine the fault type as cracks.

[0081] More specifically, accurate fault location is key to fault diagnosis. By processing primary superimposed images, the specific location of the fault on the detection target can be determined, providing accurate information for subsequent maintenance or treatment. Identifying the fault condition can help understand the nature and severity of the fault, which is helpful in developing targeted solutions. For example, different treatment measures are needed for crack faults and loosening faults.

[0082] More specifically, the fault location and identification information of each primary overlay imaging is interactively compared and verified to check whether the location and fault type judgment of the same fault point are consistent in different primary overlay imaging. For example, the location coordinates and fault type judgment of a certain fault point in three primary overlay imaging are compared. If there are differences, they can be corrected by majority voting, confidence-based methods, etc. For example, for the judgment of fault type, if two of the three primary overlay imaging judge it as crack and one judges it as loose, the fault type is corrected to crack. Different primary overlay imaging may have errors in fault location and type judgment due to various factors (such as sensor error, environmental interference, etc.). Through interactive information verification and correction, the accuracy and reliability of fault information can be improved.

[0083] More specifically, the corrected fault location identification information has more accurate feedback content compared to the original fault location information. Since the fault location identification information corresponds to the primary overlay imaging, and the primary overlay imaging is composed of multiple acoustic optimization imaging, it can be seen that the tendency of the feedback content of the corrected fault location identification information is composed of each acoustic optimization imaging. On the other hand, the same acoustic optimization imaging participates in the construction of multiple primary overlay imaging. Therefore, the processing method in this step is to analyze the content feedback tendency of each acoustic optimization imaging that participates in its construction through the primary overlay imaging, to obtain the various possibilities of the content feedback tendency of the acoustic optimization imaging, that is, the content feedback label of the acoustic optimization imaging corresponding to the primary overlay imaging. Then, the various possibilities of the content feedback tendency of the acoustic optimization imaging corresponding to each primary overlay imaging are statistically analyzed to obtain the accurate content feedback tendency, that is, the content feedback feature.

[0084] More specifically, if an acoustic optimization imaging plays a key role in fault location, a location-key label is added to it; if it makes a significant contribution to fault type determination, a type-key label is added. A comprehensive analysis is performed on the various imaging content feedback labels possessed by each acoustic optimization imaging, and the number and weight of different labels for each acoustic optimization imaging are counted to obtain the content feedback characteristics of each acoustic optimization imaging. For example, if an acoustic optimization imaging has two location-key labels and one type-key label, its comprehensive score in location location and type determination is calculated based on the label weights.

[0085] More specifically, the imaging content feedback labels and content feedback features can intuitively reflect the role and contribution of each acoustic optimization imaging in fault analysis, helping to determine which imaging is more valuable in fault location and fault type identification. By analyzing the content feedback features, guidance can be provided for subsequent acoustic optimization imaging screening and combination, which helps to select the most effective imaging information to generate fault diagnosis acoustic imaging.

[0086] More specifically, based on content feedback characteristics, various acoustic optimization images are screened, and those that make a high contribution to fault location and fault type judgment are selected. The screened acoustic optimization images are then combined using methods such as image stitching and fusion to form a fault diagnosis acoustic image. For example, key fault-related areas from different acoustic optimization images can be stitched together, or they can be fused together using a weighted fusion method. Three acoustic optimization images are selected based on content feedback characteristics, and they are fused into a new image using a weighted fusion method to obtain the fault diagnosis acoustic image. Screening and combining acoustic optimization images can integrate the effective feedback content from each image, avoid interference from redundant information, and generate a comprehensive, complete, and accurate fault diagnosis acoustic image that reflects the fault status of the detected target. The final fault diagnosis acoustic image can provide maintenance personnel or relevant decision-makers with clear and intuitive fault information, facilitating their rapid diagnosis and decision-making.

[0087] Reference Figure 2 As shown, in a second aspect, the present invention provides a real-time sound field imaging system based on FPGA and ARM heterogeneous computing, used to implement the real-time sound field imaging method based on FPGA and ARM heterogeneous computing as described in any one of the first aspects, comprising: The signal acquisition module is used to acquire acoustic signal data obtained by the acoustic sensor array based on the FPGA module, and simultaneously acquire image signal data and infrared signal data through the DVP interface and USB interface; The data fusion module is used to fuse acoustic signal data, image signal data and infrared signal data acquired by the FPGA module based on the ARM computing unit to obtain acoustic fusion imaging; An imaging optimization module is used to allocate processing logic to the FPGA module and ARM computing unit based on the fault detection information fed back by the acoustic fusion imaging, so as to update the acoustic fusion imaging and obtain several acoustic optimization images. An imaging integration module is used to perform fault analysis on the detected target based on each of the acoustic optimization imaging methods, so as to guide the integration of each of the acoustic optimization imaging methods into fault diagnosis acoustic imaging.

[0088] In this embodiment, the specific implementation of each module in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time sound field imaging method based on FPGA and ARM heterogeneous computing, characterized in that, include: Acoustic signal data acquired by the acoustic sensor array is collected using an FPGA module, and image signal data and infrared signal data are simultaneously acquired through the DVP interface and USB interface. The acoustic signal data, image signal data and infrared signal data acquired by the FPGA module are fused using the ARM computing unit to obtain acoustic fusion imaging; Based on the fault detection information fed back by the acoustic fusion imaging, the FPGA module and ARM computing unit are allocated processing logic to update the acoustic fusion imaging and obtain several acoustic optimization images. Based on the aforementioned acoustic optimization imaging, fault analysis is performed on the detected target to guide the combination of the aforementioned acoustic optimization imaging into fault diagnosis acoustic imaging.

2. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 1, wherein, The steps for acquiring acoustic signal data from an acoustic sensor array using an FPGA module, and simultaneously acquiring image and infrared signal data via DVP and USB interfaces, include: The FPGA module sends initial detection commands to the acoustic sensor array, optical sensor, and infrared sensor to drive each sensor to collect data from the target. The system receives acoustic signal data from the acoustic sensor array via 128 sampling channels, and simultaneously receives image signal data from the optical sensor and infrared signal data from the infrared sensor via the DVP interface and USB interface.

3. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 1, wherein, The acoustic sensor array contains 128 acoustic sensors, each corresponding to one of the 128 sampling channels.

4. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 1, wherein, The steps for fusing acoustic signal data, image signal data, and infrared signal data acquired by the FPGA module using the ARM computing unit to obtain acoustic fusion imaging include: The image parsing algorithm configured on the ARM computing unit is used to interpret the content of the image signal data in order to identify the target classification information of the detected target; Based on the target classification information, the parameters of the heterogeneous data fusion algorithm configured in the ARM computing unit are adjusted to fuse acoustic signal data, image signal data, and infrared signal data to obtain acoustic fusion imaging.

5. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 4, wherein, The heterogeneous data fusion algorithm includes the following data fusion steps for acoustic signal data, image signal data, and infrared signal data: Based on the image signal data, an original digital model of the target is established, and several key recognition regions are divided on the original digital model. By combining the acoustic signal data and the infrared signal data, the key identification regions on the original digital model are matched to obtain the data mapping relationship between the acoustic signal data and the infrared signal data relative to the original digital model; Based on the data mapping relationship, the acoustic signal data and the infrared signal data are decomposed into data units and mapped to generate acoustic feedback units and infrared feedback units at various locations on the original digital model, thereby obtaining acoustic fusion imaging.

6. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 5, wherein, In the process of matching key identification regions on the original digital model by combining the acoustic signal data and the infrared signal data, the process also includes digital morphology reconstruction of the detected target by combining the acoustic signal data and the infrared signal data, so as to correct the structural details of the original digital model.

7. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing according to claim 1, wherein, The steps of allocating processing logic to the FPGA module and ARM computing unit based on the fault detection information fed back by the acoustic fusion imaging to update the acoustic fusion imaging and obtain several acoustically optimized imaging include: Fault detection is performed on the acoustic fusion imaging, and the fault detection results are annotated on the acoustic fusion imaging to generate fault detection information; The integrity of the feedback of the fault detection information is evaluated, and the value analysis of various optimization methods of the acoustic signal sensing mode of the acoustic sensor array is performed based on the evaluation results to determine several feasible optimization methods. The acoustic signal sensing mode is then optimized based on the optimization methods to obtain an optimized sensing strategy. Based on the optimized perception strategies described above, corresponding processing logic is allocated to the FPGA module and ARM computing unit in sequence to issue working instructions to the acoustic sensor array. The working instructions drive the acoustic sensor array to acquire acoustic signal data of the detection target in a specified acoustic signal perception mode. The ARM computing unit deploys a corresponding heterogeneous data fusion algorithm for the newly acquired acoustic signal data to update the acoustic fusion imaging based on the acoustic signal data, thereby obtaining acoustically optimized imaging.

8. The real-time sound field imaging method based on FPGA and ARM heterogeneous computing as described in claim 1, characterized in that, The steps for performing fault analysis on the detected target based on the aforementioned acoustic optimization imaging, so as to guide the combination of the aforementioned acoustic optimization imaging into fault diagnosis acoustic imaging, include: Based on the acoustic optimization imaging described above, the detection target is subjected to multiple imaging superposition processes to obtain a primary superposition imaging corresponding to several acoustic optimization imaging combinations. Based on the primary superimposed imaging, the fault points of the detected target are located and the fault conditions are identified, so as to obtain several fault location and identification information of the detected target. Interactive information verification is performed on the fault location and identification information of each of the primary superimposed imaging methods to correct each fault location and identification information. Based on the corrected fault location identification information, the imaging content feedback tendency of each acoustic optimization image that makes up the primary superimposed imaging is analyzed to generate corresponding content feedback tags for each acoustic optimization image, and the imaging content feedback tags of each acoustic optimization image are comprehensively analyzed to obtain the content feedback characteristics of each acoustic optimization image. Based on the aforementioned content feedback features, effective feedback content is selected and combined for each acoustic optimization imaging to generate fault diagnosis acoustic imaging.

9. A real-time sound field imaging system based on FPGA and ARM heterogeneous computing, characterized in that, A real-time sound field imaging method based on FPGA and ARM heterogeneous computing as described in any one of claims 1-8 includes: The signal acquisition module is used to acquire acoustic signal data obtained by the acoustic sensor array based on the FPGA module, and simultaneously acquire image signal data and infrared signal data through the DVP interface and USB interface; The data fusion module is used to fuse acoustic signal data, image signal data and infrared signal data acquired by the FPGA module based on the ARM computing unit to obtain acoustic fusion imaging; An imaging optimization module is used to allocate processing logic to the FPGA module and ARM computing unit based on the fault detection information fed back by the acoustic fusion imaging, so as to update the acoustic fusion imaging and obtain several acoustic optimization images. An imaging integration module is used to perform fault analysis on the detected target based on each of the acoustic optimization imaging methods, so as to guide the integration of each of the acoustic optimization imaging methods into fault diagnosis acoustic imaging.

Images