Multi-modal pipeline inspection data acquisition system and method based on FPGA architecture
The multimodal pipeline inspection system based on FPGA architecture solves the problems of inconsistent ultrasonic echo acquisition and image processing delay, and realizes high-precision real-time acquisition and synchronous transmission. It is suitable for the inspection of metal or non-metal pipelines in industries such as petroleum, natural gas and chemical.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-29
AI Technical Summary
Existing pipeline inspection systems suffer from a lack of precise timing control in ultrasonic echo acquisition, inconsistent phases across multiple channels, significant image processing delays, absence of hardware-level synchronization mechanisms for multimodal data, and insufficient communication interface bandwidth, making it difficult to meet the demands for high-precision real-time acquisition and large-volume data transmission.
Employing an FPGA architecture, independent ultrasonic and image acquisition modules are designed. Time alignment is achieved through external synchronization signals. Combined with SPI and fiber optic heterogeneous communication, it supports high-precision timing control and high-throughput image processing, realizes hardware-level synchronous triggering, and outputs raw data streams.
It achieves high-precision real-time acquisition and synchronous transmission of multimodal data, supports real-time image processing at 1080p@30fps, reduces the burden on the main control device, has strong compatibility, and is suitable for various main control platforms.
Smart Images
Figure CN122109306A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of industrial non-destructive testing and embedded system technology, and in particular relates to a multimodal pipeline inspection data acquisition system and method based on FPGA architecture. It is applicable to the synchronous acquisition, buffering and reliable transmission of raw ultrasonic and visual image data required during the inspection of metal or non-metal pipelines in industries such as petroleum, natural gas and chemical. Background Technology
[0002] In existing pipeline inspection systems, a single sensor paired with a general-purpose processor is typically used to complete data acquisition tasks. However, with increasing requirements for detection accuracy, traditional solutions face the following problems: First, ultrasonic echo acquisition lacks precise timing control, resulting in phase inconsistencies among multiple channels; second, image processing latency is large, making it difficult to meet the real-time acquisition requirements of 1080p@30fps; third, there is no hardware-level synchronization mechanism between different modal data, leading to difficulties in subsequent registration; and fourth, insufficient communication interface bandwidth limits the effective transmission of large amounts of images.
[0003] Although some systems have attempted to improve performance using FPGAs, they generally have not achieved functional decoupling design and still suffer from problems such as resource contention and processing bottlenecks.
[0004] Therefore, there is an urgent need for a dedicated independent subsystem for multimodal data acquisition, which has high-precision acquisition capabilities, strong real-time performance and heterogeneous communication support, to provide high-quality raw data streams for upper-level analysis. Summary of the Invention
[0005] The purpose of this invention is to address the problems mentioned in the background art by proposing a multimodal pipeline detection data acquisition system and method based on FPGA architecture. This system aims to solve problems such as asynchronous ultrasonic excitation and echo acquisition, large delay in image acquisition and processing, lack of hardware-level time alignment mechanism for multimodal data, and the inability of a single communication path to meet the demands of both small data volume and high frequency and large data volume and high speed.
[0006] To achieve the objectives of this invention, a multimodal pipeline inspection data acquisition system based on an FPGA architecture is disclosed, comprising an ultrasonic acquisition module and an image acquisition and processing module. The ultrasonic acquisition module generates eight 40kHz pulse excitation signals to produce ultrasonic signals, controls the AD9226 to acquire echo data, and outputs raw sampled data through an SPI interface. The image acquisition and processing module acquires image data output by an OV5640 sensor, performs distortion correction and edge enhancement preprocessing, transmits image frames through an optical fiber interface, writes them to a DDR3 buffer, and finally displays them on an HDMI display. The ultrasonic acquisition module and the image acquisition and processing module achieve time alignment of the acquisition actions through an external synchronization signal.
[0007] Furthermore, the ultrasonic acquisition module integrates the ODDR2 primitive to generate a symmetrical sampling clock for the AD9226 and synchronously triggers the pulse excitation and AD9226 sampling window through the same enable signal.
[0008] Furthermore, the SPI interface communication rate reaches over 800kHz, and the output data packet includes a channel identifier and a timestamp; the system also includes a status feedback mechanism, which integrates the AD9226 analog-to-digital converter sampling completion signal and the SPI transmission completion signal to generate a high-level pulse as a "complete acquisition transaction of this round" flag.
[0009] Furthermore, the image acquisition and processing module also includes grayscale transformation, filtering, Sobel edge detection, and morphological manipulation units; the image sensor configuration is capable of capturing clear images even in poor lighting conditions.
[0010] Furthermore, the image acquisition and processing module also includes a morphological processing unit. The morphological processing unit adopts a 3×3 square structural element and performs dilation and erosion operations in sequence. The dilation operation is used to fill small holes in the image and connect broken edges, while the erosion operation is used to eliminate isolated noise and burrs in the image.
[0011] Furthermore, the morphological processing unit is implemented using a three-stage pipeline architecture, including: a first-stage multi-line buffer unit for simultaneously buffering multiple lines of image data; a second-stage bitwise operation unit for performing bitwise logical operations on the buffered image data; and a third-stage synchronous output unit for outputting the morphologically processed image data. Among these, the dilation operation is implemented by performing a bitwise OR operation in the second-stage bitwise operation unit, and the erosion operation is implemented by performing a bitwise AND operation in the second-stage bitwise operation unit.
[0012] Furthermore, the image acquisition and processing module deploys a Sobel edge detection IP core with a four-stage pipeline structure, with a resource utilization rate of less than 5% LUT; the DDR3 cache capacity is no less than 128MB, and it supports dual-frame ping-pong caching.
[0013] Furthermore, the fiber optic interface between the image acquisition and processing module and the receiving device adopts a custom frame protocol, including field synchronization markers, data segment start / end markers, and padding codes; the fiber optic interface uses 8B / 10B encoding, with a transmission rate of 3.125Gbps; and the image frame format is RGB565.
[0014] Furthermore, the ultrasonic acquisition module generates two clocks, 50MHz and 12.5MHz, through a PLL phase-locked loop, which are used for pulse generation and AD sampling, respectively; it supports online updating of the FPGA bitstream file via JTAG or SD card, enabling dynamic upgrading of acquisition parameters and preprocessing algorithms.
[0015] A multimodal pipeline detection data acquisition method based on FPGA architecture includes the following steps:
[0016] S1. Initialize each FPGA module and clock source;
[0017] S2. Upon receiving an external start command, ultrasonic pulse emission and image frame acquisition are triggered simultaneously.
[0018] S3. Control the AD9226 analog-to-digital converter to acquire echo signals. The raw data is cached in RAM and then uploaded in batches through the SPI interface.
[0019] S4. Preprocess the image data, and then send the image frames through the fiber optic interface.
[0020] S5: The external FPGA receives and buffers image frames, completes decoding, and outputs the image to the HDMI display.
[0021] S6. Output a "transaction completed" signal to indicate that the current round of data acquisition has been completed. The "transaction completed" signal is output in the form of a rising edge and is used for time base synchronization.
[0022] S7. Repeat S2 to S6 to continuously output the raw data stream.
[0023] Compared with existing technologies, the significant advancements of this invention are: 1) It adopts an FPGA architecture to undertake high-precision timing control and high-throughput image processing tasks, achieving multimodal acquisition requirements; 2) It uses SPI + fiber optic heterogeneous communication to balance the low latency requirements of small ultrasonic data volume and the high bandwidth requirements of large image data volume; 3) It pioneers a hardware-level synchronous triggering mechanism to achieve microsecond-level multimodal data alignment, providing a foundation for spatiotemporal registration at the main control end; 4) It designs a lightweight image preprocessing IP core that supports 1080p@30fps real-time processing, reducing the burden on the main control; 5) The system only outputs the raw data stream and does not participate in wall thickness calculation, defect identification, or alarm judgment, providing clear positioning, strong compatibility, and wide applicability to various main control platforms.
[0024] To more clearly illustrate the functional characteristics and structural parameters of the present invention, further explanation is provided below in conjunction with the accompanying drawings and specific embodiments. Attached Figure Description
[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0026] Figure 1 This is a block diagram of the overall system structure of the present invention;
[0027] Figure 2 This is a functional structure diagram of the ultrasonic acquisition module;
[0028] Figure 3 This is a data flow diagram for the image acquisition and processing module. Detailed Implementation
[0029] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] This invention provides a multimodal pipeline inspection data acquisition system based on an FPGA architecture, with the XC7A35T FPGA chip as its core. The system comprises an ultrasonic acquisition module and an image acquisition and processing module; the two modules operate independently, coordinated by an external synchronization signal, and each outputs raw data streams to the main control device.
[0031] The ultrasonic acquisition module generates a 40kHz pulse excitation signal that lasts for 8 cycles to drive the ultrasonic probe array to emit ultrasonic waves. At the same time, it controls the AD9226 analog-to-digital converter to acquire the reflected echo signal at a rate of 12.5MSPS. The digital sampled data is buffered in Block RAM and then output to the ARM or other receiving end in a fixed format through the SPI communication interface.
[0032] The image acquisition and processing module receives RGB format image data output from the OV5640 industrial CMOS image sensor, performs preprocessing operations such as grayscale transformation, median filtering, mean filtering, Sobel edge enhancement, and morphological operations, and then transmits image data packets outward at a rate of ≥3Gbps through the fiber optic interface. The receiving end writes the received image data into the DDR3 SDRAM cache, and finally reads and displays the image data through the HDMI display screen.
[0033] Specifically, the ultrasonic acquisition module integrates the ODDR2 primitive to generate a symmetrical sampling clock for the AD9226 analog-to-digital converter, ensuring that the setup / hold time meets the requirements.
[0034] Specifically, the image acquisition and processing module deploys a Sobel edge detection IP core with a four-stage pipeline structure, which can achieve real-time processing with less than 5% of logic resources.
[0035] Specifically, the fiber optic interface in the image acquisition and processing module adopts a custom frame protocol, which includes field synchronization markers, data segment start / end markers, and padding codes, supporting reliable high-speed transmission.
[0036] Specifically, the ultrasonic acquisition module synchronously triggers pulse excitation and the sampling window of the AD9226 analog-to-digital converter through the same enable signal to ensure that the two are strictly aligned.
[0037] Specifically, the system also includes a status feedback mechanism, which uses the FPGA to synthesize the AD9226 sampling completion signal and the SPI transmission completion signal to generate a high-level pulse as a "complete acquisition transaction of this round" flag, which is used to establish a time mapping relationship with the image frame capture time.
[0038] The present invention also provides a method for acquiring multimodal pipeline inspection data based on the above system, comprising the following steps:
[0039] S1. Power-on initialization, configuring the internal functional modules of the Artix-7 FPGA;
[0040] S2. Upon receiving an external start command, eight ultrasonic pulses are simultaneously emitted and image frames are acquired.
[0041] S3. The FPGA controls the AD9226 to acquire echo signals. The raw sampled data is buffered in RAM and then uploaded in batches through the SPI interface.
[0042] S4, the FPGA preprocesses the current frame image and then sends the image frame through the fiber optic interface;
[0043] S5: The external FPGA receives and buffers image frames, completes decoding, and outputs the image to the HDMI display.
[0044] S6. Output a "transaction completed" signal to the main control device to indicate that the current round of data collection has ended;
[0045] S7. Repeat steps S2 to S5 to continuously output the raw data stream and image.
[0046] Specifically, in step S3, the data packet output by the SPI interface includes a channel identifier, a timestamp, and the original sampled value.
[0047] Specifically, in step S4, image preprocessing includes denoising, edge enhancement, and morphological operations to improve the quality of subsequent processing.
[0048] Specifically, in step S5, the "transaction completed" signal is output in the form of a rising edge, which can be used for interruption notification or time base synchronization.
[0049] Example
[0050] like Figure 1As shown, in one embodiment, the multimodal pipeline inspection data acquisition system of the present invention includes two independent modules: an ultrasonic acquisition module and an image acquisition and processing module. Both are installed at the front end of the pipeline inspection probe and are connected via an external synchronization signal. Each transmits raw data to a host computer or main control device via SPI or an optical fiber interface.
[0051] See Figure 2 In one embodiment, the overall design of the ultrasonic acquisition module is as follows: The FPGA generates independent 40kHz pulse signals, which are then used by a PWM controller to produce eight consecutive square waves with edge jitter of less than 5ns. The excitation signal drives the ring-shaped ultrasonic probes to emit ultrasonic waves. The echoes are pre-amplified and conditioned before being input to an AD9226 analog-to-digital converter for analog-to-digital conversion. The converted 12-bit digital signal is buffered in the FPGA's Block RAM and then packaged and uploaded by the SPI master controller. After the upload is complete, the FPGA outputs a high-level pulse as a "transaction complete" status feedback signal.
[0052] See Figure 3 In one embodiment, the overall design of the image acquisition and processing module is as follows: The OV5640 sensor outputs 1080p@30fps RGB image data, which undergoes grayscale transformation, filtering, Sobel edge detection, and morphological operations sequentially. The processed image is packaged into a custom frame protocol (including VS_POSE / DATA_START / END markers) by the sfp_encoder module, sent to the receiving device via the fiber optic interface, written to the DDR3 SDRAM cache, and displayed on an HDMI display.
[0053] In one embodiment, both modules receive a unified "acquisition start" signal from the host device via an external GPIO pin. This signal simultaneously triggers ultrasonic pulse transmission and image frame capture, ensuring that the two data sets share a common time start.
[0054] In one embodiment, the SPI data stream output by the ultrasonic acquisition module contains the raw sampled values; the image acquisition and processing module outputs a substantially enhanced image frame.
[0055] This device is characterized by its simple operation, high measurement accuracy, portability, and applicability to various testing applications.
[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0057] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multimodal pipeline inspection data acquisition system based on FPGA architecture, characterized in that, It includes an ultrasonic acquisition module and an image acquisition and processing module. The ultrasonic acquisition module generates eight 40kHz pulse excitation signals to produce ultrasonic signals, controls the AD9226 to acquire echo data, and outputs raw sampled data through the SPI interface. The image acquisition and processing module acquires image data output by the OV5640 sensor, performs distortion correction and edge enhancement preprocessing, and then transmits the image frames through the fiber optic interface to write them into the DDR3 buffer, and finally displays them on the HDMI display. The ultrasonic acquisition module and the image acquisition and processing module achieve time alignment of the acquisition actions through an external synchronization signal.
2. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The ultrasonic acquisition module integrates the ODDR2 primitive to generate a symmetrical sampling clock for the AD9226 and synchronously triggers the pulse excitation and AD9226 sampling window through the same enable signal.
3. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The SPI interface has a communication rate of over 800kHz, and the output data packet includes a channel identifier and a timestamp. The system also includes a status feedback mechanism, which combines the AD9226 analog-to-digital converter sampling completion signal and the SPI transmission completion signal to generate a high-level pulse as a "complete acquisition transaction of this round" flag.
4. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The image acquisition and processing module also includes grayscale transformation, filtering, Sobel edge detection and morphological manipulation units; the image sensor configuration is capable of capturing clear images even in poor lighting conditions.
5. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The image acquisition and processing module also includes a morphological processing unit. The morphological processing unit uses a 3×3 square structural element and performs dilation and erosion operations in sequence. The dilation operation is used to fill small holes in the image and connect broken edges, while the erosion operation is used to eliminate isolated noise and burrs in the image.
6. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 5, characterized in that, The morphological processing unit is implemented using a three-stage pipeline architecture, including: a first-stage multi-line buffer unit for simultaneously buffering multiple lines of image data; a second-stage bitwise operation unit for performing bitwise logical operations on the buffered image data; and a third-stage synchronous output unit for outputting the morphologically processed image data. The dilation operation is implemented by performing a bitwise OR operation in the second-stage bitwise operation unit, and the erosion operation is implemented by performing a bitwise AND operation in the second-stage bitwise operation unit.
7. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The image acquisition and processing module deploys a Sobel edge detection IP core with a four-stage pipeline structure, with a resource utilization rate of less than 5% LUT; the DDR3 cache capacity is not less than 128MB, and it supports dual-frame ping-pong caching.
8. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The optical fiber interface between the image acquisition and processing module and the receiving device adopts a custom frame protocol, which includes field synchronization markers, data segment start / end markers and padding codes; the optical fiber interface adopts 8B / 10B encoding and the transmission rate reaches 3.125Gbps; the image frame format is RGB565.
9. The multimodal pipeline inspection data acquisition system based on FPGA architecture according to claim 1, characterized in that, The ultrasonic acquisition module generates two clocks, 50MHz and 12.5MHz, through a PLL phase-locked loop, which are used for pulse generation and AD sampling, respectively. It supports online updates of the FPGA bitstream file via JTAG or SD card, enabling dynamic upgrades of acquisition parameters and preprocessing algorithms.
10. A method for acquiring multimodal pipeline inspection data based on an FPGA architecture, the method being based on a multimodal pipeline inspection data acquisition system based on an FPGA architecture as described in any one of claims 1-9, characterized in that, Includes the following steps: S1. Initialize each FPGA module and clock source; S2. Upon receiving an external start command, ultrasonic pulse emission and image frame acquisition are triggered simultaneously. S3. Control the AD9226 analog-to-digital converter to acquire echo signals. The raw data is cached in RAM and then uploaded in batches through the SPI interface. S4. Preprocess the image data, and then send the image frames through the fiber optic interface. S5: The external FPGA receives and buffers image frames, completes decoding, and outputs the image to the HDMI display. S6. Output a "transaction complete" signal to indicate that the current round of data collection has been completed. The "transaction complete" signal is output on a rising edge and is used for time base synchronization. S7. Repeat S2 to S6 to continuously output the raw data stream.