Image registration parallel acceleration optimization method based on TMS320C6678 multi-core DSP

Abstract

The application discloses a kind of based on TMS320C6678 multi-core DSP's heterogeneous image registration parallel acceleration method, system and storage medium.The method faces SAR and optical image and so on heterogeneous image registration task, and registration procedure is divided into feature extraction, multi-channel feature descriptor generation, FFT frequency domain transformation, frequency domain correlation calculation, correlation result fusion, IFFT inverse transformation and matching peak search stage;Master-slave core collaborative scheduling model is established on TMS320C6678 multi-core DSP, and core 0 executes task distribution, process control and result summary, and remaining kernel executes parallel computing task;Through MSMC shared memory, inter-core communication structure body is set, and task trigger, state feedback and stage synchronization are realized;Through EDMA, ping-pong double buffering mechanism is built between off-chip DDR3 and each kernel L2 local buffer, and the overlapping execution of data prefetching, result backwriting and kernel calculation is realized;And in combination with multi-level storage mapping, VLIW and SIMD instruction level optimization improve feature extraction, FFT and frequency domain correlation calculation efficiency.The application can reduce off-chip memory access waiting and inter-core synchronization overhead, improve multi-core DSP resource utilization, and be suitable for embedded image registration scene constrained by power consumption, volume and real-time performance.

Description

Technical Field

[0001] This invention belongs to the fields of digital signal processing, embedded parallel computing and image registration technology. Specifically, it relates to a heterogeneous image registration parallel acceleration method, system and storage medium based on the TMS320C6678 multi-core digital signal processor, which can be applied to scenarios such as SAR and optical image registration, remote sensing image processing, UAV aerial image processing and industrial visual inspection. Background Technology

[0002] With the rapid development of high-resolution imaging technology and embedded systems, image registration, as a core preprocessing step in computer vision, remote sensing mapping, and medical image processing, is becoming increasingly widely used. Image registration aims to align the spatial positions of two or more images acquired at different times, from different perspectives, or from different sensors. However, modern applications pose a dual challenge to the accuracy and real-time performance of registration, especially in fields such as UAV aerial photography, remote sensing, and industrial online inspection, where feature extraction, matching, and transformation model estimation of high-resolution images are often required to be completed within milliseconds.

[0003] Currently, image registration algorithms (such as feature-based SIFT, SURF, ORB, or region-based cross-correlation algorithms) are computationally intensive, and traditional single-core processors (such as general-purpose CPUs or single-core DSPs) struggle to meet real-time processing requirements. While GPUs offer high throughput in parallel computing, their deployment cost and environmental adaptability are limited in embedded scenarios such as aerospace, unmanned platforms, and underwater exploration, where power consumption, size, and heat dissipation are constraints. The C6678 series multi-core digital signal processors (DSPs), based on the KeyStone architecture, integrate eight C66x fixed-point and floating-point cores with a maximum clock speed of 1.4 GHz. The TMS320C6678, based on the KeyStone architecture, integrates eight C66x DSP cores and is configured with MSMC shared memory, an EDMA controller, and a multi-level on-chip / off-chip memory structure, making it suitable for executing embedded image processing tasks with data parallelism.

[0004] However, when developing a high-performance image registration system based on C6678, existing technical solutions still have the following bottlenecks:

[0005] 1. Uneven load distribution across multiple cores: For image registration algorithms based on multi-channel feature descriptors and FFT correlation calculations, the data size, number of channels, and computational density vary significantly across different processing stages. Using a static task allocation method with single granularity can easily cause some cores to wait for synchronization, reducing the efficiency of multi-core parallelism.

[0006] 2. Low storage architecture utilization: The C6678 has a multi-level storage architecture (L1, L2, MSMC, and off-chip DDR3). The amount of relevant feature data to be extracted during image processing and registration is huge, usually at the [W×H] pixel level. If block scheduling and data reuse design are not implemented in conjunction with the local memory capacity of each core's L2 memory, MSMC shared memory, and off-chip DDR3 access bandwidth, frequent access to off-chip memory will introduce high latency, becoming a major bottleneck for real-time processing.

[0007] 3. Inter-core communication and synchronization overhead: During parallel registration, each core needs to frequently exchange feature descriptors or synchronize intermediate matching results. Traditional methods based on semaphores or hard synchronization incur significant processing overhead under high-frequency multi-core interaction, reducing the system's real-time response capability.

[0008] In summary, how to combine the hardware architecture characteristics of C6678 to design a parallel acceleration scheme that includes efficient task scheduling, multi-level storage optimization, and low-loss inter-core communication to achieve fast and accurate registration of high-resolution images is a key issue that urgently needs to be addressed in the current embedded vision field. Summary of the Invention

[0009] To address the problems of low efficiency, high external memory access latency, and uneven load distribution on C6678 multi-core DSPs in existing image registration algorithms, this invention provides a parallel acceleration and optimization method for multimodal image registration based on C6678 multi-core DSPs. This invention achieves real-time and efficient registration of high-resolution images through deep hardware architecture adaptation.

[0010] The technical solution adopted by this invention to solve its technical problem is:

[0011] Step 1: Modular Decomposition of Multimodal Image Registration Process

[0012] The image registration algorithm is divided into processing stages according to data dependencies, including feature extraction, multi-channel feature descriptor generation, FFT frequency domain transformation, frequency domain correlation calculation, multi-channel correlation result fusion, inverse IFFT transformation, and matching peak search, and the appropriate parallel granularity for each stage is determined.

[0013] Step Two: Task Division and Master-Slave Core Coordination

[0014] Based on the partitioned parallel computing regions, a master-slave core collaboration model is adopted. Core 0 of the C6678 is used as the master core to be responsible for global task allocation, instruction parsing and result aggregation, while cores 1 to 7 are used as slave cores. After determining the task granularity according to the amount of computation, number of channels, cache capacity and data dependency relationship at each stage, parallel computing tasks are allocated.

[0015] Step 3: Inter-core communication and synchronization based on MSMC shared memory

[0016] In the MSMC shared memory of C6678, shared communication variables are allocated. Each kernel confirms the task status by updating and polling the status flag field in the shared communication structure, and completes task triggering, progress feedback and stage synchronization between kernels. The status flag is ensured to be visible between multiple kernels by caching off or caching consistency maintenance, thereby reducing the overhead of operating system-level semaphores or complex synchronization mechanisms.

[0017] Step 4: Data transmission based on EDMA dual-buffering mechanism

[0018] The C6678's EDMA is used to perform the prefetching of the next batch of data or the writing back of the previous batch of results in the background. The completion of data transfer is confirmed by the completion flag, interrupt or status bit. A ping-pong double buffer mechanism is built between the off-chip DDR3 and the on-chip L2 SRAM. When one buffer is used for the current kernel calculation, the other buffer completes the prefetching and transmission of the next batch of data synchronously through EDMA, realizing the parallel overlap of data transmission and kernel calculation.

[0019] Step 5: Multi-level storage architecture adaptation and memory optimization

[0020] Image data and large-capacity feature descriptors in the algorithm are allocated to the DDR3, shared variables for inter-core communication are allocated to the MSMC multi-core shared space, and local computational variables and hotspot data required for kernel computation are allocated to the independent L2 SRAM of each core.

[0021] As a preferred specific implementation, for the image registration algorithm based on AWOG descriptors and FFT, the present invention adopts a fine division of computing power space in task partitioning: in the feature extraction stage, the main core distributes the data to cores 1 to 4 to complete AWOG feature extraction in parallel; in the FFT and related calculation stage, the full computing power of cores 0 to 7 is used to perform FFT transformation and related calculations in parallel; in the accumulation and inverse transformation stage, core 0 is solely responsible for the accumulation and fusion of multi-channel related results and the inverse IFFT transformation.

[0022] The advantages of this invention are:

[0023] 1. Alleviating the bottleneck of off-chip memory access: To address the problem of low memory architecture utilization, a dual-cache (ping-pong cache) mechanism based on EDMA transmission is designed, which makes the DSP core calculation and EDMA background data transfer highly overlapped, effectively hiding the huge latency of frequent access to off-chip DDR3 memory.

[0024] 2. Reduce inter-core communication overhead: By utilizing the MSMC multi-core shared space integrated within the chip, lightweight inter-core state representation variables are deployed. Each core only needs to poll the value of the shared variable to confirm the task status, abandoning the traditional semaphore or hard synchronization mechanism and greatly reducing communication latency.

[0025] 3. Improved multi-core load balancing: The designed "master-slave" core collaboration multi-level task scheduling strategy dynamically adjusts the number of slave cores participating in the computation based on the computational characteristics of the registration algorithm at different stages (such as feature extraction and FFT transformation), ensuring that the task granularity of the eight cores is basically aligned, effectively shortening the worst-case processing time of the system and meeting the stringent requirements of industrial-grade real-time vision systems.

[0026] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0027] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0028] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0029] Figure 1 This is a diagram showing the algorithm-hardware collaborative architecture and data flow based on C6678 in this embodiment.

[0030] Figure 2 This is a diagram of the TMS320C6678 multi-core DSP multi-level storage and communication architecture. Detailed Implementation

[0031] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that these embodiments are provided so that this disclosure is thorough and complete, and that the scope of the disclosure is fully conveyed to those skilled in the art.

[0032] This invention provides a detailed deployment method and optimization scheme for synthetic aperture radar and optical image registration on C6678. It should be noted that (1) the embodiments described are only used to explain the technical solutions of this invention and are not used to limit the scope of protection of this invention; (2) under the condition of satisfying multi-channel feature extraction and frequency domain correlation calculation, this invention can also be applied to other heterogeneous or homogeneous image registration tasks; (3) this embodiment adopts a SAR-optical fast matching algorithm based on AWOG descriptors and fast Fourier transform. Among them, AWOG stands for angular weighted orientated gradients, which is used to characterize the local directional gradient information of the image; SOFM stands for SAR-to-Optical Fast Matching, which is used to describe the fast registration process between SAR images and optical images. Specifically, it includes the following steps:

[0033] S1. Based on the C6678 multi-level storage structure, the data segment allocation of important variables in the algorithm is carried out reasonably through the link command file.

[0034] In S1, the following principles should be followed when allocating data segments for important variables in the algorithm:

[0035] (1) For the chip-integrated 4MB MSMC multi-core shared SRAM, priority should be given to allocating status characterization variables for communication between the eight cores and confirmation of the progress of each core, as well as other important variables in small memory;

[0036] (2) For DDR3 memory that can meet the storage requirements of large amounts of data, allocate feature descriptors and other intermediate large-capacity process variables with larger required space;

[0037] (3) For small data contents such as temporary variables and EDMA transfer control parameters required for local calculation of each core, allocate them to the independent 512KB L2 SRAM of each core to reduce external memory access latency.

[0038] S2. Acquire synthetic aperture radar and optical images.

[0039] In this embodiment, an optical image is used as the reference image, and a SAR image is used as the image to be registered. Downsampling is performed when reading the image. All image data is first stored in the off-chip DDR3 memory of the C6678, awaiting subsequent multi-level memory scheduling and parallel computing processing. The image data format adopts a single-channel grayscale format to meet the extraction requirements of the AWOG descriptor.

[0040] S3. Based on the "Master-Slave" core collaboration model, the task coordination and scheduling strategy is deployed in core 0, and a cyclic processing framework is built.

[0041] Core 0 serves as the master core, undertaking the core functions of global task scheduling, instruction parsing, flow control, and result aggregation. Cores 1 through 7 serve as slave cores, performing parallel computations according to the tasks assigned by the master core. The overall registration process is implemented based on a polling loop.

[0042] 1. Inter-core communication rules: Define a communication structure in the MSMC shared space, containing the status flags of each core (e.g., 0 for idle, 1 for working, 2 for completed). The master core sends a task trigger signal to the slave cores by modifying the task type and data address and setting the corresponding status flag to 1.

[0043] 2. Slave Core Response Rules: Each slave core obtains its own core number through the DSP's built-in function `cpu_id_get()` and polls the corresponding status flag in the MSMC in real time. Upon detecting a trigger signal, it immediately starts calculation, and updates the status flag to 2 upon completion. The master core polls the status of all slave cores; if all are 2, it proceeds to the next stage.

[0044] S4. Optimization processing of feature extraction from images to be registered based on multi-core parallel processing.

[0045] For the image to be registered (SAR image), AWOG feature extraction is performed. To fully utilize the multi-core computing power of the C6678, multi-core parallel optimization is only performed on the feature extraction stage of the image to be registered. The feature extraction task of the image to be registered is assigned to cores 1-4, while cores 5-7 are temporarily idle. The specific extraction process is finely designed in combination with EDMA data transfer and L2 cache space constraints.

[0046] 1. Data Slicing and EDMA Transfer: The main core (core 0) divides the image to be registered into multiple row blocks according to the image row direction, and retains the boundary overlap region corresponding to the radius of the AWOG feature extraction window between adjacent row blocks to avoid edge feature breakage caused by image slicing. Cores 1 to 4 process the corresponding row blocks respectively, and remove duplicate overlapping regions when summarizing features. Cores 1 to 4 use the EDMA channel to build a "ping-pong double cache" mechanism, transferring image row data adapted to the cache limit to the local L2 cache each time.

[0047] 2. Local feature extraction computation: When data in a cache is being computed by the kernel (x, y direction gradient calculation, feature descriptor construction, etc.), the next batch of data is prefetched in the background through EDMA, and the transfer is confirmed by completion flag, interrupt or status bit, realizing the tight integration of "computation and IO".

[0048] 3. Feature Summary: Cores 1-4 will transfer the calculated 9-channel AWOG features back to DDR3 via EDMA and update their status to "Completed".

[0049] S5, Parallel computation of 9-channel feature FFT based on eight-core full computing power

[0050] To fully unleash the computing potential of the C6678 eight-core processor, the FFT stage for the 9-channel AWOG features is allocated with full core alignment. The specific process is as follows:

[0051] 1. Perform joint partitioning of the 9-channel AWOG feature data at the channel level and frequency domain block level. Some channels are further split according to the spectrum row blocks or column blocks, so that the number of task blocks after partitioning is not less than the number of kernels participating in the calculation. The main kernel allocates them to kernels 0 to 7 for FFT transformation according to the number of task blocks and the data size.

[0052] 2. Cores 0-7 move the initial feature data in DDR3 to the local L2 cache via EDMA.

[0053] 3. Fully utilize the VLIW (Very Long Instruction Word) and SIMD (Single Instruction Multiple Data) features of the C66x core, call the reconstructed FFT operator using intrinsics to perform calculations, and optimize the loop body through software pipeline technology. The calculated frequency characteristics are then transferred back to DDR3 via EDMA.

[0054] S6. Multi-core parallel correlation calculation of frequency characteristics

[0055] If the image being processed in this polling loop is the image to be registered, after completing the 9-channel feature FFT transformation, the correlation calculation of the frequency domain features of the image to be registered and the frequency domain features of the reference image will be started immediately. This step follows the eight-core allocation strategy of FFT transformation, combining EDMA data transfer and L2 cache local calculation throughout to maximize computational efficiency. The specific process is as follows:

[0056] 1. The main core 0 issues instructions, and cores 0 to 7 inherit the channel data processed in the FFT stage and perform relevant calculations.

[0057] 2. Each core moves two data paths via EDMA: synchronously moves the "frequency domain features of the image to be registered" and the corresponding "main frequency features of the reference image" it is responsible for from DDR3 to L2 Cache, ensuring that all cross-correlation calculations are completed in high-speed memory.

[0058] S7. Accumulation of relevant calculation results and IFFT transformation

[0059] After nine sets of independent related calculation results are generated, the main core (core 0) performs the accumulation and fusion independently. Core 0 performs an inverse IFFT transformation on the fused frequency domain results in the L2 cache (also using VLIW and SIMD instruction-level optimizations) to convert them into a spatial domain registration correlation map, which is then stored back to DDR3 via EDMA, avoiding unnecessary inter-core synchronization overhead at this stage.

[0060] S8. Solving and Outputting Registration Results

[0061] The master core (core 0) searches for the peak coordinates of the correlation map in the L2 cache to determine the optimal registration displacement of the SAR image to be registered relative to the optical reference image. The master core outputs the coordinates and matching confidence to the external system, resets all slave cores to "idle" status, and waits for the next frame instruction.

[0062] To verify the parallel performance of the acceleration optimization method described in this invention in a real engineering environment, this embodiment uses the following dataset for testing:

[0063] Test data: The first part is the publicly available QXS-SAROPT dataset (containing 20,000 pairs of SAR and optical images covering scenes such as cities, mountains, and ports); the second part is a non-public high-resolution complex terrain dataset (containing terrain pairs with huge differences such as farmland, airports, and rivers).

[0064] Test metrics and results: By recording single-core runtime ( ) and multi-core parallel running time ( ), calculate speedup ratio ( ) and parallel efficiency ( Experimental results show that, in the aforementioned dataset, the hardware-software collaborative strategy of this invention effectively releases the multi-core computing power of the C6678, greatly reducing data access latency and inter-core synchronization losses. While ensuring registration accuracy, it achieves extremely high computational energy efficiency and speedup, meeting the real-time requirements of high-resolution image heterogeneous registration in embedded scenarios such as UAV aerial photography and remote sensing monitoring.

[0065] In this embodiment, all EDMA-based data transfer operations adopt a "ping-pong caching" optimization strategy, establishing a dual-cache structure between the off-chip DDR3 and the on-chip L2 cache. This allows the DSP core to process the current data block while the EDMA synchronously prefetches the next data block in the background, achieving deep hiding of computation and I / O operations and minimizing data transfer latency. Simultaneously, operators for all core computational steps, such as feature extraction, FFT transformation, and correlation calculation, are reconstructed using the VLIW and SIMD features of the C66x core. Software pipeline technology optimizes loop bodies, improving instruction execution efficiency and fully leveraging the hardware computing potential of the C6678 multi-core DSP to achieve real-time and efficient image registration.

Claims

1. A parallel acceleration method for multimodal image registration based on a TMS320C6678 (hereinafter referred to as C6678) multi-core DSP, characterized in that, Includes the following steps: S1. Obtain the reference image and the image to be registered, and divide the image registration process into the following stages according to data dependencies: feature extraction stage, multi-channel feature descriptor generation stage, FFT frequency domain transformation stage, frequency domain correlation calculation stage, multi-channel correlation result fusion and IFFT inverse transformation stage, and matching peak search stage. S2. Establish a master-slave core collaborative scheduling model on the TMS320C6678 multi-core DSP, where core 0 is the master core, used for task distribution, process control and result aggregation, and cores 1 to 7 are slave cores, used to execute parallel computing tasks allocated by the master core; S3. Determine the task granularity based on the computational load, number of channels, cache capacity, and data dependencies of each stage, and allocate the corresponding computational tasks to different DSP cores by using image row block partitioning, feature channel partitioning, or frequency domain data block partitioning respectively. S4. Set up an inter-core communication structure in the MSMC shared memory. The inter-core communication structure includes at least a task type, source data address, target data address, data length, and kernel status identifier field. Each kernel completes task triggering, execution status feedback, and stage synchronization by updating and polling the kernel status identifier field. S5. Utilize EDMA to move data between off-chip DDR3 and the L2 local buffer of each core, and build a ping-pong double buffering mechanism to enable the current buffer data calculation and the next buffer data prefetch or the previous buffer result write-back to be executed in parallel. S6. Store image data and large-capacity intermediate feature data in off-chip DDR3, store the inter-core communication structure in MSMC shared memory, and store the local variables, data block buffers and hotspot data required for each core calculation in the L2 local memory of the corresponding core. S7. Perform an inverse IFFT transform on the fused correlation results to obtain a spatial domain correlation map, and determine the registration displacement or registration coordinates of the image to be registered relative to the reference image based on the peak position of the spatial domain correlation map.

2. The method according to claim 1, characterized in that, For the SAR and optical image registration algorithm based on AWOG descriptors and FFT, the task granularity determination and task allocation in step S3 include: During the feature extraction stage, the main core divides the image to be registered into rows and blocks, and retains the boundary overlap area corresponding to the AWOG feature extraction window between adjacent rows and blocks. The main core moves each row and block to the L2 local buffer of core 1 to core 4 in batches through EDMA, and the multi-channel AWOG feature extraction is completed in parallel by core 1 to core 4. During the FFT and frequency domain correlation calculation stage, the main core divides the multi-channel AWOG feature data into tasks according to feature channels and frequency domain data blocks, and allocates them to cores 0 to 7 to perform FFT transformation and correlation calculation between the frequency domain features of the image to be registered and the frequency domain features of the reference image in parallel. In the relevant result fusion and inverse transformation stage, kernel 0 accumulates and fuses the frequency domain correlation results of each channel, and performs inverse IFFT transformation to obtain the spatial domain correlation map.

3. The method according to claim 1, characterized in that, The inter-core communication structure in step S4 includes a task stage number, task type, source data address, target data address, data length, participating kernel number, and the status identifier field of the corresponding kernel. The status identifier includes at least three states: idle, working, and completed. The master core sends a task trigger signal to the slave core by writing the task type, data address, and data length, and setting the status flag of the corresponding slave core to the working state. Each slave core obtains its own core number and polls the corresponding status identifier field. After detecting the working status, it executes the corresponding task and updates the corresponding status identifier to the completed status after the calculation is completed. The storage area containing the inter-core communication structure is cache disabled or cache consistency maintained to ensure that the state information read by each kernel is consistent.

4. The method according to claim 1, characterized in that, The ping-pong double buffering mechanism in step S5 includes: Set up a first buffer and a second buffer in the L2 local memory of each DSP core participating in the calculation; Data transfer is performed between off-chip DDR3 and the first and second buffers using EDMA logical channels or EDMA parameter sets; The size of each data block moved is set to not exceed the available capacity of the corresponding L2 local buffer and is matched with the feature extraction window or the frequency domain data block size. When the DSP kernel performs calculations on the data in the first buffer, the EDMA performs prefetching of the next data block or writing back the calculation result of the previous data block on the second buffer, thereby achieving overlapping execution of data transfer and kernel calculation.

5. The method according to claim 1, characterized in that, Also includes: By leveraging the VLIW (Very Long Instruction Word) architecture and SIMD (Single Instruction Multiple Data) operation characteristics of the C66x kernel, instruction-level reconstruction based on inline functions is performed on the loop computation operators in feature descriptor generation, FFT transform, frequency domain correlation calculation, and inverse IFFT transform. Furthermore, software pipelines are used to improve the instruction-level parallelism and computational throughput of the loop body.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the TMS320C6678 multi-core DSP processor, it implements the method of any one of claims 1 to 5.

Images