A method for matching heterogeneous images based on spatial information constraint and GPU implementation

Abstract

The present disclosure is a method for heterogeneous scene matching based on spatial information constraint and GPU implementation, comprising: memory allocation and thread allocation of GPU; preprocessing of reference image and real-time image during heterogeneous image matching; performing convolution processing to obtain gradient reference image and gradient real-time image respectively; performing spatial block processing on the gradient real-time image to obtain a plurality of gradient real-time image subgraphs; sequentially matching the plurality of gradient real-time image subgraphs with the gradient reference image to obtain the matching position of the gradient real-time image subgraph; and performing information fusion on the gradient real-time image subgraph and its corresponding relative position relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image. The embodiment can effectively avoid image mismatch caused by image gray nonlinear change, effectively screen out matching key areas, improve image matching degree and robustness, and realize rapid processing of each subgraph and reference image matching by using GPU parallel processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for heterogeneous scene matching based on spatial information constraints and its GPU implementation. Background Technology

[0002] Scene matching technology refers to the process where, after an aircraft reaches a certain navigation position, it uses its imaging sensors to image a predetermined area and then matches the real-time image data with pre-stored image data based on certain criteria to determine the position of the real-time image data within a known image. Scene matching technology includes three steps: preprocessing, scene matching, and post-processing. The image preprocessing module extracts useful information from the image for effective use by the matching module, while the image scene matching module determines the relative positions of the two images.

[0003] The core of scene matching technology is to solve the spatial transformation relationship and grayscale mapping relationship between two images of the same scene to be matched. Through this correspondence, the complementary information of the two images can be comprehensively utilized to obtain a more complete representation of the scene and the target. Heterogeneous image matching uses an optical image as the reference image and a SAR image as the real-time image. The relative positional relationship between the two images is determined by matching them. However, due to problems such as nonlinear mapping between grayscale values, grayscale-based matching algorithms have a high probability of mismatch. In addition, the feature information contained in a single SAR image is limited, resulting in low relative positional accuracy and unstable results after matching with the reference image. It is easy for the relative position to deviate significantly from the true position. Therefore, it is necessary to improve the matching method to increase the accuracy and stability of the matching.

[0004] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0005] The purpose of this invention is to provide a heterogeneous scene matching method based on spatial information constraints and its GPU implementation, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0006] This invention first provides a heterogeneous scene matching method based on spatial information constraints and its GPU implementation, including: S1, performs memory allocation and thread allocation for the GPU; S2, using the GPU to preprocess the reference image and real-time image during heterogeneous image matching, to obtain the processed reference image and real-time image; S3, the GPU uses the Sobel operator to convolve the processed baseline image and the real-time image to obtain the gradient baseline image and the gradient real-time image, respectively. S4. Spatial block processing is performed on the real-time gradient image to obtain multiple real-time gradient image sub-images. S5, match multiple real-time gradient image sub-images with the gradient reference image in sequence to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. S6, fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

[0007] In this invention, S2 includes the following steps: S201 uses the GPU kernel function rgbtogray() to convert the reference image into a grayscale image; S202 uses the GPU kernel function floatconv2float() to perform a convolution operation between the Gaussian operator and the real-time image in the time domain to obtain the filtered real-time image.

[0008] In this invention, step S3 includes the following steps: S301, store the Sobel vertical and horizontal operators into the CPU memory, and use the two-dimensional convolution function floatconv2float() to convolve the Sobel vertical and horizontal operators with the reference image and the real-time image converted to grayscale images respectively to obtain the corresponding gradient components. S302, the magnitudes of the gradient components corresponding to the reference image and the real-time image are added together to obtain the gradient reference image and the gradient real-time image.

[0009] In this invention, in step S4, the real-time gradient image is divided into blocks along the horizontal and vertical directions.

[0010] In this invention, in S5, the image matching method is: the maximum correlation value method, the gradient-based correlation surface search method, or the amplitude and gradient-based correlation surface search method.

[0011] In this invention, step S6 includes the following steps: S601, Sort the distance positions in the relative positional relationship in ascending order to obtain the sorting result and index value, and rearrange the azimuth positions and the maximum correlation coefficient in the relative positional relationship according to the index value to obtain the rearranged results of the distance positions and azimuth positions; S602, perform difference calculation on the rearranged results of distance position and orientation position, and filter out the results whose difference value is less than the set threshold; S603: Select the indexes whose distance and azimuth differences simultaneously meet the threshold conditions, take the union of the indexes with the indexes with the maximum correlation coefficient, and take the average of the distance and azimuth corresponding to the indexes as the final matching positions; S604 If no difference result that simultaneously meets the threshold condition is found, the index corresponding to the maximum correlation coefficient is directly selected, and the distance position and orientation position of the index are used as the final matching position.

[0012] The present invention further provides a heterogeneous scene matching and GPU implementation device based on spatial information constraints, comprising: The allocation module is used for memory allocation and thread allocation for the GPU; The preprocessing module uses the GPU to preprocess the reference image and real-time image during heterogeneous image matching to obtain the processed reference image and real-time image. The convolution module uses the Sobel operator on the processed reference image and the real-time image to obtain the gradient reference image and the gradient real-time image, respectively. The block processing module is used to perform spatial block processing on the real-time gradient image to obtain multiple real-time gradient image sub-images. The matching module is used to match multiple real-time gradient image sub-images sequentially with the gradient reference image to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. The fusion module is used to fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

[0013] The technical solution provided by this invention may include the following beneficial effects: This invention discloses a heterogeneous scene matching method based on spatial information constraints and its GPU implementation. By extracting gradient information from a reference image and a real-time image, the gradient information is used to minimize image mismatch caused by nonlinear changes in image grayscale. After spatially decomposing the gradient map of the real-time image, the decomposed sub-images are matched with the reference image to obtain the relative positions of each sub-image and the reference image. Information fusion of the relative positions of each sub-image can effectively filter out key matching regions, improving image matching accuracy and robustness. GPU parallel processing is used to achieve fast processing of matching each sub-image with the reference image. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0015] Figure 1A flowchart illustrating a heterogeneous scene matching and GPU implementation method based on spatial information constraints in an exemplary embodiment of this disclosure is shown. Figure 2 A flowchart illustrating a heterogeneous scene matching and GPU implementation method based on spatial information constraints in yet another exemplary embodiment of this disclosure; Figure 3 This illustrates the optical SAR image matching results in an exemplary embodiment of this disclosure; Figure 4 In the exemplary embodiments shown in this disclosure Figure 3 Matching detail diagram; Figure 5 This diagram illustrates the block matching results in an exemplary embodiment of this disclosure. Figure 6 The block variation curves in an exemplary embodiment of this disclosure are shown. Detailed Implementation

[0016] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0017] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0018] This example implementation first provides a heterogeneous scene matching method based on spatial information constraints and its GPU implementation. Please refer to [reference needed]. Figure 1 This method may include: S1-S6, as follows: S1, performs memory allocation and thread allocation for the GPU; S2, using the GPU to preprocess the reference image and real-time image during heterogeneous image matching, to obtain the processed reference image and real-time image; S3, the GPU uses the Sobel operator to convolve the processed baseline image and the real-time image to obtain the gradient baseline image and the gradient real-time image, respectively. S4. Spatial block processing is performed on the real-time gradient image to obtain multiple real-time gradient image sub-images. S5, match multiple real-time gradient image sub-images with the gradient reference image in sequence to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. S6, fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

[0019] In this embodiment, gradient information is extracted from the reference image and the real-time image to minimize image mismatch caused by nonlinear changes in image grayscale. After spatially splitting the gradient map of the real-time image, the split sub-images are matched with the reference image to obtain the relative positions of each sub-image and the reference image. Information fusion of the relative positions of each sub-image can effectively filter out key matching regions, improving image matching degree and robustness. GPU parallel processing is used to achieve fast processing of matching each sub-image with the reference image.

[0020] Scene matching aims to determine the spatial transformation and grayscale mapping relationships between two images of the same scene. Through this correspondence, complementary information from the two images can be comprehensively utilized to obtain a more complete representation of the scene and the target. Assuming an intensity image... and visible light images For example, let the visible light image As the reference image, intensity image If the images to be matched are given, then the matching model between the two images can be represented as:

[0021] In the formula, and These represent the grayscale transformation function and the spatial transformation function, respectively.

[0022] The key to heterogeneous image matching lies in solving the one-dimensional grayscale transformation function and the two-dimensional spatial transformation function, with the core issue being the calculation of the spatial transformation function. For heterogeneous images of the same scene, due to differences in imaging principles and spectral bands, the grayscale distributions between images exhibit significant nonlinear differences, making the grayscale transformation function extremely complex and difficult to express uniformly with a simple function. In practical applications, feature spaces that are invariant to nonlinear grayscale changes, such as gradient space or phase space, can be selected based on image characteristics to avoid calculating the grayscale transformation function. Furthermore, the probability of mismatch is high when matching a single image with a reference image. To address these issues, this invention proposes a heterogeneous scene matching method based on spatial information constraints and its GPU implementation. The specific steps of this invention are further elaborated below.

[0023] Analysis of memory allocation and thread allocation in the GPU implementation process in S1.

[0024] (1) CPU and GPU memory pre-allocation A Graphics Processing Unit (GPU) is a highly parallelized multi-core processor that utilizes a large number of processing units for parallel computing, improving operational efficiency. Computing Unified Device Architecture (CUDA) refers to a hardware and software architecture proposed by NVIDIA that uses GPUs as data-parallel computing components, implemented using the C programming language for general-purpose computing.

[0025] The frequent allocation and deallocation of CPU / GPU memory during GPU implementation is time-consuming. Therefore, GPU programs can be optimized by roughly calculating the amount of memory space required by the program. After the program runs, the globally used memory is pre-allocated to form a memory pool. During subsequent program execution, pointers can be used to point to the memory pool based on the actual amount of memory used. The specific memory allocation and data copying methods are as follows: GPU memory allocation method: Use the cudaMalloc() function in the CUDA function library to allocate storage space in GPU memory; CPU memory allocation methods: Use the cudaMallocHost() function in the CUDA function library to allocate storage space in CPU memory or use the malloc() function in the C language library; GPU data copying method: The cudaMemcpy() function from the CUDA library is used to copy data between the host (CPU memory) and the device (GPU memory), or between devices; where cudaMemcpyKindkind specifies the direction of the copy. cudaMemcpyHostToHost means copying from CPU to CPU (generally not used, memcpy can be used directly). cudaMemcpyHostToDevice means copying from the CPU to the GPU; cudaMemcpyDeviceToHost means copying from the GPU to the CPU. cudaMemcpyDeviceToDevice means copying from GPU to GPU (copying data between GPUs). (2) Analysis of GPU kernel functions, thread blocks, threads, thread block indices, and thread indices A kernel function is a function that is called by the host on the device. A kernel function is a single step that is executed in parallel within a program.

[0026] The kernel is organized in the form of a thread grid. Each grid consists of several thread blocks, and each block consists of several threads. Thread blocks and threads are the first and second level parallel units for kernel functions to perform parallel computations. Thread blocks cannot communicate with each other, but threads within the same thread block can share data.

[0027] The thread block index refers to the position of a thread block within the thread grid. It is a built-in variable in CUDA. The first, second, and third-dimensional indices of a thread block are blockIdx.x, blockIdx.y, and 1, respectively. The thread index refers to the position of a thread within a thread block. It is a built-in variable in CUDA. The first, second, and third-dimensional indices of a thread are threadIdx.x, threadIdx.y, and threadIdx.z, respectively.

[0028] Please refer to Figure 2 S2 includes the following steps: S201 uses the GPU kernel function rgbtogray() to convert the reference image into a grayscale image; S202 uses the GPU kernel function floatconv2float() to perform a convolution operation between the Gaussian operator and the real-time image in the time domain to obtain the filtered real-time image.

[0029] S2 is the baseline image. With real-time images The preprocessing process. Specifically, during heterogeneous image matching, the reference image... Optical images, real-time images This refers to SAR images. Optical images are passive imaging, where the camera passively receives information in the visible light band of natural light reflected from the surface of an object. Due to their wide spectral range and sufficient integrated energy, these images have rich grayscale levels, a wide dynamic range, and, in color imaging, also possess rich color information. Radar SAR images, on the other hand, are active imaging, receiving radar pulse information reflected from the surface of an object. Their spectral range is very narrow, and their energy is weak, resulting in monotonous grayscale levels, a narrow dynamic range, and an inability to reproduce the color information of the scene. Therefore, when using optical images as reference images, they need to be converted into grayscale images.

[0030] In visible light imaging, the ample and uniform illumination of sunlight results in sufficient and evenly distributed reflected energy from objects, leading to low noise and high image quality. However, in radar SAR imaging, the uniformity of radar radiation energy distribution is less than that of sunlight, and the radar's power is limited, resulting in insufficient and uneven reflected energy from objects. Consequently, SAR images have higher noise levels and lower image quality. Image noise significantly impacts image matching performance; therefore, noise suppression is a crucial step in SAR image preprocessing. When using SAR images as real-time images, Gaussian filtering can effectively reduce the impact of noise on the matching process, making the noise levels in different regions of the SAR image more consistent, addressing the differences in noise between different source images.

[0031] In the GPU implementation, optical image grayscale processing is achieved using the kernel function `rgbtogray()`, which converts a three-channel RGB image into a single-channel grayscale image. The GPU implementation of Gaussian filtering for SAR images involves calculating and solving the Gaussian operator, with the operator dimension selected as [value missing]. The Gaussian operator can be convolved with the data to be filtered in the time domain. However, generating the operator online on the GPU is very time-consuming. Instead, the Gaussian operator can be generated in Matlab and then stored in memory as a constant dataset. The kernel function floatconv2float() is designed to implement the convolution function.

[0032] The following explains the process of thread and thread block division in GPU implementation. Since SAR images exist in two dimensions, the first and second dimensions are the range dimension and the azimuth dimension, respectively, and the corresponding number of points is represented as follows: and Similarly, thread blocks and threads in the thread grid are also divided in two dimensions. Therefore, the number of threads in the first dimension of the thread grid can be equal to the number of distance points, and the number of threads in the second dimension can be equal to the number of azimuth points. The specific steps for thread division are as follows: First, determine the number of threads. and The number of threads allocated in the first and second dimensions of the thread block is set sequentially, and the number is set to an integer multiple of 8, while not exceeding the range specified by the GPU thread hardware. Then, the number of thread blocks is determined. and The numbers represent the number of thread blocks allocated in the first and second dimensions of the thread grid, respectively. Image distance to number of points and jointly determined, From image orientation to point count and The calculation formula is determined together as follows:

[0033] A new reference image can be obtained by performing grayscale processing on the optical image and Gaussian filtering on the SAR image as described above. and real-time images .

[0034] S3 is the step in gradient processing of the baseline image and the real-time image. S3 includes the following steps: S301, store the Sobel vertical and horizontal operators into the CPU memory, and use the two-dimensional convolution function floatconv2float() to convolve the Sobel vertical and horizontal operators with the reference image and the real-time image converted to grayscale images respectively to obtain the corresponding gradient components. S302, the magnitudes of the gradient components corresponding to the reference image and the real-time image are added together to obtain the gradient reference image and the gradient real-time image.

[0035] Since there is a non-linear mapping between gray levels in heterogeneous images, the use of gradient information can minimize the image mismatch problem caused by non-linear changes in gray levels of heterogeneous images.

[0036] For input real-time images and benchmark images First, the gradient maps of each Sobel operator are extracted. and The gradient map acquisition operation can be represented by the following formula:

[0037] in, and These represent the Sobel vertical and horizontal operator templates, respectively. This represents the convolution operation.

[0038] GPU implementation process: First, Sobel vertical and horizontal operators are generated in MATLAB and stored in CPU memory as constants. Then, the Sobel vertical and horizontal operators are sequentially convolved with the reference image using the 2D convolution function floatconv2float() in S2. Data obtained by convolution processing and , in turn and The gradient reference image is obtained by summing the amplitude values. Similarly, real-time images Gradient processing is also performed to obtain a real-time gradient image. ;Will and The outer edge of the data matrix is ​​set to 0 to eliminate the influence of data boundaries on the matching results.

[0039] S4 is a real-time gradient image. Process of spatial block processing

[0040] When a single-frame real-time image is matched with a reference image, there are fewer regions in the image that play a decisive role in the matching. The spatial information of the image can be used to screen out the key regions related to the matching. Therefore, first, the gradient real-time image is spatially partitioned. The most direct splitting method is to divide the image into blocks along the horizontal (range direction) and vertical (azimuth direction). From S2, it can be seen that the number of points in the range direction and azimuth direction of the real-time image are respectively and . Let the block length be blockLen, the number of blocks in the range direction of the image be , and the number of blocks in the azimuth direction . Therefore, the gradient real-time image can be partitioned into blockNum = blockNum_Nr * blockNum_Na sub-images in two dimensions, and the sub-images are represented as . For the convenience of GPU processing, blockLen and blockNum should be selected as powers of 2 .

[0041] S5. The sub-images of the gradient real-time image are sequentially matched with the gradient reference image

[0042] Specifically, the sub-images of the split gradient real-time image are sequentially matched with the gradient reference image . The matching results of the sub-images can be expressed as follows

[0043]

[0044] Where represents the th sub-image after splitting the gradient real-time image, represents the matching position of the th sub-image and the gradient reference image, represents the matching function. The GPU implementation process is as follows (1) The size of the sub-image of the gradient real-time image is , and the size of the gradient reference image is . The size difference between the gradient reference image and the sub-image is , where ; (2) Calculate the mean value of the sub-image . Use the mean value to zero-mean the sub-image (subtract the mean value from the pixel values of the sub-image ). The result is , since the size of the sub - figure is smaller than the size of the reference image ( ), for the convenience of unified GPU calculation, it is necessary to expand the size of to and obtain , with the filling value being 0; (3) For the result after filling the real - time image and the gradient reference image , use the cufftExecR2C() function in the CUDA library to implement range - direction FFT in sequence.

[0045] Design the kernel function Trans_cplx() to change the storage arrangement of image data from range - dimension arrangement to azimuth - dimension arrangement. The specific design process of the Trans_cplx() function is as follows: Let the number of threads in the thread block and be TransDim, and TransDim can be 16 or 32; Use __shared__ to construct the shared memory cache[TransDim][TransDim]; Before image transposition, the index is: x_in = threadIdx.x+blockIdx.x*blockDim.x, y_in = threadIdx.y+blockIdx.y*blockDim.y, the number of points in the image size is XLength = gridDim.x*TransDim, YLength = gridDim.y*TransDim. Under the condition of if((x_in<XLength)&&(y_in<YLength)), assign the data Src[y_in*XLength + x_in] before image transposition to the shared memory cache[threadIdx.x][threadIdx.y]; Use the __syncthreads() function to synchronize all threads to ensure that all threads are assigned; After image transposition, the position index is x_out = threadIdx.x+blockIdx.y*blockDim.y, y_out = threadIdx.y+blockIdx.x*blockDim.x. Under the condition of if((x_out<YLength)&&(y_out<XLength)), assign the shared memory cache[threadIdx.y][threadIdx.x] to the data Dst[y_out*YLength + x_out] after image transposition to complete the image data transposition.

[0046] ​​Implement azimuth FFT using the cufftExecR2C() function in the CUDA library; obtain the result after real-time image filling. With gradient reference image FTsmall_i and FTbig after 2D FFT transformation.

[0047] Multiplying the conjugates of FTbig and FTsmall_i yields the cross-correlation coefficient FTR (frequency domain correlation coefficient). The cross-correlation coefficient FTR is then subjected to an azimuth IFFT using the cufftExecC2C() function in the CUDA library. Subsequently, the azimuth IFFT result is transformed from azimuth continuous to range continuous using the Trans_cplx() function. Finally, a range IFFT is performed using the cufftExecC2R() function, yielding the two-dimensional IFFT result of the cross-correlation coefficient FTR as corr.

[0048] Calculate the current subgraph Zero-mean result The standard deviation T_sum is used to calculate the correlation coefficient corr. The subplot can be obtained by dividing corr by the standard deviation T_sum. With gradient reference image Matching position .

[0049] The following will provide three methods for finding the matching position. Method: 1) Maximum Correlation Value Based Method The coordinates corresponding to the maximum value of the correlation coefficient (corr), Maxcorr, are the matching locations. .

[0050] 2) Gradient-based correlation surface search method Construct prewitt vertical and horizontal operators, and then use the two-dimensional convolution function floatconv2float() in S2 to convolve the prewitt vertical and horizontal operators with the correlation coefficient corr to obtain the data. and , in turn and After taking the amplitude values ​​and summing them, the correlation coefficient (corr) is obtained. The gradient coefficient is corr_gradient. The coordinate position corresponding to the maximum value (Maxcorr) is the matching position. .

[0051] 3) Surface search method based on magnitude and gradient correlation Using the method in 2), the correlation coefficient corr is gradientped to obtain the coefficient corr_gradient. The correlation coefficient corr and the gradient coefficient corr_gradient are combined in a certain proportion to form the amplitude-gradient joint coefficient corrPlusGradient. The specific method is as follows: Normalize the correlation coefficient corr and the gradient coefficient corr_gradient to obtain corr_nor and corr_gradient_nor respectively. Set the scaling factor alpha, and corrPlusGradient = alpha * corr_gradient_nor + (1-alpha) * corr_nor. The coordinate position corresponding to the maximum value of the corrPlusGradient coefficient Maxcorr is the matching position. .

[0052] Subgraphs are sequentially processed via S5. With gradient reference image Matching yields the matching position. and the maximum correlation coefficient MaxCorr_i Since it is a real-time image, the image is divided into blocks along the horizontal (range) and vertical (azimuth) directions, and the number of range points and azimuth points in the real-time image are respectively... and Let the block length be blockLen, and the number of distance blocks in the image be... Orientation of blocks Therefore, the matching position The spatial relationship between distance and orientation and the partitions

[0053] To facilitate subsequent use of the matching positions of each subgraph, the matching positions of each subgraph can be determined based on the spatial relationships of the subgraphs. Transforming to the same subgraph yields the matching relative positional relationships as follows: .

[0054] S6, Subgraphs Relative positional relationship Information fusion.

[0055] Based on the relative positional relationships of the subgraphs obtained in S5 By fusing information from the image and extracting useful information, the optimal matching position of the current real-time image can be determined.

[0056] S6 includes the following steps: S601, Sort the distance positions in the relative positional relationship in ascending order to obtain the sorting result and index value, and rearrange the azimuth positions and the maximum correlation coefficient in the relative positional relationship according to the index value to obtain the rearranged results of the distance positions and azimuth positions; S602, perform difference calculation on the rearranged results of distance position and orientation position, and filter out the results whose difference value is less than the set threshold; S603: Select the indexes whose distance and azimuth differences simultaneously meet the threshold conditions, take the union of the indexes with the indexes with the maximum correlation coefficient, and take the average of the distance and azimuth corresponding to the indexes as the final matching positions; S604 If no difference result that simultaneously meets the threshold condition is found, the index corresponding to the maximum correlation coefficient is directly selected, and the distance position and orientation position of the index are used as the final matching position.

[0057] Specifically, the GPU implementation process is as follows: (1) Distance position Sort the data in ascending order to obtain the sorted result SortA and the index value SortA_idx. Similarly, assign the location... The maximum correlation coefficient MaxCorr_i is rearranged according to the index value SortA_idx to obtain SortB and SortC; (2) The difference between SortA and SortB, which are rearranged based on distance and orientation, is obtained. and , ; for the difference results and The result that takes a value less than minidx (minidx can be 2 to 10) is denoted as and Select and A result that simultaneously satisfies a value less than minidx is denoted as... , ;from ,make , combine IDX with The union of the two sets yields a new IDX. The average of SortA(IDX) and SortB(IDX) is then used as the final output matching position. They considered this result to be the position where the real-time image and the reference image had the highest matching degree.

[0058] If not found and If all the difference results in SortC are less than minidx, then find the index IDX of the largest value in SortC, and use SortA(IDX) and SortB(IDX) as the matching positions in the final output. They considered this result to be the position where the real-time image and the reference image had the highest matching degree.

[0059] The effectiveness of the proposed method will be verified through experimental results below. Due to the significant differences in imaging principles between optical images and SAR images, images from different sensors exhibit substantial differences in amplitude information. Matching algorithms based on grayscale information cannot meet the requirements of application scenarios. The proposed method utilizes gradient information for block-based heterogeneous scene matching, as shown in the following results. Figure 3 and Figure 4 As shown.

[0060] from Figure 3 and Figure 4 The results of heterogeneous matching based on spatial information constraints show that the optical image and the SAR image have a high degree of edge matching and good road continuity at the image connection points. Therefore, the matching accuracy of the block matching algorithm is high. At the same time, it can be found that when the real-time image is not completely contained within the reference image, the block matching algorithm can still achieve accurate matching results.

[0061] Because the reflectivity of ground objects varies greatly across different spectral bands, the brightness of the same object differs between optical and SAR images. However, the object's outline, edges, and other features remain relatively intact. Therefore, the gradient-based block matching algorithm can effectively avoid grayscale differences between heterogeneous images. The above conclusions will be discussed in detail below.

[0062] When the image size is large, the effective matching information in the image will be further diluted. If the whole image matching approach is adopted, the matching difficulty will be further increased. Therefore, the block matching approach can effectively avoid the above situation, minimize the influence of information in the image that is not conducive to matching, and avoid its impact on the matching position.

[0063] The impact of the number of blocks on the matching results is compared and analyzed below. The reference satellite optical image size is 4385×6177. In the experiment, the SAR image (image sheet 3142×3142) was divided into 1, 4, 9, 16, 25, and 36 blocks. The algorithm was implemented in Matlab simulation software under the above experimental conditions. The algorithm was also implemented on the NAVIDA Orin NX embedded GPU platform. The results of the Matlab and NAVIDA Orin NX programs were consistent. The obtained block matching results are as follows: Figure 5 As shown in Table 1, the number of correctly matched blocks (whether they are correctly matched) and the matching time used by the program on the Matlab / NX platform are statistically analyzed.

[0064] Table 1. Statistics on Block Matching Accuracy and Time

[0065] Analysis of the experimental data above shows that the heterogeneous scene matching algorithm based on spatial information constraints proposed in this invention has strong robustness and can be applied to various terrain scenarios; it has inherent parallel computing advantages, fully leveraging the multi-threaded parallel computing capabilities of the GPU platform; it has a certain resistance to rotational distortion, and the final matching result is obtained by fusing information from multiple correctly matched blocks. Figure 6 The block variation curve shows that the more blocks there are, the higher the matching precision, but the computational load will increase exponentially. Since the proportion of effective matching regions in the same scene is relatively fixed, continuously increasing the number of block regions has limited effect on improving matching performance.

[0066] This disclosure also provides a heterogeneous scene matching and GPU implementation device based on spatial information constraints, including: The allocation module is used for memory allocation and thread allocation for the GPU; The preprocessing module uses the GPU to preprocess the reference image and real-time image during heterogeneous image matching to obtain the processed reference image and real-time image. The convolution module uses the Sobel operator on the processed reference image and the real-time image to obtain the gradient reference image and the gradient real-time image, respectively. The block processing module is used to perform spatial block processing on the real-time gradient image to obtain multiple real-time gradient image sub-images. The matching module is used to match multiple real-time gradient image sub-images sequentially with the gradient reference image to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. The fusion module is used to fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

[0067] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0068] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0069] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0070] It should be noted that the installation of image acquisition and personal identification equipment in public places involved in this application is necessary for maintaining public safety, complies with relevant national regulations, and is accompanied by prominent warning signs. The collected personal images and identification information can only be used for the purpose of maintaining public safety and not for other purposes; or the images, personal identification data, etc. in this application are all legally and compliantly obtained or collected with the individual's separate consent.

Claims

1. A method for spatial information constraint based multi-sensor image matching and GPU implementation, characterized in that, include: S1, performs memory allocation and thread allocation for the GPU; S2, using the GPU to preprocess the reference image and real-time image during heterogeneous image matching, to obtain the processed reference image and real-time image; S3, the GPU uses the Sobel operator to convolve the processed baseline image and the real-time image to obtain the gradient baseline image and the gradient real-time image, respectively. S4. Spatial block processing is performed on the real-time gradient image to obtain multiple real-time gradient image sub-images. S5, match multiple real-time gradient image sub-images with the gradient reference image in sequence to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. S6, fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

2. The method of claim 1, wherein, S2 includes the following steps: S201 uses the GPU kernel function rgbtogray() to convert the reference image into a grayscale image; S202 uses the GPU kernel function floatconv2float() to perform a convolution operation between the Gaussian operator and the real-time image in the time domain to obtain the filtered real-time image.

3. The method of claim 2, wherein, S3 includes the following steps: S301, store the Sobel vertical and horizontal operators into the CPU memory, and use the two-dimensional convolution function floatconv2float() to convolve the Sobel vertical and horizontal operators with the reference image and the real-time image converted to grayscale images respectively to obtain the corresponding gradient components. S302, the magnitudes of the gradient components corresponding to the reference image and the real-time image are added together to obtain the gradient reference image and the gradient real-time image.

4. The method of claim 1, wherein, In S4, the real-time gradient image is divided into blocks along the horizontal and vertical directions.

5. The method of claim 1, wherein, In S5, the image matching methods are: maximum correlation value method, gradient-based correlation surface search method, or magnitude and gradient-based correlation surface search method.

6. The method of claim 1, wherein, S6 includes the following steps: S601, Sort the distance positions in the relative positional relationship in ascending order to obtain the sorting result and index value, and rearrange the azimuth positions and the maximum correlation coefficient in the relative positional relationship according to the index value to obtain the rearranged results of the distance positions and azimuth positions; S602, perform difference calculation on the rearranged results of distance position and orientation position, and filter out the results whose difference value is less than the set threshold; S603: Select the indexes whose distance and azimuth differences simultaneously meet the threshold conditions, take the union of the indexes with the indexes with the maximum correlation coefficient, and take the average of the distance and azimuth corresponding to the indexes as the final matching positions; S604 If no difference result that simultaneously meets the threshold condition is found, the index corresponding to the maximum correlation coefficient is directly selected, and the distance position and orientation position of the index are used as the final matching position.

7. A device for spatial information constraint based multi-sensor image matching and GPU implementation, characterized in that, include: The allocation module is used for memory allocation and thread allocation for the GPU; The preprocessing module uses the GPU to preprocess the reference image and real-time image during heterogeneous image matching to obtain the processed reference image and real-time image. The convolution module uses the Sobel operator on the processed reference image and the real-time image to obtain the gradient reference image and the gradient real-time image, respectively. The block processing module is used to perform spatial block processing on the real-time gradient image to obtain multiple real-time gradient image sub-images. The matching module is used to match multiple real-time gradient image sub-images sequentially with the gradient reference image to obtain the matching position of the real-time gradient image sub-images, and transform all matching results to the relative positional relationship under the same reference. The fusion module is used to fuse the gradient real-time image sub-image with its corresponding relative positional relationship to obtain the matching position with the highest matching degree between the real-time image and the reference image.

Images