An opencl-based panoramic image perspective transformation method

Abstract

The application discloses a panoramic image perspective transformation method based on OpenCL, and comprises the following steps: firstly, OpenCL related initialization is completed, and OpenCL is used to perform steps two to five in parallel; secondly, a three-dimensional sphere is established; thirdly, a mapping relationship between a panoramic image and a surface of the three-dimensional sphere is established by using a perspective projection function, and the panoramic image is projected onto the surface of the three-dimensional sphere established in the second step; fourthly, the sphere is rotated by using a sphere rotation function; and fifthly, the panoramic image on the sphere after rotation is projected back to a screen by using an inverse projection function. The application has the characteristics of adjustable perspective, high real-time performance, good visual effect and the like.

Description

Technical Field

[0001] This invention relates to the fields of panoramic image stitching and parallel computing, specifically to a panoramic image viewpoint transformation method based on OpenCL. Background Technology

[0002] Panoramic video stitching is a technique that uses real-world images to create a panoramic space, stitching multiple images together to form a large-scale image or a 360-degree panoramic view. Panoramic video technology involves computer vision, computer graphics, digital image processing, and mathematical tools. The basic steps of panoramic stitching mainly include: camera calibration, sensor image distortion correction, image projection transformation, panoramic video stitching (fusion), and brightness and color equalization processing. To ensure the visual effect of the final stitched image, the common practice is to project the image captured by each camera onto a three-dimensional surface, such as a cylinder, sphere, or cube. Different projection methods result in different panoramic image effects, with spheres being a commonly used method.

[0003] In practical applications, panoramic stitching is often deployed on embedded platforms, such as autonomous driving systems in cars and drones. Therefore, to ensure the real-time performance of the stitching, most panoramic stitching uses a stitching table method. That is, after the camera positions are installed, the stitching data is first obtained according to the normal process and saved locally as a file. In subsequent actual use, the parameter file is directly read for stitching.

[0004] OpenCL is a programming framework for heterogeneous platforms, which can consist of CPUs, GPUs, or other types of processors. Its core is the OpenCL-C language (based on C99), a language for writing kernels (functions that run on OpenCL devices), and a set of APIs for defining and controlling the platform. OpenCL's data-parallel computing mechanisms provide a highly efficient implementation for algorithms like panoramic stitching, which are simple tasks but involve large amounts of data. Summary of the Invention

[0005] The purpose of this invention is to provide a panoramic image perspective transformation method based on OpenCL. Traditional panoramic stitching systems may not have perspective adjustment capabilities, meaning users can only observe the stitched image from the angle provided by a single manufacturer, which to some extent limits the scope of panoramic applications. This invention first defines a three-dimensional sphere and then builds a spherical model on this sphere for perspective transformation. The panoramic stitched image is projected onto this sphere, and then the angle of the panoramic stitched image on the sphere model is adjusted according to usage requirements. Finally, the transformed sphere surface is back-projected onto the screen to obtain the image after perspective adjustment. The algorithm is implemented using the OpenCL programming framework and features high real-time performance and good visual effects.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A panoramic image viewpoint transformation method based on OpenCL, characterized by the following steps:

[0008] Step 1: First, complete the OpenCL-related initializations, and then use OpenCL to execute steps 2-5 in parallel.

[0009] Step 2: Create a custom 3D sphere;

[0010] Step 3: Use the orthographic projection function to establish the mapping relationship between the panoramic image and the surface of the 3D sphere, and project the panoramic image onto the surface of the 3D sphere established in Step 2;

[0011] Step 4: Rotate the sphere using the sphere rotation function;

[0012] Step 5: Use the back projection function to project the panoramic image on the rotated sphere back onto the screen.

[0013] Preferably, step one includes: first, querying which platforms on the host support OpenCL by calling OpenCL API functions, selecting the platform for algorithm execution, and creating a platform object; querying heterogeneous computing devices under the platform, selecting the device and creating a device object GPU, and creating an OpenCL context Clcontext, which is used to manage the platform, heterogeneous devices, program objects, and kernel; then, creating a command queue clCommandQueue on the GPU for program scheduling; next, generating a computation kernel object, writing OpenCL kernel functions, including custom 3D sphere, orthographic / reverse projection functions, rotation matrix calculation functions, sphere rotation functions, and other auxiliary functions; and finally, executing steps two through five in parallel.

[0014] Preferably, when using OpenCL to execute steps two through five in parallel, it is necessary to determine the device status and whether platform initialization, device initialization, and kernel compilation are completed. If the device status is abnormal or platform initialization, device initialization, and kernel compilation are not completed, the original image is output. Otherwise, if the device is normal and OpenCL initialization is completed, the view adjustment function is provided.

[0015] Preferably, in step two, a Cartesian coordinate system is established with the center of the three-dimensional sphere as the origin, and three coordinate axes are defined according to the placement and orientation of the multiple cameras. The radius of the sphere is the average focal length of the multiple cameras.

[0016] Preferably, if the three-dimensional sphere defined in step two is a cylinder, then the orthographic projection function in step three is as follows: Represent the surface of the three-dimensional sphere using polar coordinates, where the coordinates of any point on the sphere are (X, Y, Z). Assigning the horizontal and vertical coordinates (U, V) of the panoramic image to (X, Y, Z) gives the projection of the image onto the sphere. The mapping relationship between the panoramic image and the sphere is as follows:

[0017]

[0018] Where s is the average focal length of the camera, K is the camera intrinsic parameter matrix, and R is the rotation matrix.

[0019] In some preferred embodiments, in step four, a rotation matrix is ​​constructed using a rotation matrix calculation function, and the sphere is rotated according to the rotation matrix using a sphere rotation function.

[0020] Preferably, if the three-dimensional sphere defined in step two is a cylinder, then the rotation matrix calculation function in step four is as follows: the rotation of the sphere is represented using Euler angles (α, β, γ), that is, the rotation of the sphere is decomposed into rotations along the coordinate axes in sequence. The rotation angles α, β, and γ correspond to pitch, raw, and roll, which are commonly used in panoramic stitching images. The rotation matrix is ​​then calculated using the following formula:

[0021]

[0022]

[0023] The sphere rotation function is: multiply the sphere coordinates by the rotation matrix to obtain the rotated coordinates.

[0024] In some preferred embodiments, the rotation in step four is either a rotation around the coordinates or a calculation using quaternions.

[0025] The beneficial effects of this invention are as follows:

[0026] This invention provides a panoramic image perspective transformation method based on OpenCL, which uses a custom three-dimensional sphere to complete the perspective transformation of the panoramic image, and adopts parallel computing for matrix operations and image projection involved in the algorithm based on OpenCL to ensure the running efficiency of the algorithm. Attached Figure Description

[0027] Figure 1 This is the overall flowchart of the present invention.

[0028] Figure 2 This is a schematic diagram of the coordinates of a three-dimensional sphere.

[0029] Figure 3 This is the default stitched image output.

[0030] Figure 4 An image with a horizontally adjusted viewing angle.

[0031] Figure 5 This is an image after multiple adjustments to the viewing angle. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0033] This invention proposes a panoramic image viewpoint transformation method based on OpenCL. This method achieves panoramic image viewpoint adjustment by using a custom three-dimensional sphere, and simultaneously uses OpenCL to accelerate the algorithm and ensure its real-time performance. The main steps are as follows:

[0034] Step 1: First, complete the OpenCL-related initialization. By calling the OpenCL API function (clGetPlatformIDs), query which platforms on the current host support OpenCL, select the platform for algorithm execution, create a platform object, query the heterogeneous computing devices under that platform, select the device and create a device object GPU, and create an OpenCL context Clcontext. This context is used to manage OpenCL resources such as platform, heterogeneous devices, program objects, and kernel. Then, create a command queue clCommandQueue on the GPU for program scheduling. Next, generate the computation kernel object, write the OpenCL kernel functions, including the functional functions required by the algorithm such as custom 3D sphere, orthographic / reverse projection functions, rotation matrix calculation functions, sphere rotation functions, and other auxiliary functional functions. Finally, execute steps 2-5 in parallel.

[0035] When using OpenCL for parallel computing, the process includes steps such as platform initialization, device initialization, compiling and running kernel functions, and resource release. During viewpoint changes, it's necessary to check the device status and whether platform initialization, device initialization, and kernel function compilation are complete. If the device status is abnormal or platform initialization, device initialization, and kernel function compilation are incomplete, the original image is output. Conversely, if the device is normal and OpenCL initialization is complete, viewpoint adjustment functionality is provided.

[0036] Step 2: Create a custom 3D sphere. For example... Figure 2 A Cartesian coordinate system is established with the center of the three-dimensional sphere as the origin. Based on the actual situation of the panoramic stitching system (the placement and orientation of the multiple cameras), three coordinate axes are defined, and the radius of the sphere is the average focal length of the multiple cameras.

[0037] A three-dimensional sphere can be a cylinder, a cube, or other commonly used three-dimensional spheres, or other custom three-dimensional spheres. Steps three to five below are illustrated using a cylinder as an example. The steps for other three-dimensional sphere shapes are exactly the same, only the mathematical formulas differ.

[0038] Step 3: Use the orthographic projection function to establish the mapping relationship between the panoramic image and the surface of the 3D sphere, and project the panoramic image onto the surface of the 3D sphere established in Step 2.

[0039] The orthographic projection function is as follows: Represent the surface of the three-dimensional sphere using polar coordinates. The coordinates of any point on the sphere can then be represented as (X, Y, Z). Mapping the panoramic image's x and y coordinates (U, V) to (X, Y, Z) gives the projection of the image onto the sphere. The mapping relationship between the panoramic image and the sphere is as follows:

[0040]

[0041] Where s is the average focal length of the camera, K is the camera intrinsic parameter matrix, and R is the rotation matrix.

[0042] Step 4: Construct the rotation matrix using the rotation matrix calculation function, and rotate the sphere according to the rotation matrix using the sphere rotation function.

[0043] After projecting the panoramic image onto the surface of the sphere, rotating the sphere while keeping the viewpoint unchanged will change the viewing angle of the panoramic image. Since the center of the custom coordinate system in step two coincides with the center of the sphere, the positional relationship of the sphere before and after rotation can be easily represented by a rotation matrix.

[0044] The rotation matrix calculation function is as follows: the rotation of the sphere is represented using Euler angles (α, β, γ), that is, the rotation of the sphere is decomposed into rotations along the coordinate axes in sequence. The rotation angles α, β, and γ correspond to pitch, raw, and roll, which are commonly used in panoramic image stitching. The rotation matrix can then be directly calculated using the following formula:

[0045]

[0046]

[0047] The sphere rotation function is obtained by multiplying the sphere coordinates by the rotation matrix.

[0048] As a preferred embodiment of the present invention, the rotation of the three-dimensional sphere in step four is not limited to the rotation matrix method, but includes decomposing the general rotation into rotation around the coordinates or using quaternions for calculation.

[0049] As a preferred embodiment of the present invention, the rotation of the three-dimensional sphere in step four is implemented using OpenGL, that is, the viewing angle is adjusted when the position of the three-dimensional sphere is determined.

[0050] Step 5: Use the back projection function to project the panoramic image on the rotated sphere back onto the screen.

[0051] After the sphere completes its rotation, the panoramic image projected onto the sphere's surface also rotates by a certain angle. At this point, the back-projection relationship from the sphere to the image is calculated using the projection relationship from step three, and the image after back-projection is obtained. This image is the image after the viewpoint transformation.

[0052] Steps two through five involve a large number of matrix calculations, such as the projection of a two-dimensional image onto a three-dimensional sphere, the real-time calculation of the rotation matrix, and the back projection of three-dimensional coordinates onto two-dimensional coordinates. The real-time performance of the algorithm can be guaranteed by using parallel computing.

[0053] To verify the panoramic image perspective transformation method based on OpenCL proposed in this invention, this embodiment uses 6 channels of simulation data to simulate a panoramic stitching system of multiple cameras for testing. Figure 2 This is the default stitched image output. Figure 3 For images with horizontally adjusted viewing angles, Figure 4 The image, after multiple perspective adjustments, is accelerated by OpenCL parallel computing, enabling smooth and dynamic browsing of panoramic images from different perspectives.

[0054] It is understood that those skilled in the art can make equivalent substitutions or modifications to the technical solution and inventive concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the appended claims.

Claims

1. A panoramic image viewpoint transformation method based on OpenCL, characterized in that... The steps are as follows: Step 1: First, complete the OpenCL-related initializations, and then use OpenCL to execute steps 2-5 in parallel. Step 2: Create a custom 3D sphere; Step 3: Use the orthographic projection function to establish the mapping relationship between the panoramic image and the surface of the 3D sphere, and project the panoramic image onto the surface of the 3D sphere established in Step 2; Step 4: Rotate the sphere using the sphere rotation function; Step 5: Use the back projection function to project the panoramic image on the rotated sphere back onto the screen.

2. The panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... Step one involves first calling OpenCL API functions to query which platforms on the host machine support OpenCL, selecting the platform for algorithm execution, and creating a platform object; then querying heterogeneous computing devices under that platform, selecting the device and creating a device object GPU, and creating an OpenCL context Clcontext, which is used to manage the platform, heterogeneous devices, program objects, and kernel; then creating a command queue clCommandQueue on the GPU for program scheduling; next, generating a computation kernel object, writing OpenCL kernel functions, including custom 3D sphere, orthographic / reverse projection functions, rotation matrix calculation functions, sphere rotation functions, and other auxiliary functions; and finally, executing steps two through five in parallel.

3. The panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... When using OpenCL to execute steps two through five in parallel, it is necessary to determine the device status and whether platform initialization, device initialization, and kernel compilation are complete. If the device status is abnormal or platform initialization, device initialization, and kernel compilation are incomplete, the original image is output. Otherwise, if the device is normal and OpenCL initialization is complete, the view adjustment function is provided.

4. The panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... In step two, a Cartesian coordinate system is established with the center of the three-dimensional sphere as the origin, and three coordinate axes are defined according to the placement and orientation of the multiple cameras. The radius of the sphere is the average focal length of the multiple cameras.

5. A panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... If the 3D sphere defined in step two is a cylinder, then the orthographic projection function in step three is as follows: Represent the surface of the 3D sphere using polar coordinates. The coordinates of any point on the sphere are then represented as (X, Y, Z). Mapping the horizontal and vertical coordinates (U, V) of the panoramic image to (X, Y, Z) gives the projection of the image onto the sphere. The mapping relationship between the panoramic image and the sphere is as follows: Where s is the average focal length of the camera, K is the camera intrinsic parameter matrix, and R is the rotation matrix.

6. The panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... In step four, the rotation matrix is ​​constructed using the rotation matrix calculation function, and the sphere is rotated according to the rotation matrix using the sphere rotation function.

7. A panoramic image perspective transformation method based on OpenCL according to claim 6, characterized in that... If the 3D sphere defined in step two is a cylinder, then the rotation matrix calculation function in step four is as follows: Represent the sphere rotation using Euler angles (α, β, γ), that is, decompose the sphere rotation into rotations sequentially along the coordinate axes. The rotation angles α, β, and γ correspond to pitch, raw, and roll in the panoramic stitched image. The rotation matrix is ​​then calculated using the following formula: The sphere rotation function is: multiply the sphere coordinates by the rotation matrix to obtain the rotated coordinates.

8. A panoramic image perspective transformation method based on OpenCL according to claim 1, characterized in that... In step four, the rotation can be performed by rotating around the coordinates or by using quaternions for calculation.

Images