Virtual sphere-based fish-eye image morphing method

Abstract

The application provides a virtual sphere-based fisheye image morphing algorithm, comprising: defining a real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2, wherein a virtual small sphere is formed by taking O as a sphere center and a radius of 1, a virtual large sphere is formed by taking O2 as a sphere center and a radius of n, and the large sphere is inscribed in the small sphere; the fisheye image is converted between the real camera coordinate system and the virtual camera coordinate system by means of the two spheres, and is projected in the virtual camera coordinate system; and the correction degree of line bending in the morphing image of the fisheye image is adjusted by the horizontal field of view angle or the vertical field of view angle of the virtual camera. The application makes the line bending degree in the morphing image smaller under the premise of keeping the horizontal field of view angle or the vertical field of view angle of the real camera, and improves the fisheye image morphing visual effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a fisheye image morphing method based on a virtual sphere. BACKGROUND

[0002] A fisheye lens is a short focal length, large field of view (field of view angle FOV is about 180 degrees) camera lens, the front lens looks like a fish eye, and the fisheye lens is named accordingly. The lens of the human eye is flat and round, so it can see things farther away, while the lens of a fish eye is spherical, so it can only see objects close by, but has a larger field of view, that is, it can see more widely.

[0003] The fisheye lens is widely used in various fields due to its own advantages, but the fisheye image has very serious distortion. If you want to use these fisheye images with serious distortion, you need to improve or eliminate the distortion of the fisheye image, improve the viewing angle effect, make it more in line with people's visual habits, and achieve the purpose of practical application. SUMMARY

[0004] The purpose of the present application is to provide a fisheye image morphing method based on a virtual sphere, which can make the line bending degree of the morphing image smaller while keeping the horizontal field of view angle or the vertical field of view angle of the real camera, and improve the fisheye image morphing visual effect.

[0005] To solve the above technical problems, the present application provides a fisheye image morphing method based on a virtual sphere, comprising:

[0006] A real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2 are defined, wherein O is the center of a virtual small sphere with a radius of 1, and O2 is the center of a virtual large sphere with a radius of n, and the large sphere is inscribed in the small sphere;

[0007] The fisheye image is converted between the real camera coordinate system and the virtual camera coordinate system by means of the two spheres, and the fisheye image is projected in the virtual camera coordinate system; and

[0008] The correction degree of line bending in the morphing image of the fisheye image is adjusted by the horizontal field of view angle or the vertical field of view angle of the virtual camera.

[0009] Optionally, the coordinate conversion process of the fisheye image includes forward transformation and inverse transformation.

[0010] The forward transformation comprises: obtaining coordinates (x0, y0) of the fisheye image; obtaining a 3D unit vector (i0, j0, k0) in a real camera coordinate system according to a camera model; obtaining a 3D unit vector (i1, j1, k1) in a virtual camera coordinate system according to a coordinate conversion between a large sphere and a small sphere; and obtaining coordinates (x1, y1) in the morphed image according to a projection transformation.

[0011] The inverse transformation comprises: obtaining coordinates (x1, y1) in the morphed image; obtaining a 3D unit vector (i1, j1, k1) in the virtual camera coordinate system according to an inverse projection transformation; obtaining a 3D unit vector (i0, j0, k0) in the real camera coordinate system according to a coordinate conversion between the large sphere and the small sphere; and obtaining coordinates (x0, y0) of the fisheye image according to the camera model.

[0012] Optionally, in the inverse transformation, the method for obtaining (i0, j0, k0) from (i1, j1, k1) comprises:

[0013] obtaining coordinates (i1*n, j1*n, k1*n) of an intersection point of (i1, j1, k1) and the large sphere in a coordinate system O2-X2Y2Z2;

[0014] obtaining a vector (i1*n, j1*n, k1*n+(n-1)) of the intersection point in a coordinate system O-XYZ; and

[0015] normalizing the vector into a unit vector to obtain (i0, j0, k0).

[0016] Optionally, in the forward transformation, (i1, j1, k1) is obtained from (i0, j0, k0) according to formula (1);

[0017] (1)

[0018] wherein, (2).

[0019] Optionally, formula (3) is obtained according to a relationship between the two spheres, formula (4) is calculated according to formula (3), and formula (2) and formula (1) are calculated according to formula (4).

[0020] (3)

[0021] (4).

[0022] Optionally, the projection comprises a cylindrical projection, an equirectangular projection or an equidistant projection; and a relationship between a horizontal field of view angle and a vertical field of view angle of the virtual camera is determined by the projection mode and an aspect ratio of the morphed image.

[0023] Optionally, the horizontal field of view angle of the real camera is set as 2δ, the horizontal field of view angle of the virtual camera is set as 2β, formula (5) is obtained according to the relationship between the two spheres, and the calculation formula (6) of the large sphere radius n is obtained according to formula (5);

[0024] (5);

[0025] (6);

[0026] And δ and β need to satisfy formula (7) to make the large sphere radius n>1;

[0027] (7).

[0028] Optionally, the width of the deformation image is set as W, the height is set as H, formula (8) is obtained according to the projection algorithm when the cylindrical projection is adopted, and the vertical field of view angle 2α of the virtual camera is obtained according to formula (8);

[0029] (8);

[0030] Formula (9) is obtained according to the projection algorithm when the equirectangular projection or the equidistant projection is adopted, and the vertical field of view angle 2α of the virtual camera is obtained according to formula (9);

[0031] (9).

[0032] Optionally, formula (10) is obtained according to the relationship between the two spheres, formula (11) is obtained according to formula (10) and formula (8) or formula (9), and the vertical field of view angle 2θ of the real camera is obtained according to formula (11);

[0033] (10);

[0034] (11);

[0035] Wherein, val is a temporary variable.

[0036] Optionally, the vertical field of view angle of the real camera is set as 2θ, the vertical field of view angle of the virtual camera is set as 2α, and the calculation formula (12) of the large sphere radius n is obtained according to the relationship between the two spheres;

[0037] (12);

[0038] And θ and α need to satisfy formula (13) to make the large sphere radius n>1;

[0039] (13).

[0040] Optionally, the width of the deformation image is set as W, and the height is set as H, formula (14) is obtained according to a projection algorithm when a cylindrical projection is adopted, and the horizontal field of view angle 2β of the virtual camera is obtained according to formula (14);

[0041] (14);

[0042] Formula (15) is obtained according to a projection algorithm when an equirectangular projection or equidistant projection is adopted, and the horizontal field of view angle 2β of the virtual camera is obtained according to formula (15);

[0043] (15).

[0044] Optionally, formula (16) is obtained according to the relationship between the two spheres and formula (14) or formula (15), and the horizontal field of view angle 2δ of the real camera is obtained according to formula (16);

[0045] (16);

[0046] Wherein, val is a temporary variable.

[0047] The virtual camera coordinate system O2-X2Y2Z2 is defined, wherein O is the center of a virtual small sphere with a radius of 1, and O2 is the center of a virtual large sphere with a radius of n, and the large sphere is inscribed in the small sphere, then the conversion of the fisheye image between the real camera coordinate system and the virtual camera coordinate system is realized by means of the two spheres, the projection of the fisheye image in the virtual camera coordinate system is performed, the correction degree of the line bending in the deformation image of the fisheye image is adjusted through the horizontal field of view angle or the vertical field of view angle of the virtual camera, the line bending degree in the deformation image is smaller under the premise of keeping the horizontal field of view angle or the vertical field of view angle of the real camera, and the visual effect of the fisheye image deformation is improved. Meanwhile, the field of view angle range that can be supported is 0 degrees to 360 degrees, and all fisheye lenses can be covered.

[0048] Further, the fisheye image deformation method based on the virtual sphere provided by the application supports cylindrical projection, equirectangular projection or equidistant projection, and can be combined with multiple projection methods, so that different deformation styles can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0049] Those skilled in the art should understand that the provided drawings are used to better understand the application, and do not constitute any limitation on the scope of the application.

[0050] Figure 1 is a schematic diagram of the position relationship between the real camera coordinate system and the virtual camera coordinate system provided by an embodiment of the application.

[0051] Figure 2 is Figure 1 schematic view in XOZ plane.

[0052] Figure 3 is Figure 1 schematic view in YOZ plane.

[0053] Figures 4a to 4d is a schematic view of a morphed image using cylindrical projection.

[0054] Figures 5a to 5d is a schematic view of a morphed image using equirectangular projection.

[0055] Figures 6a to 6d is a schematic view of a morphed image using equirectangular projection. DETAILED DESCRIPTION

[0056] In order to make the objects, advantages and features of the present application more clearly, the following further describes the present application in conjunction with the drawings and specific embodiments. It should be noted that the drawings are very simplified and not drawn to scale, and are only used to facilitate and clearly assist the purpose of describing the embodiments of the present application. In addition, the structures shown in the drawings are often a part of the actual structures. In particular, the emphasis of each drawing needs to be different, and sometimes different scales are used.

[0057] As used in the present application, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. The term "or" is generally employed in its sense of "and / or" unless the content clearly dictates otherwise. The term "at least two" is generally employed in its sense of "two or more" unless the content clearly dictates otherwise. In addition, the terms "first," "second," "third," etc. are used only to describe the objects and are not to be construed as indicating or implying relative importance or a specific number of the technical features indicated. Thus, the features defined as "first," "second," "third" can explicitly or implicitly include one or at least two of the features, unless the content clearly indicates otherwise.

[0058] The present application provides a virtual sphere-based fish-eye image morphing method, comprising:

[0059] defining a real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2, wherein O is the center of a virtual small sphere with a radius of 1, and O2 is the center of a virtual large sphere with a radius of n, and the large sphere is inscribed in the small sphere;

[0060] realizing the conversion of the fish-eye image between the real camera coordinate system and the virtual camera coordinate system by means of the two spheres, and projecting the fish-eye image in the virtual camera coordinate system; and

[0061] The degree of correction of the line bending in the deformation image of the fisheye image is adjusted by a horizontal field of view angle or a vertical field of view angle of the virtual camera.

[0062] Figure 1 is a schematic diagram of a positional relationship between a real camera coordinate system and a virtual camera coordinate system provided by an embodiment of the present application. As shown in Figure 1 The real camera coordinate system O-XYZ forms a virtual small sphere with O as the center and a radius of 1, and the virtual camera coordinate system O2-X2Y2Z2 forms a virtual large sphere with O2 as the center and a radius of n, and the large sphere and the small sphere are tangent at point C.

[0063] The coordinate conversion process of the fisheye image includes forward transformation and inverse transformation. The forward transformation includes obtaining the coordinates (x0, y0) of the fisheye image; obtaining the 3D unit vector (i0, j0, k0) in the real camera coordinate system according to the camera model; obtaining the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system according to the coordinate conversion between the large sphere and the small sphere; and obtaining the coordinates (x1, y1) in the deformation image according to the projection transformation.

[0064] In the forward transformation, the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system can be solved from the 3D unit vector (i0, j0, k0) in the real camera coordinate system according to formula (1), and the specific calculation process includes: first, formula (3) is obtained according to the relationship between the two spheres, then formula (4) is calculated according to formula (3), then formula (2) is calculated according to formula (4), and finally formula (1) is calculated according to formula (2), and each formula is as follows:

[0065] (3)

[0066] (4)

[0067] (2)

[0068] (1)

[0069] The inverse transformation includes obtaining the coordinates (x1, y1) in the deformation image; obtaining the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system according to the inverse projection transformation; obtaining the 3D unit vector (i0, j0, k0) in the real camera coordinate system according to the coordinate conversion between the large sphere and the small sphere; and obtaining the coordinates (x0, y0) of the fisheye image according to the camera model.

[0070] The method for solving the 3D unit vector (i0, j0, k0) in the real camera coordinate system from the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system in the inverse transformation comprises the following steps: firstly, obtaining the intersection point (i1*n, j1*n, k1*n) of the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system and the big sphere in the coordinate system O2-X2Y2Z2, for example, the P point in Figure 1 ; then obtaining the vector (i1*n, j1*n, k1*n+ (n-1)) of the intersection point (i.e. the P point) in the coordinate system O-XYZ; and finally normalizing the vector into a unit vector to obtain (i0, j0, k0).

[0071] In the forward transformation, the 3D unit vector (i0, j0, k0) in the real camera coordinate system is obtained according to the camera model, and the coordinates (x1, y1) in the morphed image are obtained according to the projection transformation; in the inverse transformation, the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system is obtained according to the inverse projection transformation, and the coordinates (x0, y0) of the fisheye image are obtained according to the camera model, which can be obtained or calculated by using the method known to those skilled in the art, and the present application does not make a specific description.

[0072] The horizontal field of view (HFOV) and the vertical field of view (VFOV) of the real camera, one of which is determined according to the needs of the customer, and the other of which is solved according to other parameters, for example, according to the horizontal field of view, the vertical field of view of the virtual camera or the radius of the big sphere and other parameters. The relationship between the horizontal field of view and the vertical field of view of the virtual camera is determined by the projection method and the aspect ratio of the morphed image, and the horizontal field of view or the vertical field of view of the virtual camera can be used to adjust the correction degree of the line bending of the morphed image, and the width and height of the morphed image are determined by the needs of the customer. In the embodiment, the projection comprises cylindrical projection, equirectangular projection or equidistant projection.

[0073] The following is an example of setting the horizontal field of view 2δ of the real camera, the horizontal field of view 2β of the virtual camera, to solve the vertical field of view 2θ of the real camera, the vertical field of view 2α of the virtual camera and the radius n of the big sphere.

[0074] Figure 2 is Figure 1 In the XOZ plane, please refer to Figure 2As shown in FIG. 1, the horizontal field of view angle of the real camera is set as 2δ, and the horizontal field of view angle of the virtual camera is set as 2β. When δ and β are both acute angles, according to the relationship between the two spheres, sinδ=HQ / HO, cosδ=QO / HO, sinβ=HQ / HO2, cosβ=QO2 / HO2, and QO2=QO+n-1 can be obtained, so formula (5) is obtained, and then formula (6) for calculating the radius n of the large sphere is obtained by formula (5). Formula (5) and formula (6) are as follows:

[0075] (5);

[0076] (6).

[0077] When δ and β are obtuse angles or right angles respectively, the above formula (5) and formula (6) can be obtained by the same method. In addition, δ and β need to satisfy formula (7) to make the radius n of the large sphere greater than 1;

[0078] (7)

[0079] Figure 3 is Figure 1 In the schematic view of YOZ plane, please refer to FIG. 2, Figure 3 As shown in FIG. 2, the vertical field of view angle of the real camera is set as 2θ, and the vertical field of view angle of the virtual camera is set as 2α. The width of the deformation image is set as W, and the height is set as H. According to the projection algorithm, formula (8) can be obtained when the cylindrical projection is adopted, and the vertical field of view angle 2α of the virtual camera is obtained according to formula (8);

[0080] (8);

[0081] According to the projection algorithm, formula (9) can be obtained when the equirectangular projection or equidistant projection is adopted, and the vertical field of view angle 2α of the virtual camera is obtained according to formula (9);

[0082] (9).

[0083] When θ and α are both acute angles, according to the relationship between the two spheres, sinθ=KF / KO, cosθ=FO / KO, sinα=KF / KO2, cosα=FO2 / KO2, and FO2=FO+n-1 can be obtained, so formula (10) is obtained, and formula (11) can be obtained according to formula (10) and formula (7) or formula (8), so the vertical field of view angle 2θ of the real camera is obtained. Formula (10) and (11) are as follows:

[0084] (10);

[0085] (11);

[0086] where val is a temporary variable. When θ and α are obtuse angles or right angles, respectively, the same method can be used to obtain the above formula (10) and formula (11).

[0087] The above embodiment is to solve the vertical field of view angle when the horizontal field of view angle is known, that is, the horizontal field of view angle 2δ of the real camera and the horizontal field of view angle 2β of the virtual camera are known, and the vertical field of view angle 2θ of the real camera and the vertical field of view angle 2α of the virtual camera and the large sphere radius n are calculated. When the vertical field of view angle is known, the horizontal field of view angle can be calculated by the same method. That is, the vertical field of view angle 2θ of the real camera and the vertical field of view angle 2α of the virtual camera are known, and the horizontal field of view angle 2δ of the real camera and the horizontal field of view angle 2β of the virtual camera and the large sphere radius n are calculated.

[0088] The vertical field of view angle of the real camera is set to 2θ, and the vertical field of view angle of the virtual camera is set to 2α. According to the relationship between the two spheres, the calculation formula (12) of the large sphere radius n is obtained;

[0089] (12);

[0090] And θ and α need to satisfy formula (13) to make the large sphere radius n>1;

[0091] (13).

[0092] The width of the deformation image is set to W, and the height is set to H. When cylindrical projection is used, formula (14) is obtained according to the projection algorithm, and the horizontal field of view angle 2β of the virtual camera is obtained according to formula (14);

[0093] (14);

[0094] When equirectangular projection or equidistant projection is used, formula (15) is obtained according to the projection algorithm, and the horizontal field of view angle 2β of the virtual camera is obtained according to formula (15);

[0095] (15).

[0096] According to the relationship between the two spheres and formula (14) or formula (15), formula (16) is calculated, and the horizontal field of view angle 2δ of the real camera is obtained according to formula (16);

[0097] (16);

[0098] where val is a temporary variable.

[0099] Compared with basic perspective projection method, cube projection method, cylindrical projection method, rectangular projection method and the like, the application adds the conversion between the two camera coordinate systems of the big sphere and the small sphere, so as to change the deformation effect. In the fish-eye image deformation method based on virtual sphere provided by the application, first, a real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2 are defined, wherein O is the center of a virtual small sphere with a radius of 1, and O2 is the center of a virtual big sphere with a radius of n, and the big sphere is inscribed in the small sphere; then the conversion of the fish-eye image between the real camera coordinate system and the virtual camera coordinate system is realized by means of the two spheres, the fish-eye image is projected in the virtual camera coordinate system, the correction degree of the line bending in the deformation image of the fish-eye image is adjusted through the horizontal field of view angle or the vertical field of view angle of the virtual camera, the line bending degree is smaller under the premise of keeping the horizontal field of view angle or the vertical field of view angle of the real camera, and the fish-eye image deformation visual effect is improved. Meanwhile, the supported field of view angle range is 0 degrees to 360 degrees, and all fish-eye lenses can be covered.

[0100] Further, the fish-eye image deformation method based on virtual sphere provided by the application supports cylindrical projection, equi-rectangular projection or equidistant projection, and can be used in combination with multiple projection methods, so as to realize different deformation styles.

[0101] Figures 4a to 4d is a schematic view of a deformation image obtained by using cylindrical projection. Wherein, Figure 4a is a schematic view of a deformation image obtained by using cylindrical projection. Wherein, Figure 4b is a schematic view of a deformation image obtained by using cylindrical projection. Wherein, Figure 4c is a schematic view of a deformation image obtained by using cylindrical projection. Wherein, Figure 4d is a deformation image obtained by using the prior art, and the horizontal field of view angle of the real camera is 140 degrees. Figures 4a to 4d It can be seen that, compared with the deformation image (i.e. Figure 4d ) of the prior art, after the virtual camera is set, the line bending degree in the deformation image is smaller, and the fish-eye image deformation visual effect is improved; and after the virtual camera is set, under the premise of keeping the horizontal field of view angle of the real camera, with the decrease of the horizontal field of view angle of the virtual camera, the line bending degree in the deformation image gradually becomes smaller, and the fish-eye image deformation visual effect is continuously improved. That is to say, Figure 4a has the best visual effect, and Figure 4d has the relatively worst visual effect.

[0102] Figures 5a to 5d is a schematic view of a deformation image obtained by using equi-rectangular projection. Wherein, Figure 5ais the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 80 degrees; Figure 5b is the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 100 degrees; Figure 5c is the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 120 degrees; Figure 5d is the morphed image obtained by using the prior art, and the horizontal field of view angle of the real camera is 140 degrees. Figures 5a to 5d It can be seen that, compared with the morphed image (i.e. Figure 5d ) of the prior art, after setting the virtual camera, the degree of line bending in the morphed image is smaller, and the visual effect of the morphed fisheye image is improved. Figure 5a The visual effect of is the best, and the visual angle effect of is relatively the worst. Figure 5d

[0103] Figures 6a to 6d is a schematic diagram of the morphed image obtained by using the equirectangular projection. Wherein, Figure 6a is the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 80 degrees; Figure 6b is the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 100 degrees; Figure 6c is the horizontal field of view angle of the real camera is 140 degrees, and the horizontal field of view angle of the virtual camera is 120 degrees; Figure 6d is the morphed image obtained by using the prior art, and the horizontal field of view angle of the real camera is 140 degrees. Figures 6a to 6d It can be seen that, compared with the morphed image (i.e. Figure 6d ) of the prior art, after setting the virtual camera, the degree of line bending in the morphed image is smaller, and the visual effect of the morphed fisheye image is improved. Figure 6a The visual effect of is the best, and the visual angle effect of is relatively the worst. Figure 6d

[0104] The fisheye image morphing method based on a virtual sphere provided by the application can be applied in OA798, OA805, OA8000, OA4600 and other related products containing a DCPC module chip, and only the coordinate mapping table generated by the method is used in firmware, and the code of the method is not embedded in the product.

[0105] ​​The fisheye image morphing method aims to generate a coordinate mapping table before and after transformation according to the customer's requirements for morphing, and the product loads the coordinate mapping table when the DCPC module is running, so as to realize the specified morphing function. Compared with the traditional fisheye image morphing method, the application can reduce the bending degree of the line under the premise of maintaining the field of view angle, and improve the visual effect of the image before and after morphing.

[0106] In summary, in the fisheye image morphing method based on virtual sphere provided by the application, first, a real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2 are defined, wherein O is the center of a virtual small sphere with a radius of 1, and O2 is the center of a virtual large sphere with a radius of n, and the large sphere is inscribed in the small sphere, then the conversion of the fisheye image between the real camera coordinate system and the virtual camera coordinate system is realized by means of the two spheres, the fisheye image is projected in the virtual camera coordinate system, and the correction degree of the line bending in the morphing image of the fisheye image is adjusted through the horizontal field of view angle or the vertical field of view angle of the virtual camera, so that the line bending degree in the morphing image is smaller under the premise of maintaining the horizontal field of view angle or the vertical field of view angle of the real camera, and the fisheye image morphing visual effect is improved. At the same time, the supported field of view angle range is 0-360 degrees, which can cover all fisheye lenses.

[0107] Further, the fisheye image morphing method based on virtual sphere provided by the application supports cylindrical projection, equirectangular projection or equidistant projection, and can be used in combination with multiple projection methods, so as to realize different morphing styles.

[0108] The above description is only a description of the preferred embodiments of the application, and does not limit the scope of the application in any way, and any modification or modification of the application by a person skilled in the art according to the above disclosure is within the protection scope of the claims.

Claims

1. A fisheye image deformation method based on a virtual sphere, characterized in that, The deformation method includes: Define a real camera coordinate system O-XYZ and a virtual camera coordinate system O2-X2Y2Z2, where a virtual small sphere is formed with O as the center and a radius of 1, and a virtual large sphere is formed with O2 as the center and a radius of n, and the large sphere is internally tangent to the small sphere. Two spheres are used to transform the fisheye image between the real camera coordinate system and the virtual camera coordinate system, and the fisheye image is projected into the virtual camera coordinate system; and, Adjust the degree of line curvature correction in the fisheye image by adjusting the horizontal or vertical field of view of the virtual camera; The coordinate transformation process for fisheye images includes: forward transformation and inverse transformation; The forward transformation includes: obtaining the coordinates (x0, y0) of the fisheye image; obtaining the 3D unit vector (i0, j0, k0) in the real camera coordinate system according to the camera model; obtaining the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system according to the coordinate transformation between the large sphere and the small sphere; and obtaining the coordinates (x1, y1) in the deformed image according to the projection transformation. The inverse transformation includes: obtaining the coordinates (x1, y1) in the deformed image; obtaining the 3D unit vector (i1, j1, k1) in the virtual camera coordinate system according to the inverse projection transformation; obtaining the 3D unit vector (i0, j0, k0) in the real camera coordinate system according to the coordinate transformation between the large sphere and the small sphere; and obtaining the coordinates (x0, y0) of the fisheye image according to the camera model.

2. The fisheye image deformation method based on a virtual sphere as described in claim 1, characterized in that, In the inverse transformation, the methods for obtaining (i0, j0, k0) from (i1, j1, k1) include: Obtain the coordinates (i1*n, j1*n, k1*n) of the intersection point of (i1, j1, k1) and the large sphere in the coordinate system O2-X2Y2Z2. Obtain the vector (i1*n, j1*n, k1*n+(n-1)) of the intersection point in the coordinate system O-XYZ; and The vector is normalized to a unit vector to obtain (i0, j0, k0).

3. The fisheye image deformation method based on a virtual sphere as described in claim 1, characterized in that, In the positive transformation, (i1, j1, k1) is obtained by solving (i0, j0, k0) according to formula (1); （1） in, (2).

4. The fisheye image deformation method based on a virtual sphere as described in claim 3, characterized in that, Formula (3) is obtained based on the relationship between the two spheres. Formula (4) is calculated based on Formula (3). Formula (2) and Formula (1) are then calculated based on Formula (4). （3） （4）。 5. The fisheye image deformation method based on a virtual sphere as described in claim 1, characterized in that, The projection includes cylindrical projection, rectangular projection, or equidistant projection; the relationship between the horizontal and vertical field of view of the virtual camera is determined by the projection method and the aspect ratio of the distorted image.

6. The fisheye image deformation method based on a virtual sphere as described in claim 5, characterized in that, Set the horizontal field of view of the real camera to 2δ and the horizontal field of view of the virtual camera to 2β. Based on the relationship between the two spheres, we get formula (5). Based on formula (5), we get the formula (6) for calculating the radius n of the large sphere. （5）； （6）； Furthermore, δ and β need to satisfy formula (7) so that the radius n of the large sphere is greater than 1; （7）。 7. The fisheye image deformation method based on a virtual sphere as described in claim 6, characterized in that, The width of the deformed image is set to W and the height to H. When using cylindrical projection, formula (8) is obtained according to the projection algorithm. The vertical field of view 2α of the virtual camera is obtained according to formula (8). （8）； When using rectangular projection or equidistant projection, formula (9) is obtained according to the projection algorithm, and the vertical field of view 2α of the virtual camera is obtained according to formula (9); （9）。 8. The fisheye image deformation method based on a virtual sphere as described in claim 7, characterized in that, Formula (10) is obtained based on the relationship between the two spheres. Formula (11) is obtained by calculating formula (11) with formula (8) or formula (9). The vertical field of view 2θ of the real camera is obtained based on formula (11). （10）； （11）； Here, val is a temporary variable.

9. The fisheye image deformation method based on a virtual sphere as described in claim 5, characterized in that, The vertical field of view of the real camera is set to 2θ, and the vertical field of view of the virtual camera is set to 2α. Based on the relationship between the two spheres, the formula for calculating the radius n of the larger sphere is obtained (12). （12）； Furthermore, θ and α need to satisfy formula (13) so that the radius n of the large sphere is greater than 1; （13）。 10. The fisheye image deformation method based on a virtual sphere as described in claim 9, characterized in that, The width of the deformed image is set to W and the height to H. When using cylindrical projection, formula (14) is obtained according to the projection algorithm. The horizontal field of view 2β of the virtual camera is obtained according to formula (14). （14）； When using rectangular projection or equidistant projection, formula (15) is obtained according to the projection algorithm, and the horizontal field of view 2β of the virtual camera is obtained according to formula (15); （15）。 11. The fisheye image deformation method based on a virtual sphere as described in claim 10, characterized in that, Formula (16) is obtained by calculating the relationship between the two spheres and formula (14) or formula (15), and the horizontal field of view 2δ of the real camera is obtained by formula (16). （16）； Here, val is a temporary variable.

Images