Fisheye image simulation method, system and equipment

Abstract

The invention relates to a fisheye image simulation method, system and equipment, and the method comprises the steps: obtaining the parameters of a fisheye camera, and the parameters of the fisheye camera comprise an output image resolution, a camera internal parameter, a fisheye distortion model parameter and a field angle; generating a fisheye image mapping table based on the parameters of the fisheye camera, the fisheye image mapping table storing sampling parameters of a fisheye camera lens surface corresponding to each pixel in an imaging plane of the fisheye camera; in the simulated three-dimensional scene, using a virtual camera to render a target scene, constructing a panoramic image, and mapping the panoramic image to a unit spherical surface; according to the sampling parameter corresponding to each pixel, sampling a corresponding point on the unit spherical surface to obtain color information of each pixel; and performing image synthesis based on the color information of each pixel to generate a simulated fisheye image of the target scene.

Description

Technical Field

[0001] The invention relates to a fisheye image simulation method, system and equipment, and belongs to the field of computer technology. Background Art

[0002] Fisheye image simulation is a technology that simulates the effect of fisheye lens shooting. Fisheye lens is an ultra-wide-angle lens with a field of view of more than 180 degrees. This lens has a significant distortion effect. Therefore, in order to make the simulated image more realistic, it is necessary to add the same distortion effect as that of real camera shooting to the simulated image.

[0003] Fisheye lenses are widely used in the field of environmental perception of autonomous vehicles. During the development and testing phase of autonomous driving algorithms, fisheye image simulation technology can simulate the environmental perception data obtained by autonomous vehicles in different scenarios, allowing developers to evaluate the performance and reliability of autonomous driving systems in a simulated environment without having to conduct dangerous and time-consuming tests on actual roads. In addition, in the field of machine learning, fisheye image simulation technology can also be used to generate diverse training data to improve the ability of intelligent agents to recognize and process complex environments.

[0004] The existing technology still lacks fisheye image simulation technology that can meet both accuracy and high real-time requirements, which makes it difficult to meet the needs of real-time application scenarios such as autonomous driving. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a fisheye image simulation method, system and device, which ensure the accuracy of the simulation results through camera parameters, and provide real-time simulated fisheye image generation in real-time rendering through a pre-built fisheye image mapping table, which can simultaneously meet the accuracy and high real-time requirements of fisheye image simulation.

[0006] According to a first aspect of the present invention, a fisheye image simulation method is provided, the method comprising:

[0007] Obtaining parameters of a fisheye camera, wherein the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters, and field of view angle;

[0008] Based on the parameters of the fisheye camera, a fisheye image mapping table is generated, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera;

[0009] In a simulated three-dimensional scene, a virtual camera is used to render a target scene, a panoramic image is constructed, and the panoramic image is mapped onto a unit sphere;

[0010] According to the sampling parameter corresponding to each pixel, sampling corresponding points on the unit sphere to obtain color information of each pixel;

[0011] Image synthesis is performed based on the color information of each pixel to generate a simulated fisheye image of the target scene.

[0012] According to one embodiment, the sampling parameter includes a three-dimensional unit vector of an incident angle of an incident light ray corresponding to the pixel; the incident angle is acquired according to an azimuth angle and a pitch angle of the incident light ray.

[0013] According to one embodiment, the sampling parameters further include whether the pixel exceeds the field of view angle boundary of the fisheye camera.

[0014] According to one embodiment, using a virtual camera to render a target scene and construct a panoramic image includes:

[0015] Six virtual cameras with a field of view angle of 90 degrees are used to render the target scene in six vertical directions of up, down, left, right, front and back to generate images of six surfaces, and the images of the six surfaces are synthesized into the panoramic image.

[0016] According to a second aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the fisheye image simulation method described in any one of the first aspects of the present invention.

[0017] According to a third aspect of the present invention, there is provided a fisheye image simulation device, comprising a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the fisheye image simulation method as described in any one of the first aspects of the present invention.

[0018] According to a fourth aspect of the present invention, a fisheye image simulation system is provided, the system comprising:

[0019] An input module, used to obtain parameters of a fisheye camera, wherein the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters, and field of view angle;

[0020] A mapping table generating module, used to generate a fisheye image mapping table based on the parameters of the fisheye camera, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera;

[0021] A panoramic image rendering module is used to render a target scene using a virtual camera in a simulated three-dimensional scene, construct a panoramic image, and map the panoramic image onto a unit sphere;

[0022] A sampling module, used for sampling corresponding points on the unit sphere according to the sampling parameters corresponding to each pixel, so as to obtain color information of each pixel;

[0023] The output module is used to perform image synthesis based on the color information of each pixel to generate a simulated fisheye image of the target scene.

[0024] According to one embodiment, the sampling parameter includes a three-dimensional unit vector of an incident angle of an incident light ray corresponding to the pixel; the incident angle is acquired according to an azimuth angle and a pitch angle of the incident light ray.

[0025] According to one embodiment, the sampling parameters further include whether the pixel exceeds the field of view angle boundary of the fisheye camera.

[0026] According to one embodiment, using a virtual camera to render a target scene and construct a panoramic image includes:

[0027] Six virtual cameras with a field of view angle of 90 degrees are used to render the target scene in six vertical directions of up, down, left, right, front and back to generate images of six surfaces, and the images of the six surfaces are synthesized into the panoramic image.

[0028] The present invention constructs a fisheye image mapping table through the parameters of a fisheye camera, and the mapping table stores the sampling parameters of the fisheye camera lens surface corresponding to each pixel in the imaging plane of the fisheye camera, so that the distortion characteristics of the fisheye camera actually used and the influence of the field angle boundary on the image are directly incorporated into the sampling parameters. In a simulated three-dimensional scene, a virtual camera is used to render the target scene in real time, a panoramic image is constructed and the panoramic image is mapped onto a unit sphere. Through a pre-constructed fisheye image mapping table, according to the sampling parameters corresponding to each pixel in the imaging plane of the fisheye camera, the sampling points of each pixel on the unit sphere are obtained, and the color information of each pixel can be obtained in real time and a simulated fisheye image of the target scene can be synthesized. The fisheye image simulation method of the present invention ensures the accuracy of the simulation result by calibrating the camera parameters, and provides real-time simulated fisheye image generation in real-time rendering through a pre-constructed fisheye image mapping table, which can simultaneously meet the accuracy and high real-time requirements of fisheye image simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart of the steps of a fisheye image simulation method provided by an embodiment of the present invention.

[0030] Figure 2 A flow chart of step S120 of a fisheye image simulation method provided by an embodiment of the present invention.

[0031] Figure 3 Schematic diagram of the fisheye projection model.

[0032] Figure 4 A schematic diagram of modules of a fisheye image simulation system provided by an embodiment of the present invention.

[0033] Figure 5 A structural schematic diagram of a fisheye image simulation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.

[0035] Technologies, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the above-mentioned technologies, methods and equipment should be considered as part of the specification.

[0036] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0037] like Figure 1 As shown, a fisheye image simulation method includes the following steps S110 to S150.

[0038] Step S110, obtaining parameters of the fisheye camera, the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters and field of view angle.

[0039] Among them, the image resolution determines the quality and clarity of the output image. The output image resolution of the fisheye camera includes: image horizontal resolution H and image vertical resolution V. The total number of pixels of the output image is the product of the image horizontal resolution and vertical resolution, that is, H*V.

[0040] The fisheye camera internal parameters are the parameters inherent in the camera, which determine how the camera projects points in three-dimensional space onto the two-dimensional image plane. The fisheye camera internal parameters include: x , c y ) and focal length (f x , f y ). The principal point is the intersection of the lens optical axis and the imaging plane, c x and c y Respectively represent the x-axis and y-axis coordinates of the principal point in the image coordinate system. Focal length is a parameter that describes the ability of a lens to focus light. Focal length is expressed as two components f x and f y , corresponding to the x-axis and y-axis of the image coordinate system respectively, these two components describe the focusing ability of the lens in the x-axis and y-axis directions.

[0041] Among them, the fisheye distortion model parameters are mathematical parameters used to describe the image distortion during the imaging process of the fisheye camera. Due to its ultra-wide-angle characteristics, fisheye lenses often produce serious image distortion, including barrel distortion and radial distortion. In order to correct these distortions, it is necessary to establish a mathematical model to describe the distortion law and calibrate the parameters of these models through experiments. The fisheye distortion model parameters include: k 1 , k 2 , k 3 , k 4 , these parameters are radial distortion coefficients, which are used to describe the offset of pixels at different positions in the image due to radial distortion. Radial distortion is the distortion distributed along the radius of the lens and is the main type of distortion in fisheye cameras.

[0042] The FOV (Field of View) of a fisheye camera refers to the angle formed by the two edges of the maximum range of the image of the object being measured that can be seen through the lens, with the optical center of the camera lens as the vertex. The FOV is used to measure the field of view that the camera can capture. The FOV of a fisheye camera is usually much larger than that of an ordinary camera, and may be close to or exceed 180 degrees.

[0043] The above fisheye camera parameters are determined by the model of the fisheye camera and are corrected through internal calibration to ensure that an accurate projection model from three-dimensional space to a two-dimensional image plane is established.

[0044] Step S120: generating a fisheye image mapping table based on the parameters of the fisheye camera, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera.

[0045] In some examples, such as Figure 2 As shown, S120 may include the following steps S121 to S124.

[0046] Step S121, constructing a fisheye image mapping table of the fisheye camera, the mapping table includes H*V rows and 4 columns, each row corresponds to a pixel in the imaging plane of the fisheye camera, and the 4 columns correspond to 4 parameters of each pixel.

[0047] Step S122, based on the parameters of the fisheye camera, obtaining the azimuth angle and the pitch angle of the incident light on the lens surface of the fisheye camera corresponding to each pixel in the imaging plane of the fisheye camera.

[0048] In one implementation, let the current pixel coordinates on the imaging plane be (u, v), firstly transform the pixel coordinates into the image coordinate system through the intrinsic matrix A.

[0049]

[0050] In the image coordinate system, the coordinates corresponding to the pixel are (x′, y′):

[0051] x′=(uc x ) / f x y′=(vc y ) / f y

[0052] The azimuth of the incident light can be obtained as

[0053]

[0054] Secondly, if Figure 3 The fisheye projection model shown in the figure shows that for the pixel distance from the optical center, r d The relationship between the pitch angle θ of the incident light of the fisheye camera is determined according to a certain projection function based on the fisheye distortion model. In this embodiment, r d The 5th-order Taylor series expansion approximation about θ is expressed as:

[0055] r d =θ(1+k 1 θ 2 +k 2 θ 4 +k 3 θ 6 +k 4 θ 8 )

[0056] but:

[0057]

[0058] For the pixel (u, v), the pitch angle θ of the incident light of the fisheye camera exists in the above one-dimensional high-order equation, which is solved using the numerical analysis method (Newton iteration method), assuming:

[0059]

[0060] f′(θ)=1+3k 1 γθ 2 +5k 2 θ 4 +7k 3 θ 6 +9k 4 θ 8

[0061] Newton iteration formula:

[0062] θ k+1 =θ k -f(θ) / f′(θ) (k=0,1,2,3...)

[0063] Among them, k is the number of iterations, when θ k+1 -θ k →0, that is, when the result converges, the pitch angle θ of the incident light corresponding to the pixel is solved. Similarly, the azimuth and pitch angle of the incident light corresponding to each pixel in the imaging plane of the fisheye camera can be obtained, and the direction of the incident light in three-dimensional space can be determined by these two angles.

[0064] Step S123, obtaining the three-dimensional unit vector of the incident angle of the incident light corresponding to each pixel in the imaging plane of the fisheye camera according to the azimuth angle and the pitch angle, and storing them as the first three parameters of the corresponding pixel in the fisheye image mapping table.

[0065] The incident angle refers to the angle between the incident light and the normal of the incident surface. When the azimuth and elevation of the incident light are known, the incident angle of the incident light can be calculated based on geometric relationships and trigonometric functions. Specifically, according to the azimuth and elevation of the incident light corresponding to each pixel in the imaging plane of the fisheye camera obtained in the above steps, the incident angle of the incident light corresponding to each pixel is calculated, the incident angle is converted into a three-dimensional unit vector, and the three components of the three-dimensional unit vector are stored as the first three parameters of the corresponding pixel in the fisheye image mapping table.

[0066] Step S124, judging whether each pixel in the imaging plane of the fisheye camera exceeds the boundary of the field angle of the fisheye camera according to the incident angle, and storing the judgment result as the fourth parameter of the corresponding pixel in the fisheye image mapping table.

[0067] Specifically, according to the incident angle of the incident light corresponding to each pixel in the imaging plane of the fisheye camera, the angle between the incident light and the normal of the imaging plane of the fisheye camera is calculated, and it is determined whether the angle exceeds the field angle boundary of the fisheye camera, and the determination result is stored as the fourth parameter of the corresponding pixel in the fisheye image mapping table.

[0068] Step S130: In the simulated three-dimensional scene, a virtual camera is used to render the target scene, a panoramic image is constructed, and the panoramic image is mapped onto a unit sphere.

[0069] The simulated 3D scene is a virtual environment composed of 3D models, lighting, materials, textures and other elements to simulate real scenes, including various objects and interactive elements. In the simulated 3D scene, the virtual camera is used to simulate the position and perspective of the camera in the real world. When using the virtual camera for rendering, the 3D rendering engine will calculate the 2D image from the camera's perspective based on the camera parameters and the information in the scene, using computer graphics algorithms and techniques.

[0070] In some examples, in a simulated three-dimensional scene, a real-time 3D rendering engine (such as Unreal Engine, Unity, etc.) is used, and six virtual cameras with a FOV of 90 degrees are used to render the target scene in six vertical directions of up, down, left, right, front and back, to generate images of six surfaces, which are synthesized into a panoramic Cube image, and each surface of the panoramic Cube image is mapped to a corresponding part of the unit sphere.

[0071] Step S140 , sampling corresponding points on the unit sphere according to the sampling parameters corresponding to each pixel in the imaging plane of the fisheye camera to obtain color information of each pixel.

[0072] In some examples, as described above, each pixel in the fisheye image mapping table corresponds to 4 sampling parameters, and the pixels within the field of view are determined as pixels to be sampled based on information about whether each pixel exceeds the field of view boundary of the fisheye camera. The specific position of the corresponding sampling point on the unit sphere is calculated based on the three component information of the three-dimensional unit vector corresponding to each pixel to be sampled, and the corresponding points on the unit sphere are sampled to obtain the color information of each pixel to be sampled.

[0073] It is worth mentioning that the range of sampling pixels is determined according to the field angle boundary of the actual fisheye camera, so that the generated simulated fisheye image can be closer to the shooting effect of a real camera.

[0074] Step S150, performing image synthesis based on the color information of each pixel in the imaging plane of the fisheye camera to generate a simulated fisheye image of the target scene.

[0075] The embodiment of the present invention constructs a fisheye image mapping table through the parameters of the fisheye camera, and the mapping table stores the sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera. In the simulated three-dimensional scene, a virtual camera is used to render the target scene in real time, a panoramic cube image is constructed and the panoramic cube image is mapped onto a unit sphere. Through the pre-constructed fisheye image mapping table, according to the sampling parameters corresponding to each pixel in the imaging plane of the fisheye camera, the sampling point of the pixel on the unit sphere is obtained, and fast sampling can be performed to obtain the color information of each pixel in real time and synthesize the simulated fisheye image of the target scene.

[0076] like Figure 4 As shown, an embodiment of the present invention further provides a fisheye image simulation system, the system comprising:

[0077] An input module 401 is used to obtain parameters of a fisheye camera, wherein the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters, and field of view angle;

[0078] A mapping table generating module 402, configured to generate a fisheye image mapping table based on the parameters of the fisheye camera, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens surface corresponding to each pixel in the imaging plane of the fisheye camera;

[0079] A panoramic image rendering module 403 is used to render a target scene using a virtual camera in a simulated three-dimensional scene, construct a panoramic image, and map the panoramic image onto a unit sphere;

[0080] The sampling module 404 is used to sample corresponding points on the unit sphere according to the sampling parameters corresponding to each pixel in the imaging plane of the fisheye camera to obtain color information of each pixel;

[0081] The output module 405 is used to perform image synthesis based on the color information of each pixel in the imaging plane of the fisheye camera to generate a simulated fisheye image of the target scene.

[0082] In some examples, the sampling parameters of the fisheye camera lens plane include a three-dimensional unit vector of an incident angle of an incident light ray corresponding to a pixel, wherein the incident angle is obtained according to an azimuth angle and a pitch angle of the incident light ray.

[0083] In some examples, the sampling parameters of the fisheye camera lens plane also include whether the pixel exceeds the field of view angle boundary of the fisheye camera.

[0084] In some examples, a virtual camera is used to render a target scene and construct a panoramic image, including:

[0085] Six virtual cameras with a field of view of 90 degrees are used to render the target scene in six vertical directions: up, down, left, right, front and back, to generate images of six surfaces, which are then synthesized into a panoramic image.

[0086] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and the computer program implements any of the methods in the aforementioned embodiments of the present invention when executed by a processor. Optionally, the computer-readable storage medium may be a non-transitory storage medium, but is not limited to this, and it may also be a temporary storage medium. A computer-readable storage medium is a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media may include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above.

[0087] like Figure 5 As shown, the embodiment of the present invention also provides a fisheye image simulation device, such as Figure 5As shown, the device 500 may include a memory 501 and a processor 502. The memory 501 may be used to store computer instructions, and the processor 502 may be used to call computer instructions from the memory 501 to execute all or part of the steps of any method in the aforementioned embodiments of the present invention. The processor may be one or more, and the one or more processors may execute instructions individually or together, and the memory may be one or more, and the one or more memories may store the above-mentioned computer instructions individually or together.

[0088] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the methods, systems, devices and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions. It should be noted that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0089] It should be understood that the above is only an exemplary embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Those skilled in the art can easily conceive of modifications, replacements or adaptive changes without departing from the principle and essence of the present invention, which should all be covered within the protection scope of the present invention.

Claims

1. A fisheye image simulation method, characterized in that: include: Obtaining parameters of a fisheye camera, wherein the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters, and field of view angle; Based on the parameters of the fisheye camera, a fisheye image mapping table is generated, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera; In a simulated three-dimensional scene, a virtual camera is used to render a target scene, a panoramic image is constructed, and the panoramic image is mapped onto a unit sphere; According to the sampling parameter corresponding to each pixel, sampling corresponding points on the unit sphere to obtain color information of each pixel; Image synthesis is performed based on the color information of each pixel to generate a simulated fisheye image of the target scene.

2. The fisheye image simulation method according to claim 1, characterized in that: The sampling parameters include a three-dimensional unit vector of an incident angle of an incident light ray corresponding to the pixel; the incident angle is acquired according to an azimuth angle and a pitch angle of the incident light ray.

3. The fisheye image simulation method according to claim 1, characterized in that: The sampling parameters also include whether the pixel exceeds the field of view angle boundary of the fisheye camera.

4. The fisheye image simulation method according to claim 1, characterized in that: The method of using a virtual camera to render the target scene and construct a panoramic image includes: Six virtual cameras with a field of view angle of 90 degrees are used to render the target scene in six vertical directions of up, down, left, right, front and back to generate images of six surfaces, and the images of the six surfaces are synthesized into the panoramic image.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is performed.

6. A fisheye image simulation device, characterized in that: The system comprises a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call the computer instructions from the memory to execute the method according to any one of claims 1 to 4.

7. A fisheye image simulation system, characterized in that: include: An input module, used to obtain parameters of a fisheye camera, wherein the parameters of the fisheye camera include output image resolution, camera intrinsic parameters, fisheye distortion model parameters, and field of view angle; A mapping table generating module, used to generate a fisheye image mapping table based on the parameters of the fisheye camera, wherein the fisheye image mapping table stores sampling parameters of the fisheye camera lens plane corresponding to each pixel in the imaging plane of the fisheye camera; A panoramic image rendering module is used to render a target scene using a virtual camera in a simulated three-dimensional scene, construct a panoramic image, and map the panoramic image onto a unit sphere; A sampling module, used for sampling corresponding points on the unit sphere according to the sampling parameters corresponding to each pixel, so as to obtain color information of each pixel; The output module is used to perform image synthesis based on the color information of each pixel to generate a simulated fisheye image of the target scene.

8. The fisheye image simulation system according to claim 7, characterized in that: The sampling parameters include a three-dimensional unit vector of an incident angle of an incident light ray corresponding to the pixel; the incident angle is acquired according to an azimuth angle and a pitch angle of the incident light ray.

9. The fisheye image simulation system according to claim 7, characterized in that: The sampling parameters also include whether the pixel exceeds the field of view angle boundary of the fisheye camera.

10. The fisheye image simulation system according to claim 7, characterized in that: The method of using a virtual camera to render the target scene and construct a panoramic image includes: Six virtual cameras with a field of view angle of 90 degrees are used to render the target scene in six vertical directions of up, down, left, right, front and back to generate images of six surfaces, and the images of the six surfaces are synthesized into the panoramic image.

Images