A fisheye camera model construction method and system based on virtual engine

Abstract

The present invention provides a method and system for constructing a fisheye camera model based on a virtual engine, the method comprising: constructing a multi-camera model, matching rendering targets for each camera in the multi-camera model respectively; constructing multiple UV spherical patches, and splicing multiple UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; constructing a fisheye camera model, the field of view of the fisheye camera model is adapted to the UV mesh body, so as to obtain a fisheye image of the UV mesh body. The present invention allows 3D rendering software similar to UE to capture the problem of images with a field of view angle of more than 180°, and the image has a fisheye distortion effect, so that the camera used by games or simulation software can be closer to the real effect.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle simulation, and more specifically, to a method and system for constructing a fisheye camera model based on a virtual engine. Background Art

[0002] With the rapid development of autonomous driving technology, vehicle simulation testing technology plays a vital role in the development and verification of autonomous driving systems. Simulation testing technology can conduct a large number of tests on autonomous driving algorithms in a safe and controllable virtual environment, reduce the risks and costs of actual road testing, and improve development efficiency and system reliability.

[0003] In vehicle simulation, building camera sensors based on simulation software (such as virtual engine UE5, Unreal Engine5) is an essential process in building simulation models. The implementation method of a simulated fisheye camera is usually to use a camera to capture the scene, and then post-process the scene to generate a distortion effect. Due to the limitations of the rendering engine, this method has the problem of not being able to capture images with a field of view of more than 180°, and there is no fisheye distortion effect. However, on real vehicles, the field of view of a surround fisheye camera is approximately between 140° and 250°, so a solution is needed to solve the problem that the field of view cannot be expanded to more than 180° when building a model. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a method and system for constructing a fisheye camera model based on a virtual engine, so as to solve the problem that a simulated fisheye camera in a simulated environment cannot expand the field of view angle to greater than 180°.

[0005] According to a first aspect of the present invention, a method for constructing a fisheye camera model based on a virtual engine is provided, comprising: Constructing a multi-camera model, and matching a rendering target to each camera in the multi-camera model; Constructing a plurality of UV spherical patches, and splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; A fisheye camera model is constructed, wherein the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0006] Based on the above technical solution, the present invention can also make the following improvements.

[0007] Optionally, the constructing a multi-camera model and matching a rendering target for each camera in the multi-camera model respectively includes: Construct multiple camera models, set one of the camera models in the center, and arrange the other camera models around the camera model in the center, and set the projection type of each camera model to perspective projection; Construct a plurality of rendering targets, and bind the plurality of rendering targets to the plurality of camera models in a one-to-one correspondence, so that the rendering targets receive input data of the camera models.

[0008] Optionally, in the multi-camera model, any two adjacent cameras are arranged perpendicular to each other.

[0009] Optionally, constructing a plurality of UV spherical patches includes: Constructing a spherical surface, and respectively intercepting spherical surface patches corresponding to the size ratios of the respective rendering targets from the spherical surface; UV is set on the inner concave surface of each spherical patch to obtain a UV spherical patch.

[0010] Optionally, the step of splicing a plurality of the UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body includes: A plurality of static meshes matching the field of view of each camera in the multi-camera model are constructed, and the plurality of UV spherical patches are imported into the plurality of static meshes in a one-to-one correspondence to obtain a spliced ​​UV mesh.

[0011] Optionally, rendering the UV mesh volume using the rendering target includes: According to the correspondence between the rendering targets and the spherical patches, the multiple rendering targets are used as materials and mapped one-to-one to the UV mesh volume to render the UV mesh volume.

[0012] Optionally, the constructing of a fisheye camera model, wherein the field of view of the fisheye camera model is adapted to the UV grid body, so as to obtain a fisheye image of the UV grid body, includes: A fisheye camera model is constructed within the UV grid body so that the field of view of the fisheye camera model covers the UV grid body; and the fisheye camera model is used to record the image of the UV grid body and form an image or video stream.

[0013] According to a second aspect of the present invention, a fisheye camera model construction system based on a virtual engine is provided, comprising: A multi-camera construction module, used to construct a multi-camera model and match a rendering target to each camera in the multi-camera model; A mesh body construction module, used for constructing a plurality of UV spherical patches, splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; The fisheye camera construction module is used to construct a fisheye camera model so that the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0014] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the processor is configured to implement the steps of the above-mentioned method for constructing a fisheye camera model based on a virtual engine when executing a computer management program stored in the memory.

[0015] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the above-mentioned method for constructing a fisheye camera model based on a virtual engine are implemented.

[0016] The present invention provides a method, system, electronic device and storage medium for constructing a fisheye camera model based on a virtual engine. The multi-camera model constructed collects images of multiple fields of view and performs splicing and mapping to obtain a fisheye image with a large field of view angle. The fisheye image is directly recorded by a fisheye camera, which solves the problem that the simulated fisheye camera in a simulation environment such as UE5 cannot expand the field of view angle to more than 180°. The obtained image has a fisheye distortion effect, which can make the camera used in games or simulation software closer to the real effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a method for constructing a fisheye camera model based on a virtual engine provided by the present invention; Figure 2 A schematic diagram of a multi-camera model constructed in a certain embodiment; Figure 3 A schematic diagram of a spherical patch constructed in a certain embodiment; Figure 4 A schematic diagram of the effect of obtaining a fisheye image through a fisheye camera model in a certain embodiment; Figure 5 A block diagram of a fisheye camera model building system based on a virtual engine provided by the present invention; Figure 6 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention; Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0018] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0019] Figure 1 A flow chart of a method for constructing a fisheye camera model based on a virtual engine provided by the present invention is as follows: Figure 1 As shown, the method includes steps S1 to S3: S1, constructing a multi-camera model, and matching a rendering target for each camera in the multi-camera model; S2, constructing a plurality of UV spherical patches, and splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; S3, constructing a fisheye camera model, wherein the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0020] It is understandable that, based on the defects in the background technology, an embodiment of the present invention proposes a method for constructing a fisheye camera model based on a virtual engine. In the past, due to the rendering architecture, the camera field of view of 3D rendering software often could not exceed 180°, and there was no fisheye distortion effect. This embodiment collects images of multiple fields of view through the constructed multi-camera model and stitches and maps them to obtain a fisheye image with a large field of view angle range, and uses a fisheye camera to directly record the fisheye image, which solves the problem that the simulated fisheye camera in the simulation environment such as UE5 cannot expand the field of view to more than 180°, and the obtained image has a fisheye distortion effect, which can make the camera used in the game or simulation software closer to the real effect.

[0021] In a possible implementation manner, in step S1, constructing a multi-camera model and matching a rendering target for each camera in the multi-camera model respectively include steps S101-S102: S101, construct multiple camera models, set one of the camera models in the center, and arrange the other camera models around the camera model in the center, and set the projection type of each camera model to perspective projection; preferably, in the multi-camera model, any two adjacent cameras are arranged perpendicular to each other.

[0022] In order to explain the present invention more intuitively, Figure 2 As shown, the following embodiments use 5 cameras to build a multi-camera model as an example. The field of view of a single camera is 90°. One camera is located in the front, and the remaining 4 cameras are arranged in the upper, lower, left, and right directions around the camera in the front, so that any two adjacent cameras remain vertical. The following embodiments use UE5 software as an example. Figure 2 The total field of view of the multi-camera model shown can be greater than 180°.

[0023] In the specific operation, in the virtual engine UE5, generate an actor. The actor specifically refers to the class object that can be placed in the UE scene. Bind five cameras to the root node of the actor. The capture directions of the five cameras are up, down, left, right, and center. The field of view of the five cameras is set to 90° respectively, and the projection of the five cameras is set to perspective projection.

[0024] S102, constructing a plurality of rendering targets, and binding the plurality of rendering targets to the plurality of camera models in a one-to-one correspondence, so that the rendering targets receive input data of the camera models.

[0025] During the specific operation in UE5, 5 cameras are bound to 5 rendering targets respectively. Preferably, the size x and size y of the resolution of each rendering target are consistent, that is, the horizontal pixel size and the vertical pixel size of the rendering target are consistent, so that the edge of the rendering target is square.

[0026] In a possible embodiment, in step S2, constructing a plurality of UV spherical patches includes: S201, constructing a spherical surface, and respectively intercepting spherical surface patches corresponding to the size ratios of the respective rendering targets from the spherical surface; UV is set on the inner concave surface of each spherical patch to obtain a UV spherical patch.

[0027] When operating in the simulation software, use 3D software to create a cube, and each face of the cube corresponds to the size ratio of the rendering target, for example, the edge of the rendering target is square. Subdividing the surface of the cube can convert a cube into a sphere, and the 6 vertices of the cube are located on the spherical surface of the sphere. Based on the cube, the spherical surface can be divided into six patches, corresponding to the 6 faces of the cube. Keep one patch corresponding to the field of view of a camera in the multi-camera model, delete the remaining 5 patches, and you will get a Figure 3 The spherical patch shown. Set the UV of the inner concave surface of the retained spherical patch. Since the spherical patch is subdivided from the surface of a cube, the UV of its inner concave surface corresponds to a square projection. Import the obtained UV spherical patch into UE5. Repeat the above steps until you get 5 UV spherical patches corresponding to the field of view of each camera in the multi-camera model.

[0028] In a possible implementation manner, in step S2, stitching a plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh volume includes: S202, constructing a plurality of static meshes that match the field of view of each camera in the multi-camera model, and importing the plurality of UV spherical patches into the plurality of static meshes in a one-to-one correspondence to obtain a spliced ​​UV mesh.

[0029] During the specific operation in the simulation software, a second actor is generated in UE5, and five static mesh components are bound to the root node of the second actor. The five static mesh components are placed one by one at the top, bottom, left, right, and center according to the field of view of each camera in the multi-camera model, and the five static mesh components are all set to the UV spherical patches imported in step S201. The five UV spherical patches in step S201 are assembled into the second actor to obtain a spliced ​​UV mesh.

[0030] In a possible embodiment, in step S2, rendering the UV mesh volume using the rendering target includes: S203 , according to the correspondence between the rendering targets and the spherical patches, use the multiple rendering targets as materials and map them one-to-one to the UV mesh volume to render the UV mesh volume.

[0031] In the specific operation, the five rendering targets set in step S102 are mapped as materials to the UV mesh in UE5, so that the images captured by the five cameras in the multi-camera model are merged into the second actor for overall display.

[0032] In a possible embodiment, in step S3, constructing a fisheye camera model, wherein the field of view of the fisheye camera model is adapted to the UV grid body, so as to obtain a fisheye image of the UV grid body, includes: A fisheye camera model is constructed within the UV grid body so that the field of view of the fisheye camera model covers the UV grid body; and the fisheye camera model is used to record the image of the UV grid body and form an image or video stream.

[0033] It can be understood that after merging the images captured by the five cameras into the second actor, a camera (i.e., the fisheye camera model) is set in the second actor to capture the image composed of the five static meshes, and a fisheye image with a fisheye distortion effect can be obtained, and the field of view angle exceeds 180°. The effect of generating a fisheye image is as follows: Figure 5 shown.

[0034] Preferably, in step S3, the x-axis and / or y-axis size scaling of the fisheye camera model in the second actor can be modified. When the x-axis scaling and / or y-axis scaling are modified, the fisheye camera intrinsic parameters and distortion parameters will also change accordingly. According to the scaling ratio of the x-axis and y-axis, the camera intrinsic parameters and camera distortion parameters under the current scaling condition can be derived, and the relationship between the x-axis and y-axis and the camera intrinsic parameters and distortion parameters is recorded.

[0035] Then, a vehicle model is set up, and multiple fisheye camera models in step S3 are bound to the vehicle model. The fisheye images recorded by the fisheye camera model are used as a camera video stream channel. The video channel can be transmitted to the hardware under test using hardware devices such as optical fiber and HDMI.

[0036] Figure 5 A structure diagram of a fisheye camera model building system based on a virtual engine provided by an embodiment of the present invention, such as Figure 5 As shown, a fisheye camera model construction system based on a virtual engine includes a multi-camera construction module, a mesh body construction module and a fisheye camera construction module, wherein: A multi-camera construction module, used to construct a multi-camera model and match a rendering target to each camera in the multi-camera model; A mesh body construction module, used for constructing a plurality of UV spherical patches, splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; The fisheye camera construction module is used to construct a fisheye camera model so that the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0037] It can be understood that the fisheye camera model construction system based on a virtual engine provided by the present invention corresponds to the fisheye camera model construction method based on a virtual engine provided in the aforementioned embodiments. The relevant technical features of the fisheye camera model construction system based on a virtual engine can refer to the relevant technical features of the fisheye camera model construction method based on a virtual engine, which will not be repeated here.

[0038] See also Figure 6 , Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, the following steps are implemented: Constructing a multi-camera model, and matching a rendering target to each camera in the multi-camera model; Constructing a plurality of UV spherical patches, and splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; A fisheye camera model is constructed, wherein the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0039] See also Figure 7 , Figure 7 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 7 As shown, this embodiment provides a computer-readable storage medium 700, on which a computer program 711 is stored. When the computer program 711 is executed by a processor, the following steps are implemented: Constructing a multi-camera model, and matching a rendering target to each camera in the multi-camera model; Constructing a plurality of UV spherical patches, and splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; A fisheye camera model is constructed, wherein the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

[0040] The embodiments of the present invention provide a method, system, electronic device and storage medium for constructing a fisheye camera model based on a virtual engine. The multi-camera model constructed collects images of multiple fields of view and performs stitching and mapping to obtain a fisheye image with a large field of view angle. The fisheye image is directly recorded using a fisheye camera, which solves the problem that the simulated fisheye camera in a simulation environment such as UE5 cannot expand the field of view angle to more than 180°. The obtained image has a fisheye distortion effect, which can make the camera used in games or simulation software closer to the real effect.

[0041] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0042] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0044] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0046] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0047] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for constructing a fisheye camera model based on a virtual engine, characterized in that: include: Constructing a multi-camera model, and matching a rendering target to each camera in the multi-camera model; Constructing a plurality of UV spherical patches, and splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; A fisheye camera model is constructed to obtain a fisheye image of the UV mesh body.

2. The method for constructing a fisheye camera model based on a virtual engine according to claim 1, characterized in that: The step of constructing a multi-camera model and matching a rendering target to each camera in the multi-camera model includes: Construct multiple camera models, set one of the camera models in the center, and arrange the other camera models around the camera model in the center, and set the projection type of each camera model to perspective projection; Construct a plurality of rendering targets, and bind the plurality of rendering targets to the plurality of camera models in a one-to-one correspondence, so that the rendering targets receive input data of the camera models.

3. The method for constructing a fisheye camera model based on a virtual engine according to claim 2, characterized in that: In the multi-camera model, any two adjacent cameras are arranged perpendicular to each other.

4. A method for constructing a fisheye camera model based on a virtual engine according to any one of claims 1 to 3, characterized in that: The constructing of multiple UV spherical patches includes: Constructing a spherical surface, and respectively intercepting spherical surface patches corresponding to the size ratios of the respective rendering targets from the spherical surface; UV is set on the inner concave surface of each spherical patch to obtain a UV spherical patch.

5. The method for constructing a fisheye camera model based on a virtual engine according to claim 4, characterized in that: The step of splicing a plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body comprises: A plurality of static meshes matching the field of view of each camera in the multi-camera model are constructed, and the plurality of UV spherical patches are imported into the plurality of static meshes in a one-to-one correspondence to obtain a spliced ​​UV mesh.

6. The method for constructing a fisheye camera model based on a virtual engine according to claim 5, characterized in that: The using the rendering target to render the UV mesh volume includes: According to the correspondence between the rendering targets and the spherical patches, the multiple rendering targets are used as materials and mapped one-to-one to the UV mesh volume to render the UV mesh volume.

7. The method for constructing a fisheye camera model based on a virtual engine according to claim 1, characterized in that: The constructing of a fisheye camera model, wherein the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body, includes: A fisheye camera model is constructed within the UV grid body so that the field of view of the fisheye camera model covers the UV grid body; and the fisheye camera model is used to record the image of the UV grid body and form an image or video stream.

8. A fisheye camera model construction system based on a virtual engine, characterized in that: include: A multi-camera construction module, used to construct a multi-camera model and match a rendering target to each camera in the multi-camera model; A mesh body construction module, used for constructing a plurality of UV spherical patches, splicing the plurality of UV spherical patches according to the field of view of each camera in the multi-camera model to obtain a UV mesh body, and rendering the UV mesh body using the rendering target; The fisheye camera construction module is used to construct a fisheye camera model so that the field of view of the fisheye camera model is adapted to the UV grid body to obtain a fisheye image of the UV grid body.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the processor is used to implement the steps of the method for constructing a fisheye camera model based on a virtual engine as described in any one of claims 1 to 7 when executing a computer management program stored in the memory.

10. A computer-readable storage medium, characterized in that: A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of the method for constructing a fisheye camera model based on a virtual engine as described in any one of claims 1 to 7 are implemented.

Images