A spatial curved screen projection visual mapping and simulation method, electronic device and storage medium

Abstract

The present invention discloses a spatial curved screen projection visual mapping and simulation method, an electronic device and a storage medium, the method comprising the following steps: building a simulation device for panoramic visual perception perspective mapping of an automobile; data collection; projection image processing and correction; mapping a virtual camera with a real camera. The present invention simulates the panoramic perspective imaging of a virtual automobile and projects the image onto each curved screen, collects the curved screen image through the real camera of the automobile, and then establishes data mapping from the real camera to the virtual camera, thereby obtaining the virtual camera imaging effect, and then generates multiple virtual camera images, realizing the function of spatial three-dimensional perception in a virtual scene, and achieving the effect of simulating the panoramic visual perception of an automobile in a simulation environment. The method has the characteristics of low cost, high precision, and strong versatility, and is of great significance to the improvement and improvement of the software and hardware systems of automobile automatic driving; by designing curved screens of different sizes and shapes, imaging test requirements in different scenes can be met.

Description

Technical Field

[0001] The invention belongs to the field of optoelectronic testing technology, and in particular relates to a spatial curved screen projection visual mapping and simulation method, an electronic device and a storage medium. Background Art

[0002] Ground testing provides a verification platform for testing the various functions of the unmanned driving simulation system. Through testing, the control and safety of the unmanned driving system can be verified, and technical optimization can be carried out. Simulation testing is a more advanced testing method that simulates various driving scenarios and road conditions through computers and conducts a large number of tests and data analysis. With the rapid development of technologies such as machine vision and optical display, they are widely used in the automated testing of unmanned driving with their advantages of non-contact, high precision, and large range, effectively improving the efficiency and quality of detection. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, the purpose of the present invention is to provide a spatial curved screen projection visual mapping and simulation method, an electronic device and a storage medium.

[0004] In order to achieve the above purpose and the above technical effect, the technical solution adopted by the present invention is:

[0005] A spatial curved screen projection visual mapping and simulation method comprises the following steps:

[0006] Step 1: Build a car panoramic visual perception perspective mapping simulation device;

[0007] Step 2: Start the simulation video playback and collect data;

[0008] Step 3: Projection image processing and correction;

[0009] Step 4: Map the virtual camera to the real camera, thereby completing the mapping of the real camera image to the virtual camera image.

[0010] Furthermore, in step 2, the steps of starting the simulation video playback and collecting data include:

[0011] The computer of the car panoramic visual perception perspective mapping simulation device simulates the virtual car panoramic perspective imaging and projects the image onto each curved screen. The cameras of the car to be tested respectively image the curved screen projected by the corresponding virtual camera; the simulation video playback is started, and real-time data collection is performed through the real cameras of the car.

[0012] Furthermore, in step 3, the projected image processing and correction steps include:

[0013] Each real camera is calibrated in the early stage to obtain the internal parameters of each camera and use them as system parameter configuration;

[0014] Let the undistorted feature point image coordinates be p = [u, v, 1] T , the unit is pixel, and its normalized coordinate is p n =[x n ,y n ,1] T , the unit is millimeter; let q = [x, y, z, 1] T is the coordinate of the feature point in the target coordinate system; according to the pinhole imaging model, we have:

[0015]

[0016] Among them, ρ is a non-zero coefficient, is the camera internal parameter matrix, u0, v0 are the image plane principal point coordinates, f x 、f y are the proportional coefficients in the u and v directions respectively, γ is the tilt factor of the two axes of the image, R = [r1 r2 r3] 3×3 and t are the rotation matrix and translation vector between the target coordinate system and the camera coordinate system respectively;

[0017] Let p d =[u d ,v d ,1] T is the homogeneous coordinate of the distorted image point, p n =[x n y n ,1] T To normalize the homogeneous coordinates of the image points, the camera distortion model is expressed as:

[0018]

[0019] in, k1 and k2 are the radial distortion coefficients of the lens;

[0020] After the real camera collects images of various curved screens, it is substituted into the camera de-distortion model to obtain a distortion-free image.

[0021] Furthermore, in step 4, the steps of mapping the virtual camera to the real camera, thereby completing the mapping of the real camera image to the virtual camera image include:

[0022] Based on the virtual camera images collected by the real camera in step 3, the mapping chain is used to complete the mapping from the real camera image to the virtual camera image, which includes the following four steps:

[0023] Step 1: Virtual camera projection and curved screen mapping;

[0024] Step 2: Calibration of curved screen and real camera external parameters;

[0025] Step 3: transform the image captured by the real camera to the curved screen image;

[0026] Step 4: The curved screen image is transformed into the virtual camera image.

[0027] Furthermore, the configuration parameters of the virtual camera are set according to the simulation environment, including the focal length f of the virtual camera. v , resolution w v ×h v ; The configuration parameters of the curved screen include resolution w s ×h s , the pixel size is s, the unit is mm.

[0028] Furthermore, in the second step, the steps of calibrating the curved screen and the real camera external parameters include:

[0029] First, the checkerboard target is projected onto the curved screen, and the real camera collects the curved screen image. The curved screen checkerboard target point coordinate sequence q is obtained by aiming and measuring with a laser tracker. The ideal checkerboard point sequence is Q, and the checkerboard image point coordinate p collected by the real camera is obtained according to the mapping relationship between the two. gs (Q), where H gs Represents the quadratic mapping equation of the space surface, which is F(x,y,z)=0,p=H sc Q, H sc Represents the transformation matrix between the ideal chessboard and the real camera acquisition.

[0030] Furthermore, in the third step, the step of transforming the image captured by the real camera into the image on the curved screen includes:

[0031] When the scene image I projected by the virtual camera corresponding to the curved screen is collected r When the image data I corresponding to the curved screen is obtained according to the calibration parameters in the second step s , and both satisfy

[0032] Furthermore, in the fourth step, the step of transforming the curved screen image into the virtual camera image includes:

[0033] According to the curved screen image I s Imaging with Virtual Camera I v The transformation relationship is transformed to obtain virtual imaging, and the two satisfy the mapping relationship from the space surface to the virtual plane, which is expressed as So far, I have been established r To I v The transformation chain is obtained by transforming the image acquired by the actual real camera to obtain the corresponding virtual camera image Iv .

[0034] The present invention also discloses an electronic device, comprising:

[0035] at least one processor;

[0036] a memory communicatively coupled to the at least one processor;

[0037] The memory stores a computer program executable by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the spatial curved screen projection visual mapping and simulation method as described above.

[0038] The present invention also discloses a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the spatial curved screen projection visual mapping and simulation method as described above is implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention is based on the principle of photoelectric testing. It simulates the panoramic perspective imaging of a virtual car and projects the image onto each curved screen, collects the curved screen image through the real camera of the car, and then establishes data mapping from the real camera to the virtual camera, so as to obtain the virtual camera imaging effect, and then generate multi-channel virtual camera images, thereby realizing the function of spatial three-dimensional perception in a virtual scene, and achieving the effect of simulating the panoramic visual perception of the car in a simulation environment. It can be used to produce a data set for automatic driving of a car, evaluate the automatic driving algorithm of a car and other demand scenarios, and has the characteristics of low cost, high precision and strong versatility, and is of great significance to the enhancement and improvement of the software and hardware systems of automatic driving of a car.

[0041] The present invention uses the real camera of the car to collect images projected by the virtual camera onto the curved screen, constructs transformation relationship chains from the virtual camera to the curved screen, and from the curved screen to the real camera, and then obtains the image generated by the virtual camera through the image collected by the real camera of the car, thereby completing the function of the real camera of the car to obtain video data of the simulated scene, and providing an algorithmic basis for evaluating the algorithm of the car's automatic driving system.

[0042] The present invention can achieve one-to-one mapping between a real camera of a car and a camera in a virtual scene by designing curved screens of different sizes and shapes, and can meet imaging test requirements in different scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0045] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0046] A spatial curved screen projection visual mapping and simulation method comprises the following steps:

[0047] Step 1: Construction of the car panoramic visual perception perspective mapping simulation device

[0048] like Figure 1 As shown, the car panoramic visual perception perspective mapping simulation device includes a real camera 1, a display 2 and a computer, wherein the real camera 1 is a panoramic camera, the display 2 is a curved screen, and the real camera 1 and the display 2 are respectively connected to the computer. The simulated car of the present invention has a working mode of 8 sets of panoramic cameras, and the number and installation positions of the cameras are consistent with the number and installation positions of the cameras on the real car. By simulating the parameters and imaging of multiple cameras, each imaging image is projected onto the curved screen for display, a real camera is set up to align with the curved screen and collect images, and the images collected by the real camera are transformed and mapped to the virtual camera images, so as to obtain the virtual camera imaging effect, and then generate multiple virtual camera images. With the aid of a controller, etc., automatic playback, automatic collection, and real-time data mapping functions are realized.

[0049] Step 2: Start simulation video playback and data collection

[0050] The computer simulates the panoramic imaging of the virtual car and projects the image onto each curved screen. The cameras of the car to be tested respectively image the curved screen projected by the corresponding virtual camera. Start the simulation video playback and collect real-time data through the real cameras of the car.

[0051] Step 3: Projection image processing and correction

[0052] Each real camera needs to be calibrated with internal parameters in the early stage. The present invention adopts Zhang Zhengyou's checkerboard target calibration method to obtain the internal parameters of each camera and use them as system parameter configuration;

[0053] Let the undistorted feature point image coordinates be p = [u, v, 1] T , the unit is pixel, and its normalized coordinate is p n =[x n,y n ,1] T , in millimeters. q=[x,y,z,1] T is the coordinate of the feature point in the target coordinate system. According to the pinhole imaging model, the target point q and its corresponding image point p can be expressed as:

[0054]

[0055] Among them, ρ is a non-zero coefficient, is the camera internal parameter matrix, u0, v0 are the image plane principal point coordinates, f x 、f y are the proportional coefficients in the u and v directions respectively, γ is the tilt factor of the two axes of the image, R = [r1 r2 r3] 3×3 and t are the rotation matrix and translation vector between the target coordinate system and the camera coordinate system respectively. Let p d =[u d ,v d ,1] T is the homogeneous coordinate of the distorted image point, p n =[x n y n ,1] T is the homogeneous coordinate of the normalized image point. The camera distortion model can be expressed as:

[0056]

[0057] in, k1 and k2 are the radial distortion coefficients of the lens. Usually, the internal parameters of the camera need to be accurately calibrated before measurement, mainly including [f x ,f y ,γ,u0,v0,k1,k2];

[0058] After the real camera collects images of various curved screens, it is substituted into the camera de-distortion model to obtain a distortion-free image.

[0059] Step 4: Mapping the virtual camera to the real camera

[0060] Based on the virtual camera images captured by the real camera in step 3, a transformation relationship chain is constructed from the virtual camera to the curved screen, and from the curved screen to the real camera. The mapping chain is used to complete the mapping from the real camera image to the virtual camera imaging. It is mainly divided into the following four steps:

[0061] Step 1: Virtual camera projection and curved screen mapping. The virtual camera configuration parameters are set according to the simulation environment, including the virtual camera focal length f v , resolution w v ×h v . Curved screen configuration parameters include resolution ws ×h s , the pixel size is s, the unit is mm;

[0062] Step 2: Calibration of external parameters of curved screen and real camera. First, the checkerboard target is projected onto the curved screen. The real camera collects the image of the curved screen and measures it by aiming with the laser tracker to obtain the coordinate sequence q of the curved screen checkerboard target point. The ideal checkerboard point sequence is Q. The coordinates of the checkerboard image point collected by the real camera are p. According to the mapping relationship between the two, q = H gs (Q), where H gs Represents the quadratic mapping equation of the space surface, which is F(x,y,z)=0,p=H sc Q, H sc Represents the transformation matrix between the ideal chessboard and the real camera acquisition;

[0063] Step 3: Convert the image captured by the real camera to the curved screen image. According to the calibration parameters in the second step, when the scene image I projected by the virtual camera corresponding to the curved screen is captured r When the image data I corresponding to the curved screen is obtained through the above transformation relationship s , and both satisfy

[0064] Step 4: The curved screen image is transformed into a virtual camera image. s Imaging with Virtual Camera I v The transformation relationship is transformed to obtain virtual imaging. The two satisfy the mapping relationship from the space surface to the virtual plane, which can be expressed by the equation: So far, I have been established r To I v The transformation chain is obtained by transforming the image acquired by the actual real camera to obtain the corresponding virtual camera image I v , which can be used for subsequent research and evaluation of autonomous driving algorithms.

[0065] Through the transformation of the above steps, the image captured by the real camera 1 of the car can be transformed into the virtual camera imaging. At the same time, the cameras of various parts of the real car can be mapped using this method, and then the panoramic image of the car in the virtual scene can be obtained by capturing images with the real camera. The real road conditions can be simulated to evaluate and improve the relevant algorithms.

[0066] Parts or structures not specifically described in the present invention may adopt existing technologies or existing products and will not be described in detail here.

[0067] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A spatial curved screen projection visual mapping and simulation method, characterized in that: The following steps are involved: Step 1: Build a car panoramic visual perception perspective mapping simulation device; Step 2: Start the simulation video playback and collect data; Step 3: Projection image processing and correction; Step 4: Map the virtual camera to the real camera, thereby completing the mapping of the real camera image to the virtual camera image.

2. A spatial curved screen projection visual mapping and simulation method according to claim 1, characterized in that: In step 2, the steps of starting the simulation video playback and collecting data include: The computer of the car panoramic visual perception perspective mapping simulation device simulates the virtual car panoramic perspective imaging and projects the image onto each curved screen. The cameras of the car to be tested respectively image the curved screen projected by the corresponding virtual camera; the simulation video playback is started, and real-time data collection is performed through the real cameras of the car.

3. The spatial curved screen projection visual mapping and simulation method according to claim 1, characterized in that: In step 3, the steps of projected image processing and correction include: Each real camera is calibrated in the early stage to obtain the internal parameters of each camera and use them as system parameter configuration; Let the undistorted feature point image coordinates be p = [u, v, 1] T , the unit is pixel, and its normalized coordinate is p n =[x n ,y n ,1] T , the unit is millimeter; let q = [x, y, z, 1] T is the coordinate of the feature point in the target coordinate system; according to the pinhole imaging model, we have: Among them, ρ is a non-zero coefficient, is the camera internal parameter matrix, u0, v0 are the image plane principal point coordinates, f x 、f y are the proportional coefficients in the u and v directions respectively, γ is the tilt factor of the two axes of the image, R = [r1 r2 r3] 3×3 and t are the rotation matrix and translation vector between the target coordinate system and the camera coordinate system respectively; Let p d =[u d ,v d ,1] T is the homogeneous coordinate of the distorted image point, p n =[x n y n ,1] T To normalize the homogeneous coordinates of the image points, the camera distortion model is expressed as: in, k1 and k2 are the radial distortion coefficients of the lens; After the real camera collects images of various curved screens, it is substituted into the camera dedistortion model to obtain a distortion-free image.

4. The spatial curved screen projection visual mapping and simulation method according to claim 1, characterized in that: In step 4, the virtual camera is mapped to the real camera, so that the steps of mapping the real camera image to the virtual camera image include: Based on the virtual camera images collected by the real camera in step 3, the mapping chain is used to complete the mapping from the real camera image to the virtual camera imaging, which includes the following four steps: Step 1: Virtual camera projection and curved screen mapping; Step 2: Calibration of curved screen and real camera external parameters; Step 3: transform the image captured by the real camera into the curved screen image; Step 4: The curved screen image is transformed into the virtual camera image.

5. A spatial curved screen projection visual mapping and simulation method according to claim 4, characterized in that: The configuration parameters of the virtual camera are set according to the simulation environment, including the virtual camera focal length f v , resolution w v ×h v ; The configuration parameters of the curved screen include resolution w s ×h s , the pixel size is s, the unit is mm.

6. The spatial curved screen projection visual mapping and simulation method according to claim 4, characterized in that: In the second step, the steps of calibrating the curved screen and the real camera extrinsic parameters include: First, the checkerboard target is projected onto the curved screen, and the real camera collects the curved screen image. The curved screen checkerboard target point coordinate sequence q is obtained by aiming and measuring with a laser tracker. The ideal checkerboard point sequence is Q, and the checkerboard image point coordinate p collected by the real camera is obtained according to the mapping relationship between the two. gs (Q), where H gs Represents the quadratic mapping equation of the space surface, which is F(x,y,z)=0,p=H sc Q, H sc Represents the transformation matrix between the ideal chessboard and the real camera acquisition.

7. The spatial curved screen projection visual mapping and simulation method according to claim 4, characterized in that: In the third step, the steps of transforming the image captured by the real camera into the curved screen image include: When the scene image I projected by the virtual camera corresponding to the curved screen is collected r When the image data I corresponding to the curved screen is obtained according to the calibration parameters in the second step s , and both satisfy 8. The spatial curved screen projection visual mapping and simulation method according to claim 4, characterized in that: In the fourth step, the steps of transforming the curved screen image into the virtual camera image include: According to the curved screen image I s Imaging with Virtual Camera I v The transformation relationship is transformed to obtain virtual imaging, and the two satisfy the mapping relationship from the space surface to the virtual plane, which is expressed as So far, I have been established r To I v The transformation chain is obtained by transforming the image acquired by the actual real camera to obtain the corresponding virtual camera image I v .

9. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute a spatial curved screen projection visual mapping and simulation method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a spatial curved screen projection visual mapping and simulation method as described in any one of claims 1 to 8 is implemented.

Images