Camera calibration evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a camera calibration evaluation method and device, electronic equipment and a storage medium. The camera calibration evaluation method comprises the steps that first intermediate images corresponding to all cameras on target equipment are acquired; determining a mapping relationship between the first intermediate image corresponding to each camera and a virtual projection image of the virtual camera; based on the mapping relationship, mapping the first intermediate image corresponding to each camera into the virtual projection image to obtain a second intermediate image corresponding to each camera; splicing the second intermediate images corresponding to the cameras to obtain a spliced projection image; and evaluating respective calibration external parameters of each camera based on the spliced projection image. The method and the device are used for solving the problem that multi-camera calibration external parameters cannot be quickly and intuitively evaluated.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to a camera calibration evaluation method, device, electronic device and storage medium. Background Art

[0002] Camera calibration is a crucial part of the offline process of driverless vehicles, mobile robots and other machine vision products, and plays a core role in ensuring the precise operation of the vision system. Camera calibration lays a solid foundation for subsequent tasks such as 3D reconstruction, target detection, and path planning. Especially in complex autonomous driving scenarios, high-precision camera calibration can significantly improve the system's environmental perception capabilities, thereby enhancing the navigation accuracy and safety of vehicles or robots.

[0003] However, despite the importance of camera calibration technology for machine vision products, the industry still faces considerable challenges in dealing with multi-camera systems, especially in the evaluation of multi-camera calibration extrinsics. Multi-camera systems often require extremely high precision in the spatial relationship between cameras to achieve seamless stitching of the surrounding environment and accurate acquisition of depth information. However, most existing calibration methods focus on the calibration of single-camera systems. For systems containing multiple cameras, there is currently no good method for evaluating multi-camera calibration extrinsics, and it is impossible to quickly and intuitively evaluate multi-camera calibration extrinsics. Summary of the invention

[0004] The present application provides a camera calibration evaluation method, device, electronic device and storage medium to solve the problem that it is impossible to quickly and intuitively evaluate the external parameters of multi-camera calibration.

[0005] According to a first aspect of an embodiment of the present application, a camera calibration evaluation method is provided, comprising:

[0006] Acquire a first intermediate image corresponding to each camera on the target device;

[0007] Determine a mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection images of the virtual camera;

[0008] Based on the mapping relationship, mapping the first intermediate images corresponding to the cameras to the virtual projection image to obtain the second intermediate images corresponding to the cameras;

[0009] splicing the second intermediate images corresponding to the cameras to obtain a spliced ​​projection image;

[0010] Based on the stitched projection image, the calibration extrinsic parameters of each of the cameras are evaluated.

[0011] According to a second aspect of an embodiment of the present application, a camera calibration and evaluation device is provided, including:

[0012] An acquisition unit, used to acquire a first intermediate image corresponding to each camera on the target device;

[0013] A processing unit, used to determine a mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection image of the virtual camera;

[0014] A mapping unit, configured to map the first intermediate images corresponding to the cameras to the virtual projection image based on the mapping relationship, so as to obtain the second intermediate images corresponding to the cameras;

[0015] A stitching unit, used for stitching the second intermediate images corresponding to the cameras to obtain a stitched projection image;

[0016] An evaluation unit is used to evaluate the calibration extrinsic parameters of each of the cameras based on the spliced ​​projection image.

[0017] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;

[0018] The memory is connected to the processor and is used to store programs;

[0019] The processor is used to implement the camera calibration and evaluation method as described in the first aspect by running the program in the memory.

[0020] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the camera calibration and evaluation method as described in the first aspect is implemented.

[0021] In the present application, the first intermediate image corresponding to each camera on the target device is obtained, and the mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera is determined. Based on the mapping relationship, the first intermediate image corresponding to each camera is mapped to the virtual projection image to obtain the second intermediate image corresponding to each camera. The first intermediate image corresponding to each camera is converted to the perspective of the virtual camera to obtain the second intermediate image corresponding to each camera under the perspective of the virtual camera. The second intermediate images corresponding to each camera are spliced ​​to obtain a spliced ​​projection image. Based on the spliced ​​projection image, the calibration external parameters of each camera are evaluated. By observing the spliced ​​projection image under the perspective of the virtual camera, the calibration external parameters of each camera on the target device can be quickly and intuitively evaluated. It is convenient, simple and easy to operate, and solves the problem of not being able to quickly and intuitively evaluate the calibration external parameters of multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a camera calibration evaluation method provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the positional relationship between an image coordinate system and a pixel coordinate system of a virtual projection image provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of a three-dimensional coordinate system of a camera, an image coordinate system of a first intermediate image corresponding to the camera, and a world coordinate system provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a process of step 102 provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a process of step 402 provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a process of step 502 provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of a process of step 403 provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of a first projection light path and a second projection light path provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of a stitched projection image with qualified calibration accuracy of calibration extrinsic parameters of each camera provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of a spliced ​​projection image in which the calibration accuracy of the calibration extrinsic parameters of each camera provided in an embodiment of the present application is unqualified;

[0033] Fig.11 A schematic diagram of a spliced ​​projection image in which the calibration accuracy of the calibration extrinsic parameters of each camera provided in an embodiment of the present application is unqualified;

[0034] Fig.12 A schematic diagram of the structure of a camera calibration and evaluation device provided in an embodiment of the present application;

[0035] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] Exemplary Implementation Environment

[0038] The camera calibration evaluation method according to the embodiment of the present application can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0039] In some embodiments, the camera calibration evaluation method in the present application supports online, offline and remote methods.

[0040] The online method means that the on-board equipment of the smart driving vehicle establishes a connection with the smart driving vehicle through technical means such as Ethernet and WIFI (a wireless local area network communication technology). After obtaining security verification, the on-board equipment of the smart driving vehicle can directly trigger the camera calibration evaluation method in this application.

[0041] The offline method means that when the external environment of the intelligent driving vehicle meets the preset conditions, the original images and internal parameters, distortion parameters, and calibration external parameter files of each camera are manually or automatically collected, and the original images and internal parameters, distortion parameters, and calibration external parameter files of each camera are copied to another electronic device that deploys the camera calibration evaluation method in this application, and the camera calibration evaluation method in this application is run.

[0042] The remote method refers to establishing a connection between the electronic device and the smart driving vehicle through technical means such as 4G, 5G, and dedicated satellite communications. After obtaining security verification, the remote command triggers the camera calibration and evaluation method in this application, or the electronic device automatically triggers the camera calibration and evaluation method in this application when the external environment of the smart driving vehicle meets the preset conditions, remotely reads the spliced ​​projected image, and quickly evaluates the calibration external parameters of each camera.

[0043] Exemplary Methods

[0044] See also Figure 1 In an exemplary embodiment, a camera calibration evaluation method is provided. Figure 1 As shown in the figure, the process of the camera calibration evaluation method mainly includes:

[0045] Step 101: Acquire a first intermediate image corresponding to each camera on a target device.

[0046] In the exemplary embodiment, the target device refers to a device equipped with multiple cameras. For example, the target device may be a vehicle or a mobile robot. The target device is explained as a vehicle in this application, but it is not limited to this. Specifically, the target device may be an intelligent driving vehicle.

[0047] In an exemplary embodiment, the camera model of each camera on the target device may be a pinhole camera model, a fisheye camera model, or other types of camera imaging models, and the present application is not limited thereto.

[0048] In some embodiments, the intrinsic parameters, calibrated extrinsic parameters, and distortion parameters of each camera on the target device are obtained. The intrinsic parameters and distortion parameters of each camera are provided by the camera manufacturer or other upstream modules, and are not obtained by calibration after loading and unloading. It is assumed that the intrinsic parameters and distortion parameters of the camera after loading and unloading are qualified. The calibrated extrinsic parameters of each camera are the calibration parameters that need to be evaluated in this application. They are obtained after the production line calibration when loading and unloading or calibration during after-sales maintenance, and describe the pose relationship between the three-dimensional coordinate system of the camera and the three-dimensional coordinate system of the vehicle.

[0049] In some embodiments, the first intermediate image corresponding to each camera may be an image corresponding to the same moment.

[0050] In some embodiments, the first intermediate image corresponding to each camera may be an original image, or an image obtained by dedistorting the original image.

[0051] In some embodiments, when the first intermediate image corresponding to each camera is an image after the original image is dedistorted, step 101 includes: obtaining the original image corresponding to each camera on the target device; based on the intrinsic parameters and distortion parameters of each camera, performing a dedistortion operation on the original image corresponding to each camera to obtain the first intermediate image after dedistortion corresponding to each camera.

[0052] In some embodiments, obtaining the original image corresponding to each camera on the target device can be by obtaining the original image captured by each camera on the vehicle at the same time when the external environment of the vehicle meets the preset conditions; or it can be by extracting the original image corresponding to each camera at the same time when the external environment meets the preset conditions from the video recorded by each camera on the vehicle.

[0053] In some embodiments, the preset conditions satisfied by the external environment specifically include the following conditions: (1) the vehicle is stationary or running smoothly without any bumps or large vibrations, and the tire pressure is basically consistent, and the air suspension is turned off (vehicles without air suspension do not need to turn off the air suspension); (2) the ground on which the vehicle is located is flat and has no undulations (for example, the ground must be flat within 100 meters in front and behind the vehicle and within 15 meters on the left and right sides); (3) the imaging field of view of each camera on the vehicle contains common and continuous straight line features parallel to the ground on which the vehicle is located (for example, generally, a straight lane line or a flat road with a road extension is selected; according to the actual application, preferably, a straight open flat road with three lanes or more is selected).

[0054] In some embodiments, when the original image is collected or extracted, the vehicle is parallel to the lane line and is within the lane.

[0055] In some embodiments, the format of the original image can be RAW (raw image file, also called digital negative), YUV (a color encoding method), RGB (a color mode represented by three channels of red, green and blue), etc., and digital analysis is supported.

[0056] In some embodiments, the first intermediate images corresponding to each camera are a set of undistorted images that are consistent with the number of cameras and have a resolution consistent with the corresponding original images. The first intermediate images can support intuitive evaluation of the camera's intrinsic parameters and distortion parameters.

[0057] Step 102: Determine a mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera.

[0058] In some embodiments, the virtual projection image may be a BEV (Bird's Eye View, also known as a bird's-eye view) projection image.

[0059] In some embodiments, the resolution of the virtual projection image of the virtual camera is preset.

[0060] In an exemplary embodiment, the resolution of the virtual projection image of the virtual camera is set according to the hardware computing power and the running time. For example, the greater the hardware computing power and the higher the computing performance, the greater the resolution of the virtual projection image of the virtual camera; the shorter the running time requirement, the smaller the resolution of the virtual projection image of the virtual camera.

[0061] In some embodiments, the origin Oego_rfu of the vehicle's three-dimensional coordinate system is the vertical projection ground point of the center of the vehicle's rear axle, the positive direction of the Xego_rfu axis of the vehicle's three-dimensional coordinate system is directly to the right of the vehicle, the positive direction of the Yego_rfu axis of the vehicle's three-dimensional coordinate system is directly in front of the vehicle, and the positive direction of the Zego_rfu axis of the vehicle's three-dimensional coordinate system is directly above the vehicle.

[0062] In some embodiments, the virtual camera is a distortion-free camera.

[0063] In some embodiments, the origin of the image coordinate system of the virtual projection image and the origin of the image coordinate system of the first intermediate image corresponding to each camera are uniformly represented by Oimg, the two coordinate axes of the image coordinate system of the virtual projection image and the two coordinate axes of the image coordinate system of the first intermediate image are uniformly represented by x-axis and y-axis, the origin of the pixel coordinate system of the virtual projection image and the origin of the pixel coordinate system of the first intermediate image corresponding to each camera are uniformly represented by Opixel, the two coordinate axes of the pixel coordinate system of the virtual projection image and the two coordinate axes of the pixel coordinate system of the first intermediate image are uniformly represented by u-axis and v-axis, and the positional relationship between the image coordinate system and the pixel coordinate system of the virtual projection image is consistent with the positional relationship between the image coordinate system and the pixel coordinate system of the first intermediate image. Figure 2 , which is a schematic diagram of the positional relationship between the image coordinate system and the pixel coordinate system of the virtual projection image. The origin Oimg of the image coordinate system of the virtual projection image is the point (u0, v0) in the pixel coordinate system of the virtual projection image, the x-axis is parallel to the u-axis, and the y-axis is parallel to the v-axis.

[0064] In some embodiments, the origin Oimg of the image coordinate system of the virtual projection image is the projection point of the origin Oego_rfu of the three-dimensional coordinate system of the vehicle in the virtual projection image, which is regarded as the reference target point.

[0065] In some embodiments, the field of view of the virtual camera is preset based on the preset conditions satisfied by the external environment. The field of view of the virtual camera is the size of the rectangular visible area projected from a top-down perspective of the real 3D space captured by the virtual camera perpendicular to the ground where the vehicle is located.

[0066] Generally, the field of view of the virtual camera takes the origin Oego_rfu of the vehicle's three-dimensional coordinate system as the center point of the field of view, and the aspect ratio of the field of view of the virtual camera is consistent with the aspect ratio of the resolution of the virtual projection image. Assuming that the length of the side perpendicular to the lane line direction in the field of view of the virtual camera is Width_, the length of the side parallel to the lane line direction in the field of view of the virtual camera is Height_; assuming that the pixels perpendicular to the lane line direction in the resolution of the virtual projection image are width_imgpixel, the pixels parallel to the lane line direction in the resolution of the virtual projection image are height_imgpixel, Width_ / Height_=width_imgpixel / height_imgpixel. Generally, Height_ takes a value of 30 to 150 meters. Preferably, Width_ takes the value of the actual distance between the two lane lines at the outermost edges on both sides of all lane lines in the external environment.

[0067] In some embodiments, a virtual camera capable of realizing a virtual projection image is constructed according to preset conditions satisfied by the external environment, the resolution of the virtual projection image, and the field of view of the virtual camera.

[0068] In some embodiments, the preset height of the virtual camera is H vir_cam , H vir_cam is the vertical height from the virtual camera to the ground where the vehicle is located. vir_cam The minimum value of is not less than the vertical height from the highest camera among all cameras to the ground where the vehicle is located.

[0069] In some embodiments, the internal parameters of the virtual camera are as follows:

[0070]

[0071] In some embodiments, the origin Ovir of the three-dimensional coordinate system of the virtual camera is the position of the virtual camera, the Xvir axis of the three-dimensional coordinate system of the virtual camera is parallel to the x-axis of the image coordinate system of the virtual projection image, the positive direction of the Xvir axis of the three-dimensional coordinate system of the virtual camera is consistent with the positive direction of the x-axis of the image coordinate system of the virtual projection image, the Yvir axis of the three-dimensional coordinate system of the virtual camera is parallel to the y-axis of the image coordinate system of the virtual projection image, the positive direction of the Yvir axis of the three-dimensional coordinate system of the virtual camera is consistent with the positive direction of the y-axis of the image coordinate system of the virtual projection image, the Zvir axis of the three-dimensional coordinate system of the virtual camera is a straight line formed by the origin Ovir of the three-dimensional coordinate system of the virtual camera and the origin Oimg of the image coordinate system of the virtual projection image, and the positive direction of the Zvir axis of the three-dimensional coordinate system of the virtual camera is from the origin Ovir of the three-dimensional coordinate system of the virtual camera to the origin Oimg of the image coordinate system of the virtual projection image.

[0072] In some embodiments, the origin Oc of the three-dimensional coordinate system of the camera is the position of the camera, the Xc axis of the three-dimensional coordinate system of the camera is parallel to the x axis of the image coordinate system of the first intermediate image corresponding to the camera, the positive direction of the Xc axis of the three-dimensional coordinate system of the camera is consistent with the positive direction of the x axis of the image coordinate system of the first intermediate image corresponding to the camera, the Yc axis of the three-dimensional coordinate system of the camera is parallel to the y axis of the image coordinate system of the first intermediate image corresponding to the camera, the positive direction of the Yc axis of the three-dimensional coordinate system of the camera is consistent with the positive direction of the y axis of the image coordinate system of the first intermediate image corresponding to the camera, the Zc axis of the three-dimensional coordinate system of the camera is a straight line formed by the origin Oc of the three-dimensional coordinate system of the camera and the origin Oimg of the image coordinate system of the first intermediate image corresponding to the camera, and the positive direction of the Zc axis of the three-dimensional coordinate system of the camera is from the origin Oc of the three-dimensional coordinate system of the camera to the origin Oimg of the image coordinate system of the first intermediate image corresponding to the camera. Figure 3 The figure shows a schematic diagram of the three-dimensional coordinate system of the camera, the image coordinate system of the first intermediate image corresponding to the camera, and the world coordinate system. Figure 3 In the figure, the origin of the world coordinate system is Ow, and the three coordinate axes of the world coordinate system are Xw axis, Yw axis and Zw axis respectively. Figure 3 In the figure, the three-dimensional coordinates of the three-dimensional point in the three-dimensional coordinate system of the camera are P(Xc, Yc, Zc), the two-dimensional coordinates of the imaging point of the three-dimensional point in the image coordinate system of the first intermediate image corresponding to the camera are p(x, y), point B is the intersection of the perpendicular line from point P to the plane XcOc Zc and the plane XcOc Zc, and point A is the intersection of the perpendicular line from point B to the Zc axis and the Zc axis.

[0073] In some embodiments, Figure 4 As shown, step 102 includes:

[0074] The following operations are performed for any preset pixel point in the virtual projection image of the virtual camera:

[0075] Step 401 : determining a first projection light path projected on the preset pixel point by a virtual camera based on the first two-dimensional coordinate of the preset pixel point.

[0076] In an exemplary embodiment, the first two-dimensional coordinate of the preset pixel point is denoted by p vir_cam (u, v), according to the first two-dimensional coordinates of the preset pixel point and the intrinsic parameter IntrinsicM of the virtual camera vir_cam , get the normalized point coordinates p of the preset pixel point vir_cam (x,y), normalized point coordinates p based on the preset pixel point vir_cam (x, y), determine the first projection light path projected on the preset pixel point by the virtual camera.

[0077] Step 402: determine the first three-dimensional coordinates of the target space three-dimensional point on the first projection light path in the three-dimensional coordinate system of the virtual camera.

[0078] In some embodiments, Figure 5 As shown, step 402 includes:

[0079] Step 501: determine a first normalized vector of a first projection light path in a three-dimensional coordinate system of a virtual camera.

[0080] In an exemplary embodiment, the normalized point coordinates p of a preset pixel point are vir_cam (x, y), determine the first normalized vector Vec of the first projection light path in the three-dimensional coordinate system of the virtual camera vir_cam (x,y,1).

[0081] Step 502: determine a first conversion relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera.

[0082] In some embodiments, Figure 6 As shown, step 502 includes:

[0083] Step 601: Determine a third transformation relationship from the three-dimensional coordinate system of the virtual camera to the world coordinate system based on the external parameters of the virtual camera.

[0084] In an exemplary embodiment, the external parameters of the virtual camera are transformed into a third transformation relationship from the three-dimensional coordinate system of the virtual camera to the world coordinate system. The third transformation relationship is a homogeneous translation and rotation matrix. The formula of the third transformation relationship is as follows:

[0085]

[0086] The third conversion relationship The upper left corner 3*3 (that is, the first 3 rows and first 3 columns) represents the rotation relationship, and the third conversion relationship The first three rows of the fourth column represent the translation relationship, and the third conversion relationship The 4th row of represents a homogeneous mapping.

[0087] Step 602: determine a fourth transformation relationship from the three-dimensional coordinate system of the target device to the world coordinate system.

[0088] In an exemplary embodiment, the target device is a vehicle, and the formula of the fourth transformation relationship from the three-dimensional coordinate system of the vehicle to the world coordinate system is specifically as follows:

[0089]

[0090] Step 603: Determine a first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera based on the third transformation relationship and the fourth transformation relationship.

[0091] In the exemplary embodiment, based on the third conversion relationship And the fourth conversion relationship Determine a first transformation relationship from the three-dimensional coordinate system of the vehicle to the three-dimensional coordinate system of the virtual camera. The formula of the first transformation relationship is as follows:

[0092]

[0093] Based on the external parameters of the virtual camera, the third transformation relationship from the three-dimensional coordinate system of the virtual camera to the world coordinate system is determined, and the fourth transformation relationship from the three-dimensional coordinate system of the target device to the world coordinate system is determined. Based on the third transformation relationship and the fourth transformation relationship, the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera is determined. With the help of the world coordinate system, the first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera can be accurately obtained.

[0094] Step 503: Determine a second normalized vector of the first projection light path in the three-dimensional coordinate system of the target device based on the first normalized vector and the first conversion relationship.

[0095] In an exemplary embodiment, based on the first normalized vector Vec vir_cam (x,y,1) and the first transformation relationship Determine the second normalized vector of the first projection light path in the three-dimensional coordinate system of the target device. The formula of the second normalized vector is as follows:

[0096]

[0097] in, means The first 3 rows and first 3 columns of .

[0098] Step 504: determine a scale factor based on the first conversion relationship and the second normalized vector.

[0099] In an exemplary embodiment, the unit vector in the positive direction of the Zego_rfu axis of the three-dimensional coordinate system of the vehicle is denoted as VecGround ego_rfu (0,0,1), find VecGround ego_rfu (0,0,1) and the second normalized vector Vec ego_rfu The angle θ between (x′, y′, z′) is taken as The value in the third row and fourth column is Scale h , based on the angle θ, Scale h and the second normalized vector Vec ego_rfu (x′, y′, z′), find the scale factor Scale H, scale factor Scale H The specific calculation formula is as follows:

[0100]

[0101] Where ||Vec ego_rfu (x′,y′,z′)|| Lx is the second normalized vector Vec ego_rfu The norm of (x′, y′, z′), preferably, the second normalized vector Vec ego_rfu L2 norm of (x′,y′,z′).

[0102] In an exemplary embodiment, the scale factor may be understood as a proportional relationship between the image of the virtual camera and the field of view of the virtual camera.

[0103] Step 505 : determining first three-dimensional coordinates of the three-dimensional point in the target space on the first projection light path in the three-dimensional coordinate system of the virtual camera based on the first normalized vector and the scale factor.

[0104] In an exemplary embodiment, based on the first normalized vector Vec vir_cam (x,y,1) and the scale factor Scale H , determine the first three-dimensional coordinate P of the target space three-dimensional point on the first projection light path in the three-dimensional coordinate system of the virtual camera vir_cam (x*Scale H ,y*Scale H ,Scale H ). The first three-dimensional coordinate can also be expressed as P vir_cam (x*Scale H ,y*Scale H ,Scale H ,1), in order to maintain the homogeneity of subsequent matrix calculations.

[0105] In an exemplary embodiment, a preset pixel point corresponds to a fixed three-dimensional point in the target space.

[0106] Determine a first normalized vector of the first projection light path in the three-dimensional coordinate system of the virtual camera, determine a first conversion relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera, determine a second normalized vector of the first projection light path in the three-dimensional coordinate system of the target device based on the first normalized vector and the first conversion relationship, determine a scale factor based on the first conversion relationship and the second normalized vector, and determine a first three-dimensional coordinate of the three-dimensional point in the target space on the first projection light path in the three-dimensional coordinate system of the virtual camera based on the first normalized vector and the scale factor. The spatial position of the first projection light path does not change, but the normalized vector of the first projection light path in different three-dimensional coordinate systems will be different, determine a scale factor based on the first conversion relationship and the second normalized vector, and accurately determine the first three-dimensional coordinate of the three-dimensional point in the target space on the first projection light path in the three-dimensional coordinate system of the virtual camera based on the first normalized vector and the scale factor in the three-dimensional coordinate system of the virtual camera.

[0107] Step 403 : determining the second three-dimensional coordinates of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera based on the calibrated external parameters of the target camera and the first three-dimensional coordinates.

[0108] The target camera is at least one camera among all cameras corresponding to the three-dimensional point in the target space.

[0109] In an exemplary embodiment, based on the first three-dimensional coordinates and the view cones of each camera, the target camera corresponding to the three-dimensional point in the target space can be determined in each camera. Specifically, one three-dimensional point in the target space may correspond to one camera, or one three-dimensional point in the target space may correspond to two cameras, because the actual three-dimensional space captured by different cameras may have partial overlap.

[0110] In some embodiments, Figure 7 As shown, step 403 includes:

[0111] Step 701: determine a first transformation relationship from a three-dimensional coordinate system of a target device to a three-dimensional coordinate system of a virtual camera.

[0112] In the exemplary embodiment, the first conversion relationship is

[0113] Step 702: Determine a second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device based on the calibrated external parameters of the target camera.

[0114] In an exemplary embodiment, based on the calibration external parameters of the target camera, a second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device is determined. Among them, i represents the i-th camera, that is, the target camera.

[0115] Step 703: Determine the second three-dimensional coordinates of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera based on the first conversion relationship, the second conversion relationship and the first three-dimensional coordinates.

[0116] In an exemplary embodiment, based on the first conversion relationship The second conversion relationship and the first three-dimensional coordinate P vir_cam (x*Scale H ,y*Scale H ,Scale H ,1), determine the second three-dimensional coordinates of the target space three-dimensional point in the three-dimensional coordinate system of the target camera. The specific calculation formula of the second three-dimensional coordinates is as follows:

[0117]

[0118] Determine a first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera, determine a second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device based on the calibrated external parameters of the target camera, determine the second three-dimensional coordinates of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera based on the first transformation relationship, the second transformation relationship and the first three-dimensional coordinates, and can accurately obtain the transformation relationship between the three-dimensional coordinate system of the target camera and the three-dimensional coordinate system of the virtual camera with the help of the three-dimensional coordinate system of the target device, and then accurately determine the second three-dimensional coordinates of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera based on the first three-dimensional coordinates.

[0119] Step 404 : Based on the second three-dimensional coordinates, determine a second projection light path for projecting the three-dimensional point in the target space through the target camera.

[0120] In an exemplary embodiment, based on the second three-dimensional coordinate P real_cam_i (x i ,y i , z i ), determine the third normalized vector of the second projection light path projected by the target camera to the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera, and the formula of the third normalized vector is as follows:

[0121] Vec real_cam_i (x i ′,y i ′, 1) = Vec real_cam_i (x i / z i ,y i / z i , z i / z i )

[0122] In some embodiments, before step 404, the camera calibration evaluation method further includes: determining, based on the second three-dimensional coordinates, that the three-dimensional point in the target space can be imaged by the target camera.

[0123] In an exemplary embodiment, based on the second three-dimensional coordinate, determining that the three-dimensional point in the target space can be imaged by the target camera may be performed by converting the second three-dimensional coordinate P real_cam_i (x i ,y i , z i )Zc axis coordinate z i The negative target space 3D points are deleted to ensure that the remaining target space 3D points can be imaged by the target camera.

[0124] Before determining the second projection light path for projecting the three-dimensional point in the target space through the target camera based on the second three-dimensional coordinates, determine whether the three-dimensional point in the target space can be imaged by the target camera based on the second three-dimensional coordinates, delete the three-dimensional points in the target space that cannot be imaged by the target camera, and then determine the second projection light path for projecting the three-dimensional point in the target space through the target camera based on the second three-dimensional coordinates, thereby reducing the amount of calculation for determining the second projection light path and further reducing the amount of calculation for subsequent steps.

[0125] Step 405 , based on the second projection optical path, the internal parameters of the target camera and the internal parameters of the virtual camera, determine the second two-dimensional coordinates of the actual pixel point projected from the three-dimensional point in the target space to the virtual projection image of the virtual camera.

[0126] In an exemplary embodiment, based on the third normalized vector Vec of the second projection light path in the three-dimensional coordinate system of the target camera real_cam_i (x i ′,y i ′, 1), the internal parameters of the target camera and the internal parameters of the virtual camera IntrinsicM vir_cam , determine the second two-dimensional coordinate p of the actual pixel point projected from the three-dimensional point in the target space to the virtual projection image of the virtual camera vir_img_i (u i , v i ).

[0127] Step 406: Determine a mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera based on the second two-dimensional coordinates.

[0128] In an exemplary embodiment, based on the second two-dimensional coordinates, determining the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera may be performed by transforming the second two-dimensional coordinates p vir_img_i (u i , v i ) i and v iAfter decomposing these two parameters, we get the mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera in the x-axis direction and the y-axis direction, which can be named mapx and mapy. i As mapx, v i As mapy.

[0129] The following operations are performed for any preset pixel point in the virtual projection image of the virtual camera: based on the first two-dimensional coordinates of the preset pixel point, a first projection light path projected on the preset pixel point by the virtual camera is determined, the first three-dimensional coordinates of the target space three-dimensional point on the first projection light path in the three-dimensional coordinate system of the virtual camera are determined, based on the calibrated external parameters of the target camera and the first three-dimensional coordinates, the second three-dimensional coordinates of the target space three-dimensional point in the three-dimensional coordinate system of the target camera are determined, based on the second three-dimensional coordinates, a second projection light path for projecting the target space three-dimensional point through the target camera is determined, based on the second projection light path, the internal parameters of the target camera and the internal parameters of the virtual camera, the second two-dimensional coordinates of the actual pixel point in the virtual projection image of the virtual camera projected onto the target space three-dimensional point are determined, based on the second projection light path, the internal parameters of the target camera and the internal parameters of the virtual camera, and based on the second two-dimensional coordinates, a mapping relationship of the first intermediate image corresponding to the target camera projected to the virtual projection image of the virtual camera is determined, so that the mapping relationship of the first intermediate image corresponding to each camera projected to the virtual projection image of the virtual camera can be obtained.

[0130] The target space 3D point is on both the first projection optical path projected by the virtual camera and the second projection optical path projected by the target camera. The second projection optical path can be obtained based on the first projection optical path, and then based on the second projection optical path, the intrinsic parameters of the target camera and the intrinsic parameters of the virtual camera, the second two-dimensional coordinates of the actual pixel point in the virtual projection image of the virtual camera projected from the target space 3D point are determined, and based on the second two-dimensional coordinates, the mapping relationship of the first intermediate image corresponding to the target camera projected to the virtual projection image of the virtual camera is determined. Moreover, the calibration extrinsic parameters of the target camera are used in the calculation process, that is, the mapping relationship of the first intermediate image corresponding to the target camera projected to the virtual projection image of the virtual camera is related to the calibration extrinsic parameters of the target camera.

[0131] In an exemplary embodiment, Figure 8 The figure shows a schematic diagram of a first projection light path and a second projection light path. Figure 8, the three-dimensional coordinates of the target space three-dimensional point in the world coordinate system are P(Xw, Yw, Zw), pvir is the preset pixel point, Ovir is the position of the virtual camera, P(Xw, Yw, Zw), P1, P2, P3 are the points on the first projection light path projected by the virtual camera on the preset pixel point pvir, Ocl is the position of the first camera, Ocl P is the second projection light path of the first camera projecting the target space three-dimensional point P, P(Xw, Yw, Zw) is the intersection of the first projection light path and the second projection light path, pc1 is the imaging point of P(Xw, Yw, Zw) in the image of the first camera, and Tc1-vir is the conversion relationship from the three-dimensional coordinate system of the first camera to the three-dimensional coordinate system of the virtual camera.

[0132] Step 103: Based on the mapping relationship, the first intermediate image corresponding to each camera is mapped into the virtual projection image to obtain the second intermediate image corresponding to each camera.

[0133] Step 104: stitching the second intermediate images corresponding to the cameras to obtain a stitched projection image.

[0134] In an exemplary embodiment, the second intermediate images corresponding to each camera are stitched together to obtain a stitched projection image. This may be accomplished by stitching the second intermediate images corresponding to each camera at the resolution of the virtual projection image to obtain a stitched projection image.

[0135] Step 105 : Based on the spliced ​​projection image, evaluate the calibration extrinsic parameters of each camera.

[0136] In some embodiments, step 105 includes: evaluating the calibration accuracy of the calibration extrinsic parameters of each camera based on the geometric feature relationship in the real external environment and the geometric feature relationship in the spliced ​​projection image.

[0137] In an exemplary embodiment, evaluating the calibration accuracy of the calibration extrinsic parameters of each camera may be evaluating whether the calibration accuracy of the calibration extrinsic parameters of each camera is qualified.

[0138] In the exemplary embodiment, the stitched projection image with qualified calibration accuracy of the calibration external parameters of each camera satisfies the following conditions: all straight line features, preferably lane lines, form continuous straight lines in the stitched projection image, and the relative position relationship such as parallel or vertical is consistent with the relative position relationship between geometric features in the real external environment; the same straight line features in the projection area of ​​each camera overlap with each other without obvious misalignment. Fig. 9 As shown, it is a schematic diagram of a stitched projection image with qualified calibration accuracy of the calibration extrinsic parameters of each camera.

[0139] In an exemplary embodiment, Fig.10, which is a schematic diagram of a spliced ​​projection image in which the calibration accuracy of the calibration extrinsic parameters of each camera is unqualified. Fig.10 There are 6 cameras in total, namely the front view camera, left front view camera, right front view camera, left rear view camera, right rear view camera, and rear view camera. The projected image is divided into 6 areas, namely the front view area, left front view area, right front view area, left rear view area, right rear view area, and rear view area. According to the images of each area in the spliced ​​projected image, it can be judged which camera's calibration accuracy of the calibrated external parameters is unqualified. Fig.10 In the image, for the left and right rear view areas, the parallel lane lines in the real environment are skewed in the spliced ​​projection image and no longer maintain a parallel relationship; for the rear view area, due to insufficient pitch angle calibration accuracy, the parallel lane lines in the real environment appear in an "eight" shape in the spliced ​​projection image, and the lane lines on the left and right sides of the vehicle tend to cross after being extended, and no longer maintain a parallel relationship; for the front view, left front view and left rear view areas, the projections of the same lane line at the outermost edges do not overlap and cross each other, which means that the calibration accuracy does not meet the standard.

[0140] In an exemplary embodiment, Fig.11 , which is a schematic diagram of a spliced ​​projection image in which the calibration accuracy of the calibration extrinsic parameters of each camera is unqualified. Fig.11 There are 6 cameras in total, namely the front view camera, left front view camera, right front view camera, left rear view camera, right rear view camera, and rear view camera. The projected image is divided into 6 areas, namely the front view area, left front view area, right front view area, left rear view area, right rear view area, and rear view area. According to the images of each area in the spliced ​​projected image, it can be judged which camera's calibration accuracy of the calibrated external parameters is unqualified. Fig.11 For the right front view area, the outer lane line is skewed and no longer coincides with the same straight lane line in the front view; for the left rear view, similarly, the outer lane line is skewed and no longer coincides with the same straight lane line in the left front view and rear view; for the rear view, due to insufficient accuracy of the translation position calibration, the lane line shifts to the right, and due to insufficient accuracy of the pitch angle calibration, the parallel lane lines in the real environment appear in an inverted "eight" open shape in the BEV projection image, especially the outermost lane line at the right rear is the most obvious.

[0141] In this application, the second intermediate images corresponding to each camera are stitched together to obtain a stitched projection image. Based on the stitched projection image, the calibration external parameters of each camera are evaluated. By observing the stitched projection image from the perspective of the virtual camera, the calibration external parameters of each camera on the target device can be quickly and intuitively evaluated. This is convenient, simple and easy to operate, and solves the problem of being unable to quickly and intuitively evaluate the calibration external parameters of multiple cameras. Moreover, it has little dependence on the external environment and only requires parallel straight line elements on a flat road surface.

[0142] In summary, in the present application, the first intermediate image corresponding to each camera on the target device is obtained, and the mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera is determined. Based on the mapping relationship, the first intermediate image corresponding to each camera is mapped to the virtual projection image to obtain the second intermediate image corresponding to each camera. The first intermediate image corresponding to each camera is converted to the perspective of the virtual camera to obtain the second intermediate image corresponding to each camera under the perspective of the virtual camera. The second intermediate images corresponding to each camera are spliced ​​to obtain a spliced ​​projection image. Based on the spliced ​​projection image, the calibration external parameters of each camera are evaluated. By observing the spliced ​​projection image under the perspective of the virtual camera, the calibration external parameters of each camera on the target device can be quickly and intuitively evaluated. It is convenient, simple and easy to operate, and solves the problem of not being able to quickly and intuitively evaluate the calibration external parameters of multiple cameras.

[0143] Exemplary Devices

[0144] Accordingly, the present application also provides a camera calibration evaluation device, such as Fig.12 As shown, the camera calibration evaluation device includes:

[0145] An acquiring unit 1201 is used to acquire a first intermediate image corresponding to each camera on a target device;

[0146] A processing unit 1202 is used to determine a mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection image of the virtual camera;

[0147] A mapping unit 1203 is used to map the first intermediate image corresponding to each of the cameras to the virtual projection image based on the mapping relationship to obtain a second intermediate image corresponding to each of the cameras;

[0148] A stitching unit 1204 is used to stitch the second intermediate images corresponding to the cameras to obtain a stitched projection image;

[0149] The evaluation unit 1205 is used to evaluate the calibration extrinsic parameters of each of the cameras based on the spliced ​​projection image.

[0150] The camera calibration evaluation device provided in this embodiment belongs to the same application concept as the camera calibration evaluation method provided in the above embodiments of this application, and can execute the camera calibration evaluation method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the camera calibration evaluation method. For technical details not fully described in this embodiment, please refer to the specific processing content of the camera calibration evaluation method provided in the above embodiments of this application, and will not be repeated here.

[0151] The functions implemented by the above acquisition unit 1201, processing unit 1202, mapping unit 1203, splicing unit 1204 and evaluation unit 1205 may be implemented by the same or different processors respectively, which is not limited in the embodiment of the present application.

[0152] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0153] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0154] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0155] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0156] Exemplary Electronic Devices

[0157] An embodiment of the present application provides an electronic device, see Fig.13 As shown, the device includes:

[0158] Memory 200 and processor 210;

[0159] The memory 200 is connected to the processor 210 and is used to store programs;

[0160] The processor 210 is used to implement the camera calibration and evaluation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0161] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0162] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0163] A bus may include a pathway that transfers information between components of a computer system.

[0164] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0165] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0166] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0167] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0168] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0169] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0170] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any camera calibration and evaluation method provided in the above embodiments of the present application.

[0171] Exemplary computer program products and storage media

[0172] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the processor executes the steps of the camera calibration evaluation method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0173] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0174] In addition, the embodiment of the present application may also be a storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the camera calibration evaluation method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0175] Step 101, obtaining a first intermediate image corresponding to each camera on the target device;

[0176] Step 102, determining a mapping relationship between the first intermediate image corresponding to each camera and the virtual projection image of the virtual camera;

[0177] Step 103, based on the mapping relationship, mapping the first intermediate image corresponding to each camera to the virtual projection image to obtain the second intermediate image corresponding to each camera;

[0178] Step 104, stitching the second intermediate images corresponding to the cameras to obtain a stitched projection image;

[0179] Step 105 : Based on the spliced ​​projection image, evaluate the calibration extrinsic parameters of each camera.

[0180] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0181] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0182] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0183] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0187] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0188] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0189] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A camera calibration evaluation method, characterized in that: include: Acquire a first intermediate image corresponding to each camera on the target device; Determine a mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection images of the virtual camera; Based on the mapping relationship, mapping the first intermediate images corresponding to the cameras to the virtual projection image to obtain the second intermediate images corresponding to the cameras; splicing the second intermediate images corresponding to the cameras to obtain a spliced ​​projection image; Based on the stitched projection image, the calibration extrinsic parameters of each of the cameras are evaluated.

2. The camera calibration evaluation method according to claim 1, characterized in that: The determining of the mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection image of the virtual camera includes: The following operations are performed for any preset pixel point in the virtual projection image of the virtual camera: Based on the first two-dimensional coordinates of the preset pixel point, determining a first projection light path projected by the virtual camera onto the preset pixel point; Determine a first three-dimensional coordinate of a three-dimensional point in the target space on the first projection light path in the three-dimensional coordinate system of the virtual camera; Determine, based on the calibrated extrinsic parameters of the target camera and the first three-dimensional coordinates, a second three-dimensional coordinate of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera, wherein the target camera is at least one camera among the cameras corresponding to the three-dimensional point in the target space; Based on the second three-dimensional coordinates, determining a second projection optical path for projecting the three-dimensional point in the target space through the target camera; Determine, based on the second projection optical path, the internal parameters of the target camera and the internal parameters of the virtual camera, a second two-dimensional coordinate of an actual pixel point projected from the three-dimensional point in the target space to the virtual projection image of the virtual camera; Based on the second two-dimensional coordinates, a mapping relationship between the first intermediate image corresponding to the target camera and the virtual projection image of the virtual camera is determined.

3. The camera calibration evaluation method according to claim 2, characterized in that: The determining of the first three-dimensional coordinates of the target space three-dimensional point on the first projection light path in the three-dimensional coordinate system of the virtual camera includes: Determine a first normalized vector of the first projection light path in the three-dimensional coordinate system of the virtual camera; Determine a first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera; Determine, based on the first normalized vector and the first conversion relationship, a second normalized vector of the first projection light path in the three-dimensional coordinate system of the target device; Determine a scale factor based on the first conversion relationship and the second normalized vector; Based on the first normalized vector and the scale factor, a first three-dimensional coordinate of a three-dimensional point in the target space on the first projection light path in the three-dimensional coordinate system of the virtual camera is determined.

4. The camera calibration evaluation method according to claim 2, characterized in that: The step of determining the second three-dimensional coordinates of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera based on the calibrated external parameters of the target camera and the first three-dimensional coordinates includes: Determine a first transformation relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera; Determine, based on the calibrated extrinsic parameters of the target camera, a second transformation relationship from the three-dimensional coordinate system of the target camera to the three-dimensional coordinate system of the target device; Based on the first conversion relationship, the second conversion relationship and the first three-dimensional coordinates, a second three-dimensional coordinate of the three-dimensional point in the target space in the three-dimensional coordinate system of the target camera is determined.

5. The camera calibration evaluation method according to claim 3 or 4, characterized in that: The determining of a first conversion relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera includes: Determine a third transformation relationship from the three-dimensional coordinate system of the virtual camera to the world coordinate system based on the external parameters of the virtual camera; Determining a fourth transformation relationship from the three-dimensional coordinate system of the target device to the world coordinate system; Based on the third conversion relationship and the fourth conversion relationship, a first conversion relationship from the three-dimensional coordinate system of the target device to the three-dimensional coordinate system of the virtual camera is determined.

6. The camera calibration evaluation method according to claim 2, characterized in that: Before determining a second projection light path for projecting the three-dimensional point in the target space through the target camera based on the second three-dimensional coordinate, the method further includes: Based on the second three-dimensional coordinates, it is determined that the three-dimensional point in the target space can be imaged by the target camera.

7. The camera calibration evaluation method according to claim 1, characterized in that: The step of evaluating the calibration extrinsic parameters of each camera based on the spliced ​​projection image comprises: Based on the geometric feature relationship in the real external environment and the geometric feature relationship in the spliced ​​projection image, the calibration accuracy of the calibration extrinsic parameters of each of the cameras is evaluated.

8. A camera calibration evaluation device, characterized in that: include: An acquisition unit, used to acquire a first intermediate image corresponding to each camera on the target device; A processing unit, used to determine a mapping relationship between the first intermediate images corresponding to the cameras and the virtual projection image of the virtual camera; A mapping unit, configured to map the first intermediate images corresponding to the cameras to the virtual projection image based on the mapping relationship, so as to obtain the second intermediate images corresponding to the cameras; A stitching unit, used for stitching the second intermediate images corresponding to the cameras to obtain a stitched projection image; An evaluation unit is used to evaluate the calibration extrinsic parameters of each of the cameras based on the spliced ​​projection image.

9. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the camera calibration and evaluation method according to any one of claims 1 to 7 by running the program in the memory.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the camera calibration and evaluation method according to any one of claims 1 to 7 is implemented.

Images