A multi-camera extrinsic parameter calibration method and device, and an electronic device

Abstract

The application discloses a multi-camera extrinsic parameter calibration method and device and electronic equipment, and relates to the technical field of intelligent driving. The multi-camera extrinsic parameter calibration method comprises the following steps: determining initial extrinsic parameters corresponding to each camera respectively, performing perspective transformation on a first image collected by each camera according to a perspective transformation matrix corresponding to each initial extrinsic parameter, obtaining a second image, identifying all first target corner points of a calibration board in each second image, and jointly adjusting each initial extrinsic parameter based on first image coordinates corresponding to each first target corner point. Through the above method, the perspective of the collected image is corrected by performing the perspective transformation operation on the image collected by each camera, and the identification accuracy of the corner points of the calibration board is improved, so that the accuracy of the extrinsic parameter calibration of each camera is higher.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a multi-camera extrinsic parameter calibration method, device, and electronic device. Background Technology

[0002] With the rapid development of intelligent driving vehicles, the automotive industry's demand for driver assistance functions based on vision solutions has surged. As a result, multi-camera surround view systems, such as 360-degree and 540-degree systems, have become a standard requirement for intelligent vehicles. Parameter calibration of multiple cameras is the foundation of surround view systems, and the quality of parameter calibration directly affects the final performance of the driver assistance system.

[0003] Generally, camera parameter calibration involves two parts. The first part is the conversion from the camera coordinate system to the image coordinate system, which is a transformation from 3D points to 2D points. This includes camera intrinsic parameters, which describe the camera's physical characteristics, such as focal length and resolution. The second part is the conversion from the world coordinate system to the camera coordinate system, which is a transformation from 3D points to 3D points. This includes camera extrinsic parameters, which determine the camera's position and orientation in a specific 3D space, such as rotation and translation parameters.

[0004] During camera extrinsic calibration, the calibration boards are laid flat on the ground. The shooting angles of each camera in the surround-view system are at certain angles to the calibration boards, causing stretching in the captured images. For example, the edge lines of calibration boards closer to the cameras are relatively longer, while those farther away are relatively shorter. In this situation, the significant difference between the acquired calibration board images and the standard images severely affects the accuracy of subsequent identification of corner points on the calibration boards, leading to inaccurate extrinsic parameter calibration for each camera. Summary of the Invention

[0005] This application discloses a multi-camera extrinsic parameter calibration method, apparatus, and electronic device. By performing perspective transformation on the images acquired by each camera, the recognition accuracy of the corner points of the calibration board is improved, thereby improving the accuracy of the extrinsic parameter calibration of each camera.

[0006] Firstly, this application provides a method for calibrating the extrinsic parameters of multiple cameras, the method comprising:

[0007] Determine the initial extrinsic parameters for each camera;

[0008] Based on the perspective transformation matrix corresponding to each initial extrinsic parameter, the first image acquired by each camera is transformed by perspective to obtain each second image.

[0009] Identify all first target corner points of the calibration board in each second image;

[0010] Based on the first image coordinates corresponding to each first target corner point, the initial extrinsic parameters are jointly adjusted.

[0011] By performing perspective transformation on the images acquired by each camera, the viewpoint of the acquired images is corrected, thereby improving the recognition accuracy of the calibration board corner points and making the calibration accuracy of the external parameters of each camera higher.

[0012] In one possible design, determining the initial parameters for each camera includes:

[0013] The third images acquired by each camera are subjected to distortion correction to obtain the fourth images.

[0014] Determine the coordinates of the second image corresponding to the preset corner points of the calibration plate in each of the fourth images;

[0015] Based on the arrangement order between each preset corner point, the first world coordinates corresponding to each second image coordinate are determined, wherein the first world coordinates are three-dimensional reference coordinates;

[0016] Based on the second image coordinates and the first world coordinates, the initial extrinsic parameters corresponding to each camera are determined.

[0017] Using the above method, the initial extrinsic parameters of each camera are initially calculated, providing a basis for subsequent adjustments to the extrinsic parameters of each camera.

[0018] In one possible design, the step of performing perspective transformation on the first images acquired by each camera according to the perspective transformation matrix corresponding to each initial extrinsic parameter to obtain each second image includes:

[0019] Calculate the perspective transformation matrix for each camera based on the initial extrinsic parameters;

[0020] The perspective transformation is performed on the first images acquired by each camera using the transformation matrix to obtain the second images.

[0021] By using the above method, the perspective transformation matrix is ​​used to correct the viewpoint of the images acquired by each camera, thereby improving the corner point recognition accuracy in the calibration board.

[0022] In one possible design, identifying all first target corner points of the calibration board in each of the second images includes:

[0023] Identify all corner points of the calibration board in each second image;

[0024] Among all the corner points, abnormal corner points corresponding to the common viewing area are identified, wherein the common viewing area is the area where the acquisition range of the two cameras overlaps;

[0025] The abnormal corner points are removed from all the corner points to obtain all the first target corner points.

[0026] By using the above method, abnormal corner points corresponding to the common viewing area of ​​each calibration plate are eliminated, which helps to improve the calibration accuracy of the external parameters of each camera and prevents ghosting in the common viewing area of ​​two adjacent cameras.

[0027] In one possible design, identifying the abnormal corner points corresponding to the common viewing area among all corner points includes:

[0028] In each of the second images, the first common viewing point corresponding to the common viewing area is determined;

[0029] Match each first common viewpoint with each second common viewpoint corresponding to the common view area in the world coordinate system;

[0030] If any of the first common viewpoints contains a corner point that cannot be matched, then the corner point that cannot be matched is identified as the abnormal corner point.

[0031] The above method can identify each abnormal corner point corresponding to the common viewing area, which helps to improve the calibration accuracy of the extrinsic parameters of each camera and prevent ghosting in the common viewing area of ​​two adjacent cameras.

[0032] In one possible design, the joint adjustment of the initial extrinsic parameters based on the first image coordinates corresponding to each first target corner point includes:

[0033] Determine the first image coordinates corresponding to each first target corner point;

[0034] Based on the arrangement order of the first target corner points, the second world coordinates corresponding to each first image coordinate are determined;

[0035] Based on the input of each second image coordinate and each second world coordinate into the preset model, the optimization parameters corresponding to each camera are obtained;

[0036] Each initial extrinsic parameter is adjusted in conjunction with the other optimization parameters.

[0037] By using the above methods, the initial external parameters are adjusted to further improve the calibration accuracy of each camera's external parameters.

[0038] Secondly, this application provides a multi-camera extrinsic parameter calibration device, the device comprising:

[0039] The determination module is used to determine the initial extrinsic parameters for each camera.

[0040] The transformation module is used to perform perspective transformation on the first images acquired by each camera according to the perspective transformation matrix corresponding to each initial extrinsic parameter, so as to obtain each second image.

[0041] The recognition module is used to identify all the first target corner points of the calibration board in each second image;

[0042] The adjustment module is used to jointly adjust the initial extrinsic parameters based on the first image coordinates corresponding to each first target corner point.

[0043] In one possible design, the determining module is specifically used for:

[0044] The third images acquired by each camera are subjected to distortion correction to obtain the fourth images.

[0045] Determine the coordinates of the second image corresponding to the preset corner points of the calibration plate in each of the fourth images;

[0046] Based on the arrangement order between each preset corner point, the first world coordinates corresponding to each second image coordinate are determined, wherein the first world coordinates are three-dimensional reference coordinates;

[0047] Based on the second image coordinates and the first world coordinates, the initial extrinsic parameters corresponding to each camera are determined.

[0048] In one possible design, the transformation module is specifically used for:

[0049] Calculate the perspective transformation matrix for each camera based on the initial extrinsic parameters;

[0050] The perspective transformation is performed on the first images acquired by each camera using the transformation matrix to obtain the second images.

[0051] In one possible design, the identification module is specifically used for:

[0052] Identify all corner points of the calibration board in each second image;

[0053] Among all the corner points, abnormal corner points corresponding to the common viewing area are identified, wherein the common viewing area is the area where the acquisition range of the two cameras overlaps;

[0054] The abnormal corner points are removed from all the corner points to obtain all the first target corner points.

[0055] In one possible design, the identification module is further used for:

[0056] In each of the second images, the first common viewing point corresponding to the common viewing area is determined;

[0057] Match each first common viewpoint with each second common viewpoint corresponding to the common view area in the world coordinate system;

[0058] If any of the first common viewpoints contains a corner point that cannot be matched, then the corner point that cannot be matched is identified as the abnormal corner point.

[0059] In one possible design, the adjustment module is specifically used for:

[0060] Determine the first image coordinates corresponding to each first target corner point;

[0061] Based on the arrangement order of the first target corner points, the second world coordinates corresponding to each first image coordinate are determined;

[0062] Based on the input of each second image coordinate and each second world coordinate into the preset model, the optimization parameters corresponding to each camera are obtained;

[0063] Each initial extrinsic parameter is adjusted in conjunction with the other optimization parameters.

[0064] Thirdly, this application provides an electronic device, comprising:

[0065] Memory, used to store computer programs;

[0066] When the processor executes the computer program stored in the memory, it implements the steps of the above-described multi-camera extrinsic parameter calibration method.

[0067] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described multi-camera extrinsic parameter calibration method.

[0068] The multi-camera extrinsic parameter calibration method described above obtains the initial extrinsic parameters for each camera by performing a distortion correction operation on the images acquired by each camera. Based on the initial extrinsic parameters, a perspective transformation operation is performed on the distorted images to correct the viewing angle of the acquired images, thereby improving the recognition accuracy of the calibration board corner points and making the calibration accuracy of each camera extrinsic parameter higher.

[0069] The technical effects of each of the second to fourth aspects mentioned above, as well as the technical effects that each aspect may achieve, are described above with reference to the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, and will not be repeated here. Attached Figure Description

[0070] Figure 1 This application provides a schematic diagram of a surround-view system camera layout;

[0071] Figure 2 A schematic diagram of a multi-camera field of view is provided for this application;

[0072] Figure 3 A schematic diagram of a calibration plate layout is provided for this application;

[0073] Figure 4 A flowchart of a multi-camera extrinsic parameter calibration method provided in this application;

[0074] Figure 5 An example diagram of perspective transformation provided for this application;

[0075] Figure 6 An example diagram of corner matching provided in this application;

[0076] Figure 7 A schematic diagram of the structure of a multi-camera extrinsic parameter calibration device provided in this application;

[0077] Figure 8 This is a schematic diagram of an electronic device structure provided in this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0079] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0080] The automotive industry's demand for vision-based driver assistance systems has surged, making 360-degree and 540-degree multi-camera surround view systems standard equipment in smart cars. For example... Figure 1 The diagram shows a common camera layout for a surround-view system. A represents the rear-view camera, B the left-side camera, C the front-view camera, and D the right-side camera. The field of view obtained by the four cameras A, B, C, and D is shown below. Figure 2 As shown, there is a shared viewing area between every two adjacent visual areas.

[0081] Calibrate the parameters of each camera in the surround view system. This is fundamental to the system, and the quality of the calibration directly affects the final performance of the driver assistance system. Generally, the camera parameter calibration process consists of two parts. The first part is the conversion from the camera coordinate system to the image coordinate system, which is a conversion from three-dimensional points to two-dimensional points. This includes camera intrinsic parameters, which describe the camera's physical characteristics, such as focal length and resolution. The second part is the conversion from the world coordinate system to the camera coordinate system, which is a conversion from three-dimensional points to three-dimensional points. This includes camera extrinsic parameters, which determine the camera's position and orientation in a specific three-dimensional space, such as rotation and translation parameters.

[0082] During the camera extrinsic parameter calibration process, all calibration plates are laid flat on the ground. From a frontal view, the layout diagram of the calibration plates is as follows: Figure 3 As shown, the calibration boards for each common viewing area are composed of 3×3 black and white squares, while the calibration boards for the four directional viewing areas (front, back, left, and right) are also composed of 3×3 black and white squares.

[0083] However, because the shooting angles of the cameras in the aforementioned automotive surround view system are not directly above the calibration plates, but rather at an angle, the captured images of the calibration plates will exhibit stretching. For example, the edge lines of the calibration plates closer to the camera will be relatively longer, while those farther away will be relatively shorter. In this case, the significant difference between the acquired calibration plate images and the standard images will severely affect the accuracy of subsequent identification of corner points on the calibration plates, leading to inaccurate calibration of the camera extrinsic parameters.

[0084] To address the aforementioned problems, this application provides a multi-camera extrinsic parameter calibration method. By performing perspective transformation on images acquired by each camera, the method corrects the viewing angle of the acquired images, thereby improving the recognition accuracy of corner points on the calibration board and resulting in higher accuracy for the calibration of extrinsic parameters of each camera. The methods and apparatus described in the embodiments of this application are based on the same technical concept. Since the principles by which the methods and apparatus solve the problems are similar, embodiments of the apparatus and methods can be referred to mutually, and repeated details will not be elaborated further.

[0085] like Figure 4 The diagram shown is a flowchart of a multi-camera extrinsic parameter calibration method provided in this application, which specifically includes the following steps:

[0086] S41, determine the initial extrinsic parameters for each camera;

[0087] In this embodiment of the application, when calibrating the extrinsic parameters of each camera in the vehicle surround view system, such as Figure 3As shown, various calibration plates need to be placed around the vehicle. Then, a world coordinate system is established with a preset point as the origin. The preset point can be the point corresponding to the projection of the rear axle center point of the vehicle onto the ground, or the point corresponding to the projection of the front axle center point of the vehicle onto the ground. The specific point is set according to the actual situation.

[0088] After creating the world coordinate system, the first world coordinates of all corner points are determined in the world coordinate system by combining the distances from all corner points to the origin of the coordinate system and the dimensions of each calibration plate. The first world coordinates are the three-dimensional reference coordinates.

[0089] Next, control each camera to acquire images, obtaining various third images. For example... Figure 3 As shown, camera A is responsible for capturing images of the vehicle's rear view area, camera B is responsible for capturing images of the vehicle's left view area, camera C is responsible for capturing images of the vehicle's front view area, and camera D is responsible for capturing images of the vehicle's right view area. Each area has a corresponding calibration plate. Therefore, each acquired third image contains the corner points of the calibration plate.

[0090] Typically, the third images captured by each camera in a surround-view system exhibit some distortion. For example, the bottom edge of the calibration board in the third image may appear as a distorted arc instead of a straight line. Therefore, it is necessary to perform distortion correction operations on each third image captured by each camera to restore the lines in the third image, resulting in a fourth image corresponding to each third image. Furthermore, the coordinates of the second image corresponding to the preset corner points of the calibration board in each fourth image are determined. These preset corner points can be corner points in the middle area of ​​the calibration board, or corner points in the edge area; no specific limitation is made here.

[0091] After obtaining the second image coordinates corresponding to each preset corner point, it is necessary to determine the first world coordinates corresponding to each second image coordinate in the world coordinate system. In this embodiment, when arranging the calibration plates around the vehicle, each corner point in each calibration plate can be numbered. The specific numbering order can be from left to right, from top to bottom, or other methods to ensure that each corner point number is unique. In this case, it can be guaranteed that the number corresponding to the same preset corner point in the fourth image is the same as the number corresponding to the same preset corner point in the world coordinate system.

[0092] Furthermore, based on the number corresponding to each preset corner point, a correspondence is established between the second image coordinates corresponding to each preset corner point and the first world coordinates corresponding to each preset corner point, and the initial extrinsic parameters corresponding to each camera are determined based on the correspondence.

[0093] Furthermore, to enable rapid identification of each preset corner point, a preset corner point model can be set based on the size of the preset corner point and its relative position to world coordinates. The preset corner point model includes information on the number, arrangement, and size of the corner points.

[0094] S42, based on the perspective transformation matrix corresponding to each initial extrinsic parameter, perform perspective transformation on the first image acquired by each camera to obtain each second image;

[0095] After obtaining the initial extrinsic parameters corresponding to each camera, the perspective transformation matrix is ​​determined based on the initial extrinsic parameters. The perspective transformation matrix is ​​then used to perform perspective transformation on the first image acquired by each camera, thereby realizing the viewpoint correction operation on the first image and converting the first image into a second image under the camera's frontal view angle.

[0096] For example, such as Figure 5 The image shown is a schematic diagram of a perspective transformation provided in this application. Figure 5 In the image M, the first image captured by the camera is shown. Side 1 in image M represents the camera's location, and side 2 represents the location furthest from the camera. As shown in image M, the size of side 1 is larger than that of side 2. Therefore, the first image captured by the camera is not a standard rectangle; that is, the first image M is distorted due to the acquisition angle. Therefore, to improve the accuracy of the calibration board corner point recognition in the first image, a perspective transformation operation needs to be performed on the first image. The second image obtained after the perspective transformation is shown in image N. In image N, the size of side 1 is equal to that of side 2, forming a standard rectangle, indicating that a perspective correction has been performed on image M.

[0097] S43, Identify all first target corner points of the calibration board in each second image;

[0098] After obtaining each second image, to avoid ghosting between images acquired from two adjacent shared viewing areas, in this embodiment, firstly, all corner points of the calibration board in each second image need to be identified; then, abnormal corner points corresponding to the shared viewing area are determined among all corner points; finally, these abnormal corner points are removed from all corner points to obtain all first target corner points. The shared viewing area is the area where the acquisition ranges of the two cameras overlap.

[0099] In the above process, the methods for identifying abnormal corner points in the shared viewing area include:

[0100] In each second image, the first common viewpoints corresponding to the common view area are identified; each first common viewpoint is matched with each second common viewpoint corresponding to the common view area in the world coordinate system. During the matching process, a matching base point needs to be selected so that the first common viewpoints and the second common viewpoints can be successfully matched; if there are corner points in each first common viewpoint that cannot be matched, the corner points that cannot be matched are identified as abnormal corner points.

[0101] For example, such as Figure 6 The image shown is a schematic diagram illustrating the matching of the first and second common viewpoints. Figure 6 In the diagram, solid dots represent the first common viewpoints, and hollow dots represent the second common viewpoints, which serve as reference corner points for matching. When matching the first and second common viewpoints, a matching base point must first be determined to ensure that more corner points within the first common viewpoints can successfully match with the second common viewpoints. In the example, the upper right corner of each first common viewpoint is used as the matching base point. It is found that five first common viewpoints still cannot match successfully; these five unmatched first common viewpoints are considered abnormal corner points. Finally, the abnormal corner points are removed from the first common viewpoints, resulting in all the first target corner points.

[0102] S44, based on the first image coordinates corresponding to each first target corner point, jointly adjust each initial extrinsic parameter.

[0103] After determining all the first target corner points in the second image, the initial extrinsic parameters are jointly adjusted based on the first image coordinates corresponding to each first target corner point. Specifically:

[0104] First, determine the first image coordinates corresponding to each first target corner point;

[0105] As a preferred approach, before determining the first image coordinates corresponding to each first target corner point, sub-pixel recognition can be performed on all first target corner points to improve the corner point recognition accuracy.

[0106] Furthermore, since the calibration board corner points in each first image are numbered according to a preset order when the camera captures each first image (e.g., from top to bottom, from left to right), the second world coordinates corresponding to the coordinates of each first image are determined based on the arrangement order of the first target corner points. For example, the first image coordinates corresponding to the corner point numbered N in the second image are determined to be corresponding to the calibration board corner point numbered N in the world coordinates.

[0107] Furthermore, the corresponding second image coordinates and second world coordinates are input into a preset model to obtain the optimization parameters for each camera. The preset model can be a Global Optimization (GO) model. Finally, the initial extrinsic parameters are jointly adjusted according to the optimization parameters.

[0108] By performing distortion correction on the images acquired by each camera, the initial extrinsic parameters for each camera are obtained. Based on the initial extrinsic parameters, perspective transformation is performed on the distorted images to correct the viewing angle of the acquired images, thereby improving the accuracy of the calibration board corner point recognition and making the calibration of the extrinsic parameters of each camera more accurate.

[0109] Based on the same inventive concept, this application also provides a multi-camera extrinsic parameter calibration device, such as... Figure 7 The diagram shown is a structural schematic of a multi-camera extrinsic parameter calibration device according to this application. The device includes:

[0110] Module 71 is used to determine the initial extrinsic parameters corresponding to each camera.

[0111] Transformation module 72 is used to perform perspective transformation on the first images acquired by each camera according to the perspective transformation matrix corresponding to each initial extrinsic parameter, so as to obtain each second image.

[0112] The recognition module 73 is used to identify all the first target corner points of the calibration board in each second image;

[0113] The adjustment module 74 is used to jointly adjust the initial extrinsic parameters based on the first image coordinates corresponding to each first target corner point.

[0114] In one possible design, the determining module 71 is specifically used for:

[0115] The third images acquired by each camera are subjected to distortion correction to obtain the fourth images.

[0116] Determine the coordinates of the second image corresponding to the preset corner points of the calibration plate in each of the fourth images;

[0117] Based on the arrangement order between each preset corner point, the first world coordinates corresponding to each second image coordinate are determined, wherein the first world coordinates are three-dimensional reference coordinates;

[0118] Based on the second image coordinates and the first world coordinates, the initial extrinsic parameters corresponding to each camera are determined.

[0119] In one possible design, the transformation module 72 is specifically used for:

[0120] Calculate the perspective transformation matrix for each camera based on the initial extrinsic parameters;

[0121] The perspective transformation is performed on the first images acquired by each camera using the transformation matrix to obtain the second images.

[0122] In one possible design, the identification module 73 is specifically used for:

[0123] Identify all corner points of the calibration board in each second image;

[0124] Among all the corner points, abnormal corner points corresponding to the common viewing area are identified, wherein the common viewing area is the area where the acquisition range of the two cameras overlaps;

[0125] The abnormal corner points are removed from all the corner points to obtain all the first target corner points.

[0126] In one possible design, the identification module 73 is further used for:

[0127] In each of the second images, the first common viewing point corresponding to the common viewing area is determined;

[0128] Match each first common viewpoint with each second common viewpoint corresponding to the common view area in the world coordinate system;

[0129] If any of the first common viewpoints contains a corner point that cannot be matched, then the corner point that cannot be matched is identified as the abnormal corner point.

[0130] In one possible design, the adjustment module 74 is specifically used for:

[0131] Determine the first image coordinates corresponding to each first target corner point;

[0132] Based on the arrangement order of the first target corner points, the second world coordinates corresponding to each first image coordinate are determined;

[0133] Based on the input of each second image coordinate and each second world coordinate into the preset model, the optimization parameters corresponding to each camera are obtained;

[0134] Each initial extrinsic parameter is adjusted in conjunction with the other optimization parameters.

[0135] The multi-camera extrinsic parameter calibration device described above performs distortion correction on the images acquired by each camera to obtain the initial extrinsic parameters corresponding to each camera. Based on the initial extrinsic parameters, perspective transformation is performed on the distorted images to correct the viewing angle of the acquired images, thereby improving the recognition accuracy of the calibration board corner points and making the calibration accuracy of each camera extrinsic parameter higher.

[0136] Based on the same inventive concept, this application also provides an electronic device that can realize the functions of the aforementioned multi-camera extrinsic parameter calibration method device. (Refer to...) Figure 8 The electronic device includes:

[0137] At least one processor 81 and a memory 82 connected to the at least one processor 81. In this embodiment, the specific connection medium between the processor 81 and the memory 82 is not limited. Figure 8 The example shown is the connection between processor 81 and memory 82 via bus 80. Bus 80 is... Figure 8 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Bus 80 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 8 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 81 can also be called a controller; there is no restriction on the name.

[0138] In this embodiment, memory 82 stores instructions executable by at least one processor 81. By executing the instructions stored in memory 82, at least one processor 81 can perform the multi-camera extrinsic calibration method discussed above. Processor 81 can implement... Figure 7 The functions of each module in the device shown.

[0139] The processor 81 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 82 and calling data stored in memory 82, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0140] In one possible design, processor 81 may include one or more processing units. Processor 81 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 81. In some embodiments, processor 81 and memory 82 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0141] Processor 81 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the multi-camera extrinsic parameter calibration method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0142] Memory 82, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 82 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 82 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 82 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0143] By designing and programming the processor 81, the code corresponding to the multi-camera extrinsic parameter calibration method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during runtime. Figure 4 The steps of the multi-camera extrinsic parameter calibration method in the illustrated embodiment are as follows. How to design and program the processor 81 is a technique well-known to those skilled in the art and will not be described further here.

[0144] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the multi-camera extrinsic parameter calibration method described above.

[0145] In some possible implementations, various aspects of the multi-camera extrinsic parameter calibration method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the multi-camera extrinsic parameter calibration method according to the various exemplary embodiments of this application described above.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for calibrating extrinsic parameters of multiple cameras, characterized in that, The method includes: Determine the initial extrinsic parameters for each camera; Based on the perspective transformation matrix corresponding to each initial extrinsic parameter, the first image acquired by each camera is transformed by perspective to obtain each second image. Identify all first target corner points of the calibration board in each second image; Based on the first image coordinates corresponding to each first target corner point, the initial extrinsic parameters are jointly adjusted, including: Determine the first image coordinates corresponding to each first target corner point; Based on the arrangement order of the first target corner points, the second world coordinates corresponding to each first image coordinate are determined; Based on the input of each second image coordinate and each second world coordinate into the preset model, the optimization parameters corresponding to each camera are obtained; Each initial extrinsic parameter is adjusted in conjunction with the other optimization parameters.

2. The method as described in claim 1, characterized in that, The determination of the initial extrinsic parameters for each camera includes: The third images acquired by each camera are subjected to distortion correction to obtain the fourth images. Determine the coordinates of the second image corresponding to the preset corner points of the calibration plate in each of the fourth images; Based on the arrangement order between each preset corner point, the first world coordinates corresponding to each second image coordinate are determined, wherein the first world coordinates are three-dimensional reference coordinates; Based on the second image coordinates and the first world coordinates, the initial extrinsic parameters corresponding to each camera are determined.

3. The method as described in claim 1, characterized in that, The step of performing perspective transformation on the first images acquired by each camera based on the perspective transformation matrix corresponding to each initial extrinsic parameter to obtain each second image includes: Calculate the perspective transformation matrix for each camera based on the initial extrinsic parameters; The perspective transformation is performed on the first images acquired by each camera using the transformation matrix to obtain the second images.

4. The method as described in claim 1, characterized in that, The identification of all first target corner points of the calibration board in each second image includes: Identify all corner points of the calibration board in each second image; Among all the corner points, abnormal corner points corresponding to the common viewing area are identified, wherein the common viewing area is the area where the acquisition range of the two cameras overlaps; The abnormal corner points are removed from all the corner points to obtain all the first target corner points.

5. The method as described in claim 4, characterized in that, The step of identifying the abnormal corner points corresponding to the common viewing area among all corner points includes: In each of the second images, the first common viewing point corresponding to the common viewing area is determined; Match each first common viewpoint with each second common viewpoint corresponding to the common view area in the world coordinate system; If any of the first common viewpoints contains a corner point that cannot be matched, then the corner point that cannot be matched is identified as the abnormal corner point.

6. A multi-camera extrinsic parameter calibration device, characterized in that, The device includes: The determination module is used to determine the initial extrinsic parameters for each camera. The transformation module is used to perform perspective transformation on the first images acquired by each camera according to the perspective transformation matrix corresponding to each initial extrinsic parameter, so as to obtain each second image. The recognition module is used to identify all the first target corner points of the calibration board in each second image; The adjustment module is used to jointly adjust the initial extrinsic parameters based on the first image coordinates corresponding to each first target corner point, including: Determine the first image coordinates corresponding to each first target corner point; Based on the arrangement order of the first target corner points, the second world coordinates corresponding to each first image coordinate are determined; Based on the input of each second image coordinate and each second world coordinate into the preset model, the optimization parameters corresponding to each camera are obtained; Each initial extrinsic parameter is adjusted in conjunction with the other optimization parameters.

7. The apparatus as claimed in claim 6, characterized in that, The determining module is specifically used for: The third images acquired by each camera are subjected to distortion correction to obtain the fourth images. Determine the coordinates of the second image corresponding to the preset corner points of the calibration plate in each of the fourth images; Based on the arrangement order between each preset corner point, the first world coordinates corresponding to each second image coordinate are determined, wherein the first world coordinates are three-dimensional reference coordinates; Based on the second image coordinates and the first world coordinates, the initial extrinsic parameters corresponding to each camera are determined.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method steps of any one of claims 1-5.

Images