Extrinsic parameter calibration method and application suitable for multi-sensor camera multi-angle application

Abstract

The application provides an external parameter calibration method and device suitable for a multi-sensing camera scene and application. The external parameter calibration is performed on a multi-sensing camera scene formed by at least four peripheral sensing cameras arranged at different positions and a central sensing camera arranged at a central position. The method comprises the following steps: local calibration is performed on the peripheral sensing cameras and the central sensing camera to obtain external parameters of the peripheral sensing cameras relative to the central sensing camera; a total error is obtained based on the sum of the re-projection errors of the plurality of peripheral sensing cameras relative to each other; and the external parameters corresponding to the minimum total error are solved. The method is particularly suitable for external parameter calibration of a multi-sensing camera arranged in a bowl structure.

Description

Technical Field

[0001] This application relates to the field of calibration, and in particular to an external parameter calibration method and application suitable for multi-sensor camera multi-angle applications. Background Technology

[0002] The primary function of a camera sensor is to acquire images of the real world and convert them into images. This transformation from 3D points in the world coordinate system to pixels in the image requires the use of both the world coordinate system and the camera coordinate system. The parameters R and T needed for this transformation are called camera extrinsic parameters. These parameters enable a rigid transformation from the world coordinate system to the camera coordinate system. Camera extrinsic parameters relate to the attitude and positional relationships between points in the 3D space corresponding to the world coordinate system and the camera corresponding to the camera coordinate system; their values ​​are uncertain in different application scenarios. The transformation from the camera coordinate system to pixels in the image requires camera intrinsic parameters. Camera intrinsic parameters are related to the camera's physical characteristics and are often represented as matrices containing focal lengths fx and fy and optical center offsets cx and cy.

[0003] In image measurement and machine vision applications, it is necessary to know the camera's extrinsic and intrinsic parameters. The process of solving for these parameters is called camera calibration. In other words, camera calibration restores points in an image to a real 3D space by solving for the camera's intrinsic and extrinsic parameters, thus accurately reflecting the actual relative spatial relationships between points. Current camera calibration methods mainly include three types: 1. Traditional calibration methods: using a well-defined, high-precision 3D reference object, establishing a camera model through the correspondence between points and image points, and using optimization algorithms to obtain camera parameters; 2. Self-calibration methods: not requiring any external special calibration objects, but only utilizing the information of corresponding points in the image to directly complete the calibration task; 3. Active vision calibration methods: linearly solving for the camera model parameters based on autonomously acquired image data. However, regardless of the specific method, all camera calibration methods focus on calibrating single camera parameters and are not well-suited for multi-sensor camera applications.

[0004] Specifically, current methods for calibrating the extrinsic parameters of multi-sensor cameras, taking binocular cameras as an example, require two cameras to simultaneously observe the same calibration board. The left and right cameras observe corner points on the calibration board to determine the positioning of the two cameras in the calibration board's coordinate system, thus obtaining initial extrinsic parameters. These initial extrinsic parameters are then used as optimization variables to minimize the reprojection error of each camera to obtain accurate extrinsic parameters. However, this method is only suitable for calibrating a small number of multi-sensor cameras. As the number of cameras increases, the calculation process becomes extremely cumbersome. Furthermore, it is not applicable to calibrating the extrinsic parameters of multiple cameras in different orientations, and only considers the transformation along a single axis of the sensor camera during calibration. Summary of the Invention

[0005] This application provides a method and application for extrinsic parameter calibration suitable for multi-sensor cameras in multi-angle applications. It is particularly suitable for extrinsic parameter calibration of multi-sensor cameras arranged in a bowl shape, reducing incomplete registration phenomena such as ghosting between data collected by multiple sensor cameras.

[0006] In a first aspect, embodiments of this application provide an extrinsic parameter calibration method applicable to multi-sensor cameras, the method comprising:

[0007] Local calibration: Obtain the first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain the second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct the relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship.

[0008] Global calibration: Select any two peripheral sensors and the center sensor to form a triangular line of sight area. Calculate the target pixel coordinates on the three sensors corresponding to the physical position of any target within the triangular line of sight area. Calculate the reprojection error corresponding to the three sensors based on the target pixel coordinates.

[0009] The total error is obtained by statistically analyzing all reprojection errors corresponding to all triangular line-of-sight regions. The extrinsic parameters corresponding to the local calibration are then calculated to minimize the total error.

[0010] Secondly, embodiments of this application provide an extrinsic parameter calibration device suitable for multi-sensor cameras, comprising: a local calibration unit, configured to obtain a first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain a second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct a relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship; and a global calibration unit, configured to select any two peripheral sensor cameras and the central sensor camera to form a triangular line-of-sight region; calculate the target pixel coordinates corresponding to any target physical position in the triangular line-of-sight region on the three sensor cameras; calculate the reprojection error corresponding to the three sensor cameras based on the target pixel coordinates; statistically analyze all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error; and solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

[0011] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any of the described extrinsic parameter calibration methods applicable to multi-sensor camera scenarios.

[0012] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, the computer program including program code for controlling a process to execute the process, the process including any of the extrinsic parameter calibration methods applicable to multi-sensor camera scenarios described above.

[0013] The main contributions and innovations of this invention are as follows:

[0014] This application provides extrinsic parameter calibration specifically for multi-sensor camera scenarios arranged in a bowl shape. The peripheral sensors are distributed in four directions, while the central sensor is located at the center. The peripheral and central sensors form a bowl-shaped structure, and the viewing angles of adjacent sensors overlap. This solution first addresses the extrinsic parameter calibration of the peripheral and central sensors, then addresses the extrinsic parameter calibration between the peripheral sensors themselves. It comprehensively considers global calibration to minimize errors, optimizes external interactions between cameras, and reduces incomplete registration phenomena such as ghosting between the data collected by the bowl-shaped sensors.

[0015] In addition, this solution provides an external parameter calibration method that can use either planar or three-dimensional targets, and can perform systematic external parameter calibration on multi-sensor cameras without the need for specially structured calibration tools.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the arrangement structure of a multi-sensor camera in a multi-angle application according to the bowl-shaped structure of the application embodiment; a flowchart of the extrinsic parameter calibration method applicable to multi-sensor camera scenarios according to the embodiments of this application.

[0019] Figure 2 This is a flowchart of an external parameter calibration method applicable to multi-sensor camera multi-angle applications according to embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the logic flow of a multi-sensor camera scene arranged in a bowl-shaped structure according to an embodiment of this application;

[0021] Figure 4 This is a structural block diagram of an extrinsic parameter calibration device applicable to multi-sensor camera scenarios according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0024] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0025] Example 1

[0026] This application provides an extrinsic parameter calibration method suitable for multi-sensor camera multi-angle applications, which can realize the extrinsic parameter calibration of multi-sensor camera scenarios with bowl-shaped structure arrangement, and minimize the calibration error of each sensor camera.

[0027] Before introducing the specific details of this solution, we will first explain the multi-sensor camera scenario with a bowl-shaped structure that is particularly suitable for this solution. For example... Figure 2 As shown, four peripheral sensors are positioned at the four cardinal points of a space, while a central sensor is located at the center. These four peripheral sensors and the central sensor together form a bowl-shaped arrangement of multiple sensors. It is worth noting that the four peripheral sensors are placed on the same horizontal plane and are evenly spaced apart from each other. The central sensor is located at the center of the square space surrounded by the four peripheral sensors, and the distance between the central sensor and each peripheral sensor is the same. In a preferred embodiment, the plane containing the central sensor is at the same height as the plane containing the peripheral sensors.

[0028] The multi-sensor cameras arranged in a bowl-shaped structure provided by this solution can be used to construct virtual scenes with a full spatial perspective. Cameras placed in different locations project images onto different parts of the space to create a visually comprehensive effect. The targets captured by the multi-sensor cameras can be either two-dimensional or three-dimensional targets, depending on the specific needs.

[0029] like Figure 1 As shown, the extrinsic parameter calibration method provided in this solution, applicable to multi-sensor camera multi-angle applications, calibrates the extrinsic parameters of a multi-sensor camera scene consisting of at least four peripheral sensors placed in different locations and a central sensor camera placed in the center. The method includes the following steps:

[0030] Local calibration: Obtain the first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain the second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct the relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship.

[0031] Global calibration: Select any two peripheral sensors and the central sensor to form a triangular line-of-sight region. Calculate the target pixel coordinates on the three sensors corresponding to the physical position of any target within the triangular line-of-sight region. Calculate the reprojection error corresponding to the three sensors based on the target pixel coordinates. Statistically calculate all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error. Solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

[0032] It's worth noting that the image camera packages in the central and peripheral sensors of this solution can be either LFET-LNFRARED or RGB cameras. This solution divides the calibration of multiple sensors in multi-sensor scenarios into two steps: first, based on the extrinsic parameters of the LFET-LNFRARED or RGB cameras, the extrinsic parameter calibration between the four peripheral sensors and the central sensor is resolved; then, a global calibration is performed on the four peripheral sensors to reduce ghosting caused by overlapping viewing areas, and the overall error of the multi-sensor system is reduced through global calibration. This solution uses adjustment to reduce the error distribution during global calibration and minimize ghosting caused by overlapping viewing areas of the four peripheral sensors.

[0033] It is worth noting that for calibrated sensor cameras, the LFET-LNFRARED or RGB camera and the infrared image sensor camera built into the sensor camera are also calibrated, and the relative extrinsic parameters between the LFET-LNFRARED or RGB camera and the infrared image sensor camera are also calibrated. Therefore, this solution can convert the extrinsic parameters of the LFET-LNFRARED or RGB camera on the two calibrated sensor cameras into relative extrinsic parameters with the infrared image sensor camera as the coordinate origin, thereby achieving the calibration of the two sensor cameras. It is worth noting that calibrated sensor cameras refer to those in which both the RGB image sensor and the infrared image sensor camera have had their intrinsic parameters calibrated, and their extrinsic parameters have also been calibrated.

[0034] The sensor camera used in this solution can be RealSense, a 3D camera that supports real-time perception and computation. Of course, other types or models of sensor cameras can also be selected. This method can also be used to calibrate two or more sensors, including cameras equipped with two RGB image sensors. If RealSense is selected, the LFET-LNFRARED or RGB camera, the left infrared image sensor, and the right infrared image sensor within RealSense are already calibrated. In some embodiments, the extrinsic parameters of the LFET-LNFRARED or RGB camera can be converted to relative extrinsic parameters with the left infrared image sensor as the coordinate origin.

[0035] In the step of "obtaining the first imaging formula for the target's physical coordinates and pixel coordinates obtained by the central sensing camera with the image camera as the origin of the coordinate system",

[0036] The first pixel coordinates of the image camera of the target at the central sensor camera, the spatial orientation position of the image camera relative to the target, and the target physical coordinates in the physical coordinate system are obtained. The imaging formula of the target physical coordinates and the target pixel coordinates is constructed to obtain the first imaging formula.

[0037] In the step of "obtaining the second imaging formula for the target's physical coordinates and pixel coordinates from the surrounding sensor cameras with the center of the image camera as the origin",

[0038] The target pixel coordinates on the image camera of the surrounding sensor camera are obtained, the spatial attitude position of the image camera relative to the target and the target physical coordinates in the physical coordinate system are obtained, and the imaging formula of the target physical coordinates and the target pixel coordinates is constructed to obtain the second imaging formula.

[0039] Specifically, the spatial relationship between the camera coordinates centered on the image sensor and the target coordinates is as follows:

[0040] (1)

[0041] in To calibrate the target's physical coordinates in the physical coordinate system within the image. and This refers to the spatial pose and position of the image from the camera relative to the target. R represents the rotation relationship, and t represents the positional movement relationship. It is the target pixel coordinate of the calibration point in the calibration image obtained by the image sensor.

[0042] The imaging relationship between the target pixel coordinates centered on the image sensor and the target physical coordinates is as follows:

[0043] (2)

[0044] in These are the target pixel coordinates in the calibration image acquired by the image sensor. This represents the intrinsic parameter matrix of the corresponding sensor camera. It refers to the target's physical coordinates in the physical coordinate system within the calibration image.

[0045] The imaging formula, which combines the imaging relationship and the spatial relationship to obtain the target's physical coordinates and pixel coordinates with the center of the image sensor as the origin, is as follows:

[0046] (3)

[0047] The first imaging relationship between the target's physical coordinates and pixel coordinates for the central sensor camera is: in It refers to any physical coordinate of the target within the target physical coordinate system. The corresponding target pixel coordinates in the image of the central sensor camera;

[0048] The second imaging formula for the target's physical coordinates and pixel coordinates from the peripheral sensor camera is:

[0049]

[0050] in , This refers to the target pixel coordinates in the image of the surrounding sensor camera.

[0051] In the step of "constructing the relative coordinate transformation relationship between each of the peripheral sensing cameras and the central sensing camera based on the first imaging formula and the second imaging formula",

[0052] Using the center of the image sensor of the central sensing camera as the center coordinate, the first coordinate transformation relationship of each peripheral sensing camera relative to the central sensing camera is obtained. The first imaging formula and the second imaging formula are substituted into the first coordinate transformation relationship to obtain the relative coordinate transformation relationship.

[0053] Specifically, taking the center of the image sensor of the central sensing camera as the coordinate center, the rotation of the central sensing camera relative to the peripheral sensing cameras as R, and the translation of the central sensing camera relative to the peripheral sensing cameras as T, the relative coordinate transformation is R and T, then we have

[0054]

[0055] in This indicates the coordinate center with the sensor camera as the origin.

[0056] At this point, the first coordinate transformation relationship exists:

[0057]

[0058] in The coordinates represent the camera coordinate system coordinates of the central sensor camera. These are the coordinates of the camera coordinate system of the surrounding sensor cameras.

[0059] Substituting the first imaging formula and the second imaging formula, we obtain the relative coordinate transformation relationship:

[0060]

[0061] in ,and , Substituting, R is a 3 The 3-dimensional orthogonal matrix, after substituting the pixel coordinates corresponding to the calibration points of the target in the central and peripheral sensors into the relative coordinate transformation relationship, yields two equations. Therefore, there are 7 unknowns in the relative coordinate transformation relationship, and it is necessary to substitute at least four target pixel coordinates corresponding to the calibration points to solve the extrinsic parameters of each peripheral sensor relative to the central sensor.

[0062] The above steps have completed the extrinsic parameter calibration of any peripheral sensor camera relative to the central sensor camera. However, in practical applications, the viewing angles of the peripheral sensors overlap, so the ghosting problem still needs to be addressed. In the step of "selecting any two peripheral sensors and the central sensor camera to form a triangular viewing area", two adjacent peripheral sensors are selected. If there are four peripheral sensors placed at the four corners, then a total of four groups of two peripheral sensors can be selected.

[0063] In the step of "calculating the reprojection error corresponding to the two peripheral sensors and the center sensor based on the target pixel coordinates", the first reprojection error of any two sensors is calculated based on the target pixel coordinates, and the three first reprojection errors are combined to obtain the reprojection error.

[0064] The first projection error is obtained by squared the product of the two target pixel coordinates on the two sensor cameras and the relative coordinate transformation relationship.

[0065] For example, the formula for calculating the first projection error is as follows:

[0066]

[0067] in (i=1,2,) represents the target pixel coordinates on the sensor camera. Relative coordinate transformation relationship:

[0068] .

[0069] The formula for calculating the reprojection error by combining the three first reprojection errors is as follows:

[0070]

[0071] In the step of “statistically calculating all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error”, two adjacent peripheral sensors and the center sensor are randomly selected from the peripheral sensors to form a group, resulting in four groups of reprojection errors. The total error is obtained by summing the four groups of reprojection errors.

[0072] The formula for calculating the total projection error is as follows:

[0073] .

[0074] The extrinsic parameters corresponding to the local calibration that minimize the total error are used as the extrinsic parameters for each sensor camera.

[0075] In some preferred embodiments, in order to optimize the calibration effect, in the step of "obtaining the total error from all reprojection errors corresponding to all triangular line-of-sight regions and solving for the extrinsic parameters corresponding to the local calibration when the total error is minimized", this scheme calculates the mutual extrinsic parameter relationship between the three devices constituting each triangular line-of-sight region, and performs adjustment processing on the extrinsic parameter relationship using the principle of minimizing the reprojection error of the three sensor cameras to obtain the optimal extrinsic parameter matrix between every two sensor cameras in each triangular line-of-sight region. All optimal extrinsic parameter matrices are used as initial values ​​and substituted into the total error calculation for adjustment optimization.

[0076] Specifically, the mutual extrinsic relationship between the three sensors constituting each triangular line-of-sight region is calculated, and the extrinsic relationship is adjusted using the principle of minimizing the reprojection error of the three sensors to obtain the optimal extrinsic matrix between two sensors in the triangular line-of-sight region based on the minimum reprojection error. Using the above principle and method, the optimal extrinsic matrix between every two cameras in each triangular region is calculated.

[0077] Then, all the optimal extrinsic parameter matrices obtained above are used as initial values ​​and substituted into the total projection error formula of the entire device for adjustment optimization to obtain the optimal extrinsic parameter matrices between each pair of the five cameras, thereby completing the calibration of the entire device.

[0078] Example 2

[0079] Based on the same concept, referencing Figure 4 This application also proposes an extrinsic parameter calibration device suitable for multi-sensor camera scenarios, which calibrates the extrinsic parameters of a multi-sensor camera scenario consisting of four peripheral sensors placed around the perimeter and a central sensor camera placed in the center, including the following:

[0080] A local calibration unit is used to obtain a first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain a second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct a relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship.

[0081] The global calibration unit is used to calculate the target pixel coordinates corresponding to the physical position of any target in the triangular line-of-sight region on the three sensor cameras, calculate the reprojection error corresponding to the three sensor cameras based on the target pixel coordinates, statistically analyze all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error, and solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

[0082] For the technical content in Embodiment 2 that is the same as in Embodiment 1, please refer to the description in Embodiment 1. Repeated content will not be repeated here.

[0083] Example 3

[0084] This embodiment also provides an electronic device, see reference. Figure 5 It includes a memory 404 and a processor 402, the memory 404 storing a computer program and the processor 402 being configured to run the computer program to perform the steps in any of the above embodiments of the extrinsic parameter calibration method applicable to multi-sensor camera scenarios.

[0085] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0086] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0087] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0088] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the extrinsic parameter calibration methods applicable to multi-sensor camera scenarios in the above embodiments.

[0089] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402 and the input / output device 408 is connected to the processor 402.

[0090] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0091] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program:

[0092] Local calibration: Obtain the first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain the second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct the relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship.

[0093] Global calibration: Select any two peripheral sensors and the central sensor to form a triangular line-of-sight region. Calculate the target pixel coordinates on the three sensors corresponding to the physical position of any target within the triangular line-of-sight region. Calculate the reprojection error corresponding to the three sensors based on the target pixel coordinates. Statistically calculate all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error. Solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

[0094] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0095] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0096] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer-executable components configured to perform embodiments when the program is run. One or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted that any block in the logical flow of the figures may represent a program step, or interconnected logical circuitry, blocks and functions, or a combination of program steps and logical circuitry, blocks and functions. The software may be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0097] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for extrinsic parameter calibration applicable to multi-sensor camera scenarios, wherein the extrinsic parameters are calibrated for a multi-sensor camera scenario consisting of at least four peripheral sensors placed in different locations and a central sensor placed in the center, characterized in that... This includes the following steps: Four peripheral sensors are placed on the same horizontal plane and are evenly spaced apart from each other. The central sensor is placed at the center of the square space surrounded by the four peripheral sensors, and the distance between the central sensor and each peripheral sensor is the same. Local calibration: Obtain the first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain the second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct the relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship. Global calibration: Select any two peripheral sensors and the central sensor to form a triangular line-of-sight region. Calculate the target pixel coordinates on the three sensors corresponding to the physical position of any target within the triangular line-of-sight region. Calculate the reprojection error corresponding to the three sensors based on the target pixel coordinates. Statistically calculate all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error. Solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

2. The extrinsic parameter calibration method for multi-sensor camera scenarios according to claim 1, characterized in that, In the step of "obtaining the first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensing camera with the center of the image camera as the coordinate origin", The target pixel coordinates of the image camera at the center sensor camera, the spatial orientation position of the image camera relative to the target, and the target physical coordinates in the physical coordinate system are obtained. An imaging formula for the target physical coordinates and target pixel coordinates is constructed to obtain the first imaging formula.

3. The extrinsic parameter calibration method applicable to multi-sensor camera scenarios according to claim 1, characterized in that, In the step of "obtaining the second imaging formula for the target's physical coordinates and pixel coordinates from the surrounding sensor cameras with the center of the image camera as the origin", The target pixel coordinates on the image camera of the surrounding sensor camera are obtained, the spatial attitude position of the image camera relative to the target and the target physical coordinates in the physical coordinate system are obtained, and the imaging formula of the target physical coordinates and the target pixel coordinates is constructed to obtain the second imaging formula.

4. The extrinsic parameter calibration method for multi-sensor camera scenarios according to claim 1, characterized in that, In the step of "constructing the relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first imaging formula and the second imaging formula", the center of the image sensor of the central sensor camera is used as the center coordinate to obtain the first coordinate transformation relationship of each peripheral sensor camera relative to the central sensor camera. The first imaging formula and the second imaging formula are substituted into the first coordinate transformation relationship to obtain the relative coordinate transformation relationship.

5. The extrinsic parameter calibration method for multi-sensor camera scenarios according to claim 1, characterized in that, In the step of "calculating the reprojection error corresponding to the three sensor cameras based on the target pixel coordinates", the first reprojection error of any two sensor cameras is calculated based on the target pixel coordinates, and the three first reprojection errors are combined to obtain the reprojection error.

6. The extrinsic parameter calibration method for multi-sensor camera scenarios according to claim 4, characterized in that, The first projection error is obtained by squared the product of the two target pixel coordinates on the two sensor cameras and the relative coordinate transformation relationship.

7. The extrinsic parameter calibration method applicable to multi-sensor camera scenarios according to claim 1, characterized in that, In the step of "obtaining the total error from all reprojection errors corresponding to all triangular line-of-sight regions and solving for the local calibration corresponding to the minimum total error", the mutual extrinsic parameter relationship between the three devices constituting each triangular line-of-sight region is calculated. The extrinsic parameter relationship is then adjusted using the principle of minimizing the reprojection error of the three sensors to obtain the optimal extrinsic parameter matrix between every two sensors in each triangular line-of-sight region. All optimal extrinsic parameter matrices are used as initial values ​​and substituted into the total error calculation for adjustment optimization.

8. An extrinsic parameter calibration device suitable for multi-sensor camera scenarios, for calibrating extrinsic parameters of a multi-sensor camera scenario consisting of four peripheral sensors placed around the perimeter and a central sensor camera placed in the center, characterized in that, include: Four peripheral sensors are placed on the same horizontal plane and are evenly spaced apart from each other. The central sensor is placed at the center of the square space surrounded by the four peripheral sensors, and the distance between the central sensor and each peripheral sensor is the same. A local calibration unit is used to obtain a first imaging formula for the target physical coordinates and target pixel coordinates obtained by the central sensor camera with the center of the image camera as the coordinate origin; obtain a second imaging formula for the target physical coordinates and target pixel coordinates obtained by each peripheral sensor camera with the center of the image camera as the coordinate origin; construct a relative coordinate transformation relationship between each peripheral sensor camera and the central sensor camera based on the first and second imaging formulas; and solve for the extrinsic parameters of each peripheral sensor camera relative to the central sensor camera by substituting the pixel coordinates corresponding to at least four calibration points into the relative coordinate transformation relationship. The global calibration unit is used to select any two peripheral sensors and the central sensor to form a triangular line-of-sight region, calculate the target pixel coordinates corresponding to the physical position of any target in the triangular line-of-sight region on the three sensors, calculate the reprojection error corresponding to the three sensors based on the target pixel coordinates, statistically analyze all reprojection errors corresponding to all triangular line-of-sight regions to obtain the total error, and solve for the extrinsic parameters corresponding to the local calibration when the total error is minimized.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the extrinsic parameter calibration method for multi-sensor camera scenarios as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that is executed by a processor to perform the extrinsic parameter calibration method for multi-sensor camera scenarios as described in any one of claims 1 to 8.

Images