Global pose calibration method for cameras with non-overlapping field of view by integrating multiple constraints

Abstract

The present application proposes a global pose calibration method for cameras with non-overlapping field of view that integrates multiple constraints. The method includes: obtaining an image containing a two-dimensional code target captured by the camera to be calibrated; determining the global pose of the two-dimensional code target using a map established by an additional mobile camera in combination with SLAM; estimating the camera's pose based on the image and the global pose of the two-dimensional code target to obtain the camera's initial global pose; fusing the ORB feature points of the image detection environment captured by the camera with the pose information of the two-dimensional code target to construct a multiple constraint function, which includes a reprojection error constraint function and a plane consistency constraint function; using dynamic weights to fuse the multiple constraint functions to obtain a joint optimization objective function, and using a nonlinear optimization algorithm to solve the optimal solution of the joint optimization objective function to obtain the camera's target global pose. The present application can improve the calibration accuracy and is suitable for the dynamically changing and complex environment of underground coal mine scenes.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a global pose calibration method for a camera with non-overlapping field of view that integrates multiple constraints. Background Art

[0002] The fusion of multi-sensor data relies on accurate global pose calibration. This requires determining the camera's position and orientation in three-dimensional space to ensure consistency in the coordinate space of each sensor. Camera calibration involves both intrinsic and extrinsic parameters. The camera's intrinsic parameters determine its lens characteristics. The Zhang Zhengyou calibration method can yield relatively accurate results for the intrinsic parameters. To obtain the complete pose between multiple cameras, the extrinsic parameters must also be calculated. Multi-camera calibration can determine the relative position and pose relationships between different cameras, enabling data fusion and collaborative processing across all cameras in a common coordinate space.

[0003] Existing camera calibration methods have limitations in underground coal mine excavation scenarios, mainly in the following three aspects: (1) The overlapping fields of view between cameras are very small or even non-existent, making calibration methods based on common view features ineffective; (2) Positioning based on total stations / UWB can only provide sensor position information and cannot calibrate attitude parameters. It also relies on manual measurement, which is inefficient; (3) Existing non-overlapping field of view camera calibration methods require the pre-placement of high-precision calibration plates in a dedicated calibration room. However, the maintenance space in underground excavation tunnels is small and time-sensitive, making it difficult to meet deployment requirements. Due to the special requirements of underground scenarios, a calibration solution that does not require common view, is highly robust, and is easy to deploy is urgently needed. Summary of the Invention

[0004] The embodiments of the present application provide a global pose calibration method for cameras with non-overlapping fields of view that integrates multiple constraints, which can solve the calibration difficulties of multi-camera systems in coal mines caused by the lack of a common viewing area, dynamic scene changes, and the inefficiency of traditional calibration methods.

[0005] According to a first aspect of an embodiment of the present application, a global pose calibration method for a camera with a non-overlapping field of view that integrates multiple constraints is provided. The camera is an airborne camera in mining equipment. The calibration method includes:

[0006] Acquiring an image captured by the camera containing a QR code target, wherein the QR code target is fixed to an anchor net of a cable tie on both sides of the lane;

[0007] Determine the global pose of the QR code target using a map created by an additional mobile camera in combination with SLAM;

[0008] Estimating the camera's pose based on the image containing the two-dimensional code target and the global pose of the two-dimensional code target to obtain an initial global pose of the camera;

[0009] The ORB feature points of the image detection environment collected by the camera are fused with the pose information of the two-dimensional code target to construct a multiple constraint function, wherein the multiple constraint function includes a reprojection error constraint function and a plane consistency constraint function;

[0010] The multiple constraint functions are fused using dynamic weights to obtain a joint optimization objective function, and a nonlinear optimization algorithm is used to solve the optimal solution of the joint optimization objective function to obtain the target global pose of the camera.

[0011] According to a second aspect of an embodiment of the present application, a global pose calibration device for a camera with a non-overlapping field of view that integrates multiple constraints is provided. The camera is an airborne camera in mining equipment, and the calibration device includes:

[0012] An acquisition module is used to acquire an image captured by the camera containing a QR code target, wherein the QR code target is fixed to an anchor net of a cable tie on both sides of the lane;

[0013] A determination module, configured to determine the global pose of the QR code target using a map established by an additional mobile camera in combination with SLAM;

[0014] a pose estimation module, configured to estimate the pose of the camera based on the image containing the QR code target and the global pose of the QR code target, and obtain an initial global pose of the camera;

[0015] A construction module is used to fuse the ORB feature points of the image detection environment captured by the camera with the pose information of the two-dimensional code target to construct a multiple constraint function, wherein the multiple constraint function includes a reprojection error constraint function and a plane consistency constraint function;

[0016] The optimization module is used to fuse the multiple constraint functions using dynamic weights to obtain a joint optimization objective function, and use a nonlinear optimization algorithm to solve the optimal solution of the joint optimization objective function to obtain the target global pose of the camera.

[0017] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0018] at least one processor;

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0021] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, wherein the storage medium stores instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect above.

[0022] According to a fifth aspect of an embodiment of the present application, a program product is provided, comprising at least one of a program and an instruction, wherein when the at least one of the program and the instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0023] The technical solution of this application effectively overcomes the difficulty of calibrating cameras without a common view area by introducing environmental geometric features and a spatial reference for a QR code target, effectively facilitating efficient multi-camera calibration in confined spaces. Its advantage lies in integrating multiple constraints to improve calibration accuracy, reducing manual intervention and equipment downtime. It is suitable for the complex and dynamically changing environments of underground tunneling equipment, providing support for high-precision 3D reconstruction and navigation positioning.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 A flowchart of a method for global pose calibration of a camera with non-overlapping field of view by integrating multiple constraints provided in an embodiment of the present application;

[0027] Figure 2 A block diagram of a device for global pose calibration of a camera with non-overlapping field of view and fusion of multiple constraints provided in an embodiment of the present application;

[0028] Figure 3 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The embodiments of the present application are not exhaustive, but are merely illustrative of some embodiments and are not intended to be a specific limitation on the scope of protection of the present application. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementations in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined. For example, some or all of the steps in different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementations of other embodiments.

[0031] In each embodiment of the present application, unless otherwise specified or there is any logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form a new embodiment based on their inherent logical relationships.

[0032] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0033] In the embodiments of the present application, "plurality" refers to two or more.

[0034] It should be noted that the fusion of multi-sensor data relies on accurate global pose calibration. This requires determining the camera's position and orientation in three-dimensional space to ensure consistency in the coordinate space of each sensor. Camera calibration involves both intrinsic and extrinsic parameters. The camera's intrinsic parameters determine its lens characteristics. The Zhang Zhengyou calibration method can yield relatively accurate results using the intrinsic parameters. To obtain the complete pose between multiple cameras, the extrinsic parameters must also be calculated. Multi-camera calibration can determine the relative position and pose relationships between different cameras, enabling data fusion and collaborative processing across all cameras in a common coordinate space.

[0035] Existing camera calibration methods have limitations in underground coal mine excavation scenarios, mainly in the following three aspects: (1) The overlapping fields of view between cameras are very small or even non-existent, making calibration methods based on common view features ineffective; (2) Positioning based on total stations / UWB can only provide sensor position information and cannot calibrate attitude parameters. It also relies on manual measurement, which is inefficient; (3) Existing non-overlapping field of view camera calibration methods require the pre-placement of high-precision calibration plates in a dedicated calibration room. However, the maintenance space in underground excavation tunnels is small and time-sensitive, making it difficult to meet deployment requirements. Due to the special requirements of underground scenarios, a calibration solution that does not require common view, is highly robust, and is easy to deploy is urgently needed.

[0036] Mining equipment (such as tunneling equipment) is in a complex environment with high-frequency vibration and time-varying conditions for a long time. The external parameters of the multiple cameras it carries are prone to offset and need to be recalibrated during maintenance. These cameras are distributed around the equipment, but are blocked by the fuselage structure, and the overlapping fields of view of each camera are very small or even non-existent. In response to the lack of common view features caused by the non-overlapping fields of view of multiple cameras in underground coal mine equipment, the drift of external parameters caused by dynamic environmental interference (vibration, temperature change, etc.), and the defects of traditional calibration methods that rely on manual intervention and fixed scenes, the embodiment of the present application provides a dynamic constraint fusion calibration framework. By introducing a high-precision QR code target as a global reference, combined with plane geometric constraints, visual reprojection errors and dynamic weight adaptation mechanisms, robust calibration of cameras without common view is achieved, avoiding the dependence of traditional methods on fixed calibration scenes, achieving fast and accurate calibration in narrow spaces, and reducing equipment downtime.

[0037] The following describes a method for global pose calibration of a camera with non-overlapping field of view by fusing multiple constraints according to an embodiment of the present application with reference to the accompanying drawings.

[0038] Figure 1 This is a flow chart of a method for global pose calibration of a camera with non-overlapping field of view that incorporates multiple constraints, as provided in an embodiment of the present application. It should be noted that, in some embodiments, the camera may be an onboard camera in mining equipment, which may include tunneling equipment. Exemplarily, global pose calibration can be performed on an onboard camera in the tunneling equipment. Exemplarily, there may be one or more cameras, and each camera's field of view may include at least two QR code targets.

[0039] like Figure 1 As shown, the global pose calibration method for non-overlapping field of view cameras integrating multiple constraints may include but is not limited to the following steps.

[0040] In step 101, an image containing a QR code target captured by a camera to be calibrated is obtained. The QR code target is fixed to an anchor net tied to both sides of the tunnel.

[0041] In an embodiment of the present application, the camera to be calibrated can be an onboard camera on mining equipment (e.g., tunneling equipment). Specifically, the present embodiment relates to global pose calibration of an onboard camera on mining equipment (e.g., tunneling equipment). For example, in the confined working environment of mining equipment (e.g., tunneling equipment), multiple QR code targets can be fixed to anchor nets on both sides of the tunnel using cable ties. Each fixed onboard camera can have a field of view that includes at least two QR code targets. This process ensures that the QR codes provide preliminary constraints on position and pose.

[0042] For example, each camera can capture an image including an Apriltag (QR code) target, detect the corner points of the QR code target, and calculate their pixel coordinates. For example, a pose chain is established with the target coordinate system as the reference, and pose transformation is transferred through the QR code target to construct a global pose graph model. As an example, taking camera 1 and camera 2 as an example, a pose chain is established with the target coordinate system as the reference, and pose transformation is transferred through the QR code target to construct a global pose graph model as shown below:

[0043] (1)

[0044] (2)

[0045] in, 、 Represents the relative external parameters from camera 1 to camera 2.

[0046] In step 102 , the global pose of the QR code target is determined using a map created by an additional mobile camera in combination with SLAM.

[0047] In some embodiments, the calibration plate is kept fixed during data acquisition, and the world coordinate system can be established by additional mobile cameras in the SLAM framework. ; Obtain the pose of the QR code target in the world coordinate system from the map established by SLAM; Determine the coordinates of the corner points of the QR code target in the target's local coordinate system, and use the pose of the QR code target in the world coordinate system to convert the coordinates of the corner points of the QR code target in the target's local coordinate system to the world coordinate system to obtain the global pose of the QR code target.

[0048] In one possible implementation, a fixed camera captures an image containing a QR code target, detects the QR code target ID, and uses the SLAM map to obtain the target's location. The posture in For each ID The four corner points of the QR code target are in the local coordinate system of the target The coordinates in , using SLAM to estimate the target pose , transform the coordinates of the corner points in the target local coordinate system to the world coordinate system , thus obtaining the QR code target (ID is )’s global pose, i.e. the QR code target (ID is )'s 3D world coordinates:

[0049] (3)

[0050] in, For the QR code target (ID is ), For the QR code target (ID is ) in the world coordinate system The posture in For the QR code target (ID is ) in the target local coordinate system The coordinates in .

[0051] In step 103, the pose of the camera is estimated based on the image containing the two-dimensional code target and the global pose of the two-dimensional code target to obtain an initial global pose of the camera.

[0052] In some embodiments, based on an image containing a QR code target, the relative pose between the camera and the QR code target is determined using the camera's intrinsic parameters and the size of the QR code target. The camera's pose is estimated based on the relative pose between the camera and the QR code target, the global pose of the QR code target, and the camera's intrinsic parameter matrix to obtain the camera's initial global pose. In one possible implementation, a PnP-RANSAC algorithm can be used to estimate the camera's pose based on the relative pose between the camera and the QR code target, the global pose of the QR code target, and the camera's intrinsic parameter matrix to obtain the camera's initial global pose.

[0053] For example, the pixel coordinates of the target corner points detected by the fixed camera are , which is the 2D image point coordinates, that is, the relative posture between the camera and the QR code target. The known 3D point set can be used (i.e., the global pose of the QR code target), corresponding to the 2D point set and the camera intrinsic parameter matrix , construct the PnP problem and solve it, where the solution target is the camera pose , so that the reprojection error is minimized:

[0054] (4)

[0055] in, is the camera projection function ( is the scale factor):

[0056] (5)

[0057] For example, the PnP solution first matches the pixel coordinates of the detected target corners with their corresponding world coordinates. RANSAC is used to remove mismatched points and robustly estimate initial values. Using these initial values ​​as a starting point, the Levenberg-Marquardt algorithm is used to minimize the reprojection error. This process allows the fixed camera pose to be solved, thereby obtaining the camera's initial global pose.

[0058] In step 104, the ORB feature points of the image detection environment captured by the camera are fused with the pose information of the QR code target to construct a multi-constraint function. In some embodiments, the multi-constraint function may include a reprojection error constraint function and a plane consistency constraint function. The plane consistency constraint function may include a normal vector alignment constraint function and a plane alignment constraint function.

[0059] In some embodiments, the RANSAC algorithm can be used to segment ORB feature points and extract the plane where the QR code target is located; the plane area can be screened according to the size of the QR code target, and the target plane equation and its normal vector can be extracted; based on the position of the QR code target in the world coordinate system, the target plane equation and its normal vector estimated by SLAM are determined, wherein the position of the QR code target in the world coordinate system is obtained based on the map established by SLAM; based on the relative posture between the camera and the QR code target and the global pose of the QR code target, a reprojection error constraint function is constructed; based on the target plane equation and its normal vector extracted from the ORB feature points and the target plane equation and its normal vector estimated by SLAM, a plane consistency constraint function is constructed.

[0060] For example, the ORB feature points can be processed first, and the RANSAC algorithm can be used to segment the ORB feature points to extract the plane where the QR code target is located. ) Filter the plane area and extract the plane equation , normal vector and center point ; Secondly, process the results obtained by the mobile camera SLAM and transform the QR code target into the world coordinate system , and get the plane equation .

[0061] After obtaining the global pose of the QR code target, the ORB feature points and image data are fused to optimize the global pose of each camera. The geometric information of the plane where the target is located, including the plane equation and the normal vector, is extracted using the geometric features of the environment, and the pose of the calibration plate estimated by the SLAM system is jointly optimized. The error of the calibration plate pose estimated by SLAM is corrected by geometric consistency constraints, thereby improving the external parameter calibration accuracy of the multi-camera system. By constructing the following three constraints (i.e., camera reprojection error, normal vector alignment constraint, and plane alignment constraint), the pose of the QR code target is jointly optimized to obtain the accurate camera pose. Among them, the camera reprojection error is as follows:

[0062] (6)

[0063] This item measures the accuracy of camera pose estimation, and its core is to minimize the projection deviation between the 3D world coordinates of the target corner points and the corresponding 2D image coordinates.

[0064] The normal alignment constraint is as follows:

[0065] (7)

[0066] in, The target plane normal vector estimated by SLAM can be obtained from the target pose get; The target plane normal vector extracted from the ORB feature point. The purpose of this item is to calculate the average angle of the normal vector before and after optimization. , ensuring that the target plane normal vector estimated by SLAM is consistent with the plane normal vector extracted from the ORB feature point.

[0067] The planar alignment constraint is as follows:

[0068] (8)

[0069] in, is the target center translation vector estimated by SLAM; This is the constant term of the target plane equation extracted from the ORB feature points. The purpose of this term is to calculate the mean absolute distance from the target center to the ORB feature point plane, ensuring that the target plane estimated by SLAM coincides with the plane in the ORB feature points in spatial position.

[0070] In step 105, multiple constraint functions are fused using dynamic weights to obtain a joint optimization objective function, and a nonlinear optimization algorithm is used to solve the optimal solution of the joint optimization objective function to obtain the target global pose of the camera.

[0071] In the embodiments of the present application, a global optimization can be performed based on the preliminary constraints provided by the QR code target, ultimately obtaining the precise relative poses of all cameras. A joint optimization objective function is constructed that includes the camera reprojection error and the ORB feature point plane constraint, and the optimal camera pose is solved using a nonlinear optimization method. As an example, the construction of this joint optimization objective function is as follows:

[0072] (9)

[0073] in, , , is the weight coefficient; is the reprojection error constraint function; is the plane consistency constraint function.

[0074] Solving the target camera pose through nonlinear optimization The exact value of the camera pose Need to be parameterized as Lie algebra representation (3), used for nonlinear optimization objective function , that is, solving , using the Levenberg-Marquardt algorithm to iteratively update, each step of the iteration updates the pose through the addition of Lie algebra, and finally the target global pose of the camera can be obtained.

[0075] By implementing the embodiments of this application, it is possible to integrate the QR code target with the characteristic information in the environment, providing a high-precision and high-robustness calibration solution for the multi-camera system onboard coal mine tunneling equipment. The core contributions and main technical advantages of the embodiments of this application are as follows:

[0076] (1) To address the problem of missing common view areas between multiple cameras, a joint optimization model based on the spatial reference of the QR code target and the constraints of the environmental geometric features is proposed. The visual positioning of the QR code target is integrated with the planar constraints extracted from the environmental geometric features. The normal vector alignment error and the plane distance error are introduced to construct a unified optimization objective function to achieve global pose alignment without common view cameras.

[0077] (2) An efficient and automated calibration process was proposed. Addressing the shortcomings of traditional methods that rely on manual measurement and fixed calibration scenarios, a rapid calibration solution suitable for underground tunneling working faces was proposed. By automatically matching the geometric features of the environment with the QR code target using visual positioning, the overall calibration time was shortened and the number of manual steps was reduced. This approach is suitable for the narrow maintenance spaces and rapid calibration requirements of underground tunneling working faces in coal mines.

[0078] (3) Engineering applicability and scalability. This method has application value in multi-camera systems for underground coal mine excavation equipment, providing support for high-precision environmental reconstruction and positioning navigation. The method can be extended to other time-varying calibration scenarios to meet the needs of efficient calibration and can be extended to other sensors such as lidar and depth cameras.

[0079] Figure 2 This is a block diagram of a device for global pose calibration of cameras with non-overlapping fields of view and multiple constraints, provided in an embodiment of the present application. It should be noted that in some embodiments, the camera may be an onboard camera in mining equipment, which may include tunneling equipment. Exemplarily, global pose calibration can be performed on an onboard camera in the tunneling equipment. Exemplarily, there may be one or more cameras, each with a field of view that includes at least two QR code targets.

[0080] like Figure 2 As shown, the global pose calibration device for cameras with non-overlapping field of view and fusion of multiple constraints may include: an acquisition module 201 , a determination module 202 , a pose estimation module 203 , a construction module 204 and an optimization module 205 .

[0081] The acquisition module 201 is used to acquire an image containing a QR code target captured by the camera to be calibrated, where the QR code target is fixed to an anchor net tied to both sides of the tunnel.

[0082] The determination module 202 is configured to determine the global pose of the QR code target using a map created by an additional mobile camera in conjunction with SLAM. In some embodiments, the determination module 202 is configured to: maintain a fixed calibration plate during data acquisition, establish a world coordinate system using the mobile camera within the SLAM framework; obtain the pose of the QR code target in the world coordinate system from the map created by SLAM; determine the coordinates of the corner points of the QR code target in the target's local coordinate system, and, using the pose of the QR code target in the world coordinate system, convert the coordinates of the corner points of the QR code target in the target's local coordinate system to the world coordinate system to obtain the global pose of the QR code target.

[0083] The pose estimation module 203 is used to estimate the pose of the camera based on the image containing the QR code target and the global pose of the QR code target to obtain the initial global pose of the camera.

[0084] In some embodiments, the pose estimation module 203 is used to: determine the relative pose between the camera and the QR code target based on an image containing a QR code target using the camera's intrinsic parameters and the size of the QR code target; estimate the camera's pose based on the relative pose between the camera and the QR code target, the global pose of the QR code target, and the camera's intrinsic parameter matrix to obtain the camera's initial global pose.

[0085] In some embodiments, the pose estimation module 203 is used to: use the PnP-RANSAC algorithm to estimate the pose of the camera based on the relative pose between the camera and the QR code target, the global pose of the QR code target, and the intrinsic parameter matrix of the camera to obtain the initial global pose of the camera.

[0086] The construction module 204 is used to use the image detection environment ORB feature points captured by the camera to fuse the pose information of the two-dimensional code target and construct a multiple constraint function, which includes a reprojection error constraint function and a plane consistency constraint function.

[0087] In some embodiments, the construction module 204 is used to: use the RANSAC algorithm to segment the ORB feature points and extract the plane where the QR code target is located; screen the plane area according to the size of the QR code target and extract the target plane equation and its normal vector; determine the target plane equation and its normal vector estimated by SLAM based on the pose of the QR code target in the world coordinate system, wherein the pose of the QR code target in the world coordinate system is obtained based on the map established by SLAM; construct a reprojection error constraint function based on the relative posture between the camera and the QR code target and the global pose of the QR code target; construct a plane consistency constraint function based on the target plane equation and its normal vector extracted from the ORB feature points and the target plane equation and its normal vector estimated by SLAM.

[0088] In some embodiments, the reprojection error constraint function is formulated as follows:

[0089]

[0090] in, is the reprojection error; is the 2D image coordinate of the i-th QR code target; is the camera projection function; is the intrinsic parameter matrix of the camera; , is the camera pose; is the 3D world coordinate of the i-th QR code target; N is the number of QR code targets.

[0091] In some embodiments, the plane consistency constraint function includes a normal vector alignment constraint function and a plane alignment constraint function; wherein the formula of the normal vector alignment constraint function is expressed as follows:

[0092]

[0093] in, is the normal vector of the target plane equation estimated by SLAM; is the normal vector of the target plane equation extracted from the ORB feature points.

[0094] In some embodiments, the plane alignment constraint function is formulated as follows:

[0095]

[0096] in, is the target center translation vector estimated by SLAM; is the constant term of the target plane equation extracted from the ORB feature points.

[0097] The optimization module 205 is used to fuse multiple constraint functions using dynamic weights to obtain a joint optimization objective function, and use a nonlinear optimization algorithm to solve the optimal solution of the joint optimization objective function to obtain the target global pose of the camera.

[0098] In some embodiments, the joint optimization objective function is formulated as follows:

[0099]

[0100] in, , , is the weight coefficient; is the reprojection error constraint function; is the plane consistency constraint function.

[0101] It should be noted that the above explanation of the embodiment of the global pose calibration method for cameras with non-overlapping field of view and fusion of multiple constraints is also applicable to the global pose calibration device for cameras with non-overlapping field of view and fusion of multiple constraints in this embodiment, and will not be repeated here.

[0102] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0103] like Figure 3 , is a block diagram of an electronic device according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0104] like Figure 3As shown, the electronic device includes: one or more processors 301, a memory 302, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 301 is taken as an example.

[0105] Memory 302 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to execute the method for global pose calibration of cameras with non-overlapping field of view and multiple constraints fused together provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to execute the method for global pose calibration of cameras with non-overlapping field of view and multiple constraints fused together provided in this application.

[0106] The memory 302 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the method for global pose calibration of cameras with non-overlapping field of view by integrating multiple constraints in the embodiment of the present application (for example, the attached Figure 2 The processor 301 executes the non-transient software programs, instructions, and modules stored in the memory 302 to execute various functional applications and data processing of the server, thereby implementing the global pose calibration method for cameras with non-overlapping field of view and fusion of multiple constraints in the above method embodiment.

[0107] The memory 302 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 302 may optionally include a memory remotely located relative to the processor 301, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0108] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 302, the input device 303 and the output device 304 may be connected via a bus or other means. Figure 3 The bus connection is taken as an example.

[0109] Input device 303 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. Examples include a touch screen, keypad, mouse, trackpad, touchpad, pointing stick, one or more mouse buttons, trackball, joystick, and other input devices. Output device 304 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors). Display devices may include, but are not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0110] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a magnetic disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0114] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical servers and VPS services ("Virtual Private Servers" or "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0115] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0116] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A global pose calibration method for cameras with non-overlapping field of view integrating multiple constraints, characterized by: The camera is an airborne camera in mining equipment, and the calibration method includes: Acquiring an image captured by the onboard camera containing a QR code target, wherein the QR code target is fixed to an anchor net of a cable tie on both sides of the laneway; Determine the global pose of the QR code target using a map created by an additional mobile camera in combination with SLAM; Estimating the pose of the airborne camera based on the image containing the two-dimensional code target and the global pose of the two-dimensional code target to obtain an initial global pose of the airborne camera; The ORB feature points of the image detection environment collected by the airborne camera are fused with the pose information of the two-dimensional code target to construct a multiple constraint function, wherein the multiple constraint function includes a reprojection error constraint function and a plane consistency constraint function; The multiple constraint functions are fused using dynamic weights to obtain a joint optimization objective function, and a nonlinear optimization algorithm is used to solve the optimal solution of the joint optimization objective function to obtain the target global pose of the airborne camera.

2. The method according to claim 1, characterized in that The method of determining the global pose of the QR code target by utilizing a map established by an additional mobile camera in combination with SLAM includes: Keep the calibration plate fixed during data acquisition, and in the SLAM framework, the world coordinate system is established by the mobile camera; Obtaining the position of the QR code target in the world coordinate system from the map established by the SLAM; Determine the coordinates of the corner points of the two-dimensional code target in the target local coordinate system, and use the pose of the two-dimensional code target in the world coordinate system to transform the coordinates of the corner points of the two-dimensional code target in the target local coordinate system into the world coordinate system to obtain the global pose of the two-dimensional code target.

3. The method according to claim 1, characterized in that The estimating the pose of the airborne camera according to the image containing the two-dimensional code target and the global pose of the two-dimensional code target to obtain the initial global pose of the airborne camera includes: Based on the image containing the QR code target, determining the relative posture between the airborne camera and the QR code target using the intrinsic parameters of the airborne camera and the size of the QR code target; The pose of the airborne camera is estimated according to the relative pose between the airborne camera and the two-dimensional code target, the global pose of the two-dimensional code target, and the intrinsic parameter matrix of the airborne camera to obtain the initial global pose of the airborne camera.

4. The method according to claim 3, characterized in that The estimating the pose of the airborne camera according to the relative pose between the airborne camera and the two-dimensional code target, the global pose of the two-dimensional code target, and the intrinsic parameter matrix of the airborne camera to obtain the initial global pose of the airborne camera includes: According to the relative posture between the airborne camera and the two-dimensional code target, the global pose of the two-dimensional code target, and the intrinsic parameter matrix of the airborne camera, the PnP-RANSAC algorithm is used to estimate the pose of the airborne camera to obtain the initial global pose of the airborne camera.

5. The method according to claim 1, characterized in that The image detection environment ORB feature points collected by the airborne camera are fused with the pose information of the two-dimensional code target to construct a multiple constraint function, including: Use the RANSAC algorithm to segment the ORB feature points and extract the plane where the two-dimensional code target is located; Screen the plane area according to the size of the QR code target and extract the target plane equation and its normal vector; Determining a target plane equation and a normal vector thereof estimated by SLAM based on the pose of the two-dimensional code target in a world coordinate system, wherein the pose of the two-dimensional code target in the world coordinate system is obtained based on a map established by the SLAM; Constructing a reprojection error constraint function based on the relative posture between the onboard camera and the two-dimensional code target and the global posture of the two-dimensional code target; The plane consistency constraint function is constructed based on the target plane equation and its normal vector extracted from the ORB feature points and the target plane equation and its normal vector estimated by SLAM.

6. The method according to claim 5, characterized in that The formula of the reprojection error constraint function is as follows: in, is the reprojection error; is the 2D image coordinate of the i-th QR code target; is the projection function of the onboard camera; is the intrinsic parameter matrix of the airborne camera; , is the position of the onboard camera; is the 3D world coordinate of the i-th two-dimensional code target; N is the number of two-dimensional code targets.

7. The method according to claim 5, characterized in that The plane consistency constraint function includes a normal vector alignment constraint function and a plane alignment constraint function; wherein, The formula of the normal vector alignment constraint function is as follows: in, is the normal vector of the target plane equation estimated by the SLAM; is the normal vector of the target plane equation extracted from the ORB feature points; The formula of the plane alignment constraint function is as follows: in, is the target center translation vector estimated by the SLAM; is the constant term of the target plane equation extracted from the ORB feature points.

8. The method according to claim 1, characterized in that The formula of the joint optimization objective function is as follows: in, , , is the weight coefficient; is the reprojection error constraint function; is the plane consistency constraint function.

9. The method according to any one of claims 1 to 8, characterized in that The number of the onboard cameras is one or more, and the field of view of each of the onboard cameras includes at least two two-dimensional code targets.

10. The method according to claim 1, characterized in that The mining equipment includes tunneling equipment.

Images