Method, device, electronic device and readable storage medium for camera calibration

Abstract

The embodiments of the present disclosure disclose a method, device, electronic device and readable storage medium for camera calibration, which relates to the field of camera calibration. The method includes: when a mobile platform equipped with a camera performs planar motion, taking multiple images by the camera; based on the multiple images taken, determining at least two different motion trajectories of the camera; and based on the two different motion trajectories determined, determining the installation posture information of the camera relative to the mobile platform.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of camera calibration, and in particular to a method, an apparatus, an electronic device, and a readable storage medium for camera calibration. Background Art

[0002] Devices that move in a plane, such as sweeping robots and logistics robots, are usually equipped with cameras on mobile platforms to perform visual measurements during movement, and to achieve functions such as automatic positioning, obstacle avoidance, and information detection. When the camera is installed on the mobile platform, due to installation errors, the camera will not be in the ideal installation position relative to the mobile platform. This will cause the camera to deviate when performing visual measurements, affecting the accuracy of positioning and detection.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] According to one aspect of an exemplary embodiment of the present disclosure, a camera calibration method is provided, comprising: when a mobile platform on which a camera is installed performs planar motion, capturing multiple images of an environment through the camera at multiple trajectory points in the planar motion; determining at least two different motion trajectories of the camera in the environment based on the multiple images captured at the multiple trajectory points; and determining installation posture information of the camera relative to the mobile platform based on the at least two different motion trajectories determined.

[0005] According to another aspect of the exemplary embodiments of the present disclosure, there is provided a device for calibrating a camera, comprising: a shooting unit, configured to control the camera to capture multiple images when a mobile platform carrying the camera makes a planar motion; a motion trajectory determination unit, determining at least two different motion trajectories of the camera based on the multiple captured images; and a camera posture information determination unit, determining posture information of the camera relative to the mobile platform based on the two different motion trajectories determined.

[0006] According to another aspect of exemplary embodiments of the present disclosure, there is provided an electronic device, including: a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method according to some exemplary embodiments of the present disclosure.

[0007] According to another aspect of exemplary embodiments of the present disclosure, there is provided a computer-readable storage medium storing a program, wherein the program includes instructions, which, when executed by a processor of an electronic device, cause the processor to perform methods according to some exemplary embodiments of the present disclosure.

[0008] According to another aspect of the exemplary embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any of the aforementioned methods when executed by a processor.

[0009] The method, device, electronic device and readable storage medium for camera calibration of the exemplary embodiments of the present disclosure can determine the actual installation posture of the camera by determining at least two motion trajectories of the camera by using images captured by the camera when the mobile platform moves, thereby completing the calibration. This can simplify the calibration process for the camera installation position and avoid manual operation affecting the accuracy of the calibration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements:

[0011] Figure 1 An exemplary flow chart of a method for calibrating a camera according to an embodiment of the present disclosure is shown;

[0012] Figure 2 A schematic diagram showing a plurality of markers arranged in an environment according to an embodiment of the present disclosure is shown;

[0013] Figure 3 An exemplary flow chart of determining a motion trajectory of a camera according to an embodiment of the present disclosure is shown;

[0014] Figure 4 An exemplary flow chart of determining installation posture information of a camera relative to a mobile platform according to an embodiment of the present disclosure is shown;

[0015] Figure 5 A flow chart of determining a rotation axis of a mobile platform relative to a camera according to an embodiment of the present disclosure is shown;

[0016] Figure 6 An exemplary flow chart of determining a rotation angle of a camera relative to a mobile platform around a rotation axis according to an embodiment of the present disclosure is shown;

[0017] Figure 7 A schematic block diagram showing a device for camera calibration according to an embodiment of the present disclosure; and

[0018] Figure 8 is a block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0020] The terms used in the description of various examples described in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship.

[0021] The camera installed on the mobile platform can acquire images during the movement of the mobile platform. In order to accurately use the images acquired by the camera to obtain the surrounding information of the mobile platform, the camera needs to be calibrated after it is installed on the mobile platform to obtain the actual installation posture information of the camera relative to the mobile platform, so as to make corrections when performing subsequent visual measurement.

[0022] Devices that basically move on a plane, such as sweeping robots, logistics robots, and vehicles, may include a mobile platform that moves on a plane. Visual measurement can be performed during movement by installing a camera on the mobile platform, thereby realizing functions such as automatic positioning, obstacle avoidance, and information detection. Specifically, during the movement of the mobile platform, the camera installed on the mobile platform takes pictures as the mobile platform moves. Based on the pictures taken, visual information around the mobile platform can be obtained. If the camera is installed on the mobile platform in a predetermined manner, for example, so that the coordinate system of the camera coincides with the coordinate system of the mobile platform, the position of the mobile platform can be characterized based on the position of the camera determined based on the picture taken by the camera. In this way, the indirect determination of the position of the mobile platform is achieved.

[0023] However, when the camera is installed on a mobile platform, due to installation errors, the camera will not be in an ideal installation position relative to the mobile platform. This will result in the position of the camera determined based on the picture taken by the camera being unable to accurately represent the position of the mobile platform, thereby affecting the accuracy of positioning and detection. The related technology determines the actual installation posture information of the camera by manual measurement or calibration using external calibration objects, thereby correcting the installation error of the camera relative to the mobile platform. However, these calibration methods often require human participation and are complicated to operate, so there is a problem of limited calibration accuracy. For example, the manual measurement of the offset angle of the camera installation has poor accuracy, and the calibration method using special external calibration objects has strict requirements on the placement of the calibration object, and the calibration process is relatively cumbersome.

[0024] In this regard, the exemplary embodiments of the present disclosure provide an improved solution for camera calibration. By determining at least two motion trajectories of the camera by means of images captured by the camera when the mobile platform moves, the installation posture of the camera can be determined, thereby completing the calibration. The calibration process does not require external calibration or measurement equipment, and can be completed automatically without manual operation.

[0025] In an embodiment of the present disclosure, the installation offset of the camera on the mobile platform can be indicated by the rotation matrix between the camera coordinate system and the mobile platform coordinate system. Among them, the installation point of the camera on the mobile platform can be determined as the coordinate origin of the camera coordinate system and the mobile platform coordinate system. The mobile platform coordinate system may include the following three characteristic directions: a first platform characteristic direction corresponding to the forward direction of the mobile platform, a second platform characteristic direction perpendicular to the moving plane of the mobile platform, and a third platform characteristic direction orthogonal to the first platform characteristic direction and the second platform characteristic direction. The camera coordinate system may include the following three characteristic directions: a first camera characteristic direction corresponding to the optical axis direction, a second camera characteristic direction corresponding to the vertical direction of the imaging plane, and a third characteristic direction orthogonal to the first camera characteristic direction and the second camera characteristic direction. Among them, the mobile platform coordinate system may coincide with the world coordinate system. When the first camera characteristic direction coincides with the first platform characteristic direction, the second camera characteristic direction coincides with the second platform characteristic direction, and the third camera characteristic direction coincides with the third platform characteristic direction, the camera trajectory determined based on the image captured by the camera can reflect the moving trajectory of the mobile platform. In this case, it can be considered that there is no offset in the installation of the camera. When there is an offset in the installation of the camera on the mobile platform, the rotation matrix between the camera coordinate system and the mobile platform coordinate system can be used to indicate the installation offset of the camera.

[0026] The embodiments of the present disclosure are further described below in conjunction with the accompanying drawings.

[0027] Figure 1FIG. 1 shows an exemplary flow chart of a method 100 for camera calibration according to an embodiment of the present disclosure. Figure 1 As shown, in step S110, when a mobile platform equipped with a camera performs a planar motion, a plurality of images of the surrounding environment are captured by the camera at a plurality of track points in the planar motion.

[0028] It should be understood that the camera can be a device that forms and records images using optical imaging principles. In some embodiments, the camera can be configured as a device that performs imaging based on visible light or invisible light (such as infrared light).

[0029] In some embodiments, the image captured by the camera may be acquired from the camera in real time. In the case of acquiring the image in real time, steps S120-S130 described below may be performed in real time. In other embodiments, the multiple images captured by the camera may be stored in a storage device. Subsequently, the multiple images captured by the camera and stored in the storage device are acquired from the storage device to perform steps S120-S130 described below.

[0030] The image captured by the camera obtained in step S110 includes information that can represent the movement trajectory of the camera when it moves in a plane.

[0031] In step S120, at least two different movement trajectories of the camera in the environment may be determined based on a plurality of images captured at a plurality of trajectory points.

[0032] The motion trajectory of the camera during the planar motion of the mobile platform may be determined using information in the image captured in step S110 and predetermined surrounding environment information.

[0033] The motion trajectory of the camera during the planar motion of the mobile platform can be determined with the help of any feature information in the environment.

[0034] In some embodiments, the shooting position of the camera when taking an image can be determined with the help of multiple markers in the environment, so as to facilitate the execution of subsequent steps S120-S130. In this embodiment, each of the multiple images captured includes at least one of the multiple markers arranged in the environment. The markers have predefined sizes and / or patterns, so that the shooting position of the camera relative to each marker when taking an image can be determined based on these sizes and patterns with the help of relevant technologies. The sizes and patterns of different markers can be the same or different. In this case, the surrounding environment information may include the relative positions between the various markers in the environment. In some examples, the environmental information may include point cloud information corresponding to the various markers in the environment. The markers can be arranged in the environment by the operator in a random or predefined manner before use. Figure 2In the illustrated embodiment, the marker may be AprilTag 201-1, 201-2, etc. A plurality of AprilTags may be pre-arranged in the environment in a random manner by the operator. Without departing from the principles of the present disclosure, those skilled in the art may also use markers of any other shapes and colors, as long as the position and direction of the marker can be easily identified in the image so as to determine the relative position of the camera relative to the marker when capturing the image.

[0035] In other embodiments, any other characteristic information in the environment may be used to determine the shooting position of the camera when shooting an image. In addition to the size and pattern of the marker, the color information of the marker may also be used to assist in determining the shooting position of the camera.

[0036] As previously mentioned, in some embodiments, each of the multiple images captured includes at least one of the multiple markers arranged in the environment. Therefore, the shooting position of the camera when shooting the image can be determined with the help of the multiple markers in the environment, thereby determining at least two different motion trajectories of the camera. In some implementations, each of the multiple markers arranged in the environment is at least included in one of the at least one calibration image acquired for the environment. In this way, the camera can obtain as much information about the surrounding environment as possible during the planar motion of the mobile platform, so that the motion trajectory of the camera can be determined more accurately. For example, when determining the motion trajectory of the camera, the entire shooting of the camera can cover all the markers arranged in the environment. Using more markers helps to improve the accuracy and robustness of the determination of the camera position.

[0037] Figure 3 An exemplary flow chart of determining the motion trajectory of a camera according to an embodiment of the present disclosure is shown. Figure 3 The process shown in Figure 1 Step S120 in .

[0038] In step S310, based on the relative position relationship between the position of at least one marker in each image in the corresponding image and multiple markers in the environment, the corresponding shooting position of the camera in the environment when shooting each image is determined. Wherein, at least one of the multiple markers is at least included in one of the multiple images captured by the camera. In some embodiments, the relative position relationship between each marker in the multiple markers and other markers includes the relative rotation and displacement of the marker and other markers. In step S320, based on multiple shooting positions corresponding to the multiple images captured, at least two different motion trajectories of the camera are determined.

[0039] In some embodiments, environmental information can be determined by using a calibration image of the environment captured by the camera before the camera calibration method begins. In some implementations, environmental information can be determined by identifying information about markers in the calibration image captured by the camera. For example, the relative position relationship between multiple markers can be determined based on at least one calibration image acquired for the environment by a camera installed on a mobile platform or other photographic equipment in the environment that has completed internal reference calibration. By using the method of determining environmental information using calibration images, there is no need to limit the arrangement of markers in the environment, thereby simplifying the calibration process and reducing the impact of the accuracy of marker arrangement on the calibration accuracy. In addition, if Figure 2 As shown, the markers can be arranged on the walls and the ground in the environment, so the calibration image can also include information such as the walls and the ground of the environment. It can be understood that it can be assumed that the moving plane of the mobile platform is parallel to the ground, and it can also be assumed that the wall is perpendicular to the ground, so the expression of the coordinate system of the mobile platform in the environment can be determined based on the environmental information.

[0040] In some other embodiments, pre-stored environmental information may also be obtained from a database, wherein the environmental information includes the relative position relationship between a plurality of markers arranged in the environment for camera calibration.

[0041] Return to reference again Figure 1 In step S130, based on the determined at least two different motion trajectories, the installation posture information of the camera relative to the mobile platform is determined. In some embodiments, the installation posture information is represented as a rotation matrix between the camera coordinate system and the mobile platform coordinate system. The two different motion trajectories may include at least a first motion trajectory and a second motion trajectory. The process of determining the installation posture information of the camera using the first motion trajectory and the second motion trajectory will be described below.

[0042] Figure 4 An exemplary flow chart of determining the installation posture information of a camera relative to a mobile platform according to an embodiment of the present disclosure is shown. Figure 4 The process shown in Figure 1 Step S130 shown in FIG.

[0043] like Figure 4 As shown, determining the installation posture information of the camera relative to the mobile platform further includes steps S410 to S430.

[0044] In step S410, the rotation axis of the mobile platform relative to the camera can be determined based on the first motion trajectory. In this step, the mobile platform is moved in a plane according to the first predetermined trajectory in the environment. The plane movement includes at least three trajectory points of different lines. The specific shape of the first predetermined trajectory is not limited here. The camera can be controlled to capture images during the movement of the mobile platform according to the trajectory, and corresponding images can be obtained at at least three non-collinear shooting positions. Based on the captured images, at least three trajectory points of the camera movement can be determined, and the first motion trajectory can be formed by connecting at least three trajectory points of the camera. It should be understood that choosing to capture images at more trajectory points is conducive to improving the accuracy and robustness of determining the rotation axis.

[0045] Based on the first motion trajectory, the plane where the camera movement trajectory is located and the normal vector of the plane can be determined.

[0046] If the second camera feature direction in the camera coordinate system and the second platform feature direction in the mobile platform coordinate system coincide with each other when the camera is installed, then the plane where the first motion trajectory is located determined by step S410 and the normal vector of the ground should coincide with each other. However, if there is a deviation in the installation angle of the camera, then the plane where the first motion trajectory is located determined in step S410 and the ground do not coincide with each other, and the normal vector of the plane where the first motion trajectory is located and the normal vector of the ground do not coincide with each other. Therefore, based on the first motion trajectory, the information of the rotation axis of the mobile platform relative to the camera can be determined, and the rotation axis can be used to represent the deviation between the vertical axis of the camera and the vertical axis of the mobile platform.

[0047] Figure 5 FIG. 4 is a flowchart of determining the rotation axis of the mobile platform relative to the camera according to an embodiment of the present disclosure. Figure 5 The process shown in Figure 4 Step S410 in .

[0048] In step S510, at least three trajectory points (or first motion trajectories) are fitted to the same plane to form a first plane fitting motion trajectory. In some embodiments, at least one plane constraint may be applied to at least three first trajectory points to fit at least three first trajectory points to the same plane to form a first plane fitting motion trajectory. Since the plane constraint is applied during the fitting process, the first plane fitting motion trajectory is a plane trajectory (i.e., a plane curve). In step S520, based on the determined first plane fitting motion trajectory, the plane normal vector of the plane where the first plane fitting motion trajectory is located is determined as the rotation axis of the mobile platform relative to the camera. Among them, the expression of the plane normal vector of the first plane fitting motion trajectory in the camera coordinate system is confirmed as the rotation axis of the mobile platform relative to the camera coordinate system.

[0049] In some implementations, the plane constraint includes a nonlinear least squares optimization algorithm. In fact, those skilled in the art can use any plane constraint algorithm to implement the above plane constraint, and the specific method of applying the plane constraint is not limited here.

[0050] Return to reference Figure 4 , after determining the rotation axis of the mobile platform relative to the camera based on the first motion trajectory in step S410, execute step S420. In step S420, based on the second motion trajectory and the determined rotation axis, determine the rotation angle of the mobile platform relative to the camera around the rotation axis. In this step, the mobile platform is moved in a plane according to the second predetermined trajectory in the environment. The second predetermined trajectory includes at least two trajectory points. The camera is controlled to capture images during the movement of the mobile platform according to the second predetermined trajectory, and the corresponding images are captured at the positions of at least two trajectory points. The actual position of the camera at the two trajectory points can be determined through the images captured at at least two trajectory points, and the actual motion trajectory of the camera can be fitted. By comparing the actual motion trajectory of the camera with the motion trajectory of the mobile platform (i.e., the second predetermined trajectory), the rotation angle of the mobile platform relative to the camera around the rotation axis can be determined.

[0051] Figure 6 An exemplary flow chart of determining the rotation angle of the mobile platform relative to the camera around the rotation axis according to an embodiment of the present disclosure is shown. Figure 6 The process shown in Figure 4 Step S420 in .

[0052] In step S610, at least two second trajectory points are connected to form a second motion trajectory. In step S620, the second motion trajectory can be fitted to the plane where the mobile platform performs the second plane motion using the previously determined rotation axis to form a second plane fitting motion trajectory. For example, the second motion trajectory can be rotationally compensated using the rotation axis between the camera and the mobile platform determined based on the first motion trajectory, so that the second plane fitting motion trajectory obtained after compensation is located on the plane where the mobile platform moves. In step S630, the angle between the trajectory of the mobile platform performing the second plane motion and the second plane fitting motion trajectory of the camera is determined as the rotation angle.

[0053] As mentioned above, the rotation axis determined in step S410 can be used to determine the deviation between the second camera characteristic direction of the camera and the second platform characteristic direction of the mobile platform, but the above rotation axis cannot reflect the deviation between the optical axis direction of the camera and the forward direction of the mobile platform. Therefore, it is necessary to use the second motion trajectory to determine the deviation between the optical axis direction of the camera and the forward direction of the mobile platform, so as to obtain a complete rotation matrix between the camera coordinate system and the mobile platform coordinate system. After the second motion trajectory is rotationally compensated using the rotation axis determined in step S410, the second plane fitting motion trajectory corresponds to the camera movement trajectory corresponding to the image captured by the camera when the vertical direction of the camera coordinate system and the vertical direction of the mobile platform coincide. In this case, the angle between the second plane fitting motion trajectory and the second predetermined trajectory of the mobile platform can represent the offset between the optical axis direction of the camera and the forward direction of the mobile platform.

[0054] In some embodiments, the second motion trajectory is a straight line. In the case where the second motion trajectory is a straight line, the angle between the two second predetermined trajectories and the second plane fitting motion trajectory of the camera can be conveniently obtained by comparing the motion trajectory of the mobile platform (i.e., the second predetermined trajectory) and the direction of the determined actual motion trajectory of the camera, and the angle is used as the rotation angle. It is understandable that the second motion trajectory can also be a trajectory of any other shape, and the rotation angle can be determined by comparing the angle between the tangential direction of any point on the second plane fitting motion trajectory corresponding to the second motion trajectory (corresponding to the forward direction of the camera) and the tangential direction of the trajectory position corresponding to the same time on the second predetermined trajectory of the mobile platform (corresponding to the forward direction of the mobile platform).

[0055] Return to reference Figure 4 After obtaining the rotation axis and rotation angle of the mobile platform relative to the camera, step S430 is executed. In step S430, based on the determined rotation axis and rotation angle, the rotation matrix between the camera coordinate system and the mobile platform coordinate system is determined as the installation posture information. The rotation matrix here can be used to represent the rotation transformation between the camera coordinate system and the mobile platform coordinate system. In one embodiment, the determined rotation axis is Where x, y, and z are the parameters of the rotation axis in the mobile platform coordinate system, and the determined rotation angle is θ. The rotation matrix of the camera relative to the mobile platform can be obtained by the angle-axis expression of the rotation transformation. As shown in formula (1):

[0056]

[0057] In other embodiments, formula (1) can also be transformed into Euler angle expression, and the three parameters of yaw angle, pitch angle and roll angle are used to represent the installation posture information of the camera on the mobile platform. In fact, those skilled in the art can use any method to represent the rotation transformation according to actual conditions.

[0058] After the rotation matrix of the camera relative to the mobile platform is determined, the position of the mobile platform can be determined by the actual position of the camera with the help of the rotation matrix, thereby compensating for the installation error of the camera relative to the mobile platform.

[0059] Although the above method only describes the process of using a first motion trajectory to determine the rotation axis of the mobile platform relative to the camera, and using a second motion trajectory to determine the rotation angle of the mobile platform relative to the camera, those skilled in the art can understand that the rotation axis of the mobile platform relative to the camera can also be determined using the fitting results of multiple first motion trajectories, and the rotation angle of the mobile platform relative to the camera can also be determined using the fitting results of multiple second motion trajectories. In addition, the first motion trajectory and the second motion trajectory can also be obtained by performing a planar motion.

[0060] The above describes the method for calibrating a camera according to an exemplary embodiment of the present disclosure. Although the various operations are depicted in the drawings as being in a specific order, this should not be understood as requiring that these operations must be performed in the specific order shown or in a sequential order, nor should it be understood as requiring that all the operations shown must be performed to obtain the desired results.

[0061] Figure 7 FIG. 7 is a schematic block diagram of a camera calibration apparatus 700 according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 for camera calibration may include a photographing unit 710 , a motion trajectory determining unit 720 , and a camera posture information determining unit 730 .

[0062] Figure 7 The device 700 shown in the figure can be used to implement the present disclosure in combination with the above Figure 1-Figure 6 In some embodiments, the device 700 can be implemented as a robot that performs planar motion, such as a sweeping robot, a logistics robot, etc.

[0063] The shooting unit 710 may be configured to control the camera to shoot multiple images when the mobile platform carrying the camera moves in a plane. The motion trajectory determination unit 720 may be configured to determine at least two different motion trajectories of the camera based on the multiple images captured. The camera attitude information determination unit 730 may be configured to determine the attitude information of the camera relative to the mobile platform based on the two different motion trajectories determined.

[0064] Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules, and / or at least some functions of multiple modules may be combined into a single module. The specific module discussed herein performing an action includes the specific module itself performing the action, or alternatively the specific module calling or otherwise accessing another component or module that performs the action (or performing the action together with the specific module). Therefore, the specific module that performs an action may include the specific module itself that performs the action and / or another module that the specific module calls or otherwise accesses to perform the action.

[0065] An exemplary embodiment of the present disclosure provides an electronic device, which may include a processor; and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the aforementioned methods.

[0066] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium storing a program, wherein the program includes instructions, and when the instructions are executed by a processor of an electronic device, the processor is caused to execute any of the aforementioned methods.

[0067] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein the computer program implements any of the aforementioned methods when executed by a processor.

[0068] See also Figure 8 , an electronic device 800 will now be described, which is an example of a hardware device (electronic device) that can be applied to various aspects of the present disclosure. The electronic device 800 can be any machine configured to perform processing and / or computing, and can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal digital assistant, a robot, a smart phone, a vehicle-mounted computer, or any combination thereof. The above-mentioned camera calibration method can be implemented in whole or in part by the electronic device 800 or a similar device or system.

[0069] The electronic device 800 may include an element connected to or in communication with the bus 802 (possibly via one or more interfaces). For example, the electronic device 800 may include a bus 802, one or more processors 804, one or more input devices 806, and one or more output devices 808. The one or more processors 804 may be any type of processor, and may include, but are not limited to, one or more general-purpose processors and / or one or more special-purpose processors (e.g., special processing chips). The input device 806 may be any type of device capable of inputting information to the electronic device 800, and may include, but are not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 808 may be any type of device capable of presenting information, and may include, but are not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The electronic device 800 may also include a non-transitory storage device 810, which may be any storage device that is non-transitory and can enable data storage, including but not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, an optical disk or any other optical medium, a ROM (read-only memory), a RAM (random access memory), a cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions and / or code. The non-transitory storage device 810 may be removable from the interface. The non-transitory storage device 810 may have data / programs (including instructions) / code for implementing the above-mentioned methods and steps. The electronic device 800 may also include a communication device 812. The communication device 812 may be any type of device or system that enables communication with an external device and / or with a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset, such as Bluetooth TM devices, 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0070] The electronic device 800 may also include a working memory 814, which may be any type of working memory that may store programs (including instructions) and / or data useful for the operation of the processor 804, and may include but is not limited to random access memory and / or read-only memory devices.

[0071] The software elements (programs) may be located in the working memory 814, including but not limited to an operating system 816, one or more applications 818, drivers and / or other data and codes. Instructions for executing the above methods and steps may be included in one or more applications 818, and the above camera calibration method may be implemented by the processor 804 reading and executing the instructions of one or more applications 818. More specifically, in the above camera calibration method, steps S110-S130 may be implemented, for example, by the processor 804 executing an application 818 having instructions for steps S110-S130. In addition, other steps in the above camera calibration method may be implemented, for example, by the processor 804 executing an application 818 having instructions for executing the corresponding steps. The executable code or source code of the instructions of the software element (program) may be stored in a non-transitory computer-readable storage medium (such as the above storage device 810), and may be stored in the working memory 814 when executed (possibly compiled and / or installed). The executable code or source code of the instructions of the software element (program) may also be downloaded from a remote location.

[0072] It should also be understood that various modifications may be made according to specific requirements. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. For example, some or all of the disclosed methods and apparatus may be implemented by programming hardware (e.g., programmable logic circuits including field programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) in assembly language or hardware programming languages ​​(such as VERILOG, VHDL, C++) using logic and algorithms according to the present disclosure.

[0073] It should also be understood that the foregoing method can be implemented through a server-client model. For example, the client can receive data input by the user and send the data to the server. The client can also receive data input by the user, perform a portion of the processing in the foregoing method, and send the processed data to the server. The server can receive data from the client, and execute the foregoing method or another portion of the foregoing method, and return the execution result to the client. The client can receive the execution result of the method from the server, and present it to the user, for example, through an output device.

[0074] It should also be understood that the components of the electronic device 800 can be distributed over a network. For example, one processor can be used to perform some processing, while other processing can be performed by another processor remote from the one processor. Other components of the computing system 800 can also be distributed similarly. In this way, the electronic device 800 can be interpreted as a distributed computing system that performs processing at multiple locations.

[0075] Although the present disclosure has been illustrated and described in detail in the drawings and in the foregoing description, such illustration and description should be considered illustrative and exemplary rather than restrictive; the present disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure and the appended claims, those skilled in the art will be able to understand and implement variations to the disclosed embodiments when practicing the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps that are not listed, the indefinite article "a" or "an" does not exclude a plurality, and the term "plurality" means two or more. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A camera calibration method, comprising: When the mobile platform equipped with the camera performs planar motion, multiple images of the environment are captured by the camera at multiple track points in the planar motion; Determining at least two different motion trajectories of the camera in the environment based on the environmental feature information in the multiple images captured at the multiple trajectory points and the predetermined surrounding environment information, wherein the at least two different motion trajectories include a first motion trajectory and a second motion trajectory; and Determining installation posture information of the camera relative to the mobile platform based on the at least two different motion trajectories determined includes: Based on the first motion trajectory, determining a rotation axis of the mobile platform relative to the camera; Determining a rotation angle of the camera relative to the mobile platform around the rotation axis based on the second motion trajectory and the determined rotation axis; and Based on the determined rotation axis and the rotation angle, a rotation matrix between the camera coordinate system and the mobile platform coordinate system is determined as the installation posture information.

2. The method of claim 1, wherein: Each of the plurality of images captured includes at least one of a plurality of markers arranged in the environment, and, Determining at least two different motion trajectories of the camera based on environmental feature information in the plurality of captured images and predetermined surrounding environment information includes: Determining a corresponding shooting position of the camera in the environment when shooting each image based on a relative positional relationship between a position of at least one marker in each image in the corresponding image and the plurality of markers in the environment; and The at least two different motion trajectories are determined based on a plurality of shooting positions corresponding to the plurality of images shot.

3. The method of claim 2, wherein: The marker has a predefined size and / or pattern.

4. The method of claim 2, wherein: The relative positional relationship between each marker among the multiple markers and other markers includes the relative rotation and displacement between the marker and the other markers.

5. The method of claim 2, wherein: The relative positional relationship between the multiple markers is determined based on at least one calibration image acquired for the environment by the camera installed on the mobile platform or other photographic equipment in the environment that has completed internal reference calibration.

6. The method of claim 5, wherein: Each of the plurality of markers is included in at least one of the at least one calibration image acquired for the environment.

7. The method of claim 2, wherein: At least one of the plurality of markers is included in at least one of the plurality of images captured by the camera.

8. The method according to any one of claims 2 to 7, wherein: The plurality of identifiers include AprilTag tags.

9. The method according to any one of claims 1 to 7, wherein: The first motion trajectory corresponds to at least three first trajectory points of at least three non-collinear shooting positions of the camera when the mobile platform performs a first planar motion, and Based on the first motion trajectory, determining a rotation axis of the mobile platform relative to the camera includes: Fitting the at least three first trajectory points to the same plane to form a first plane fitting motion trajectory; and Based on the determined first plane fitting motion trajectory, a plane normal vector of a plane where the first plane fitting motion trajectory is located is determined as a rotation axis of the mobile platform relative to the camera.

10. The method of claim 9, wherein: Based on the first motion trajectory, determining the rotation axis of the mobile platform relative to the camera includes: applying at least one plane constraint to the at least three first trajectory points to fit the at least three first trajectory points to the same plane to form a first plane fitting motion trajectory.

11. The method of claim 10, wherein: The at least one plane constraint includes a non-linear least squares optimization algorithm.

12. The method according to any one of claims 1 to 7, wherein: The second motion trajectory corresponds to at least two second trajectory points of at least two shooting positions of the camera when the mobile platform performs a second planar motion, and Based on the second motion trajectory and the determined rotation axis, determining a rotation angle of the mobile platform relative to the camera around the rotation axis includes: connecting the at least two second trajectory points to form the second motion trajectory; Using the rotation axis, fitting the second motion trajectory to the plane where the mobile platform performs the second planar motion to form a second planar fitting motion trajectory; and An angle between a trajectory of the mobile platform moving in the second plane and the second plane fitting motion trajectory of the camera is determined as the rotation angle.

13. The method of claim 12, wherein: The second plane motion of the mobile platform is a linear motion.

14. A device for calibrating a camera, comprising: A shooting unit, configured to control the camera to shoot multiple images of the environment at multiple track points in the planar motion when the mobile platform carrying the camera makes a planar motion; a motion trajectory determining unit configured to determine at least two different motion trajectories of the camera in the environment based on the environmental feature information in the multiple images captured at the multiple trajectory points and the predetermined surrounding environment information, wherein the at least two different motion trajectories include a first motion trajectory and a second motion trajectory; and A camera posture information determination unit is configured to determine the installation posture information of the camera relative to the mobile platform based on the two different motion trajectories determined, including: Based on the first motion trajectory, determining a rotation axis of the mobile platform relative to the camera; Determining a rotation angle of the camera relative to the mobile platform around the rotation axis based on the second motion trajectory and the determined rotation axis; and Based on the determined rotation axis and the rotation angle, a rotation matrix between the camera coordinate system and the mobile platform coordinate system is determined as the installation posture information.

15. An electronic device, comprising: processor; as well as A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 13. 16 . A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the processor to execute the method according to claim 1 .

17. A computer program product comprising a computer program, wherein: The computer program implements the method according to any one of claims 1 to 13 when executed by a processor.

Images