Camera calibration method, device and equipment

Abstract

The invention relates to the technical field of calibration, and provides a camera calibration method, device and equipment, and the method comprises the steps: obtaining calibration images obtained by a plurality of cameras in a calibration environment, and the calibration environment is provided with a plurality of calibration identifiers; when the calibration image comprises at least two calibration identifiers, determining a pose relationship between the at least two calibration identifiers included in the calibration image according to the calibration image; when the calibration marks included in the calibration images shot by the at least two cameras are different, according to the calibration images shot by the at least two cameras and the pose relation between the different calibration marks included in the calibration images shot by the at least two cameras, the calibration images shot by the at least two cameras are obtained; and determining a pose relationship between the at least two cameras so as to improve the convenience degree and the accuracy degree of calibration of the plurality of cameras.

Description

Technical Field

[0001] The present application relates to the field of calibration technology, and in particular to a camera calibration method, device and equipment. Background Art

[0002] At present, in application scenarios such as mapping, positioning, and perception, multiple cameras are usually deployed to observe information at different locations. For example, a near-eye display device can be configured with multiple cameras to obtain information about the eyes, face, and environment of the user wearing the near-eye display device. However, when calibrating multiple cameras, due to the positional differences between the multiple cameras, it is easy to find it difficult to calibrate the camera parameters of the multiple cameras, resulting in poor convenience and accuracy in calibrating multiple cameras. Summary of the invention

[0003] The main purpose of this application is to provide a camera calibration method, device and equipment, aiming to improve the convenience and accuracy of calibrating multiple cameras.

[0004] In a first aspect, the present application provides a camera calibration method, the camera calibration method comprising the following steps:

[0005] Acquire calibration images obtained by photographing a calibration environment with a plurality of cameras respectively, wherein the calibration environment is provided with a plurality of calibration marks;

[0006] When the calibration image includes at least two calibration marks, determining a position and posture relationship between the at least two calibration marks included in the calibration image according to the calibration image;

[0007] When the calibration marks included in the calibration images respectively taken by at least two cameras are different, the posture relationship between the at least two cameras is determined according to the calibration images respectively taken by the at least two cameras and the posture relationship between the different calibration marks included in the calibration images respectively taken by the at least two cameras.

[0008] In a second aspect, the present application further provides a camera calibration device, the camera calibration device comprising:

[0009] An acquisition module, used to acquire calibration images obtained by photographing a calibration environment by multiple cameras respectively, wherein the calibration environment is provided with multiple calibration marks;

[0010] A first determining module, configured to determine, according to the calibration image, a position and posture relationship between the at least two calibration marks included in the calibration image when the calibration image includes at least two calibration marks;

[0011] The second determination module is used to determine the posture relationship between at least two cameras according to the calibration images taken by at least two cameras and the posture relationship between different calibration identifiers included in the calibration images taken by at least two cameras when the calibration identifiers included in the calibration images taken by at least two cameras are different.

[0012] In a third aspect, the present application further provides a near-eye display device, the near-eye display device comprising a plurality of cameras, a memory, and a processor;

[0013] The memory is used to store computer programs;

[0014] The processor is used to execute the computer program and implement the camera calibration method as described above when executing the computer program.

[0015] The present application provides a camera calibration method, device and equipment, the camera calibration method comprising: obtaining calibration images obtained by each of the multiple cameras shooting a calibration environment, the calibration environment being provided with multiple calibration marks; when the calibration image includes at least two calibration marks, determining the posture relationship between the at least two calibration marks included in the calibration image according to the calibration image; when the calibration marks included in the calibration images shot by the at least two cameras are different, determining the posture relationship between the at least two cameras according to the calibration images shot by the at least two cameras and the posture relationship between the different calibration marks included in the calibration images shot by the at least two cameras. In the case of obtaining calibration images obtained by each of the multiple cameras shooting a calibration environment, the calibration images can be used to determine the posture relationship between the multiple calibration marks. Based on this, the posture relationship between the multiple cameras can be combined with the posture relationship between the multiple calibration marks to determine the posture relationship between the multiple cameras, thereby completing the calibration of the multiple cameras, which is conducive to improving the convenience and accuracy of calibrating the multiple cameras. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 is a flowchart of a camera calibration method provided in an embodiment of the present application;

[0018] Figure 2 is an application scenario diagram of a camera calibration method involved in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a calibration environment involved in an embodiment of the present application;

[0020] Figure 4 is a schematic block diagram of a camera calibration device provided in an embodiment of the present application;

[0021] Figure 5 It is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0024] The embodiments of the present application provide a camera calibration method, device and apparatus, wherein the camera calibration method can be applied to a near-eye display device.

[0025] Exemplarily, the near-eye display device may include VR glasses, AR glasses, VR helmets, AR helmets, and the like, without limitation herein.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] See also Figure 1 , Figure 1 It is a flowchart of a camera calibration method provided in an embodiment of the present application. It should be noted that the camera calibration method provided in an embodiment of the present application can be used for near-eye display devices.

[0028] like Figure 1 As shown, the camera calibration method includes steps S101 to S103.

[0029] S101, obtaining calibration images obtained by photographing a calibration environment by multiple cameras respectively, where the calibration environment is provided with multiple calibration marks.

[0030] For example, when multiple cameras are in a calibration environment, the multiple cameras can each photograph the calibration environment to obtain a calibration image.

[0031] Taking a near-eye display device as an example, the near-eye display device may include multiple cameras. For example, the multiple cameras need to respectively capture facial images, eye movement images, gesture images, and the like of a user wearing the near-eye display device, so as to determine the corresponding control information of the user on the near-eye display device through the images captured by each of the multiple cameras. If the multiple cameras set on the near-eye display device are not calibrated, it is easy to have a situation where the accuracy of determining the corresponding control information of the near-eye display device is poor. For example, if the camera is not calibrated, it is easy to have a situation where the user's facial information cannot be accurately determined based on the facial image of the user captured by the camera, or the user's eye movement information cannot be accurately determined based on the eye movement image of the user captured by the camera, or the user's gesture information cannot be accurately determined based on the gesture image of the user captured by the camera, etc., resulting in poor control effect of controlling the near-eye display device. Based on this, it is necessary to calibrate the multiple cameras set on the near-eye display device. For example, the near-eye display device can be placed in a calibration environment to obtain images obtained by each of the multiple cameras set on the near-eye display device capturing the calibration environment. Of course, it is not limited to near-eye display devices. For example, when multiple cameras are provided on an electronic device other than a near-eye display device, the multiple cameras provided on the electronic device also need to be calibrated. This is not limited here.

[0032] For example, when multiple cameras are in a calibration environment, the positions of the multiple cameras that need to be calibrated on the corresponding device can be relatively fixed. Taking a near-eye display device as an example, if multiple cameras set on the near-eye display device need to capture any one of the eye movement images and gesture images of the user wearing the near-eye display device, the camera that captures the eye movement image can be set in the device display area of ​​the near-eye display device, such as the lenses of AR glasses, the display of a VR helmet, etc., which are not limited here. Correspondingly, the camera that captures the gesture image can be set in the device frame area of ​​the near-eye display device, such as the frame of AR glasses, the helmet frame of a VR helmet, etc., which are not limited here. Since the positions of the device display area and the device frame area on the near-eye display device are relatively fixed, the positions of the multiple cameras on the near-eye display device can also be relatively fixed. In some embodiments, the camera that captures the eye movement image can be shot from the device display area of ​​the near-eye display device in a first direction toward the user's eyes, and the camera that captures the gesture image can be shot from the device frame area of ​​the near-eye display device in a second direction toward the user's hand, then the camera that captures the eye movement image and the camera that captures the gesture image can be calibrated. Taking the device display area of ​​the near-eye display device as the position reference plane as an example, the first direction can be perpendicular to the position reference plane, and the second direction can be parallel to the position reference plane. Of course, it is not limited to this and is not limited here. However, when the near-eye display device is in a calibration environment to calibrate multiple cameras set on the near-eye display device, if the angle difference between the first direction and the second direction is large, at least one of the cameras that capture eye movement images and the cameras that capture gesture images may not be able to capture the calibration mark in the calibration environment. When the calibration image captured by the corresponding camera does not include the calibration mark, for example, the camera parameters of the camera cannot be determined based on the position of the calibration mark in the calibration environment and the position of the calibration mark in the calibration image, and thus the corresponding camera cannot be calibrated. Based on this, multiple calibration marks can be set in the calibration environment.

[0033] In some embodiments, a plurality of calibration plates are provided in the calibration environment, and an angle between at least two calibration plates is greater than or equal to a preset angle threshold.

[0034] For example, Figure 2As shown, a plurality of calibration plates are provided in the calibration environment, and the angle between at least two calibration plates is greater than or equal to 90 degrees. Wherein, each calibration plate may include a plurality of different calibration marks, and each calibration mark is unique. Taking a near-eye display device as an example, when the near-eye display device is placed in the calibration environment, the shooting angle difference between the first direction and the second direction can be determined based on the same plane where the device display area of ​​the near-eye display device and the device frame area of ​​the near-eye display device are located. In the case where the shooting angle difference between the first direction and the second direction is greater than or equal to a preset angle threshold such as 90 degrees, since the angle between at least two calibration plates in the calibration environment is greater than or equal to 90 degrees, the setting position of the near-eye display device in the calibration environment can be adjusted. In response to the adjustment of the setting position of the near-eye display device in the calibration environment, the camera provided in the device display area of ​​the near-eye display device can shoot at least one of the at least two calibration plates with an angle greater than or equal to 90 degrees to obtain a calibration image. Correspondingly, the camera provided in the device frame area of ​​the near-eye display device can also shoot at least one of the at least two calibration plates with an angle greater than or equal to 90 degrees to obtain a corresponding calibration image. Based on this, setting multiple calibration marks in the calibration environment is conducive to increasing the possibility of each camera on the near-eye display device capturing the calibration mark, and improving the efficiency and effect of obtaining the calibration image. Of course, it is not limited to this and is not limited here.

[0035] In some embodiments, when obtaining calibration images taken by multiple cameras, the camera parameters of the camera corresponding to the calibration image can be determined according to the calibration mark included in the calibration image and the position of the calibration mark in the calibration environment, such as determining the camera intrinsic parameters of the camera corresponding to the calibration image. Among them, the camera intrinsic parameters may include a camera intrinsic parameter matrix, a camera distortion parameter, and the like. The camera intrinsic parameters can be used to convert the position information of the calibration mark in the camera coordinate system corresponding to the camera into the position information of the calibration mark in the pixel coordinate system corresponding to the camera. For example, the calibration image is processed by a preset calibration method such as the Zhang Zhengyou calibration method, and the like, to obtain the camera intrinsic parameters of the camera corresponding to the calibration image.

[0036] In this way, by acquiring calibration images obtained by capturing the calibration environment by multiple cameras, the calibration images can be used to determine the camera parameters of each of the multiple cameras, such as camera intrinsic parameters. In addition, the calibration images can be used for subsequent calibration of multiple cameras, such as determining the camera extrinsic parameters of each of the multiple cameras, and the camera extrinsic parameters may include a camera extrinsic parameter matrix. The camera extrinsic parameters can be used to convert the position information of the calibration mark in the world coordinate system corresponding to the camera into the position information of the calibration mark in the camera coordinate system corresponding to the camera. Based on the determination of the camera parameters of each of the multiple cameras, it is beneficial to improve the accuracy and convenience of calibrating multiple cameras. Based on the calibration environment, multiple calibration marks are provided to reduce the situation where multiple cameras cannot capture the calibration marks in the calibration environment, which is beneficial to improve the efficiency and effect of obtaining the calibration images, so as to improve the accuracy and convenience of calibrating multiple cameras.

[0037] In some embodiments, when there is a calibration image taken by at least one camera that does not include a calibration mark, at least one movement of multiple cameras and the calibration mark in the calibration environment is controlled so that during the process of at least one movement of the multiple cameras and the calibration mark in the calibration environment, the multiple cameras each take a picture of the calibration environment to obtain a calibration image including the calibration mark.

[0038] Exemplarily, the movement may include linear movement, curved movement, rotation, and the like, which are not limited here.

[0039] For example, Figure 3 As shown, taking multiple cameras set on a near-eye display device as an example, the near-eye display device can be fixedly set on a robotic arm, and correspondingly, multiple cameras are also fixedly set on the robotic arm. When there is a calibration image taken by at least one camera that does not include a calibration mark, the near-eye display device can be moved by moving the robotic arm, thereby driving the movement of multiple cameras. Multiple cameras can each capture the calibration environment to obtain a calibration image during the movement of the robotic arm, so that each camera can capture a calibration image including a calibration mark for subsequent camera calibration.

[0040] For example, when at least one camera captures a calibration image that does not include a calibration mark, the calibration mark in the calibration environment may be moved. Multiple cameras may capture the calibration environment to obtain calibration images while the calibration mark in the calibration environment is moving, so that each camera can capture a calibration image including the calibration mark for subsequent camera calibration.

[0041] Of course, it is not limited to this. For example, when there is a calibration image captured by at least one camera that does not include a calibration mark, multiple cameras and the calibration mark in the calibration environment can be controlled to move in opposite directions, or multiple cameras and the calibration mark in the calibration environment can be controlled to move at different speeds in the same moving direction. Multiple cameras can each capture the calibration environment to obtain a calibration image when the multiple cameras and the calibration mark in the calibration environment are in relative motion, so that each camera can capture a calibration image including the calibration mark for subsequent camera calibration. This is not limited here.

[0042] In this way, when there is a calibration image taken by at least one camera that does not include a calibration mark, by controlling at least one of the movement of multiple cameras and the calibration mark in the calibration environment, the multiple cameras and the calibration mark in the calibration environment are in a relative motion state, so that the camera that originally could not capture the calibration mark can, for example, capture the calibration image including the calibration mark during the relative motion, which is conducive to improving the acquisition efficiency and acquisition effect of the calibration image. In the case where the calibration image includes the calibration mark, the calibration image can be used for subsequent calibration of multiple cameras, which is conducive to improving the accuracy and convenience of calibrating multiple cameras.

[0043] S102: When the calibration image includes at least two calibration marks, determine, according to the calibration image, a position and posture relationship between the at least two calibration marks included in the calibration image.

[0044] For example, when a calibration image including a calibration mark is obtained, it is equivalent to that the camera that took the calibration image took the calibration mark. Based on the calibration image, the position of the calibration mark in the calibration image can be determined. In addition, the position of the calibration mark in the calibration environment can be obtained. In some embodiments, the position relationship between the calibration image and the calibration environment can be determined based on the position of the calibration mark in the calibration image and the position of the calibration mark in the calibration environment. Since the calibration image is taken by a camera and the calibration mark is set in the calibration environment, the position relationship between the camera and the calibration mark can be determined based on the position relationship between the calibration image and the calibration environment.

[0045] Based on this, when the calibration image captured by the camera includes at least two calibration marks, it is equivalent to that the camera that captured the calibration image captured at least two calibration marks at the same time, then the position and posture relationship between the camera that captured the calibration image and the at least two calibration marks included in the calibration image can be determined according to the calibration image. In the case where corresponding position and posture relationships are established between the same camera and at least two calibration marks, any one calibration mark can be selected from the at least two calibration marks as the calibration mark to be converted, and the other calibration marks except the calibration mark to be converted in the at least two calibration marks can be used as reference calibration marks. For example, based on the position and posture relationship between the camera and the at least two calibration marks, the position of the calibration mark to be converted in the corresponding mark coordinate system can be converted to the position in the mark coordinate system corresponding to the reference calibration mark. Based on this, the position and posture relationship between the at least two calibration marks included in the calibration image can be determined in combination with the position and posture relationship between the camera that captured the calibration image including at least two calibration marks and the at least two calibration marks included in the calibration image.

[0046] For example, during the relative movement between multiple cameras and the calibration markers in the calibration environment, the multiple cameras can be moved as a whole, and / or the calibration markers in the calibration environment can be moved as a whole, and no relative movement occurs between different calibration markers in the calibration environment. Based on this, regardless of whether the multiple cameras and the calibration markers in the calibration environment perform relative movement, since no relative movement occurs between different markers in the calibration environment, after determining the positional relationship between two calibration markers, the positional relationship between the two calibration markers will not change with time, so that the positional relationship between the at least two calibration markers included in the calibration image determined based on the calibration image including at least two calibration markers can be used later.

[0047] In some embodiments, based on the calibration image, the pose relationship between the camera that took the calibration image and at least two calibration markers included in the calibration image is determined; based on the pose relationship between the camera that took the calibration image and at least two calibration markers, the pose relationship between at least two calibration markers is determined.

[0048] For example, the positions of the at least two calibration marks included in the calibration image can be determined. In addition, the positions of the at least two calibration marks in the calibration environment are obtained. Since the calibration environment is equivalent to a three-dimensional space and the calibration image is equivalent to a two-dimensional plane, after the positions of the at least two calibration marks in the calibration image and the positions of the at least two calibration marks in the calibration environment are determined, the position and posture relationship between the calibration image corresponding to the at least two calibration marks and the calibration environment can be established by using algorithms such as Direct Linear Transform (DLT) and PnP (Perspective-n-Point) algorithm to determine the position and posture relationship between the camera corresponding to the calibration image and the at least two calibration marks. According to the position and posture relationship between the camera corresponding to the calibration image and the at least two calibration marks, the position and posture relationship between the at least two calibration marks can be determined.

[0049] For example, the calibration image includes at least two calibration marks, namely, calibration mark a1 and calibration mark a2, and the camera that took the calibration image is camera c1. The position and posture relationship between camera c1 and calibration mark a1 can be determined based on the calibration image. And determine the pose relationship between camera c1 and calibration marker a2 Among them, the posture relationship It is used to indicate the position of the camera coordinate system 1 corresponding to the camera c1 is converted to the identification coordinate system 1 corresponding to the calibration identification a1. It is used to indicate the conversion of the position of camera coordinate system 1 corresponding to camera c1 to the position of marker coordinate system 2 corresponding to calibration marker a2, and so on. And the posture relationship The position relationship between the calibration mark a1 and the calibration mark a2 can be determined Among them, the posture relationship It can be expressed as:

[0050]

[0051] Of course, it is not limited to this and no limitation is made here.

[0052] For example, when multiple cameras each capture the calibration environment to obtain calibration images during at least one of the multiple cameras and the calibration markers in the calibration environment moves, the multiple cameras may capture different calibration images at different times. However, during at least one of the multiple cameras and the calibration markers in the calibration environment moves, the positional relationship between the multiple cameras and the calibration markers in the calibration environment may change in response to the relative movement between the multiple cameras and the calibration markers in the calibration environment. For example, different cameras may capture the same at least two calibration markers at different times, and the same camera may capture the same at least two calibration markers at different times.

[0053] In some embodiments, at different shooting times, if multiple cameras are in a relative motion state with a calibration mark in a calibration environment, the pose relationship between each of the multiple cameras and the calibration mark will change in response to the relative motion between the multiple cameras and the calibration mark in the calibration environment. According to the pose relationship between each of the multiple cameras and the calibration mark at different shooting times, for example, different pose relationships between multiple calibration marks can be obtained. Based on this, it is necessary to determine the pose relationship between each of the multiple cameras and the calibration mark based on multiple calibration images with the same shooting time, so as to determine the pose relationship between the multiple calibration marks based on the pose relationship between each of the multiple cameras and the calibration mark at the same shooting time.

[0054] Taking the shooting time including time t1 and time t2 as an example, camera c1 captures a calibration image including calibration marker a1 and calibration marker a2 at time t1, and camera c2 captures a calibration image including calibration marker a1 and calibration marker a2 at time t2. Based on multiple calibration images, the position relationship between camera c1 and calibration marker a1 at time t1 can be determined respectively. The pose relationship between camera c1 and calibration marker a2 And the pose relationship between camera c2 and calibration marker a1 at time t2 The pose relationship between camera c2 and calibration marker a2 According to the posture relationship And the posture relationship The position relationship between the calibration marker a1 and the calibration marker a2 at time t1 can be determined According to the posture relationship And the posture relationship The position relationship between the calibration marker a1 and the calibration marker a2 at time t2 can be determined Accordingly, according to the posture relationship And the posture relationship Determine whether the pose relationship between the calibration marker a1 and the calibration marker a2 changes at different shooting times to verify whether the pose relationship between the multiple calibration markers will not change in response to the relative motion between the multiple cameras and the calibration markers in the calibration environment. And the posture relationship If the position of the camera c1, camera c2, or at least one of the calibration marks in the calibration environment is loose during the relative movement between the multiple calibration marks and the calibration marks in the calibration environment, the corresponding camera or calibration mark needs to be fixed to avoid the adverse effect on the accuracy of determining the position relationship between the multiple calibration marks due to the loose position of the camera or calibration mark during the relative movement between the multiple cameras and the calibration marks in the calibration environment. Verifying the position relationship between the multiple calibration marks based on calibration images taken at different times is conducive to improving the accuracy of determining the position relationship between the multiple calibration marks, which can be used to subsequently improve the accuracy of calibrating multiple cameras.

[0055] In this way, when the calibration image includes at least two calibration marks, the posture relationship between the at least two calibration marks included in the calibration image can be determined according to the calibration image. For example, the posture relationship between the camera that took the calibration image and the at least two calibration marks included in the calibration image can be determined according to the calibration image, so as to comprehensively determine the posture relationship between the camera that took the calibration image and the at least two calibration marks. Since the at least two calibration marks are not in a relative motion state, for example, the posture relationship between the at least two calibration marks will not change in response to the relative motion state between the multiple cameras and the calibration marks in the calibration environment, the posture relationship between the at least two calibration marks determined can be used for subsequent calibration of multiple cameras, which is conducive to improving the convenience and accuracy of calibration of multiple cameras.

[0056] In some embodiments, when two calibration images taken at the same time each include at least two calibration markers, and there is at least one identical calibration marker among the calibration markers included in each of the two calibration images, the posture relationship between all the calibration markers included in each of the multiple calibration images is determined based on the two calibration images.

[0057] For example, at shooting time t1, camera c1 captures a calibration image including calibration marker a1 and calibration marker a2, and camera c2 captures a calibration image including calibration marker a2 and calibration marker a3. Since they are at the same shooting time, i.e., shooting time t1, the change of the pose relationship between camera c1 and any one of calibration markers a1 and calibration marker a2 caused by the movement of camera c1 at different shooting times can be avoided. Correspondingly, the change of the pose relationship between camera c2 and any one of calibration markers a2 and calibration marker a3 caused by the movement of camera c2 at different shooting times can also be avoided. Since the calibration images captured by camera c1 and camera c2 both include calibration marker a2, the pose relationship between any two calibration markers among calibration marker a1, calibration marker a2 and calibration marker a3 can be determined through calibration marker a2. For example, based on the calibration image captured by camera c1, the pose relationship between camera c1 and calibration marker a1 can be determined. And the pose relationship between camera c1 and calibration marker a2 According to the calibration image taken by camera c2, the pose relationship between camera c2 and calibration marker a2 can be determined. And the pose relationship between camera c2 and calibration marker a3 According to the posture relationship And the posture relationship The position relationship between the calibration mark a1 and the calibration mark a2 can be determined According to the posture relationship And the posture relationship The position relationship between the calibration mark a2 and the calibration mark a3 can be determined Among them, since the posture relationship between multiple calibration markers is relatively fixed, the posture relationship It can also be expressed as And the posture relationship It can also be expressed as Based on this, we can And the posture relationship Determine the pose relationship between the calibration marker a1 and the calibration marker a3 This is equivalent to using the calibration mark a2 as the intermediate mark, combined with the pose relationship And the posture relationship Determine the pose relationship

[0058] Based on this, when camera c1 does not capture calibration mark a3 and camera c2 does not capture calibration mark a1, the position relationship between calibration mark a1 and calibration mark a3 can also be determined based on the fact that camera c1 and camera c2 capture calibration mark a2 together. It is not necessary for the camera c1 or the camera c2 to capture the calibration marks a1 to a3. Of course, this is not limited to this and is not limited here.

[0059] In this way, when two calibration images taken at the same time each include at least two calibration markers, and there is at least one identical calibration marker among the calibration markers included in each of the two calibration images, a pose relationship between all the calibration markers included in each of the two images can be established based on at least one calibration marker captured by both calibration images, without the need for any camera to capture all the calibration markers, so as to avoid the situation where the pose relationship between all the calibration markers cannot be determined due to the camera's inability to capture all the calibration markers, thereby making it impossible to calibrate the camera. This is beneficial to improving the convenience of calibrating multiple cameras.

[0060] S103: When the calibration marks included in the calibration images respectively taken by the at least two cameras are different, determine the posture relationship between the at least two cameras according to the calibration images respectively taken by the at least two cameras and the posture relationship between different calibration marks included in the calibration images respectively taken by the at least two cameras.

[0061] For example, the calibration marks included in each calibration image may be different. For example, when the calibration image captured by camera c1 includes calibration mark a1, and the calibration image captured by camera c2 includes calibration mark a2, it can be determined that the calibration marks included in the calibration images captured by camera c1 and camera c2 are different. Of course, it is not limited to this. When the calibration image captured by camera c1 includes calibration mark a1 and calibration mark a2, and the calibration image captured by camera c2 includes calibration mark a2 and calibration mark a3, since calibration mark a1 is different from calibration mark a3, it can be determined that the calibration marks included in the calibration images captured by camera c1 and camera c2 are different, and this is not limited here.

[0062] For example, camera c1 captures a calibration image including calibration mark a1, and camera c2 captures a calibration image including calibration mark a2. Since calibration mark a1 is different from calibration mark a2, the positional relationship between camera c1 and calibration mark a1 can be determined based on the calibration image captured by camera c1. And the position relationship between camera c2 and calibration mark a2 can be determined based on the calibration image taken by camera c2 Combined with the position relationship between the calibration mark a1 and the calibration mark a2 Determine the pose relationship between camera c1 and camera c2 For example, the pose relationship It can be expressed as:

[0063]

[0064] Among them, the camera c1 and the camera c2 do not need to capture the calibration mark a1 and the calibration mark a2 at the same time. It can be determined based on the calibration images including the calibration markers a1 and a2 captured by other cameras other than the cameras c1 and c2 at a certain shooting time. Since the position relationship between the calibration markers a1 and a2 is relatively fixed and has nothing to do with the shooting time, the position relationship It can be used in the subsequent determination of the position and posture relationship between camera c1 and camera c2. Of course, it is not limited to calibration mark a1 and calibration mark a2, camera c1 and camera c2, and is not limited here.

[0065] In some embodiments, at different shooting times, if multiple cameras are in a relative motion state with a calibration mark in a calibration environment, the pose relationship between each of the multiple cameras and the calibration mark will change in response to the relative motion between the multiple cameras and the calibration mark in the calibration environment. Based on this, when the calibration marks included in the calibration images shot by at least two cameras at the same shooting time are different, the pose relationship between the at least two cameras can be determined based on the calibration images shot by the at least two cameras and the pose relationship between the different calibration marks included in the calibration images shot by the at least two cameras.

[0066] Accordingly, since the multiple cameras are in a relatively static state, for example, if the multiple cameras are all set on the near-eye display device, the multiple cameras can move as a whole, such as when the near-eye display device moves with the movement of the robot arm, the multiple cameras move with the movement of the near-eye display device, and the posture relationship between the multiple cameras will not change. Based on this, the posture relationship between at least two cameras determined can be used later.

[0067] For example, the position relationship between the at least two cameras at different shooting times can be determined based on the calibration images taken by at least two cameras at different shooting times, so as to verify whether the position relationship between the multiple cameras will not change in response to the relative movement between the multiple cameras and the calibration marks in the calibration environment. If the position relationship between the at least two cameras at different shooting times changes, it may be caused by the position loosening of at least one of the at least two cameras and / or the calibration mark in the calibration environment during the relative movement between the multiple calibration marks and the calibration mark in the calibration environment. In this case, the corresponding camera or calibration mark needs to be fixed in position to avoid the adverse effect on the accuracy of determining the position relationship between the multiple cameras caused by the position loosening of the camera or calibration mark during the relative movement between the multiple cameras and the calibration mark in the calibration environment. Verifying the position relationship between the multiple cameras based on the calibration images at different shooting times is conducive to improving the accuracy of determining the position relationship between the multiple cameras.

[0068] In this way, when determining the pose relationship between multiple cameras, multiple cameras do not need to capture the same calibration mark. In the case where the setting positions or setting directions between multiple cameras are greatly different, the pose relationship between multiple cameras can still be determined by combining the calibration images captured by multiple cameras at the same shooting time and the pose relationship between different calibration marks included in the calibration images, which is conducive to improving the convenience of determining the pose relationship between multiple cameras. Based on the determination of the pose relationship between multiple cameras, the camera parameters of each of the multiple cameras can be determined according to the pose relationship between the multiple cameras, such as determining the camera extrinsic parameters of each of the multiple cameras, which is conducive to improving the convenience of calibrating multiple cameras. Based on the fact that the pose relationship between multiple cameras can be verified according to the calibration images at different shooting times, it is conducive to improving the accuracy of determining the camera extrinsic parameters of each of the multiple cameras, and it is conducive to improving the accuracy of calibrating multiple cameras.

[0069] Exemplarily, based on at least one of a pose relationship between multiple calibration markers and a pose relationship between multiple cameras, the calibration markers included in the calibration image are reprojected to determine predicted position information of the calibration markers included in the calibration image; based on the predicted position information, camera parameters of the camera that took the calibration image are determined.

[0070] In some embodiments, when the pose relationship between multiple calibration markers and the pose relationship between multiple cameras are determined, since the pose relationship between multiple calibration markers will not change with the relative movement between multiple cameras and the calibration markers in the calibration environment, and the pose relationship between multiple cameras will not change with the relative movement between multiple cameras and the calibration markers in the calibration environment, the calibration markers included in the calibration image can be reprojected according to at least one of the pose relationship between the multiple calibration markers and the pose relationship between the multiple cameras. For example, when the camera parameters of each of the multiple cameras, such as camera intrinsic parameters and camera extrinsic parameters, may be distorted, the calibration markers included in the calibration image can be reprojected according to at least one of the pose relationship between the multiple calibration markers and the pose relationship between the multiple cameras, so as to determine and optimize the camera parameters of the camera that captured the calibration image. By analogy, when multiple cameras each take a calibration image, the calibration marks included in the calibration image can be reprojected according to the calibration images taken by the multiple cameras to determine and optimize the camera parameters of the camera that took the calibration image, thereby completing the determination and optimization of the camera parameters of each of the multiple cameras.

[0071] For example, the multiple calibration marks may include calibration marks a1 and calibration marks a i The multiple cameras may include camera c1 and camera c k At shooting time t1, camera c1 can capture the calibration mark a1, and camera c k The calibration mark a can be photographed i Based on this, the calibration mark a i The calibration point P on i j It can be verified by calibration mark a1 and calibration mark a i The position relationship between Convert to the identification coordinate system corresponding to the calibration marker a1, through the pose relationship between the camera c1 and the calibration marker a1 Convert to the camera coordinate system corresponding to camera c1, and k The position relationship between Switch to camera c k The corresponding camera coordinate system, and according to the camera c k Capture the calibration mark a i The calibration image of the camera c k The corresponding camera internal parameters are used to obtain the calibration mark a i In camera c k Predicted position information in the captured calibration image To mark the mark a i Reproject to camera ck In the captured calibration image, due to the presence of noise, the predicted position information With calibration mark a i In camera c k The actual position information in the captured calibration image may be different, so the predicted position information Available for camera c k The camera parameters are determined and optimized. Of course, it is not limited to this and is not limited here.

[0072] In some embodiments, when a calibration image includes multiple calibration markers, the multiple calibration markers included in the calibration image are reprojected based on the posture information between the multiple calibration markers included in the calibration image to determine the predicted position information of each of the multiple calibration markers included in the calibration image.

[0073] For example, the multiple calibration marks may include calibration marks a1 and calibration marks a i , the camera may include camera c k At shooting time t2, camera c k The calibration mark a1 and the calibration mark a can be photographed i , then the calibration image includes multiple calibration marks. Based on this, the calibration mark a i The calibration point P on i j It can be verified by calibration mark a1 and calibration mark a i The position relationship between Convert to the identification coordinate system corresponding to the calibration identification a1, and through the camera c k The pose relationship with the calibration mark a1 Switch to camera c k The corresponding camera coordinate system, and according to the camera c k Capture the calibration mark a i The calibration image of the camera c k The corresponding camera internal parameters are used to obtain the calibration mark a i In camera c k Predicted position information in the captured calibration image To mark the mark a i Reproject to camera c k In the captured calibration image, due to the presence of noise, the predicted position information With calibration mark a i In camera c k The actual position information in the captured calibration image may be different, so the predicted position information Available for camera c kThe camera parameters of the camera c are determined and optimized. Similarly, the calibration point on the calibration mark a1 can also be determined. k The actual position information in the captured calibration image is used to calibrate the camera c k The camera parameters are determined and optimized. Of course, it is not limited to this and is not limited here.

[0074] In some embodiments, when the calibration images taken by multiple cameras at the same shooting time all include the same calibration marker, the calibration marker is reprojected according to the posture relationship between the multiple cameras to determine the predicted position information of the calibration marker in the calibration images taken by the multiple cameras respectively.

[0075] For example, the calibration mark may include a calibration mark a i , the camera may include camera c1 and camera c k At shooting time t3, camera c1 and camera c k Each calibration image taken includes a calibration mark a i , then the calibration images taken by multiple cameras at the same shooting time all include the same calibration mark. Based on this, the calibration mark a i The calibration point P on i j It can be marked by a i The pose relationship with camera c1 Convert to the camera coordinate system corresponding to camera c1, and k The position relationship between Switch to camera c k The corresponding camera coordinate system, and according to the camera c k Capture the calibration mark a i The calibration image of the camera c k The corresponding camera internal parameters are used to obtain the calibration mark a i In camera c k Predicted position information in the captured calibration image To mark the mark a i Reproject to camera c k In the captured calibration image, due to the presence of noise, the predicted position information With calibration mark a i In camera c k The actual position information in the captured calibration image may be different, so the predicted position information Available for camera c k The camera parameters are determined and optimized. Similarly, the calibration mark a can also be determined iThe actual position information of the calibration points on the image captured by the camera c1 is obtained, so as to determine and optimize the camera parameters of the camera c1. Of course, it is not limited to this and is not limited here.

[0076] In this way, when reprojecting the calibration mark included in the calibration image, the method of reprojecting the calibration mark can be flexibly determined based on the calibration images captured by different cameras at the same shooting time, the calibration images captured by the same camera and including multiple calibration marks, and the calibration images captured by different cameras at the same shooting time and including the same calibration mark, so as to determine the predicted position information of the calibration mark in the calibration image, which is conducive to improving the flexibility and convenience of determining the predicted position information. Based on the predicted position information, the camera parameters of the camera corresponding to the calibration image can be subsequently determined and optimized, which is conducive to improving the convenience and accuracy of determining and optimizing the camera parameters of multiple cameras.

[0077] In some embodiments, a reprojection error function corresponding to the camera that takes the calibration image is constructed based on the predicted position information and at least one of the posture relationship between multiple calibration markers and the posture relationship between multiple cameras; and the camera parameters of the camera that takes the calibration image are determined based on the reprojection error function.

[0078] For example, the multiple calibration marks may include calibration marks a1 and calibration marks a i The multiple cameras may include camera c1 and camera c k At shooting time t, camera c1 can capture the calibration mark a1, and camera c k The calibration mark a can be photographed i Based on this, the calibration mark a can be determined i In camera c k The predicted position information in the captured calibration image, such as the predicted position information And predict the location information With calibration mark a i In camera c k The errors between the actual position information in the captured calibration image are summed.

[0079] In some embodiments, based on the predicted position information and at least one of the posture relationship between multiple calibration marks and the posture relationship between multiple cameras, to construct a reprojection error function of the camera corresponding to the calibration image. The reprojection error function can be based on a least squares problem so that the predicted position information The reprojection error function is constructed based on the principle of minimizing the error between the actual position information and the actual position information. The reprojection error function can be expressed as a function T*:

[0080]

[0081] Where m represents the number of calibration plates in the calibration environment. n represents the number of calibration marks in each calibration plate, and the number of calibration marks in each calibration plate can be different. i j Indicates the calibration mark a i The jth calibration point of . Indicates the shooting time t, the next point P i j In camera c k The position information on the captured calibration image is equivalent to the predicted position information. Represents the position and posture relationship between camera c1 and calibration marker a1 at shooting time t. Represents camera c1 and camera c k The position relationship between them. Indicates the calibration mark a1 and the calibration mark a i The pose relationship between them. k Indicates camera c k The camera intrinsic parameters, such as the camera intrinsic parameter matrix.

[0082] In an exemplary embodiment, if camera c k The captured calibration image includes multiple calibration marks, such as calibration mark a1 and calibration mark a i When the camera c1 and camera c are not needed k The pose relationship between them determines the predicted position information Then we can delete the function T* involved This one.

[0083] In another exemplary embodiment, if camera c1 and camera c k The calibration images captured include the calibration mark a i When the calibration mark a1 and the calibration mark a are not required i The pose relationship between them determines the predicted position information Then we can delete the function T* involved This one.

[0084] For example, the reprojection error function, such as function T*, can be optimized nonlinearly. When using nonlinear optimization, it is necessary to determine the derivative of each error value with respect to the optimization variable. Based on this, the right side of function T* can be expressed as function e:

[0085]

[0086] Then we can construct an optimization function based on function e, such as function e(x):

[0087] e(x+Δx)-e(x)≈J T Δx(3)

[0088] Among them, the x involved in the function e(x) is the set of items to be optimized, and the set of items to be optimized includes K k , P i j Δx represents the increment of the item to be optimized. J is used to represent the derivative of the item to be optimized.

[0089] Here is how the derivative is calculated:

[0090] For example, based on the camera projection relationship, we can k With calibration mark a i The pose relationship between them can determine the calibration points In camera c k The projection on is as follows:

[0091]

[0092]

[0093] Among them, (X, Y, Z) represents the calibration point P i j In camera c k The corresponding pose information in the camera coordinate system. x Indicates camera c k The focal length in the x-axis direction of the image. f y Indicates camera c k The focal length in the y-axis direction of the image. x Indicates camera c k The position of the optical axis in the x-axis direction of the image. y Indicates camera c k The position of the optical axis in the y-axis direction of the image. X, Y, and Z can represent the calibration point P i j The position in space, (X, Y, Z) can also be expressed as position information Location Information It complies with the following formula:

[0094]

[0095] in, Indicates camera c k With calibration mark a i The position relationship between them.

[0096] Without considering distortion, for camera c kThe camera internal parameters are derived as follows:

[0097]

[0098] For example, for camera c k With calibration mark a i The position relationship between The derivative can be expressed as a function

[0099]

[0100] The posture relationship It can be expressed as:

[0101]

[0102] Based on this, the posture relationship for The derivative of can be expressed as a function

[0103]

[0104] Posture relationship for The derivative of can be expressed as a function

[0105]

[0106] Posture relationship for The derivative of can be expressed as a function

[0107]

[0108] Where E represents the identity matrix. Ad involved in the formula is the symbol of the adjoint matrix.

[0109] Based on formulas (1) to (12), the pose relationship can be calculated according to the chain rule: Posture relationship And the posture relationship The derivative of camera c is determined k The derivative matrix corresponding to the camera parameters of camera c is obtained by nonlinear optimization. k The corresponding derivative matrix is ​​solved to obtain the camera c k Camera parameters, such as camera c k The exact solution of the camera internal parameters and camera external parameters. Of course, it is not limited to camera c k , and so on, multiple cameras can be calibrated without limitation.

[0110] In this way, by constructing a reprojection error function of the camera corresponding to the calibration image according to the predicted position information and at least one of the posture relationships between multiple calibration markers and the posture relationships between multiple cameras, and determining and optimizing the camera parameters of the camera according to the reprojection error function, it is beneficial to improve the convenience of determining and optimizing the camera parameters of multiple cameras, so as to improve the convenience and accuracy of calibrating multiple cameras.

[0111] The camera calibration method provided in the above embodiment includes: obtaining calibration images obtained by each of the multiple cameras shooting a calibration environment, wherein the calibration environment is provided with multiple calibration marks; when the calibration image includes at least two calibration marks, determining the posture relationship between the at least two calibration marks included in the calibration image according to the calibration image; when the calibration marks included in the calibration images respectively shot by the at least two cameras are different, determining the posture relationship between the at least two cameras according to the calibration images respectively shot by the at least two cameras and the posture relationship between different calibration marks included in the calibration images respectively shot by the at least two cameras, so as to improve the convenience and accuracy of calibrating the multiple cameras.

[0112] See also Figure 4 , Figure 4 is a schematic block diagram of a camera calibration device provided in an embodiment of the present application. The camera calibration device can be configured in a server or a near-eye display device to execute the aforementioned camera calibration method.

[0113] like Figure 4 As shown, the camera calibration device includes: an acquisition module 110 , a first determination module 120 and a second determination module 130 .

[0114] The acquisition module 110 is used to acquire calibration images obtained by each of the multiple cameras shooting a calibration environment, where the calibration environment is provided with multiple calibration marks.

[0115] The first determining module 120 is configured to determine, according to the calibration image, a position and posture relationship between the at least two calibration marks included in the calibration image when the calibration image includes at least two calibration marks.

[0116] The second determination module 130 is used to determine the posture relationship between at least two cameras according to the calibration images taken by the at least two cameras and the posture relationship between different calibration identifiers included in the calibration images taken by the at least two cameras when the calibration identifiers included in the calibration images taken by the at least two cameras are different.

[0117] Exemplarily, the acquisition module 110 includes a shooting submodule.

[0118] The shooting submodule is used to control at least one movement of the multiple cameras and the calibration environment when there is a calibration image taken by at least one of the cameras that does not include a calibration mark, so that during the process of at least one movement of the multiple cameras and the calibration environment, the multiple cameras each shoot the calibration environment to obtain a calibration image including the calibration mark.

[0119] Exemplarily, the first determination module 120 includes a first pose determination submodule and a second pose determination submodule.

[0120] The first pose determination submodule is used to determine the pose relationship between the camera that photographed the calibration image and at least two calibration marks included in the calibration image according to the calibration image.

[0121] The second posture determination submodule is used to determine the posture relationship between the at least two calibration markers according to the posture relationship between the camera that shoots the calibration image and the at least two calibration markers respectively.

[0122] Exemplarily, the first determination module 120 includes a third posture determination submodule.

[0123] The third posture determination submodule is used to determine the posture relationship between all calibration markers included in multiple calibration images according to the two calibration images when the two calibration images taken at the same time each include at least two calibration markers, and there is at least one identical calibration marker among the calibration markers included in the two calibration images.

[0124] Exemplarily, the camera calibration device further includes a position determination submodule and a parameter determination submodule.

[0125] The position determination submodule is used to reproject the calibration markers included in the calibration image according to at least one of the posture relationship between the multiple calibration markers and the posture relationship between the multiple cameras, so as to determine the predicted position information of the calibration markers included in the calibration image.

[0126] The parameter determination submodule is used to determine the camera parameters of the camera used to capture the calibration image according to the predicted position information.

[0127] Exemplarily, the position determination submodule includes a first position determination submodule.

[0128] The first position determination submodule is used to, when the calibration image includes multiple calibration markers, reproject the multiple calibration markers included in the calibration image according to the posture information between the multiple calibration markers included in the calibration image, and determine the predicted position information of each of the multiple calibration markers included in the calibration image.

[0129] Exemplarily, the position determination submodule includes a second position determination submodule.

[0130] The second position determination submodule is used to reproject the calibration marker according to the posture relationship between the multiple cameras when the calibration images taken by the multiple cameras at the same shooting time all include the same calibration marker, so as to determine the predicted position information of the calibration marker in the calibration images taken by the multiple cameras respectively.

[0131] Exemplarily, the parameter determination submodule includes a function construction submodule and a function processing submodule.

[0132] A function construction submodule is used to construct a reprojection error function corresponding to the camera that shoots the calibration image according to the predicted position information and at least one of the posture relationship between the multiple calibration marks and the posture relationship between the multiple cameras.

[0133] The function processing submodule is used to determine the camera parameters of the camera used to shoot the calibration image according to the reprojection error function.

[0134] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0135] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0136] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a near-eye display device. Exemplarily, the near-eye display device may include VR glasses, AR glasses, VR helmets, AR helmets, etc., which are not limited here.

[0137] See also Figure 5 , Figure 5 It is a schematic block diagram of the structure of a near-eye display device provided in an embodiment of the present application.

[0138] like Figure 5 As shown, the near-eye display device includes multiple cameras, a memory and a processor. The memory and the processor may be connected via a system bus, and the memory may include a storage medium and an internal memory.

[0139] The storage medium may store an operating system and a computer program. When the computer program is executed, the processor may execute any camera calibration method.

[0140] The processor is used to provide computing and control capabilities to support the operation of the entire near-eye display device.

[0141] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any camera calibration method.

[0142] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the near-eye display device to which the scheme of the present application is applied. The specific near-eye display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0144] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0145] Acquire calibration images obtained by photographing a calibration environment with a plurality of cameras respectively, wherein the calibration environment is provided with a plurality of calibration marks;

[0146] When the calibration image includes at least two calibration marks, determining a position and posture relationship between the at least two calibration marks according to the calibration image;

[0147] When there are calibration images taken by at least two cameras respectively including different calibration marks, the pose relationship between the at least two cameras is determined according to the calibration images taken by the at least two cameras respectively and the pose relationship between the different calibration marks.

[0148] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the camera calibration described above can refer to the corresponding process in the aforementioned camera calibration method embodiment, and will not be repeated here.

[0149] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the camera calibration method of the present application.

[0150] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0151] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0152] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0153] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A camera calibration method, characterized in that: The camera calibration method comprises: Acquire calibration images obtained by photographing a calibration environment with a plurality of cameras respectively, wherein the calibration environment is provided with a plurality of calibration marks; When the calibration image includes at least two calibration marks, determining a posture relationship between the at least two calibration marks included in the calibration image according to the calibration image; When the calibration marks included in the calibration images respectively taken by at least two cameras are different, the posture relationship between the at least two cameras is determined according to the calibration images respectively taken by the at least two cameras and the posture relationship between the different calibration marks included in the calibration images respectively taken by the at least two cameras.

2. The camera calibration method according to claim 1, characterized in that: The step of obtaining calibration images obtained by capturing the calibration environment by multiple cameras respectively includes: When there is at least one calibration image taken by the camera that does not include a calibration mark, at least one movement of the multiple cameras and the calibration marks in the calibration environment is controlled so that during the process of at least one movement of the multiple cameras and the calibration marks in the calibration environment, the multiple cameras each take a picture of the calibration environment to obtain a calibration image including the calibration mark.

3. The camera calibration method according to claim 1, characterized in that: When the calibration image includes at least two calibration marks, determining the position and posture relationship between the at least two calibration marks included in the calibration image according to the calibration image includes: Determine, according to the calibration image, a positional relationship between a camera that photographs the calibration image and at least two calibration marks included in the calibration image; The position and posture relationship between the at least two calibration marks is determined according to the position and posture relationship between the camera that shoots the calibration image and the at least two calibration marks.

4. The camera calibration method according to claim 1, characterized in that: When the calibration image includes at least two calibration marks, determining the position and posture relationship between the at least two calibration marks included in the calibration image according to the calibration image includes: When two calibration images taken at the same time each include at least two calibration marks, and there is at least one identical calibration mark among the calibration marks included in the two calibration images, the positional relationship between all the calibration marks included in the two calibration images is determined according to the two calibration images.

5. The camera calibration method according to any one of claims 1 to 4, characterized in that: When the calibration marks included in the calibration images respectively taken by the at least two cameras are different, after determining the posture relationship between the at least two cameras according to the calibration images respectively taken by the at least two cameras and the posture relationship between the different calibration marks included in the calibration images respectively taken by the at least two cameras, the method further includes: According to at least one of the posture relationship between the plurality of calibration marks and the posture relationship between the plurality of cameras, reproject the calibration marks included in the calibration image to determine the predicted position information of the calibration marks included in the calibration image in the calibration image; According to the predicted position information, camera parameters of a camera used to capture the calibration image are determined.

6. The camera calibration method according to claim 5, characterized in that: The reprojection processing is performed on the calibration mark included in the calibration image according to at least one of the posture relationship between the plurality of calibration marks and the posture relationship between the plurality of cameras to determine the predicted position information of the calibration mark included in the calibration image in the calibration image, including: When the calibration image includes multiple calibration markers, reprojection processing is performed on the multiple calibration markers included in the calibration image according to the posture information between the multiple calibration markers included in the calibration image to determine the predicted position information of each of the multiple calibration markers included in the calibration image.

7. The camera calibration method according to claim 5, characterized in that: The reprojection processing is performed on the calibration mark included in the calibration image according to at least one of the posture relationship between the plurality of calibration marks and the posture relationship between the plurality of cameras to determine the predicted position information of the calibration mark included in the calibration image in the calibration image, including: When the calibration images taken by the multiple cameras at the same shooting time all include the same calibration marker, the calibration marker is reprojected according to the posture relationship between the multiple cameras to determine the predicted position information of the calibration marker in the calibration images taken by the multiple cameras respectively.

8. The camera calibration method according to claim 5, characterized in that: The step of determining the camera parameters of a camera used to capture the calibration image according to the predicted position information includes: Constructing a reprojection error function corresponding to the camera that photographed the calibration image according to the predicted position information and at least one of the posture relationship between the plurality of calibration marks and the posture relationship between the plurality of cameras; According to the reprojection error function, camera parameters of a camera used to capture the calibration image are determined.

9. A camera calibration device, characterized in that: The camera calibration device comprises: An acquisition module, used to acquire calibration images obtained by photographing a calibration environment by multiple cameras respectively, wherein the calibration environment is provided with multiple calibration marks; A first determining module, configured to determine, according to the calibration image, a position and posture relationship between the at least two calibration marks included in the calibration image when the calibration image includes at least two calibration marks; The second determination module is used to determine the posture relationship between at least two cameras according to the calibration images taken by at least two cameras and the posture relationship between different calibration identifiers included in the calibration images taken by at least two cameras when the calibration identifiers included in the calibration images taken by at least two cameras are different.

10. A near-eye display device, characterized in that: The near-eye display device includes a plurality of cameras, a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the camera calibration method according to any one of claims 1 to 8 when executing the computer program.

Images