Parameter calibration method for light source in wearable device and related device

Abstract

The invention provides a parameter calibration method for a light source in wearable equipment and related equipment. The method comprises the steps that a first image of the wearable device is acquired, the first image is acquired by using an external binocular camera with calibrated parameters, and the first image comprises a first light source image corresponding to a light source; determining an internal reference of the light source according to the first image; a second image of the calibration plate with the spherical mirror is acquired, the second image is acquired by using a target camera of the wearable device, and the second image comprises a second light source image generated by the light source on the spherical mirror; and according to the second image, in combination with the internal reference of the light source and the camera parameters of the target camera, determining the external reference of the light source.

Description

Technical Field

[0001] The present disclosure relates to the field of extended reality technology, and in particular, to a method for calibrating parameters of a light source in a wearable device and related devices. Background Art

[0002] Extended Reality (XR) refers to combining the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), which use hardware devices and various technical means to integrate virtual content with real scenes.

[0003] Generally, an extended reality system provides a wearable device for a user to achieve human-computer interaction, and the wearable device can be a head-mounted wearable device. In some scenarios, the wearable device can be provided with a camera for collecting eye images, and then calculations can be performed based on the eye images to achieve functions such as gaze tracking or pupil distance estimation.

[0004] In related technologies, a light source is usually arranged around the camera of the wearable device, so that the light emitted by the light source forms a light spot on the eye and is then collected into the eye image, and then calculations are performed in combination with the position of the light spot in the eye image to achieve functions such as gaze tracking or pupil distance estimation.

[0005] However, the inventors of the present disclosure have found that there are certain problems with the method for calibrating the parameters of the light source in related technologies. Summary of the Invention

[0006] The present disclosure provides a method for calibrating parameters of a light source in a wearable device and related devices to solve or partially solve the above problems.

[0007] In a first aspect of the present disclosure, there is provided a method for calibrating parameters of a light source in a wearable device, including:

[0008] Obtaining a first image of the wearable device, where the first image is collected by an external binocular camera with calibrated parameters, and the first image includes a first light source image corresponding to the light source;

[0009] Determining the internal parameters of the light source according to the first image;

[0010] Obtaining a second image of a calibration plate with a spherical mirror, where the second image is collected by a target camera of the wearable device, and the second image includes a second light source image generated by the light source on the spherical mirror;

[0011] Determining the external parameters of the light source according to the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera.

[0012] In a second aspect of the present disclosure, there is provided an apparatus for calibrating parameters of a light source in a wearable device, including:

[0013] A first acquisition module, configured to: acquire a first image of the wearable device, where the first image is acquired by an external binocular camera with calibrated parameters, and the first image includes a first light source image corresponding to the light source;

[0014] A first determination module, configured to: determine the internal parameters of the light source according to the first image;

[0015] A second acquisition module, configured to: acquire a second image of a calibration board with a spherical mirror, where the second image is acquired by a target camera of the wearable device, and the second image includes a second light source image generated by the light source on the spherical mirror;

[0016] A second determination module, configured to: determine the external parameters of the light source according to the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera.

[0017] In a third aspect of the present disclosure, there is provided a computer device, including one or more processors and a memory; and one or more programs, where the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to the first aspect.

[0018] In a fourth aspect of the present disclosure, there is provided a non - volatile computer - readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to execute the method according to the first aspect.

[0019] In a fifth aspect of the present disclosure, there is provided a computer program product, including computer program instructions, which, when run on a computer, cause the computer to execute the method according to the first aspect.

[0020] The method for calibrating parameters of a light source in a wearable device and related devices provided by the embodiments of the present disclosure can still perform parameter calibration of the light source when the camera FOV and focal length are limited, and the method is simple and easy to implement. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1A Shows a schematic diagram of an exemplary system provided by an embodiment of the present disclosure.

[0023] Figure 1B Shows a schematic diagram of an exemplary head-mounted wearable device.

[0024] Figure 2 Shows a schematic diagram of another exemplary wearable device.

[0025] Figure 3A Shows a schematic diagram of an exemplary calibration board provided by an embodiment of the present disclosure.

[0026] Figure 3B Shows a schematic diagram of another exemplary calibration board provided by an embodiment of the present disclosure.

[0027] Figure 4 Shows a schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure.

[0028] Figure 5A Shows a schematic flowchart of an exemplary method provided by an embodiment of the present disclosure.

[0029] Figure 5B Shows a schematic flowchart of an exemplary method for calculating the external parameters of a light source according to an embodiment of the present disclosure.

[0030] Figure 5C Shows a schematic flowchart of an exemplary method for calculating the external parameters of a calibration board according to an embodiment of the present disclosure.

[0031] Figure 5D Shows a schematic flowchart of another exemplary method for calculating the external parameters of a light source according to an embodiment of the present disclosure.

[0032] Figure 5E Shows a schematic flowchart of an exemplary method for calculating the external parameters of the center of a sphere in the calibration board coordinate system according to an embodiment of the present disclosure.

[0033] Figure 5F Shows a schematic flowchart of an exemplary method for calculating a second loss according to an embodiment of the present disclosure.

[0034] Figure 5G Shows a schematic flowchart of another exemplary method for calculating the external parameters of a light source according to an embodiment of the present disclosure.

[0035] Figure 5H Shows a schematic flowchart of an exemplary method for calculating a third loss according to an embodiment of the present disclosure.

[0036] Figure 6 Shows a schematic diagram of a light source internal parameter calibration scenario according to an embodiment of the present disclosure.

[0037] Figure 7 Shows a schematic diagram of an external parameter calibration scenario of a light source according to an embodiment of the present disclosure.

[0038] Figure 8 Shows a schematic diagram of the hardware structure of an exemplary computer device provided by an embodiment of the present disclosure.

[0039] Figure 9 Shows a schematic diagram of an exemplary device provided by an embodiment of the present disclosure. Detailed implementation manners

[0040] To make the objectives, technical solutions, and advantages of the present disclosure clearer and more understandable, the following further elaborates on the present disclosure in detail with reference to specific embodiments and the accompanying drawings.

[0041] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the art to which the present disclosure belongs. The "first", "second", and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or items appearing before this word cover the elements or items listed after this word and their equivalents, without excluding other elements or items. The terms such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left", and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0042] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0043] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the technical solutions of the present disclosure according to the prompt message.

[0044] As an optional but non-limiting implementation, in response to receiving an active request from a user, the way of sending a prompt message to the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0045] It can be understood that the above notification and the process of obtaining user authorization are only illustrative and do not limit the implementation of the present disclosure. Other ways that meet relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0046] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of corresponding laws, regulations and related provisions.

[0047] Figure 1A FIG. shows a schematic diagram of an exemplary extended reality system 100 provided by an embodiment of the present disclosure.

[0048] Extended Reality (XR for short) refers to combining the real and the virtual through a computer to create a virtual environment for human-computer interaction. XR (Extended Reality) technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), and uses hardware devices combined with various technical means to integrate virtual content and real scenes.

[0049] As Figure 1A shown, the system 100 may include various types of wearable devices. For example, a head-mounted wearable device (such as a VR / AR glasses or a Head-Mounted Display (HMD)) 104, an operation handle 108, etc. In some scenarios, a camera / camera 110 for taking pictures of the operator (user) 102 may also be provided. In some embodiments, when the foregoing devices do not have a processing function, the system 100 may further include an external control device 112 for providing a processing function. The control device 112 may be, for example, a computer device such as a mobile phone or a computer. In some embodiments, when any one of the foregoing devices serves as a control device or a main control device, it can implement information interaction with other devices in the system 100 through wired or wireless communication means.

[0050] In system 100, user 102 can use the head-mounted wearable device 104 and the operation handle 108 to interact with the extended reality system 100. In some scenarios, system 100 can use the images captured by the camera 110 to recognize the postures, gestures, etc. of user 102, and then complete the interaction with user 102 based on the recognized postures and gestures. In some embodiments, user 130 can also implement gesture input with bare hands. The head-mounted wearable device 104 can collect the front image in real time through a camera or the like arranged in front of the head-mounted wearable device 104, and recognize the gesture of user 130 by recognizing the image.

[0051] In some embodiments, as Figure 1A shown, system 100 can also communicate with server 114, and can obtain data from server 114, such as pictures, audio, video, etc., and can output these data through the head-mounted wearable device 104. For example, display pictures or videos on the display screen of the head-mounted wearable device 104, play the audio carried by the audio and video using the speaker of the head-mounted wearable device 104, and so on. In some embodiments, as Figure 1A shown, server 114 can retrieve the required data, such as pictures, audio, video, etc., from the database server 116 for storing data.

[0052] In some embodiments, a collection unit for collecting information can be provided on the head-mounted wearable device 104. The types of the collection unit can be various.

[0053] In some embodiments, the collection unit can further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit can be used to collect the environmental information around the wearable device 104 (e.g., in front), and the positioning and tracking unit can be used to position and track the wearable device 104. Optionally, the environment acquisition unit can include, but is not limited to, a three-color camera (e.g., RGB camera), a depth camera, a binocular camera, a laser and other photosensitive elements, and the positioning and tracking unit can include, but is not limited to, visual simultaneous localization and mapping (Visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), an ultra-wideband wireless communication technology (UWB), a laser and other modules.

[0054] In some embodiments, the head-mounted wearable device 104 may also be provided with speed sensors, acceleration sensors, angular velocity sensors (e.g., gyroscopes), etc. for collecting speed information or acceleration information of the head-mounted wearable device 104. For another example, the operating handle 108 may also be provided with speed sensors, acceleration sensors, angular velocity sensors (e.g., gyroscopes), etc. for collecting speed information or acceleration information of the operating handle 108. It should be noted that, in addition to being provided on the head-mounted wearable device 104 and the operating handle 108, the aforementioned acquisition unit may also be directly attached to the body part of the interactive user 102 without relying on hardware devices, so as to collect relevant information of the body part, such as speed or acceleration or angular velocity information, or information collected by other sensors or acquisition units (e.g., eye images (including pupil images), etc.).

[0055] In some embodiments, the head-mounted wearable device 104 may also be provided with a camera or a camera head for taking pictures of the operator (user) 102 (e.g., pictures of hands or feet) and environmental images.

[0056] In some embodiments, the system 100 can identify the posture, gestures, etc. of the user 102 through the collected information, and then can perform corresponding interactions according to the identified user postures and gestures.

[0057] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0058] As Figure 1B shown, the head-mounted wearable device 104 may include a barrel 1042, and a display screen 1044 for displaying images and an optical component 1046 for processing the optical path may be provided inside the barrel 1042. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B), and the combination of the plurality of lenses can project the light emitted by the display screen 1044 into the eye 1022, so that the eye 1022 can view the picture displayed on the display screen 1044. It can be understood that Figure 1B only the unilateral structure of the head-mounted wearable device 104 is exemplarily shown in, and in order to achieve binocular display, the head-mounted wearable device 104 may include two barrel structures arranged side by side.

[0059] In some embodiments, as Figure 1B shown, the head-mounted wearable device 104 may also be provided with a camera 1048 for collecting eye images, and the camera 1048 may be a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, etc.

[0060] Optionally, the camera 1048 may be an Eye Tracking (ET) camera, and the captured eye images can be used to implement functions such as interpupillary distance estimation and eye tracking.

[0061] As Figure 1B shown, the camera 1048 is disposed outside the lens barrel. And, in order to better capture complete eye images and not affect the user's view of the display screen 1044, the common deployment positions of the camera 1048 are generally at the outer corner of the eye or the nose wing position. Referring to Figure 1B shown, if the camera 1048 is close to the outer side of the device, then Figure 1B the shown camera deployment position is at the outer corner of the eye. If the camera 1048 is close to the inner side of the device, then Figure 1B the shown camera deployment position is at the nose wing position.

[0062] Figure 2 Fig. shows a schematic diagram of another exemplary wearable device 200.

[0063] As Figure 2 shown, different from the wearable device 104 shown in Figure 1B Fig., the camera 208 of the wearable device 200 is disposed inside the lens barrel 202. Since the camera 208 is placed inside the lens barrel 202, it will not affect the wearing of the glasses, thus improving the comfort of the wearable device 200. At the same time, as Figure 2 shown, since the camera 208 is disposed inside the lens barrel 202, the distance from the camera 208 to the eye 1022 is extended, making the installation inclination angle of the camera 208 relative to the eye 1022 smaller. Consequently, the angle β between the orientation of the camera 208 and the frontal direction of the eye 1022 is smaller than the angle α, so that the camera 208 has a better observation angle, and the captured eye images can better reflect the images of the eye's frontal angle, with better imaging quality. Moreover, since the camera 208 is placed inside the lens barrel 202, the glasses will not interfere with the imaging of the camera 208, further improving the imaging quality. The improvement of the imaging quality also improves the accuracy of algorithms such as interpupillary distance estimation or eye tracking.

[0064] It should be noted that the above examples only illustrate the case where the wearable devices 104 and 200 are provided with two cameras (corresponding to the left and right eyes respectively). It can be understood that, according to different actual needs, the number of cameras can be more.

[0065] In some embodiments, as Figure 1B and Figure 2As shown, around cameras 1048 and 208, light sources (e.g., light-emitting diodes LED) 1050 and 210 can be set. The light rays emitted by the light sources form light spots on the eyes and are then collected into the eye images. In this way, when performing gaze tracking and / or pupil distance estimation, parameters such as the position of the light spots in the eye images and the positional relationship between the cameras and the light sources can be combined to calculate the corneal center and / or pupil center, thereby realizing the gaze tracking and / or pupil distance estimation function.

[0066] It can be understood that more light sources can be set around the cameras, so as to form more light spots in the eye images, and thus achieve more accurate calculations. Therefore, the light sources corresponding to a single camera are hereinafter referred to as a group of light sources, and the number included in a group of light sources can be one or more.

[0067] In the gaze tracking and / or pupil distance estimation method based on pupil / corneal reflection, it is necessary to pre-calibrate the relevant parameters of the cameras and LEDs in advance, generally including: the internal and external parameters of the cameras, the relative positional relationship between the two cameras respectively set in two lens barrels corresponding to the left and right eyes, the relative positional relationship between the LEDs (the internal parameters of the LEDs), and the conversion relationship between the LEDs and the cameras (the external parameters of the LEDs). The internal and external parameters of the cameras can be pre-calibrated using a calibration board, and the specific calibration method is not limited. In the embodiments of the present disclosure, the discussion is carried out on the premise that the internal and external parameters of the cameras have been calibrated (i.e., the internal and external parameters of the cameras are known).

[0068] The embodiments of the present disclosure mainly discuss the parameter calibration of the LEDs. Since the LEDs are not visible in the camera field of view, the following several methods can be used for parameter calibration.

[0069] A feasible calibration method is the method of using a plane mirror for parameter calibration. Specifically, a plane mirror is placed in front of two cameras (one camera corresponding to each of the left and right eyes), so that the two cameras can simultaneously capture the virtual images of two groups of LEDs (one group of LEDs corresponding to each camera) in the mirror. After obtaining the coordinates of the virtual images in the mirror by binocular vision, the actual coordinates of the LEDs are calculated by combining multiple frames of images.

[0070] Another feasible calibration method is to use an external camera to calibrate the LED parameters. Specifically, first, an external binocular camera (an external camera, not the internal camera of the wearable device) whose internal and external parameters have been calibrated is selected. Then, by placing a double-sided calibration board between the wearable device and the external binocular camera, the external parameter matrix RT of the external binocular camera in the camera coordinate system of the wearable device is calibrated. Next, the double-sided calibration board is removed, and the external binocular camera is used to capture the LED, obtaining the internal parameters of the LED (the relationship between the LEDs) and the coordinates in the camera coordinate system of the external binocular camera. Finally, the coordinates of the LED in the camera coordinate system of the external binocular camera are converted from the camera coordinate system of the external binocular camera to the camera coordinate system of the wearable device, and the external parameters of the LED can be obtained.

[0071] However, the inventors of the present disclosure have found that when the field of view (FOV) and focal length of the camera are limited, or there is a lens in the camera optical path, the above method may not be applicable:

[0072] 1. The method of using a plane mirror for calibration has certain requirements for the FOV and focal length of the camera: In order for the virtual image of the LED to be imaged in both cameras simultaneously, the distance between the virtual image and the camera cannot be too close, otherwise the LED image may be missing in the image, or even no imaging can be obtained at all.

[0073] 2. When using the external binocular camera solution, an additional external binocular camera needs to be introduced, and the parameters of the external binocular camera need to be calibrated.

[0074] 3. In the case of a camera that does not use pinhole imaging, for example, when a convex lens is placed in the camera optical path (refer to the wearable device 200 shown in Figure 2 ), when the target is at a distance from the camera exceeding a certain range, the target cannot be imaged in both cameras, resulting in the plane mirror calibration method being unable to obtain a valid LED image. At the same time, the wearable device in the external binocular camera solution cannot calibrate the external parameters of the LED either.

[0075] In view of this, the embodiments of the present disclosure provide a method for calibrating the parameters of a light source in a wearable device, which can still calibrate the LED parameters when the camera FOV and focal length are limited.

[0076] Figure 3A The schematic diagram of an exemplary calibration board 300 provided by the embodiments of the present disclosure is shown.

[0077] As Figure 3A shown, a spherical mirror 302 is provided on the calibration board 300. Optionally, the spherical mirror 302 is a hemisphere. In other words, the center of the bottom surface of the spherical mirror 302 is also the center of the sphere, and the distance from any point on the hemispherical surface of the spherical mirror 302 to the center of the bottom surface (i.e., the center of the sphere) is equal to the radius.

[0078] In the embodiments of the present disclosure, a calibration plate with a spherical mirror 302 is used to calibrate the light source parameters, which can better solve the problems of image loss or difficult imaging existing in the plane mirror calibration method when the camera FOV and focal length are limited.

[0079] Figure 3B The schematic diagram of another exemplary calibration plate 300 provided by the embodiments of the present disclosure is shown.

[0080] Unlike Figure 3A the calibration plate 300, Figure 3B two spherical mirrors 302A and 302B are provided on the calibration plate 300. In this way, it can be further ensured that the image can be completely imaged, and better solve the problems of image loss or difficult imaging existing in the plane mirror calibration method when the camera FOV and focal length are limited.

[0081] Figure 4 The schematic diagram of the exemplary wearable device 400 provided by the embodiments of the present disclosure is shown.

[0082] As Figure 4 shown, the wearable device 400 may include lens barrels 402A and 402B corresponding to the left eye and the right eye respectively. The lens barrels 402A and 402B respectively include cameras 404A and 404B and a plurality of light sources (for example, LEDs) 406A and 406B. In some embodiments, as Figure 4 shown, the cameras 404A and 404B and the plurality of light sources (for example, LEDs) 406A and 406B on both sides may be symmetrically arranged.

[0083] The parameters of the light source to be calibrated in the embodiments of the present disclosure may be the relative position relationship between multiple light sources within a single lens barrel and the relative position relationship between each light source and the camera within the lens barrel. For example, for the multiple light sources 406A in the lens barrel 402A, the internal parameters to be calibrated may be the relative position relationship between the light sources 406A, and the external parameters to be calibrated may be the coordinates of each light source 406A in the camera coordinate system of the camera 404A. Similarly, for the multiple light sources 406B in the lens barrel 402B, the internal parameters to be calibrated may be the relative position relationship between the light sources 406B, and the external parameters to be calibrated may be the coordinates of each light source 406B in the camera coordinate system of the camera 404B.

[0084] Figure 5A The schematic flow diagram of the exemplary method 500 provided by the embodiments of the present disclosure is shown.

[0085] This method 500 can be used to calibrate the parameters of the light sources 406A and 406B of the wearable device 400, as Figure 5AAs shown, the method 500 may further include the following steps.

[0086] As an alternative embodiment, the internal parameters of the light source (i.e., the relative positional relationship between multiple light sources) may be calibrated first.

[0087] Figure 6 A schematic diagram of a light source internal parameter calibration scenario 600 according to an embodiment of the present disclosure is shown.

[0088] As Figure 6 shown, in this embodiment, an external binocular camera 602 for which the camera parameters have been calibrated may be used to calibrate the internal parameters of the light source. Optionally, the internal and external parameters of the left and right cameras of the binocular camera 602 have been calibrated and are thus known. The specific calibration method is not specifically limited.

[0089] As Figure 6 shown, the camera 602 may include two camera sensors A and B, both of which have been calibrated. Among them, the internal parameter matrix of camera A is P A , and the internal parameter matrix of camera B is P B . Optionally, P A and P B are both 3×3 matrices. Moreover, the external parameters of camera B in the camera coordinate system of camera A (i.e., the relative positional relationship between camera A and camera B) have been calibrated as R B2A , T B2A .

[0090] In step 502, a first image of the wearable device 400 may be obtained. The first image is acquired using the external binocular camera 602 with calibrated parameters. The first image includes first light source images corresponding to the light sources 406A and 406B. For example, when the light source is an LED, the first image may include an image of the LED, and this image of the LED is the light source image.

[0091] In step 504, based on the first image, the internal parameters of the light sources 406A and 406B are determined.

[0092] Optionally, the first image includes a first target image acquired by a first camera of the external binocular camera and a second target image acquired by a second camera of the external binocular camera.

[0093] The determining the internal parameters of the light source based on the first image includes:

[0094] Based on the position of the light source in the first target image, the position of the light source in the second target image, and the camera parameters of the external binocular camera, the internal parameters of the light source are determined.

[0095] It can be understood that for any point W = [x, y, z] in the camera coordinate system T , since the camera internal parameter P is known, the pixel projection coordinate can be obtained: M = [u, v, 1] T = PW.

[0096] Therefore, for any point W in the camera coordinate system of camera B B , since the internal parameter P of camera B is known B , the pixel projection coordinate in the image captured by camera B is M B = [u B , v B , 1] T = P B W B .

[0097] Since the external parameters R B2A , T B2A of camera B in the camera coordinate system of camera A are known, therefore, the coordinate W B of this point in the camera coordinate system of camera B can be converted to the camera coordinate system of camera A to obtain the corresponding coordinate W A = R B2A W B + T B2A .

[0098] Since the internal parameter P of camera A is known A , the pixel projection coordinate of this point in the image captured by camera A is: M A = [u A , v A , 1] T = P A (R B2A W B + T B2A ).

[0099] Among them, in the images captured by camera A and camera B, the pixel coordinates (u A , v A ), (u B , v B ) of the light source are known. Therefore, four equations about W B can be obtained, and there are only 3 unknowns in these four equations (that is, the 3D coordinates (x B , y B , z B , z B ) of W in the camera coordinate system of camera B). Therefore, the 3D coordinates of each light source in the camera coordinate system of camera B can be obtained according to these four equations.

[0100] Finally, the parameters of multiple light sources L = [W B0,W B1 ,...,W Bn , this parameter L can also be regarded as the internal parameter of the light source.

[0101] It can be understood that since this parameter L represents the 3D coordinates of each light source in the camera coordinate system of camera B, the relative position relationship between any two light sources in the three-dimensional space can be calculated based on these coordinates.

[0102] After obtaining the internal parameter of the light source, the external parameter of the light source (i.e., the external parameter of light source 406A in the camera coordinate system of camera 404A, and the external parameter of light source 406B in the camera coordinate system of camera 404B) can be further calibrated.

[0103] Figure 7 FIG. shows a schematic diagram of a light source external parameter calibration scenario 700 according to an embodiment of the present disclosure.

[0104] As Figure 7 shown, in this step, a calibration board 300 can be used to achieve the calibration of the external parameter of the light source.

[0105] In the calibration scenario 700, the known parameters include: the already calibrated internal parameter of the light source, the serial numbers and corresponding coordinates of the corner points (the four vertices of each square in the calibration board) of the calibration board 300 in the coordinate system of the calibration board 300, the external and internal parameters Pc of cameras 404A and 404B of the wearable device 400, and the radii R of the spherical mirrors 300A and 300B.

[0106] Taking the number of spherical mirrors provided on the calibration board 300 as two as an example (in fact, the number of spherical mirrors may not be limited to two), the unknowns to be obtained include: the external parameter T of the centers of the spherical mirrors 300A and 300B in the calibration board coordinate system bm , and the external parameter R of the light source in the camera coordinate system of the corresponding camera l ,T l (i.e., the external parameter R of light source 406A in the camera coordinate system of camera 404A lA ,T lA , and the external parameter R of light source 406B in the camera coordinate system of camera 404B lB ,T lB ).

[0107] It should be noted that since the volumes of the spherical mirrors 300A and 300B are relatively large, it is difficult to accurately locate the positions of their centers when placed on the calibration board 300. Therefore, to ensure the accuracy of the algorithm, in this embodiment, the external parameter T of the centers of the spherical mirrors 300A and 300B in the calibration board coordinate system bmSet as an unknown parameter for calculation. It can be understood that in some scenarios, when the positions of the centers of the spherical mirrors 300A and 300B on the calibration board can be determined relatively accurately, they can be used as known parameters without calculation.

[0108] In step 506, a second image of the calibration board with the spherical mirrors can be obtained. The second image is captured by the target cameras 404A and 404B of the wearable device 400, and the second image includes second light source images generated by the light sources 406A and 406B on the spherical mirrors 300A and 300B.

[0109] As Figure 7 shown, place the calibration board 300 with the spherical mirrors 300A and 300B in front of the cameras 404A and 404B whose internal parameters have been calibrated. The light rays emitted by the light sources 406A and 406B will undergo specular reflection on the spherical surfaces of the spherical mirrors 300A and 300B. The cameras 404A and 404B can capture the light source images formed by the light sources themselves on the surface of the spherical mirrors and the calibration board 300 where the spherical mirrors 300A and 300B are located. In order to obtain images of the cameras 404A and 404B and the light sources 406A and 406B in different poses, and then multiple sets of parameters for calculation can be obtained. Therefore, the wearable device 400 can be moved along the perimeter of the calibration board 300, and multiple images are captured during the movement for subsequent calculation.

[0110] In step 508, based on the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera, the external parameters of the light source can be determined.

[0111] In some embodiments, as Figure 5B shown, step 508 of determining the external parameters of the light source based on the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera, may further include the following steps.

[0112] In step 510, based on the second image and the camera parameters of the target camera, determine the calibration board external parameters of the calibration board relative to the target camera.

[0113] In some embodiments, as Figure 5C shown, step 510 of determining the calibration board external parameters of the calibration board relative to the target camera based on the second image and the camera parameters of the target camera may further include the following steps.

[0114] In step 5102, determine the initial external parameters of the calibration board.

[0115] It can be understood that an external parameter R of the calibration board 300 relative to the target camera 406A or 406B can be randomly initialized.mt , T mt 。

[0116] In step 5104, detect the positions and corresponding serial numbers of the corner points in the calibration board in the second image.

[0117] Specifically, detect the calibration board 300 from the collected second image to obtain N serial numbers (the serial numbers of N squares in the known calibration board 300) and the pixel coordinates of the four corner points corresponding to the serial number:

[0118]

[0119] In step 5106, determine the three-dimensional coordinates of the corner points in the calibration board coordinate system according to the serial numbers.

[0120] Specifically, according to the detected serial numbers, the three-dimensional coordinates of the four corner points corresponding to the serial number in the coordinate system of the calibration board 300 can be determined:

[0121]

[0122] As described above, the three-dimensional coordinates are known.

[0123] In step 5108, according to the initialized external parameters, convert the three-dimensional coordinates of the corner points in the calibration board coordinate system into the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera.

[0124] Specifically, since the initialized external parameters R mt , T mt are obtained previously, the three-dimensional coordinates of the corner points in the calibration board coordinate system can be converted into the three-dimensional coordinates in the camera coordinate system of the target camera 406A or 406B according to the external parameters R mt , T mt :

[0125]

[0126] In step 5110, determine the predicted pixel point positions of the corner points according to the internal parameters of the target camera and the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera.

[0127] Specifically, according to the internal parameters of the target camera 406A or 406B and the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera, the projected pixel points can be calculated:

[0128]

[0129] Since the projected pixel points are based on the initialized external parameters R mt , T mtCalculated, which can be called predicted pixel points.

[0130] It can be understood that for the second image corresponding to the calculated parameters, the camera parameters used in this calculation step are the camera parameters corresponding to the camera that collected the image. The same applies hereinafter and will not be elaborated.

[0131] In step 5112, according to the predicted pixel point position of the corner point and the position of the corner point detected in the second image (detected in step 5104), a first loss is determined.

[0132] Specifically, according to the coordinate difference between the predicted pixel point and the detected pixel point, a first loss loss1 is obtained:

[0133]

[0134] In step 5114, according to the first loss, the external calibration parameters of the calibration board relative to the target camera are determined.

[0135] Specifically, another second image can be selected (optionally, this second image and the second image already used before can be second images collected in different poses), and using the R mt , T mt updated in the previous iteration, repeat the foregoing steps, calculate the first loss again, and repeat the iteration and update R mt , T mt until the first loss converges.

[0136] At this time, the latest R mt , T mt are the external calibration parameters of the calibration board relative to the target camera.

[0137] After obtaining the external calibration parameters R mt , T mt of the calibration board relative to the target camera, the external calibration parameters of the light source can be further determined.

[0138] Therefore, in step 512, according to the second image, combining the external calibration parameters of the calibration board, the internal calibration parameters of the light source, and the camera parameters of the target camera, the external calibration parameters of the light source are determined.

[0139] In this step, the external calibration parameter T bm of the center of the spherical mirror in the calibration board coordinate system and the external calibration parameters R l , T l of the light source in the camera coordinate system of the target camera can be initialized first.

[0140] In some embodiments, such as Figure 5DAs shown, step 512 of determining the external parameters of the light source according to the second image, in combination with the external parameters of the calibration board, the internal parameters of the light source, and the camera parameters of the target camera, may further include the following steps.

[0141] In step 514, according to the second image, in combination with the external parameters of the calibration board, the internal parameters of the light source, and the camera parameters of the target camera, determine the external parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system.

[0142] In this step, the coordinates of the light source can be fixed first (keep its coordinates unchanged, that is, do not update this coordinate during the subsequent iterative update of the external parameters T of the center of the sphere bm ), traverse the light source images of each light source, and iterate T bm until the loss converges.

[0143] In some embodiments, as Figure 5E shown, step 514 of determining the external parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system according to the second image, in combination with the external parameters of the calibration board, the internal parameters of the light source, and the camera parameters of the target camera, may further include the following steps.

[0144] In step 5142, determine the first initialization coordinate T of the center of the spherical mirror of the calibration board in the calibration board coordinate system bm . The calculation can be performed taking the spherical mirror 302A as an example, and the subsequent similar cases will not be elaborated.

[0145] In step 5144, according to the first initialization coordinate T bm and the external parameters R of the calibration board mt , T mt , determine the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera.

[0146] Specifically, calculate the center coordinate W of the spherical mirror in the camera coordinate of the target camera at time t bc = R mt T bm + T mt . That is, calculate according to the second image collected by the target camera 406A or 406B at time t.

[0147] In step 5146, detect the position of the second light source image corresponding to the light source in the second image.

[0148] Specifically, detect the light source image coordinates in the second image and mark the corresponding light source numbers, which are (u i , v i ), where i is the LED number.

[0149] At step 5148, based on the position of the second light source image in the second image, determine the starting point and the light direction of the light source in the camera coordinate system of the target camera.

[0150] Specifically, calculate the light ray corresponding to the second light source image in the camera coordinate system of the target camera,

[0151] represented as the starting point si and the direction

[0152] It can be understood that since both the extrinsic and intrinsic parameters of the target camera are known, after obtaining the position of the second light source image in the second image, the starting point and the light direction of this light ray can be calculated by combining the camera parameters.

[0153] At step 5150, based on the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera, determine the distance between the light ray of the light source and the center of the sphere, that is:

[0154]

[0155] At step 5152, determine the second loss based on the distance.

[0156] In some embodiments, as Figure 5F shown, the step 5152 of determining the second loss based on the distance may further include the following steps.

[0157] At step 51522, in response to determining that the distance is greater than or equal to the radius of the spherical mirror, determine the second loss based on the distance and the radius.

[0158] If k≥R, then the light ray has no intersection with the spherical mirror, and determine the second loss based on the distance and the radius The loss calculation of the current light source image is completed.

[0159] At step 51524, in response to determining that the distance is less than the radius of the spherical mirror, determine the intersection point of the light ray of the light source and the spherical mirror.

[0160] If k<R, then the light ray has an intersection with the spherical mirror, and calculate the collision point G of the light ray and the sphere i , and the calculation method is as follows:

[0161] Assume that x is a point on the sphere with radius r, and the ray passes through x, then there is an equation:

[0162]

[0163] Obtained:

[0164] d is the value to be found

[0165] According to the solution of the quadratic equation of one variable In this equation:

[0166] Then, it is possible to determine whether there is an intersection point, that is, the corresponding ray length, by judging Δ.

[0167] Since the ray direction is known, the intersection point can be obtained. Since the required point is on the side closer to the ray starting point, the sign can be determined.

[0168] In step 51526, according to the three-dimensional coordinates of the intersection point and the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera, determine the normal line, that is:

[0169]

[0170] In step 51528, according to the internal parameters of the light source (the parameter L obtained previously) and the initial internal parameters of the light source, determine the second initial coordinates of the light source in the camera coordinate system of the target camera, that is:

[0171]

[0172] In step 51530, according to the second initial coordinates and the intersection point, determine the reflected ray.

[0173] It can be known that the reflected ray is the connection line between the light source and the intersection point

[0174] In step 51532, according to the normal line, the ray direction and the reflected ray, determine the second loss.

[0175] The law of reflection of light is known: the normal line bisects the incident ray and the reflected ray formed angle:

[0176] Therefore, specular reflection should satisfy that the normal line bisects the angle formed by the incident ray and the reflected ray, and calculate the second loss loss2:

[0177]

[0178] In step 5154, according to the second loss, determine the external parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system.

[0179] Specifically, add the current second loss loss2 to the total loss loss 总1 :

[0180] loss 总1 = loss1 + loss2

[0181] To minimize the total loss loss 总1 For this purpose, repeat the above steps and iterate until convergence.

[0182] At this time, the external parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system can be obtained.

[0183] In step 516, according to the second image, combining the external parameters of the calibration plate, the internal parameters of the light source, the camera parameters of the target camera, and the external parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system, determine the external parameters of the light source.

[0184] In this step, the external parameters T of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system obtained in step 514 can be bm Fixed (keep its coordinates unchanged, that is, do not update this coordinate in subsequent iterations), traverse the light source images of each light source, and iterate R l , T l Until the loss converges.

[0185] In some embodiments, as Figure 5G shown, according to the second image, combining the external parameters of the calibration plate, the internal parameters of the light source, the camera parameters of the target camera, and the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera, the step 516 of determining the external parameters of the light source can further include the following steps.

[0186] In step 5162, according to the external parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system and the external parameters of the calibration plate, determine the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera.

[0187] Specifically, calculate the center coordinate W of the spherical mirror in the camera coordinate of the target camera at time t bc = R mt T bm + T mt . That is, calculate according to the second image collected by the target camera 406A or 406B at time t.

[0188] In step 5164, detect the position of the second light source image corresponding to the light source in the second image.

[0189] Specifically, detect the light source image coordinates in the second image and label the corresponding light source numbers, which are (u i , v i ), where i is the LED number.

[0190] In step 5166, according to the position of the second light source image in the second image, determine the starting point and the light direction of the light source in the camera coordinate system of the target camera.

[0191] Specifically, calculate the light ray corresponding to the second light source image in the camera coordinate system of the target camera,

[0192] expressed as the starting point si and the direction

[0193] It can be understood that since both the external parameters and the internal parameters of the target camera are known, after obtaining the position of the second light source image in the second image, the starting point and the light direction of this light ray can be calculated by combining the camera parameters, that is:

[0194]

[0195] In step 5168, according to the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera, determine the distance between the light ray of the light source and the center of the sphere.

[0196] In step 5170, determine the third loss according to the distance.

[0197] In some embodiments, as Figure 5H shown, the step 5170 of determining the third loss according to the distance may further include the following steps.

[0198] In step 51702, in response to determining that the distance is greater than or equal to the radius of the spherical mirror, determine the third loss according to the distance and the radius.

[0199] If k≥R, then the light ray has no intersection with the spherical mirror, and determine the third loss according to the distance and the radius The loss calculation of the current light source image is completed.

[0200] In step 51704, in response to determining that the distance is less than the radius of the spherical mirror, determine the intersection point of the light ray of the light source and the spherical mirror.

[0201] If k<R, then the light ray has an intersection with the spherical mirror, calculate the collision point G of the light ray and the sphere i , and the calculation method is as follows:

[0202] Assume that point x is on the spherical surface with radius r, and the ray passes through x, then there is an equation:

[0203]

[0204] We get:

[0205] d is the value we want

[0206] According to the solution of the quadratic equation of one variable In this equation:

[0207] Then we can judge whether there is an intersection point, that is, the corresponding ray length, by judging Δ.

[0208] Since the ray direction is known, the intersection point can be obtained. Since the point we want is on the side closer to the ray starting point, the sign can be determined.

[0209] In step 51706, according to the three-dimensional coordinates of the intersection point and the center of the spherical mirror in the camera coordinate system of the target camera, determine the normal line, that is:

[0210]

[0211] In step 51708, according to the internal parameters of the light source and the initialized internal parameters of the light source, determine the second initialized coordinates of the light source in the camera coordinate system of the target camera, that is:

[0212]

[0213] In step 51710, according to the second initialized coordinates and the intersection point, determine the reflected ray.

[0214] It can be known that the reflected ray is the connection line between the light source and the intersection point

[0215] In step 51712, according to the normal line, the ray direction and the reflected ray, determine the third loss.

[0216] The known law of reflection of light: the normal line bisects the incident ray and the reflected ray to form an angle:

[0217] Therefore, specular reflection should satisfy that the normal line bisects the angle formed by the incident ray and the reflected ray, and calculate the third loss loss3:

[0218]

[0219] In step 5172, determine the extrinsic parameters of the light source according to the third loss.

[0220] Specifically, add the current third loss loss3 to the total loss loss 总2 :

[0221] loss 总2 = loss1 + loss3

[0222] For the purpose of minimizing the total loss loss 总2 Repeat the above steps iteratively until convergence.

[0223] At this time, a set of extrinsic parameters of the light source can be obtained.

[0224] To ensure accuracy, the extrinsic parameters of the light source can be fixed, and then step 514 is executed again to further update the extrinsic parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system. Then, taking the extrinsic parameters of the center of the spherical mirror of the updated calibration plate in the calibration plate coordinate system as parameters, execute step 516 to update the extrinsic parameters of the light source.

[0225] For example, executing step 514 once may include: determining the first calibration plate extrinsic parameters of the calibration plate relative to the target camera according to the first initialization coordinates of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system, in combination with the second image and the camera parameters of the target camera;

[0226] Executing step 516 once may include: fixing the first calibration plate extrinsic parameters, and determining the first extrinsic parameters of the light source according to the second image, in combination with the first calibration plate extrinsic parameters, the intrinsic parameters of the light source, and the camera parameters of the target camera;

[0227] After executing step 514 and 516 once, determining the calibration plate extrinsic parameters of the calibration plate relative to the target camera according to the second image and the camera parameters of the target camera further includes: fixing the first extrinsic parameters of the light source, and determining the second calibration plate extrinsic parameters of the calibration plate relative to the target camera according to the second image and the camera parameters of the target camera;

[0228] Determining the extrinsic parameters of the light source according to the second image, in combination with the calibration plate extrinsic parameters, the intrinsic parameters of the light source, and the camera parameters of the target camera further includes: fixing the second calibration plate extrinsic parameters, and determining the second extrinsic parameters of the light source according to the second image, in combination with the second calibration plate extrinsic parameters, the intrinsic parameters of the light source, and the camera parameters of the target camera.

[0229] Repeat steps 514 and 516 in the above manner until convergence or the number of repetitions is exceeded. It can be understood that the execution order of steps 514 and 516 can be swapped. For example, step 516 can be executed first and then step 514. Therefore, the execution order of the foregoing embodiments is not limited and can be adjusted.

[0230] Finally, the coordinates L of the light source in the device camera coordinate system can be obtained. i = R l L i + T l .

[0231] As can be seen from the above embodiments, for the method for calibrating the parameters of the light source in the wearable device and related devices provided by the embodiments of the present disclosure, when the camera FOV and focal length are limited, the parameters of the light source can still be calibrated, and the method is simple and easy to implement.

[0232] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present disclosure, and these multiple devices will interact with each other to complete the described method.

[0233] It should be noted that some embodiments of the present disclosure have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order than in the above embodiments and still achieve the desired results. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain implementations, multitasking and parallel processing are also possible or may be advantageous.

[0234] The embodiments of the present disclosure also provide a computer device for implementing the above method 500. Figure 8 The hardware structure diagram of the exemplary computer device 800 provided by the embodiments of the present disclosure is shown. The computer device 800 can be used to implement Figure 1A , Figure 1B the head-mounted wearable device 104, Figure 2 the wearable device 200, and can also be used to implement Figure 1A the external device 112, and can also be used to implement Figure 1A the server 114. In some scenarios, this computer device 800 can also be used to implement Figure 1A the database server 116.

[0235] As Figure 8As shown, the computer device 800 may include: a processor 802, a memory 804, a network module 806, a peripheral interface 808, and a bus 810. Among them, the processor 802, the memory 804, the network module 806, and the peripheral interface 808 are communicatively connected to each other inside the computer device 800 via the bus 810.

[0236] The processor 802 may be a central processing unit (CPU), a graphics processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. The processor 802 may be used to execute functions related to the technologies described in this disclosure. In some embodiments, the processor 802 may further include multiple processors integrated as a single logic component. For example, as Figure 8 shown, the processor 802 may include multiple processors 802a, 802b, and 802c.

[0237] The memory 804 may be configured to store data (e.g., instructions, computer code, etc.). As Figure 8 shown, the data stored in the memory 804 may include program instructions (e.g., program instructions for implementing the method 500 of the embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 802 may also access the program instructions and data stored in the memory 804 and execute the program instructions to operate on the data to be processed. The memory 804 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 804 may include a random access memory (RAM), a read-only memory (ROM), an optical disc, a magnetic disk, a hard disk, a solid state drive (SSD), a flash memory, a memory stick, etc.

[0238] The network interface 806 may be configured to provide communication with other external devices to the computer device 800 via a network. The network may be any wired or wireless network capable of transmitting and receiving data. For example, the network may be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC), etc.), a cellular network, the Internet, or a combination of the above. It can be understood that the type of the network is not limited to the above specific examples.

[0239] The peripheral interface 808 can be configured to connect the computer device 800 to one or more peripheral devices to enable information input and output. For example, the peripheral devices can include input devices such as keyboards, mice, touchpads, touchscreens, microphones, various sensors, etc., and output devices such as displays, speakers, vibrators, indicator lights, etc.

[0240] The bus 810 can be configured to transfer information between various components of the computer device 800 (such as the processor 802, the memory 804, the network interface 806, and the peripheral interface 808), such as internal buses (e.g., the processor - memory bus), external buses (USB ports, PCI - E buses), etc.

[0241] It should be noted that although the architecture of the computer device 800 shown above only shows the processor 802, the memory 804, the network interface 806, the peripheral interface 808, and the bus 810, in the specific implementation process, the architecture of the computer device 800 can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the architecture of the computer device 800 above can also only include the components necessary to implement the solution of the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0242] The embodiments of the present disclosure also provide a parameter calibration device for a light source in a wearable device. Figure 9 The schematic diagram of the exemplary device 900 provided by the embodiments of the present disclosure is shown. As Figure 9 shown, the device 900 can be used to implement the method 500 and can further include the following modules.

[0243] The first acquisition module 902 is configured to: acquire a first image of the wearable device, where the first image is acquired by an external binocular camera with calibrated parameters, and the first image includes a first light source image corresponding to the light source;

[0244] The first determination module 904 is configured to: determine the internal parameters of the light source according to the first image;

[0245] The second acquisition module 906 is configured to: acquire a second image of a calibration plate with a spherical mirror, where the second image is acquired by the target camera of the wearable device, and the second image includes a second light source image generated by the light source on the spherical mirror;

[0246] The second determination module 908 is configured to: determine the external parameters of the light source according to the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera.

[0247] In some embodiments, the first image includes a first target image captured by a first camera of the external binocular camera and a second target image captured by a second camera of the external binocular camera;

[0248] A first determination module 904, configured to:

[0249] Determine the internal parameters of the light source according to the position of the light source in the first target image, the position of the light source in the second target image, and the camera parameters of the external binocular camera.

[0250] In some embodiments, a second determination module 908, configured to:

[0251] Determine the calibration board external parameters of the calibration board relative to the target camera according to the second image and the camera parameters of the target camera;

[0252] Determine the external parameters of the light source according to the second image, in combination with the calibration board external parameters, the internal parameters of the light source, and the camera parameters of the target camera.

[0253] In some embodiments, a second determination module 908, configured to:

[0254] Determine the initial external parameters of the calibration board;

[0255] Detect the positions and corresponding serial numbers of the corner points in the calibration board in the second image;

[0256] Determine the three-dimensional coordinates of the corner points in the calibration board coordinate system according to the serial numbers;

[0257] Convert the three-dimensional coordinates of the corner points in the calibration board coordinate system to the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera according to the initial external parameters;

[0258] Determine the predicted pixel point positions of the corner points according to the internal parameters of the target camera and the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera;

[0259] Determine a first loss according to the predicted pixel point positions of the corner points and the positions of the corner points detected in the second image;

[0260] Determine the calibration board external parameters of the calibration board relative to the target camera according to the first loss.

[0261] In some embodiments, a second determination module 908, configured to:

[0262] Based on the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera, determine the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system;

[0263] Based on the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, the camera parameters of the target camera, and the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system, determine the extrinsic parameters of the light source.

[0264] In some embodiments, the second determination module 908 is configured to:

[0265] Determine the first initialization coordinates of the center of the spherical mirror of the calibration board in the calibration board coordinate system;

[0266] Based on the first initialization coordinates and the extrinsic parameters of the calibration board, determine the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera;

[0267] Detect the position of the second light source image corresponding to the light source in the second image;

[0268] Based on the position of the second light source image in the second image, determine the starting point and the light direction of the light source in the camera coordinate system of the target camera;

[0269] Based on the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the distance between the light of the light source and the center of the sphere;

[0270] Based on the distance, determine the second loss;

[0271] Based on the second loss, determine the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system.

[0272] In some embodiments, the second determination module 908 is configured to:

[0273] In response to determining that the distance is greater than or equal to the radius of the spherical mirror, determine the second loss based on the distance and the radius; or

[0274] In response to determining that the distance is less than the radius of the spherical mirror, determine the intersection point of the light of the light source and the spherical mirror;

[0275] Based on the intersection point and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the normal;

[0276] Determine the second initialization coordinates of the light source in the camera coordinate system of the target camera according to the internal parameters of the light source and the initialized internal parameters of the light source;

[0277] Determine the reflected light according to the second initialization coordinates and the intersection point;

[0278] Determine the second loss according to the normal, the light direction, and the reflected light;

[0279] In some embodiments, the second determination module 908 is configured to:

[0280] Determine the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera according to the external parameters of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system and the external parameters of the calibration plate;

[0281] Detect the position of the second light source image corresponding to the light source in the second image;

[0282] Determine the starting point and the light direction of the light source in the camera coordinate system of the target camera according to the position of the second light source image in the second image;

[0283] Determine the distance between the light of the light source and the center of the sphere according to the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration plate in the camera coordinate system of the target camera;

[0284] Determine the third loss according to the distance;

[0285] Determine the external parameters of the light source according to the third loss;

[0286] In some embodiments, in some embodiments, the second determination module 908 is configured to:

[0287] Determine the first calibration plate external parameters of the calibration plate relative to the target camera according to the first initialization coordinates of the center of the spherical mirror of the calibration plate in the calibration plate coordinate system, in combination with the second image and the camera parameters of the target camera;

[0288] Fix the first calibration plate external parameters, and determine the first external parameters of the light source according to the second image, in combination with the first calibration plate external parameters, the internal parameters of the light source, and the camera parameters of the target camera;

[0289] Fix the first external parameters of the light source, and determine the second calibration plate external parameters of the calibration plate relative to the target camera according to the second image and the camera parameters of the target camera;

[0290] Fix the external parameters of the second calibration board. According to the second image, in combination with the external parameters of the second calibration board, the internal parameters of the light source, and the camera parameters of the target camera, determine the second external parameters of the light source.

[0291] For the convenience of description, when describing the above device, it is divided into various modules according to functions for separate description. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0292] The device in the above embodiment is used to implement the corresponding method 500 in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0293] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method 500 described in any of the foregoing embodiments.

[0294] The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0295] The computer instructions stored in the storage medium in the above embodiment are used to cause the computer to execute the method 500 described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0296] Based on the same inventive concept, corresponding to the method 500 in any of the above embodiments, the present disclosure also provides a computer program product including computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of the computer to cause the computer and / or the processor to execute the method 500. Corresponding to the execution subject of each step in each embodiment of the method 500, the processor executing the corresponding step can belong to the corresponding execution subject.

[0297] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the method 500 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0298] Those of ordinary skill in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, and they are not provided in detail for the sake of brevity.

[0299] In addition, for the sake of simplicity of description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the accompanying drawings. In addition, the devices may be shown in block diagram form in order not to make the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be completely within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0300] Although the present disclosure has been described in connection with specific embodiments of the present disclosure, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0301] The embodiments of the present disclosure are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for parameter calibration of a light source in a wearable device, comprising: Obtain a first image of the wearable device, where the first image is captured by an external binocular camera with calibrated parameters, and the first image includes a first light source image corresponding to the light source; Determine the internal parameters of the light source based on the first image; Obtain a second image of a calibration board with a spherical mirror, where the second image is captured by the target camera of the wearable device, and the second image includes a second light source image generated by the light source on the spherical mirror; Determine the external parameters of the light source based on the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera; 2. The method according to claim 1, wherein, The first image includes a first target image captured by a first camera of the external binocular camera and a second target image captured by a second camera of the external binocular camera; Determining the internal parameters of the light source based on the first image includes: Determine the internal parameters of the light source based on the position of the light source in the first target image, the position of the light source in the second target image, and the camera parameters of the external binocular camera; 3. The method according to claim 1, wherein, Determining the external parameters of the light source based on the second image, in combination with the internal parameters of the light source and the camera parameters of the target camera, includes: Determine the calibration board external parameters of the calibration board relative to the target camera based on the second image and the camera parameters of the target camera; Determine the external parameters of the light source based on the second image, in combination with the calibration board external parameters, the internal parameters of the light source, and the camera parameters of the target camera; 4. The method according to claim 3, wherein, Determining the calibration board external parameters of the calibration board relative to the target camera based on the second image and the camera parameters of the target camera includes: Determine the initial external parameters of the calibration board; Detect the positions and corresponding serial numbers of the corner points in the calibration board in the second image; Determine the three-dimensional coordinates of the corner points in the calibration board coordinate system according to the serial numbers; Convert the three-dimensional coordinates of the corner points in the calibration board coordinate system to the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera according to the initial external parameters; Determine the predicted pixel point positions of the corner points based on the internal parameters of the target camera and the three-dimensional coordinates of the corner points in the camera coordinate system of the target camera; Determine a first loss based on the predicted pixel point positions of the corner points and the positions of the corner points detected in the second image; Determine the calibration board external parameters of the calibration board relative to the target camera based on the first loss; 5. The method according to claim 3, wherein, Determining the external parameters of the light source based on the second image, in combination with the calibration board external parameters, the internal parameters of the light source, and the camera parameters of the target camera, includes: Determine the external parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system based on the second image, in combination with the calibration board external parameters, the internal parameters of the light source, and the camera parameters of the target camera; Determine the external parameters of the light source based on the second image, in combination with the calibration board external parameters, the internal parameters of the light source, the camera parameters of the target camera, and the external parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system; 6. The method according to claim 5, wherein, Based on the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera, determine the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system, including: Determine the first initialization coordinates of the center of the spherical mirror of the calibration board in the calibration board coordinate system; Based on the first initialization coordinates and the extrinsic parameters of the calibration board, determine the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera; Detect the position of the second light source image corresponding to the light source in the second image; Based on the position of the second light source image in the second image, determine the starting point and the light direction of the light source in the camera coordinate system of the target camera; Based on the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the distance between the light of the light source and the center of the sphere; Based on the distance, determine the second loss; Based on the second loss, determine the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system.

7. The method according to claim 6, wherein, Based on the distance, determine the second loss, including: In response to determining that the distance is greater than or equal to the radius of the spherical mirror, determine the second loss based on the distance and the radius; or In response to determining that the distance is less than the radius of the spherical mirror, determine the intersection point of the light of the light source and the spherical mirror; Based on the intersection point and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the normal line; Based on the intrinsic parameters of the light source and the initialized intrinsic parameters of the light source, determine the second initialization coordinates of the light source in the camera coordinate system of the target camera; Based on the second initialization coordinates and the intersection point, determine the reflected light; Based on the normal line, the light direction, and the reflected light, determine the second loss.

8. The method according to claim 5, wherein, Based on the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, the camera parameters of the target camera, and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the extrinsic parameters of the light source, including: Based on the extrinsic parameters of the center of the spherical mirror of the calibration board in the calibration board coordinate system and the extrinsic parameters of the calibration board, determine the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera; Detect the position of the second light source image corresponding to the light source in the second image; Based on the position of the second light source image in the second image, determine the starting point and the light direction of the light source in the camera coordinate system of the target camera; Based on the starting point and the light direction of the light source in the camera coordinate system of the target camera and the three-dimensional coordinates of the center of the spherical mirror of the calibration board in the camera coordinate system of the target camera, determine the distance between the light of the light source and the center of the sphere; Based on the distance, determine the third loss; Based on the third loss, determine the extrinsic parameters of the light source.

9. The method according to claim 3, wherein, Determine the extrinsic parameters of the calibration board relative to the target camera according to the second image and the camera parameters of the target camera, including: determine the first extrinsic parameters of the calibration board relative to the target camera according to the first initialization coordinates of the center of the spherical mirror of the calibration board in the calibration board coordinate system, in combination with the second image and the camera parameters of the target camera; Determine the extrinsic parameters of the light source according to the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera, including: fix the first extrinsic parameters of the calibration board, and determine the first extrinsic parameters of the light source according to the second image, in combination with the first extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera; Determine the extrinsic parameters of the calibration board relative to the target camera according to the second image and the camera parameters of the target camera, further including: fix the first extrinsic parameters of the light source, and determine the second extrinsic parameters of the calibration board relative to the target camera according to the second image and the camera parameters of the target camera; Determine the extrinsic parameters of the light source according to the second image, in combination with the extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera, further including: fix the second extrinsic parameters of the calibration board, and determine the second extrinsic parameters of the light source according to the second image, in combination with the second extrinsic parameters of the calibration board, the intrinsic parameters of the light source, and the camera parameters of the target camera.

10. A parameter calibration device for a light source in a wearable device, comprising: The first acquisition module is configured to: acquire a first image of the wearable device, where the first image is acquired by an external binocular camera with calibrated parameters, and the first image includes a first light source image corresponding to the light source; The first determination module is configured to: determine the intrinsic parameters of the light source according to the first image; The second acquisition module is configured to: acquire a second image of a calibration board with a spherical mirror, where the second image is acquired by the target camera of the wearable device, and the second image includes a second light source image generated by the light source on the spherical mirror; The second determination module is configured to: determine the extrinsic parameters of the light source according to the second image, in combination with the intrinsic parameters of the light source and the camera parameters of the target camera.

11. A computer device, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for performing the method according to any one of claims 1-9.

12. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1-9.

13. A computer program product, comprising computer program instructions, which, when run on a computer, cause the computer to perform the method according to any one of claims 1-9.

Images