Camera parameter calibration method and related device

Abstract

Provided in the present disclosure are a camera parameter calibration method and a related device. The method comprises: receiving a plurality of images obtained by means of at least two cameras photographing a calibration reference object, wherein the at least two cameras are provided in a lens housing of a wearable device; determining a projection relationship between pixel points of the plurality of images and the calibration reference object; and determining a first parameter set and a second parameter set of the at least two cameras on the basis of the projection relationship, wherein the first parameter set comprises a plurality of first parameters, the first parameters are used for representing a target pixel point in the images captured by a camera to be subjected to calibration and a projection direction corresponding to the target pixel point, the second parameter set comprises at least one group of second parameters, and the at least one group of second parameters is used for indicating a pose relationship between the at least two cameras.

Description

Camera parameter calibration methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202311484874.4, filed on November 8, 2023, entitled "Camera Parameter Calibration Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of extended reality technology, and in particular to a method and related equipment for calibrating camera parameters. Background Technology

[0003] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0004] Generally, extended reality systems provide users with wearable devices to enable human-computer interaction; these wearable devices can be head-mounted devices. In some scenarios, wearable devices can perform calculations by capturing images of the human eye to achieve gaze tracking or interpupillary distance estimation.

[0005] However, in related technologies, the camera that acquires images is usually placed outside the lens barrel, which limits the accuracy of pupil distance estimation and gaze tracking algorithms.

[0006] Summary of the Invention

[0007] This disclosure proposes a method and related equipment for calibrating camera parameters to solve or partially solve the above-mentioned problems.

[0008] In a first aspect, this disclosure provides a method for calibrating camera parameters, comprising:

[0009] The wearable device receives multiple images of a calibration reference object captured by at least two cameras, the at least two cameras being disposed within the lens barrel of the wearable device; the wearable device further includes a binocular display module, the lens barrel being disposed on the light-emitting side of the binocular display module, an optical component being disposed within the lens barrel, and the at least two cameras being located between the binocular display module and the optical component;

[0010] Determine the projection relationship between the pixels of the multiple images and the calibration reference object;

[0011] Based on the projection relationship, determine the first parameter set and the second parameter set of the at least two cameras;

[0012] The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

[0013] A second aspect of this disclosure provides a wearable device, comprising:

[0014] Binocular display module;

[0015] Two lens barrels are disposed on the light-emitting side of the binocular display module. At least one of the two lens barrels includes at least two cameras for acquiring human eye images and an optical component disposed inside the lens barrel. The at least two cameras are located between the binocular display module and the optical component.

[0016] A third aspect of this disclosure provides a camera parameter calibration apparatus, comprising:

[0017] The receiving module is configured to receive multiple images of a calibration reference object captured by at least two cameras; the wearable device further includes a binocular display module, the lens barrel is disposed on the light-emitting side of the binocular display module, optical components are disposed inside the lens barrel, and the at least two cameras are located between the binocular display module and the optical components.

[0018] The first determining module is configured to: determine the projection relationship between the pixels of the plurality of images and the calibration reference object;

[0019] The second determining module is configured to: determine a first parameter set and a second parameter set of the at least two cameras based on the projection relationship;

[0020] The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

[0021] A fourth aspect of this disclosure provides a wearable device, comprising:

[0022] A lens barrel, wherein a display module, an optical component, and at least two cameras are disposed within the lens barrel, the at least two cameras and the optical component are located on the light-emitting side of the display module, and the at least two cameras are located between the optical component and the display module;

[0023] The processing module is electrically coupled to the at least two cameras and is configured to: acquire a first parameter set and a second parameter set obtained by the method of the first aspect, as well as an image captured by the camera; and use the first parameter set, the second parameter set, and the image to solve for the spatial position of a target region in the image.

[0024] A fifth aspect of this disclosure provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs including instructions for performing the method according to the first aspect.

[0025] A sixth aspect of this disclosure provides a non-volatile computer-readable storage medium comprising a computer program that, when executed by one or more processors, causes the processors to perform the method described in the first aspect.

[0026] A seventh aspect of this disclosure provides a computer program product including computer program instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this disclosure or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1A shows a schematic diagram of an exemplary system provided in an embodiment of this disclosure.

[0029] Figure 1B shows a schematic diagram of an exemplary head-mounted wearable device.

[0030] Figures 1C and 1D show schematic diagrams of exemplary human eye images.

[0031] Figure 2A shows a schematic diagram of an exemplary wearable device provided in an embodiment of this disclosure.

[0032] Figure 2B shows a schematic diagram of another exemplary wearable device provided in the embodiments of this disclosure.

[0033] Figure 2C shows a schematic diagram of yet another exemplary wearable device provided in an embodiment of this disclosure.

[0034] Figure 2D shows a schematic diagram of yet another exemplary wearable device provided in an embodiment of this disclosure.

[0035] Figure 3A shows a flowchart illustrating an exemplary method provided in an embodiment of this disclosure.

[0036] Figure 3B shows a schematic flowchart of an exemplary method for determining projection relationships according to an embodiment of the present disclosure.

[0037] Figure 4A shows a schematic diagram of an exemplary image acquisition scenario according to an embodiment of the present disclosure.

[0038] Figure 4B illustrates a schematic diagram of another exemplary imaging scenario according to an embodiment of the present disclosure.

[0039] Figure 4C shows a schematic diagram of the three selected target images.

[0040] Figure 4D shows a schematic diagram of the projection relationship between pixels and a calibration reference object according to an embodiment of the present disclosure.

[0041] Figure 4E shows a schematic diagram of an exemplary first parameter according to an embodiment of the present disclosure.

[0042] Figure 4F illustrates a calibration schematic of a second parameter of an exemplary in-tube camera on the opposite side according to an embodiment of the present disclosure.

[0043] Figure 4G shows a schematic diagram of the image obtained after binarization of the image acquired in an embodiment of this disclosure.

[0044] Figure 5 shows a schematic diagram of an exemplary wearable device provided in an embodiment of this disclosure.

[0045] Figure 6 shows a schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of this disclosure.

[0046] Figure 7 shows a schematic diagram of an exemplary device provided in an embodiment of this disclosure. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0048] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0050] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0051] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0052] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0053] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0054] Figure 1A shows a schematic diagram of an exemplary extended reality system 100 provided in an embodiment of the present disclosure.

[0055] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0056] As shown in Figure 1A, the system 100 may include various types of wearable devices, such as head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMDs)) 104, control handles 108, etc. In some scenarios, a camera / camera 110 may also be provided for taking photos of the operator (user) 102. In some embodiments, when the aforementioned devices do not have processing functions, the system 100 may also include an external control device 112 for providing processing functions. The control device 112 may be, for example, a mobile phone, computer, or other computer device. In some embodiments, when any of the aforementioned devices acts as a control device or a main control device, it can interact with other devices in the system 100 through wired or wireless communication methods to achieve information exchange.

[0057] In system 100, user 102 can interact with extended reality system 100 using head-mounted wearable device 104 and control handle 108. In some scenarios, system 100 can use images captured by camera / video camera 110 to recognize user 102's posture, gestures, etc., and then complete the interaction with user 102 based on the recognized posture and gestures. In some embodiments, user 130 can also perform gesture input using bare hands. Head-mounted wearable device 104 can capture images in front of it in real time using a camera or other camera positioned in front of the head-mounted wearable device 104, and recognize user 130's gestures by recognizing these images.

[0058] In some embodiments, as shown in FIG1A, system 100 can also communicate with server 114 and obtain data from server 114, such as images, audio, video, etc., and output this data through head-mounted wearable device 104, for example, displaying images or videos on the display screen of head-mounted wearable device 104, playing audio and audio carried by the audio in the video using the speaker of head-mounted wearable device 104, etc. In some embodiments, as shown in FIG1A, server 114 can retrieve the required data, such as images, audio, video, etc., from database server 116 used for storing data.

[0059] In some embodiments, a data acquisition unit for collecting information may be provided on the head-mounted wearable device 104. The type of data acquisition unit can be varied.

[0060] In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning and tracking unit. The environment acquisition unit can be used to acquire environmental information around (e.g., in front of) the wearable device 104, and the positioning and tracking unit can be used to perform positioning and tracking of the wearable device 104. Optionally, the environment acquisition unit may include, but is not limited to, photosensitive elements such as a three-color camera (e.g., an RGB camera), a depth camera, a binocular camera, or a laser. The positioning and tracking unit may include, but is not limited to, modules such as visual real-time localization and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), ultra-wideband wireless communication technology (UWB), and a laser.

[0061] In some embodiments, the head-mounted wearable device 104 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the head-mounted wearable device 104. Similarly, the operating handle 108 may also be equipped with a velocity sensor, acceleration sensor, angular velocity sensor (e.g., gyroscope), etc., for collecting velocity or acceleration information of the operating handle 108. It should be noted that, in addition to being mounted on the head-mounted wearable device 104 and the operating handle 108, the aforementioned data collection units may also be directly attached to the body parts of the user 102 without relying on hardware devices, thereby collecting relevant information of that body part, such as velocity, acceleration, or angular velocity information, or information collected by other sensors or data collection units.

[0062] In some embodiments, the head-mounted wearable device 104 may also be equipped with a camera or webcam for taking photos of the operator (user) 102 (e.g., photos of hands or feet) and environmental images.

[0063] In some embodiments, the system 100 can identify the posture, gestures, etc. of the user 102 by collecting information, and then perform corresponding interactions based on the identified user posture and gestures.

[0064] Figure 1B shows a schematic diagram of an exemplary head-mounted wearable device 104.

[0065] As shown in Figure 1B, the head-mounted wearable device 104 may include a lens barrel 1042, inside which a display screen 1044 for displaying images and an optical component 1046 for processing the light path may be disposed. Optionally, the optical component 1046 may further include multiple lenses (e.g., lenses 1046A and 1046B). The combination of multiple lenses can project the light emitted from the display screen 1044 into the human eye 1022, so that the human eye 1022 can view the image displayed on the display screen 1044. It is understood that Figure 1B only shows a single-sided structure of the head-mounted wearable device 104 as an example. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged side by side.

[0066] In some embodiments, as shown in FIG1B, the head-mounted wearable device 104 may also be provided with a camera 1048 for acquiring images of the human eye. The camera 1048 may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, etc.

[0067] Optionally, the camera 1048 can be an eye-tracking (ET) camera, and the human eye images it acquires can be used to achieve functions such as pupil distance estimation and eye-tracking.

[0068] As shown in Figure 1B, in related technologies, the camera 1048 is typically positioned outside the lens barrel, and usually only one camera is installed per lens barrel. Furthermore, to better capture a complete image of the human eye without interfering with the eye's viewing of the display screen 1044, the common deployment position of the camera 1048 is generally at the outer corner of the eye or the wing of the nose. Referring to Figure 1B, if the camera 1048 is close to the outer side of the device, the camera deployment position shown in Figure 1B is at the outer corner of the eye; if the camera 1048 is close to the inner side of the device, the camera deployment position shown in Figure 1B is at the wing of the nose.

[0069] However, the way the camera is installed in the related technology tends to result in a large installation tilt angle of the camera 1048 relative to the human eye 1022, which leads to a large angle α between the orientation of the camera 1048 and the frontal viewing direction of the human eye 1022. This makes it difficult for the acquired human eye image to reflect the frontal viewing angle of the human eye, as shown in Figures 1C and 1D.

[0070] In some cases, users may need to wear glasses before using the head-mounted wearable device 104. However, because the camera 1048 is located outside the lens barrel 1042, it protrudes above the lens barrel 1042, which can easily compress the glasses and affect the wearing comfort of the head-mounted wearable device 104. Simultaneously, the image quality of the camera 1048 is easily affected by the edges of the glasses. Light refraction at the edges of the glasses reduces the image sharpness of the camera 1048 and creates numerous refracted light spots in the image, thus affecting the accuracy of subsequent algorithms. This problem is further exacerbated when there is only one camera corresponding to the lens barrel.

[0071] In view of this, the present disclosure provides a wearable device that will have at least two cameras installed inside the lens barrel, which can solve or partially solve the above problems to a certain extent.

[0072] Figure 2A shows a schematic diagram of an exemplary wearable device 200 provided in an embodiment of the present disclosure.

[0073] As shown in Figure 2A, similarly, the wearable device 200 may also include a lens barrel 202, and a display module 204 and an optical component 206 disposed inside the lens barrel 202. The optical component 206 may further include multiple lenses (e.g., lenses 2062 and 2064), and the combination of multiple lenses can project the light emitted by the display module 204 into the human eye 1022, so that the human eye 1022 can view the image displayed by the display module 204.

[0074] Unlike the wearable device 104 shown in Figure 1B, the wearable device 200 includes two cameras 208A and 208B, both of which are housed inside the lens barrel 202. Since the cameras 208A and 208B are placed inside the lens barrel 202, they do not interfere with the wearing of the glasses, thus improving the comfort of the wearable device 200. Simultaneously, as shown in Figure 2A, because the cameras 208A and 208B are located inside the lens barrel 202, the distance between the cameras 208A and 208B and the human eye 1022 is increased. This results in a smaller installation tilt angle of the cameras 208A and 208B relative to the human eye 1022. Consequently, the angle β between the orientation of the cameras 208A and 208B and the direct viewing direction of the human eye 1022 is smaller than the angle α. Therefore, the cameras 208A and 208B have a better viewing angle, and the acquired images of the human eye better reflect the direct viewing angle, resulting in better image quality. Furthermore, since cameras 208A and 208B are housed inside the lens barrel 202, the eyeglasses do not interfere with the imaging of cameras 208A and 208B, further improving image quality. This improved image quality also enhances the accuracy of algorithms such as pupillary distance estimation and gaze tracking. In addition, the use of two cameras, 208A and 208B, provides more observational information (more images acquired), which also improves the accuracy of algorithms such as pupillary distance estimation and gaze tracking.

[0075] In some embodiments, to improve the accuracy of subsequent algorithms such as interpupillary distance estimation or gaze tracking, as shown in Figure 2A, the two cameras 208A and 208B can be symmetrically arranged within the lens barrel 202 relative to its axis (the central dashed line in Figure 2A). In this way, the images acquired by the two cameras 208A and 208B can be symmetrical, further improving the processing efficiency of subsequent algorithms.

[0076] In some embodiments, as shown in FIG2A, both cameras 208A and 208B are oriented towards the light-emitting side of the lens barrel 202, and the angles between the orientations of cameras 208A and 208B and the axis of the lens barrel 202 (the central dashed line in FIG2A) are equal, for example, the angles are both β. In this way, the images acquired by the two cameras 208A and 208B can be strictly symmetrical, and the processing efficiency of subsequent algorithms can be further improved.

[0077] Figure 2B shows a schematic diagram of another exemplary wearable device 200 provided in an embodiment of this disclosure.

[0078] As shown in Figure 2B, in some embodiments, the wearable device 200 may further include two reflective structures 210A and 210B. These reflective structures 210A and 210B can be structures with reflective surfaces, such as reflective films or mirrors. Reflective structures 210A and 210B can correspond to cameras 208A and 208B, respectively. Both cameras 208A and 208B face the display module 204. Reflective structure 210A reflects light from the human eye 1022 into camera 208A, and reflective structure 210B reflects light from the human eye 1022 into camera 208B. Through the reflection of light by reflective structures 210A and 210B, cameras 208A and 208B can still capture images of the human eye. Furthermore, because an additional reflection process is added to the optical path, the observation angle γ is further reduced, allowing cameras 208A and 208B to achieve better imaging.

[0079] In some embodiments, to improve the accuracy of subsequent algorithms such as interpupillary distance estimation or gaze tracking, as shown in Figure 2B, the two cameras 208A and 208B can be symmetrically arranged within the lens barrel 202 relative to its axis (the center dashed line in Figure 2B), and the reflective structures 210A and 210B can also be symmetrically arranged within the lens barrel 202 relative to its axis (the center dashed line in Figure 2B). In this way, the images acquired by the two cameras 208A and 208B can be symmetrical, further improving the processing efficiency of subsequent algorithms.

[0080] In some embodiments, as shown in FIG2B, both cameras 208A and 208B are oriented towards the display module 204, and the angles between the orientations of the two cameras 208A and 208B and the axis of the lens barrel 202 (the central dashed line in FIG2B) are equal, for example, the angles are both γ; simultaneously, the angles between the orientations of the reflective structures 210A and 210B and the axis of the lens barrel 202 (the central dashed line in FIG2B) are also equal. In this way, the images acquired by the two cameras 208A and 208B can be strictly symmetrical, further improving the processing efficiency of subsequent algorithms.

[0081] The foregoing embodiments are all illustrated with two cameras set in a single telescope tube. It can be understood that when the number of cameras is further increased, the observation data can be further increased, thereby further improving the accuracy of the algorithm. Therefore, embodiments with two or more cameras set in a single telescope tube should all fall within the protection scope of this disclosure.

[0082] Figures 2A and 2B only show a single-sided structure of the wearable device 200 as an example. It can be understood that, in order to achieve binocular display, the wearable device 200 may include two lens barrel structures arranged side by side.

[0083] Figure 2C shows a schematic diagram of yet another exemplary wearable device 200 provided in an embodiment of this disclosure.

[0084] As shown in Figure 2C, the wearable device 200 may include a binocular display module, which may further include a first display module 204A and a second display module 204B. The first display module 204A and the second display module 204B may respectively display an image for observation by a first eye 1022A (e.g., the right eye) and an image for observation by a second eye 1022B (e.g., the left eye).

[0085] In some embodiments, as shown in FIG2C, the wearable device 200 may further include two lens barrels disposed on the light-emitting side of the binocular display module, such as a first lens barrel 202A and a second lens barrel 202B. At least one of the two lens barrels may further include at least two cameras disposed inside the lens barrel for acquiring images of the human eye, thereby providing more observation data and improving the accuracy of subsequent algorithms.

[0086] As an optional embodiment, as shown in FIG2C, the first lens barrel 202A can be disposed on the light-emitting side of the first display module 204A. A first camera 208A and a second camera 208B for acquiring images of the first eye 1022A can be further disposed within the first lens barrel 202A. This allows for more accurate calculation results when the images of the first eye 1022A (e.g., the right eye) are subsequently used for interpupillary distance calculation or gaze tracking. In some embodiments, as shown in FIG2C, a first optical component 206A can also be disposed within the first lens barrel 202A for projecting the image displayed by the first display module 204A onto the first eye 1022A through optical principles. Optionally, the first optical component 206A can further include a first lens 2062A and a second lens 2064A. The parameters of the first lens 2062A and the second lens 2064A can be different or the same, and can be designed according to actual needs. It is understood that the type and number of lenses in the first optical component 206A are variable, and the specific type and number of lenses used can be designed according to actual needs.

[0087] Similarly, as another optional embodiment, as shown in FIG2C, the second lens barrel 202B can be disposed on the light-emitting side of the second display module 204A. A third camera 208C and a fourth camera 208D for acquiring human eye images of the second eye 1022B can be further disposed within the second lens barrel 202A. In this way, the human eye images acquired for the second eye 1022B (e.g., the left eye) can yield more accurate calculation results when subsequently used for interpupillary distance calculation or gaze tracking. Similarly, in some embodiments, as shown in FIG2C, a second optical component 206B can also be disposed within the second lens barrel 202B for projecting the image displayed by the second display module 204B onto the second eye 1022B through optical principles. Optionally, the second optical component 206B can further include a third lens 2062B and a fourth lens 2064B. The parameters of the third lens 2062B and the fourth lens 2064B can be different or the same, and can be designed according to actual needs. It is understood that the type and number of lenses in the second optical component 206B are variable, and the specific type and number of lenses used can be designed according to actual needs.

[0088] In some embodiments, as shown in FIG2C, the first camera 208A and the second camera 208B can be symmetrically arranged within the first lens barrel 202A with respect to the axis of the first lens barrel 202A. In this way, the human eye images 1022A captured by the first camera 208A and the second camera 208B can be symmetrical, and the processing efficiency of subsequent algorithms can be further improved.

[0089] Similarly, in some embodiments, as shown in FIG2C, the third camera 208C and the fourth camera 208D are symmetrically arranged within the second lens barrel 202B with respect to its axis. In this way, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can also be symmetrical, further improving the processing efficiency of subsequent algorithms.

[0090] In some embodiments, as shown in FIG2C, both the first camera 208A and the second camera 208B are oriented towards the light-emitting side of the first display module 204A, and the angles between the orientations of the first camera 208A and the second camera 208B and the axis of the first lens barrel 202A are equal, for example, the angles are both β. In this way, the human eye images of the first eye 1022A captured by the first camera 208A and the second camera 208B can be strictly symmetrical, and the processing efficiency of subsequent algorithms can be further improved.

[0091] Similarly, in some embodiments, as shown in FIG2C, the third camera 208C and the fourth camera 208D are both oriented towards the light-emitting side of the second display module 204B, and the angles between the orientations of the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, the angles are both β. In this way, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be strictly symmetrical, and the processing efficiency of subsequent algorithms can be further improved.

[0092] Figure 2D shows a schematic diagram of yet another exemplary wearable device 200 provided in an embodiment of this disclosure.

[0093] In some embodiments, as shown in FIG2D, both the first camera 208A and the second camera 208B face the first display module 204A. A first reflective structure 210A corresponding to the first camera 208A and a second reflective structure 210B corresponding to the second camera 208B are disposed within the first lens barrel 202A. The first reflective structure 210A and the second reflective structure 210B can be structures with reflective surfaces, such as reflective films or mirrors. The first reflective structure 210A and the second reflective structure 210B can respectively correspond to the first camera 208A and the second camera 208B. The first reflective structure 210A is used to reflect light from the first eye 1022A into the first camera 208A, and the second reflective structure 210B is used to reflect light from the first eye 1022A into the second camera 208B. Through the reflection of light by the first reflective structure 210A and the second reflective structure 210B, the first camera 208A and the second camera 208B can still capture the human eye image of the first eye 1022A. Furthermore, because an additional reflection process is added to the optical path, the observation angle γ is further reduced, enabling the first camera 208A and the second camera 208B to better image the first eye 1022A.

[0094] Similarly, in some embodiments, as shown in FIG2D, both the third camera 208C and the fourth camera 208D face the second display module 204B. The second lens barrel 202B contains a third reflective structure 210C corresponding to the third camera 208C and a fourth reflective structure 210D corresponding to the fourth camera 208D. The third reflective structure 210C and the fourth reflective structure 210D can be structures with reflective surfaces, such as reflective films or mirrors. The third reflective structure 210C and the fourth reflective structure 210D can respectively correspond to the third camera 208C and the fourth camera 208D. The third reflective structure 210C is used to reflect light from the second eye 1022B into the third camera 208C, and the fourth reflective structure 210D is used to reflect light from the second eye 1022B into the fourth camera 208D. The reflection of light by the third reflection structure 210C and the fourth reflection structure 210D allows the third camera 208C and the fourth camera 208D to still acquire images of the second eye 1022B. Furthermore, because an additional reflection process is added to the optical path, the observation angle γ is further reduced, enabling the third camera 208C and the fourth camera 208D to better image the second eye 1022B.

[0095] In some embodiments, to improve the accuracy of subsequent algorithms such as interpupillary distance estimation or gaze tracking, as shown in FIG2D, the first camera 208A and the second camera 208B may be symmetrically arranged within the first lens barrel 202A with respect to its axis. Similarly, the first reflective structure 210A and the second reflective structure 210B may also be symmetrically arranged within the first lens barrel 202A with respect to its axis. In this way, the human eye images 1022A captured by the first camera 208A and the second camera 208B can be symmetrical, further improving the processing efficiency of subsequent algorithms.

[0096] Similarly, in some embodiments, to improve the accuracy of subsequent algorithms such as interpupillary distance estimation or gaze tracking, as shown in Figure 2D, the third camera 208C and the fourth camera 208D can be symmetrically arranged within the second lens barrel 202B with respect to its axis. Similarly, the third reflection structure 210C and the fourth reflection structure 210D can also be symmetrically arranged within the second lens barrel 202B with respect to its axis. In this way, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be symmetrical, further improving the processing efficiency of subsequent algorithms.

[0097] In some embodiments, as shown in FIG2D, both the first camera 208A and the second camera 208B are oriented towards the first display module 204A, and the angles between the orientations of the first camera 208A and the second camera 208B and the axis of the first lens barrel 202A are equal, for example, the angles are both γ; simultaneously, the angles between the orientations of the first reflective structure 210A and the second reflective structure 210B and the axis of the first lens barrel 202A are also equal. Thus, the human eye images 1022A captured by the first camera 208A and the second camera 208B can be strictly symmetrical, further improving the processing efficiency of subsequent algorithms.

[0098] In some embodiments, as shown in FIG2D, both the third camera 208C and the fourth camera 208D are oriented towards the second display module 204B, and the angles between the orientations of the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, the angles are both γ; simultaneously, the angles between the orientations of the third reflective structure 210C and the fourth reflective structure 210D and the axis of the second lens barrel 202B are also equal. Thus, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be strictly symmetrical, further improving the processing efficiency of subsequent algorithms.

[0099] The foregoing embodiments are all described with two cameras installed in each of the two lens barrels. It is understood that one of the two lens barrels can be equipped with two or more cameras, while the other lens barrel can have only one camera. In this case, the algorithm accuracy can still be improved to a certain extent, and this should also fall within the scope of protection of this disclosure. Furthermore, the structures shown in Figures 2A and 2B can also be combined within the same wearable device 200. For example, the lens barrel corresponding to the first eye has the structure shown in Figure 2A, and the lens barrel corresponding to the second eye has the structure shown in Figure 2B. Of course, the structures corresponding to the first and second eyes can also be interchanged.

[0100] As can be seen from the above embodiments, by placing at least two cameras 208 inside at least one lens barrel of the wearable device 200, the embodiments of this disclosure can achieve better image quality and improve the comfort of the wearable device 200. In some embodiments, by placing two cameras in each of the left and right lens barrels of the wearable device 200, not only can the aforementioned adverse effects of cameras outside the lens barrel be avoided, but the two cameras can also provide more observation information and are less affected by occlusion, thus improving the accuracy of algorithms such as pupillary distance estimation or gaze tracking.

[0101] Furthermore, compared to placing the camera outside the lens barrel, placing the camera inside the lens barrel means that when the camera is imaging, it will be affected not only by the camera module itself but also by the optical components inside the lens barrel.

[0102] As shown in Figures 2A and 2B, relative to camera 1048 in Figure 1B, in the optical path from cameras 208A and 208B to the human eye 1022, there is also an optical component 206 or a part of optical component 206 inside the lens barrel (which varies depending on the relative position of cameras 208A and 208B and the lenses in optical component 206; for example, in addition to the positions shown in Figures 2A and 2B, cameras 208A and 208B can also be positioned between lenses 2062 and 2064). This causes the optical component 206 or a part of optical component 206 to affect or change the optical path from cameras 208A and 208B to the human eye 1022 when cameras 208A and 208B image the human eye 1022, resulting in the distortion of the image obtained by cameras 208A and 208B being no longer asymmetrical.

[0103] Furthermore, because the light paths of cameras 208A and 208B to the human eye 1022 are altered, cameras 208A and 208B do not have a unified projection center, making it impossible to use a parametric camera model to fit the projection process of the camera inside the lens barrel to process the distortion, thus making it difficult to calibrate the camera parameters.

[0104] Furthermore, in some cases, as shown in Figures 2C and 2D, because the different cameras are positioned differently within the wearable device 200, it is also necessary to calibrate the pose relationships between the different cameras, which is quite difficult to achieve. In particular, when the different cameras are located within different lens barrels (e.g., between the first camera 208A and the third camera 208C or the fourth camera 208D, or between the second camera 208B and the third camera 208C or the fourth camera 208D), such calibration becomes even more complex.

[0105] In view of this, embodiments of this disclosure also provide a method for calibrating camera parameters, which can use a non-parametric camera model to calibrate the parameters of a camera within a lens barrel. In some embodiments, for cameras located in different lens barrels, a camera extrinsic parameter calibration algorithm based on non-common viewpoints is proposed, thereby solving the problem of difficult calibration of cameras in different lens barrels.

[0106] Figure 3A shows a flowchart of an exemplary method 300 provided in an embodiment of this disclosure.

[0107] This method 300 can be applied to any computer device with data processing capabilities and can be used to calibrate the parameters of cameras 208A, 208B, 208C, or 208D in the wearable device 200 shown in Figures 2A, 2B, 2C, or 2D. As shown in Figure 3A, this method 300 may further include the following steps.

[0108] In step 302, multiple images of the calibration reference object captured by at least two cameras can be received. In this step, the multiple images can be images acquired by different cameras. As an optional embodiment, to calibrate the first parameter (e.g., intrinsic parameter) and the second parameter (e.g., extrinsic parameter) of the cameras, multiple images can be acquired separately for intrinsic parameter calibration and multiple images can be acquired separately for extrinsic parameter calibration.

[0109] Figure 4A shows a schematic diagram of an exemplary image acquisition scenario 400 according to an embodiment of the present disclosure.

[0110] As shown in Figure 4A, in the image acquisition scenario 400, a calibration reference object 402 can be set at a fixed position. In some embodiments, the calibration reference object 402 can be a calibration board or a chart, on which a black and white checkerboard pattern is set. After the camera captures an image of the calibration reference object 402, the corresponding camera parameters can be calculated based on the correspondence between the checkerboard pattern in the image and the checkerboard pattern of the calibration reference object 402.

[0111] When acquiring images of the calibration reference object 402, the camera to be calibrated (e.g., camera 208A) can be controlled to move along a certain trajectory, and the calibration reference object 402 can be continuously photographed during the camera's movement. This results in multiple images acquired by the camera in different poses, which are then used as data for calculating camera parameters. Since the acquired images are obtained by the camera in multiple poses, the calculated camera parameters are more robust and applicable to a wider range.

[0112] In some embodiments, if the at least two cameras in the wearable device are of different types or have different camera parameters, multiple images need to be acquired separately for each camera or each type of camera in the manner described above, for calibrating the first parameter (e.g., intrinsic parameter) of the corresponding camera. If the camera parameters of the at least two cameras in the wearable device are consistent, one camera can be selected to perform the aforementioned image acquisition operation, thereby calibrating the camera's intrinsic parameters. The calibrated intrinsic parameters can be applied to other cameras in the wearable device.

[0113] Figure 4B illustrates a schematic diagram of another exemplary imaging scenario 410 according to an embodiment of the present disclosure.

[0114] As shown in Figure 4B, similar to the image acquisition scenario 400, a calibration reference object 402 can also be set at a fixed position in the image acquisition scenario 410.

[0115] Unlike the image acquisition scenario 400, in order to calculate the pose relationship (second parameter) between at least two cameras in the wearable device 200, the at least two cameras can be installed in the lens barrel of the wearable device 200 and the wearable device 200 can be fully assembled. Then, the assembled wearable device 200 or its prototype can be used to acquire extrinsic parameters, and the corresponding extrinsic parameters (e.g., pose relationship between different cameras) can be calculated based on the acquired images.

[0116] As shown in Figure 4B, taking the example of two cameras respectively installed in the two lens barrels of the wearable device 200, when capturing images of the calibration reference object 402, the wearable device 200 can be controlled to move along a certain motion trajectory. During the movement of the wearable device 200, four cameras continuously capture images of the calibration reference object 402. Thus, while the relative pose relationship of the four cameras remains unchanged, multiple images captured by the four cameras in different poses are obtained, which serve as image data for subsequent calculation of the second parameter. Since the captured images are obtained by the cameras capturing images in multiple poses, the camera parameters calculated subsequently have better robustness and can be applied to a wider range.

[0117] Based on the above, it can be understood that in the image acquisition scenario 410, each camera can acquire multiple images. In addition to being used to calculate the relative pose relationship of the four cameras, the images acquired by each camera can also be used to calculate its own intrinsic parameters. Therefore, in some embodiments, only the image acquisition scenario 410 can be used to acquire images, and the first and second parameters of the camera can be calibrated based on these images.

[0118] In step 302, the computer device may receive multiple images acquired in the aforementioned scenario 400 and / or scenario 410 for subsequent processing.

[0119] After acquiring the required multiple images, camera parameters can be calibrated based on these images. Parameter calibration may include calibrating a first parameter and a second parameter, where the first parameter can be an intrinsic parameter of the camera, and the second parameter can be an extrinsic parameter of the camera.

[0120] In some embodiments, the camera's intrinsic parameters can be calibrated first. However, as mentioned earlier, since the camera is housed within the lens barrel, its imaging is affected not only by its own module but also by the lens of the lens barrel. This causes the imaging distortion to become non-symmetrical about the principal point of the image. Furthermore, the camera lacks a unified projection center, making it impossible to use a parametric camera model to fit the projection process of the camera within the lens barrel to handle the asymmetrical distortion. Therefore, in some embodiments, a non-parametric camera model is provided to calibrate the camera's intrinsic parameters.

[0121] Therefore, in step 304, the projection relationship between the pixels of the multiple images and the calibration reference object can be determined first. In this step, the projection relationship between the images and the calibration reference object 402 can be established based on the acquired images, thereby establishing the correspondence between pixels and projection directions (pixel-ray).

[0122] It is understood that the projection relationship between the pixels of an image and the calibration reference object represents the camera's inherent parameters (i.e., intrinsic parameters). Therefore, when calculating the projection relationship for a specific camera (e.g., camera 208A), it is necessary to use the image captured by that specific camera to establish the projection relationship. Thus, taking the image acquisition scenario shown in Figure 4B as an example, it is necessary to distinguish among the multiple images the first image captured by the first camera 208A, the multiple second images captured by the second camera 208B, the multiple third images captured by the third camera 208C, and the multiple fourth images captured by the fourth camera 208D.

[0123] The projection relationship is calculated below using the first camera 208A as an example.

[0124] In some embodiments, as shown in FIG3B, step 304 of determining the projection relationship between the pixels of the plurality of images and the calibration reference object may further include the following steps:

[0125] In step 3042, a first number of target images are selected from the plurality of images.

[0126] In this step, a certain number of target images can be selected from multiple first images (images acquired by the first camera 208A) among the multiple images to establish a projection relationship. The remaining first images can be used to supplement the parts for which no projection relationship has been established.

[0127] It is understood that the first number is sufficient to continue with subsequent steps, and its quantity is not specifically limited. As an optional embodiment, the first number can be 3. Figure 4C shows the three selected target images 412, 414, and 416. As shown in Figure 4C, target images 412, 414, and 416 respectively show images obtained by the first camera 208A in different postures of the calibration reference object 402. It is understood that by calibrating the camera parameters of images acquired in different camera postures, the robustness of the calculated camera parameters can be improved.

[0128] In step 3044, at least one local region is selected in each target image (for example, the region corresponding to a cell in a chessboard can be considered a local region), and a homography transformation matrix between each local region and the calibration reference object is established.

[0129] After selecting at least one local region of each target image, for each local region of each target image, a homography transformation matrix with the calibration reference object 402 can be constructed to establish the correspondence between each pixel point contained in the local region and the coordinate system of the calibration reference object 402.

[0130] It can be known that the coordinate system of the location of the calibration reference object 402 is known, and the checkerboard pattern on the calibration reference object 402 corresponds to the checkerboard pattern in the target image. At the same time, the coordinates of the four vertices corresponding to the local region in the camera coordinate system of the target image are also known. Based on this known information, the homography transformation matrix H between the local region and the calibration reference object 402 can be obtained.

[0131] In this way, a correspondence is established between the pixels contained in the local area and the calibration reference object 402 or the coordinate system in which the calibration reference object 402 is located.

[0132] In step 3046, the projection relationship between the pixels contained in the local region and the calibration reference object is determined according to the homography transformation matrix.

[0133] Figure 4D shows a schematic diagram of the projection relationship between pixels and a calibration reference object according to an embodiment of the present disclosure.

[0134] As shown in Figure 4D, the region corresponding to the four vertices connected by the four dashed lines in the figure is the local region. By constructing the homography transformation matrix, the projection relationship between each pixel in the local region of the target image and the corresponding point on the calibration reference object can be calculated.

[0135] In step 3048, the first number of target images are transformed into the reference coordinate system, and the projection relationship between the pixels of the multiple images and the calibration reference object is determined according to the region with established projection relationship corresponding to the local region.

[0136] As shown in Figure 4C, the three target images 412, 414, and 416 were acquired under different camera poses. In the previous step of constructing the homography transformation matrix of the local region, it was constructed using the camera coordinate system corresponding to each target image. It can be understood that in order to unify the projection relationship of each target image under the same reference coordinate system, in this step, the three target images 412, 414, and 416 can be transformed to the reference coordinate system.

[0137] After processing the three target images 412, 414, and 416 according to the aforementioned steps, a similar method can be used to select local regions of new target images in the remaining first images to construct homography transformation matrices until the projection relationship between each pixel in the entire image and the calibration reference object is completed.

[0138] As can be seen from the foregoing embodiments, in some embodiments, after all image processing and calibration are completed, some pixels may correspond to multiple projection direction data. Therefore, these data can be averaged to obtain a single projection direction, which can then be used as the projection direction for that pixel to obtain better projection direction data. Furthermore, during subsequent storage, only this averaged projection direction can be stored, thereby saving storage space.

[0139] It is understood that the aforementioned method can be used to establish the projection relationship between image pixels and calibration reference object 402 for the second camera 208B, the third camera 208C, the fourth camera 208D, and possibly more cameras (depending on the number of cameras in the wearable device), which will not be elaborated here.

[0140] After establishing the projection relationship, in step 306, a first parameter set and a second parameter set of the at least two cameras can be determined based on the projection relationship. The first parameter set includes multiple first parameters, which characterize target pixels in the image captured by the camera to be calibrated and the corresponding projection direction. The second parameter set includes at least one set of second parameters, which indicate the pose relationship between the at least two cameras.

[0141] In some embodiments, a first set of parameters for the at least two cameras may be determined first based on the projection relationship, and then a second set of parameters may be determined.

[0142] Figure 4E shows a schematic diagram of an exemplary first parameter according to an embodiment of the present disclosure.

[0143] As shown in Figure 4E, when constructing the aforementioned projection relationship, at least some pixels in the image under the reference coordinate system correspond one-to-one with some points on the calibration reference object, thus obtaining a linear equation about the pixel. The line represented by this equation passes through the corresponding pixel and has a ray as shown in the figure to represent its projection direction. This combination of pixel and projection direction can be used as the first parameter (this camera parameter can be considered as the camera's intrinsic parameter). This camera parameter is used to realize the transformation from the coordinate system of the image acquired by the camera to the camera coordinate system, which can then be used to calculate the position information of certain features (e.g., pupil) in the three-dimensional space (camera coordinate system) of the image acquired by the camera. Furthermore, since the homography transformation matrix with the calibration reference object is established by using multiple local regions of the first image during the calculation of this camera parameter, the projection direction corresponding to the obtained pixel already includes the correction for image distortion at that position when using this camera parameter for coordinate system transformation calculation, without the need for additional distortion correction.

[0144] In some embodiments, the previously obtained data can be simplified and used as the first parameters of the camera, thereby saving storage space for the first parameters. For example, a spline surface can be used to fit the initial set of the first parameters obtained from the aforementioned calibration, and then the control points of the fitted spline surface can be optimized using the bundle adjustment (BA) algorithm. The first parameters corresponding to all control points of the optimized spline surface are then used as the first parameter set of the at least two cameras.

[0145] When the first parameter is needed, the complete set of the first parameter can be obtained by interpolation algorithm based on the fitted spline surface and the optimized control points.

[0146] As can be seen, the aforementioned method only provides a way to calculate the first parameter for a single camera. For each camera in the wearable device 200, the corresponding first parameter can be calculated using the above method, which will not be elaborated here.

[0147] After obtaining the first parameter set, the second parameter set can be further calibrated.

[0148] Returning to Figure 2C, the wearable device 200 includes a first camera 208A and a second camera 208B disposed in the first lens barrel 202A for capturing images of the first eye 1022A, and a third camera 208C and a fourth camera 208D disposed in the second lens 202B for capturing images of the second eye 1022B. Since the four cameras are located in different positions within the wearable device 200, in order to determine the relative relationship between the images captured by the four cameras, it is necessary to know the pose relationship between the four cameras as a second parameter, thereby completing the calibration of the camera extrinsic parameters.

[0149] The following example, using the calibration of four cameras in a wearable device 200, illustrates how to calculate the second parameter.

[0150] As shown in Figure 2C, since the two cameras set in the same lens barrel of the wearable device 200 need to acquire images of the same human eye, these two cameras generally have a common viewpoint (or optical path intersection point). Therefore, a dual-camera positioning algorithm can be used to obtain the pose relationship between the two cameras in the same lens barrel.

[0151] Therefore, in some embodiments, the second parameter may include the pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D. The pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera are calculated based on the projection relationship using a dual-camera positioning algorithm.

[0152] Optionally, step 306, which determines the first parameter set and the second parameter set of the at least two cameras based on the projection relationship, may further include: determining the pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D using a dual-camera positioning algorithm based on the projection relationship.

[0153] Referring again to Figure 2C, it can be understood that since the cameras in different lens barrels capture images from different human eyes, the cameras in different lens barrels may not share a common viewpoint. Therefore, the binocular positioning algorithm cannot be used to calculate the pose relationship, or the pose relationship calculated using the binocular positioning algorithm may be inaccurate. Therefore, embodiments of this disclosure provide a method for calculating the pose relationship between cameras in different lens barrels.

[0154] Figure 4F illustrates a calibration schematic of a second parameter of an exemplary in-tube camera on the opposite side according to an embodiment of the present disclosure.

[0155] [Corrected according to Rule 91, 15.01.2025] As shown in Figure 4F, taking the calculation of the pose relationship between the first camera 208A and the fourth camera 208D as an example, the two cameras do not share a common viewpoint. When they move to a certain position, the relative positions of the two cameras with the calibration reference object 402 are shown in Figure 4F. The pose relationship of the first camera 208A relative to the calibration reference object 402 is T. 1,i The pose relationship between the fourth camera 208D and the calibration reference object 402 is T. 4,i The pose relationship between the first camera 208A and the fourth camera 208D is T. 14Using the observation data from the first camera 208A and the fourth camera 208D (the first image set and the fourth image set obtained in the image acquisition step), a BA optimization problem can be constructed, and the pose relationship T can be obtained by solving this problem. 14 .

[0156] Therefore, in some embodiments, the second parameter may further include the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera.

[0157] Optionally, step 306, which determines the first parameter set and the second parameter set of the at least two cameras based on the projection relationship, may further include: determining the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera based on the multiple images and the projection relationship.

[0158] Specifically, we can first construct an optimization function.

[0159] Optionally, the optimization function may include a first formula representing the error between a detection point in the first image acquired by the first camera 208A and its corresponding spatial position (the corresponding position on the calibration reference object 402) projected onto a two-dimensional point in the image coordinate system using the first parameters of the first camera 208A corresponding to the detection point. Optionally, the first formula is expressed as:

[0160] f cam1,1 =π1(T 1,i ,P cam1 )-d cam1

[0161] Where π1 is the projection function corresponding to the first camera 208A (i.e., the projection relationship between the previously obtained pixel points and the calibration reference object 402), T 1,i P represents the extrinsic parameters of the first camera 208A and the calibration reference object 402 corresponding to the i-th first image (which can be calculated based on the spatial coordinates of the first image and the calibration reference object 402). cam1 Let d be the 3D point corresponding to the i-th first image of the first camera 208A on the calibration reference object 402 (that is, the spatial coordinates of each pixel of the first image on the calibration reference object 402). cam1 These are all the detection points corresponding to the first camera 208A.

[0162] Optionally, the optimization function may further include a second formula representing the error between a detection point in the fourth image acquired by the fourth camera 208D and its corresponding spatial position (the corresponding position on the calibration reference object 402) projected onto a two-dimensional point in the image coordinate system using the first parameters of the fourth camera 208A corresponding to that detection point. Optionally, the second formula is expressed as:

[0163] f cam4,2 =π4(T) 4,i ,P cam4 )-d cam4

[0164] Where π4 is the projection function corresponding to the fourth camera 208D (i.e., the projection relationship between the previously obtained pixel points and the calibration reference object 402), T 4,i P represents the extrinsic parameters of the fourth camera 208D corresponding to the i-th fourth image and the calibration reference object 402 (which can be calculated based on the coordinate information of the fourth image and the calibration reference object 402). cam4 Let d be the 3D point corresponding to the i-th fourth image of the fourth camera 208D on the calibration reference object 402 (that is, the spatial coordinates of each pixel of the fourth image on the calibration reference object 402). cam4 These are all the detection points corresponding to the fourth camera, 208D.

[0165] Optionally, the optimization function may further include a third formula representing the error between a detection point in the fourth image acquired by the fourth camera 208D and its corresponding spatial position (corresponding position on the calibration reference object 402), projected onto a two-dimensional point in the image coordinate system using the first parameters of the first camera 208A corresponding to the detection point and the pose relationship between the first camera and the fourth camera. Optionally, the third formula is expressed as:

[0166] f cam4,1 =π4(T) 14 *T 1,i ,P cam4 )-d cam4

[0167] Among them, T 14 The pose relationship between the first camera 208A and the fourth camera 208D is the extrinsic parameters of the first camera and the fourth camera required in this embodiment of the disclosure.

[0168] Optionally, the optimization function may further include a fourth formula representing the error between a detection point in the fourth image acquired by the fourth camera 208D and its corresponding spatial position (corresponding position on the calibration reference object 402), projected onto a two-dimensional point in the image coordinate system using the first parameters of the fourth camera corresponding to the detection point and the pose relationship between the first camera and the fourth camera. Optionally, the fourth formula is expressed as:

[0169] in, The transpose matrix of the pose relationship between the first camera 208A and the fourth camera 208D is obtained by applying T... 14 We obtain it by transposing it.

[0170] In some embodiments, the detection point d cam1 and d cam4 The following methods can be used for calculation.

[0171] Figure 4G shows a schematic diagram of the image obtained after binarization of the image acquired in an embodiment of this disclosure.

[0172] As shown in Figure 4G, after binarization, the pixels in the image are either black or white. Through feature detection, the vertices of the grid can be identified, and these identified vertices can be used as detection points d. The detection points detected on the i-th image can be represented as d. i .

[0173] Optionally, the optimization function can further determine an error function based on the first, second, third, and fourth formulas described above. Optionally, this error function is expressed as:

[0174] f = f cam1,1 +f cam4,1 +f cam1,2 +f cam4,2

[0175] Finally, optionally, an optimization function can be constructed based on the error function to determine the final extrinsic parameters of the first camera 208A and the fourth camera 208D. Optionally, this optimization function is expressed as follows:

[0176] Where ρ is the loss function and f is the error function O. i Let I represent all the information contained in the i-th image, and let I be the number of input images.

[0177] As can be seen, the loss function res(π, T) is a function of pose. By solving the loss function using the Levenberg-Marquardt (LM) method, the optimal solution obtained is the final pose relation T. 14 .

[0178] It is understandable that the above method can be used to obtain the pose relationship for two cameras inside opposite lenses, so it will not be elaborated further here.

[0179] Considering the numerous possible arrangements of cameras on opposite sides, calculating the pose relationship for each pair of cameras using the aforementioned method would significantly increase the computational load. Therefore, in some embodiments, the pose relationships between the first and third cameras, the first and fourth cameras, the second and third cameras, and the second and fourth cameras, based on the multiple images and the projection relationship, include: determining the pose relationship between the first and fourth cameras based on the multiple images and the projection relationship; determining the pose relationship between the first and third cameras based on the pose relationships between the third and fourth cameras and the first and fourth cameras; determining the pose relationship between the second and third cameras based on the pose relationships between the first and second cameras and the first and third cameras; and determining the pose relationship between the second and fourth cameras based on the pose relationships between the third and fourth cameras and the second and third cameras.

[0180] In this way, after calculating the pose relationship between the first camera 208A and the fourth camera 208D, based on the already calculated pose relationship between the first camera 208A and the second camera 208B, as well as the pose relationship between the third camera 208C and the fourth camera 208D, other permutations and combinations of pose relationships can be obtained through data conversion, thereby saving computational load.

[0181] It is understandable that the above method provides a calculation method that can be used when the two cameras in the opposite lens barrel do not have a common viewpoint. However, when the cameras on the opposite sides have a common viewpoint, the dual-camera positioning algorithm can still be used to calculate the pose relationship.

[0182] It should be noted that the above example is only used to illustrate the two cameras set in the two lens barrels of the wearable device 200. It is understood that the number of cameras in the lens barrels may be less or more depending on the actual needs. However, based on the inventive concept of the embodiments provided in this disclosure, the corresponding camera parameters can still be calculated, which will not be elaborated here.

[0183] In a more specific embodiment, the camera parameter calibration method provided in this disclosure may include steps of image acquisition, intrinsic parameter calibration, extrinsic parameter calibration, optimization and saving of parameters, which can obtain better camera parameters for subsequent algorithm calculations.

[0184] As can be seen from the above embodiments, the camera parameter calibration method provided in this disclosure provides a feasible calibration scheme for the intrinsic and extrinsic parameters of a camera in scenarios where the imaging distortion of the camera inside the lens barrel is asymmetrical and there is no unified projection center.

[0185] It should be noted that the method of this disclosure embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this disclosure embodiment, and the multiple devices will interact with each other to complete the method described.

[0186] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0187] This disclosure also provides a wearable device.

[0188] Figure 5 shows a schematic diagram of an exemplary wearable device 500 provided in an embodiment of this disclosure.

[0189] As shown in Figure 5, similar to Figures 2A and 2B, the wearable device 500 includes a lens barrel 202, in which a display module 204, cameras 208A and 208B, and an optical component 206 are disposed. The cameras 208A and 208B and the optical component 206 are located on the light-emitting side of the display module 204, and the cameras 208A and 208B are located between the optical component 206 and the display module 204.

[0190] In some embodiments, the shooting direction of the cameras 208A and 208B is towards the optical component 206, as shown in FIG2A. In other embodiments, as shown in FIG2B, reflective structures 210A and 210B are provided between the display module 204 and the cameras 208A and 208B, with the reflective surfaces of the reflective structures 210A and 210B facing the light-emitting side of the display module 204, and the shooting direction of the cameras 208A and 208B facing the reflective structures 210A and 210B.

[0191] Furthermore, as shown in FIG5, the wearable device 500 further includes a processing module 502, which is electrically coupled to the camera 208 and the display module 204 respectively, and is configured to: acquire the first parameter set and the second parameter set obtained by the aforementioned method 300, as well as the images captured by the cameras 208A and 208B; and use the first parameter set, the second parameter set, and the images to solve the spatial position of the target region in the image.

[0192] As mentioned earlier, the first parameter set includes camera parameters (which can be considered intrinsic parameters of the camera) that are combinations of pixel points and projection direction. These camera parameters are used to transform the coordinate system of the image acquired by the camera to the camera coordinate system, thereby enabling the calculation of the positional information of certain features (e.g., pupils) in three-dimensional space (camera coordinate system) within the image acquired by the camera. Furthermore, since the calculation of these camera parameters includes correction for image distortion, distortion can be directly corrected when performing coordinate system transformation calculations using these camera parameters, eliminating the need for additional distortion correction.

[0193] The second parameter set includes a second parameter representing the pose relationship between different cameras. This second parameter can be used to unify the image information acquired by different cameras (e.g., unify them to the same camera coordinate system).

[0194] The wearable device provided in this disclosure places the camera inside the lens barrel, which is less affected by occlusion and can improve the accuracy of interpupillary distance estimation or gaze tracking algorithms. Furthermore, by setting at least two cameras, more observation information can be provided, further improving the algorithm accuracy. The camera parameter calibration method and related equipment provided in this disclosure provide a feasible camera parameter calibration scheme for scenarios where the imaging distortion of the camera inside the lens barrel is asymmetrical and there is no unified projection center.

[0195] This disclosure also provides a computer device for implementing the method 300 described above. FIG6 shows a schematic diagram of the hardware structure of an exemplary computer device 600 provided in this disclosure. The computer device 600 can be used to implement the head-mounted wearable device 104 of FIG1A, the wearable devices 200 of FIG2A to 2D, the external device 112 of FIG1A, or the server 114 of FIG1A. In some scenarios, the computer device 600 can also be used to implement the database server 116 of FIG1A.

[0196] As shown in Figure 6, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. The processor 602, memory 604, network module 606, and peripheral interface 608 are interconnected within the computer device 600 via the bus 610.

[0197] Processor 602 may be a central processing unit (CPU), a graphics processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 602 may also include multiple processors integrated into a single logic component. For example, as shown in FIG6, processor 602 may include multiple processors 602a, 602b, and 602c.

[0198] Memory 604 can be configured to store data (e.g., instructions, computer code, etc.). As shown in FIG6, the data stored in memory 604 may include program instructions (e.g., program instructions for implementing methods 300 or 500 of embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 602 can also access the program instructions and data stored in memory 604 and execute the program instructions to operate on the data to be processed. Memory 604 may include volatile storage devices or non-volatile storage devices. In some embodiments, memory 604 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0199] Network interface 606 can be configured to provide communication with other external devices to computer device 600 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0200] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0201] Bus 610 can be configured to transfer information between various components of computer device 600 (e.g., processor 602, memory 604, network interface 606, and peripheral interface 608), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0202] It should be noted that although the architecture of the computer device 600 described above only shows the processor 602, memory 604, network interface 606, peripheral interface 608, and bus 610, in specific implementations, the architecture of the computer device 600 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 600 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0203] This disclosure also provides a camera parameter calibration device. Figure 7 shows a schematic diagram of an exemplary device 700 provided in this disclosure. As shown in Figure 7, the device 700 can be used to implement method 300 and may further include the following modules.

[0204] The receiving module 702 is configured to receive multiple images obtained by at least two cameras capturing images of a calibration reference object; the wearable device further includes a binocular display module, the lens barrel being disposed on the light-emitting side of the binocular display module, an optical component being disposed inside the lens barrel, and the at least two cameras being located between the binocular display module and the optical component; the first determining module 704 is configured to determine the projection relationship between the pixels of the multiple images and the calibration reference object; the second determining module 706 is configured to determine a first parameter set and a second parameter set of the at least two cameras based on the projection relationship.

[0205] The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

[0206] In some embodiments, the at least two cameras include a first camera and a second camera for acquiring images of a first eye, and a third camera and a fourth camera for acquiring images of a second eye. The first camera and the second camera are disposed in a first lens barrel of the wearable device, and the third camera and the fourth camera are disposed in a second lens barrel of the wearable device. The second parameter includes the pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera. The pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera are calculated based on the projection relationship using a dual-target positioning algorithm.

[0207] In some embodiments, the second parameter further includes the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera.

[0208] In some embodiments, the second determining module 706 is configured to: determine the pose relationship between the first camera and the fourth camera based on the plurality of images and the projection relationship; determine the pose relationship between the first camera and the third camera based on the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera; determine the pose relationship between the second camera and the third camera based on the pose relationship between the first camera and the second camera and the pose relationship between the first camera and the third camera; and determine the pose relationship between the second camera and the fourth camera based on the pose relationship between the third camera and the fourth camera and the pose relationship between the second camera and the third camera.

[0209] In some embodiments, the first determining module 704 is configured to: select a first number of target images from the plurality of images; select at least one local region in each of the target images and establish a homography transformation matrix between each local region and the calibration reference object; determine the projection relationship between the pixels contained in the local region and the calibration reference object according to the homography transformation matrix; transform the first number of target images to a reference coordinate system, and determine the projection relationship between the pixels of the plurality of images and the calibration reference object according to the regions corresponding to the local regions with established projection relationships.

[0210] In some embodiments, the second determining module 706 is configured to: determine an initial first parameter set of the at least two cameras according to the projection relationship, the initial first parameter set including a plurality of first parameters corresponding one-to-one with a plurality of pixels of the image; fit the initial first parameter set with a spline surface to obtain a fitted spline surface; and optimize the control points of the fitted spline surface based on a bundle adjustment algorithm to obtain the first parameter set of the at least two cameras.

[0211] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0212] The apparatus of the above embodiments is used to implement the corresponding method 300 in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0213] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method 300 as described in any of the above embodiments.

[0214] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0215] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the method 300 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0216] Based on the same inventive concept, corresponding to the method 300 of any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the method 300. Corresponding to the execution entity for each step in each embodiment of method 300, the processor executing the corresponding step may belong to the corresponding execution entity.

[0217] The computer program products of the above embodiments are used to cause the computer and / or the processor to perform the method 300 as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0218] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0219] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this disclosure, the provided drawings may or may not show well-known power / ground connections to integrated circuit (IC) chips and other components. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this disclosure, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this disclosure will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuitry) have been set forth to describe exemplary embodiments of this disclosure, it will be apparent to those skilled in the art that the embodiments of this disclosure may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0220] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0221] This disclosure is intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for calibrating camera parameters, comprising: Receive multiple images of a calibration reference object captured by at least two cameras, wherein the at least two cameras are disposed within the lens barrel of the wearable device; The wearable device also includes a binocular display module, the lens barrel is disposed on the light-emitting side of the binocular display module, an optical component is disposed inside the lens barrel, and the at least two cameras are located between the binocular display module and the optical component; Determine the projection relationship between the pixels of the multiple images and the calibration reference object; Based on the projection relationship, determine the first parameter set and the second parameter set of the at least two cameras; The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

2. The method as described in claim 1, wherein, The at least two cameras include a first camera and a second camera for capturing images of the human eye for the first eye, and a third camera and a fourth camera for capturing images of the human eye for the second eye. The first camera and the second camera are disposed in the first lens barrel of the wearable device, and the third camera and the fourth camera are disposed in the second lens barrel of the wearable device. The second parameter includes the pose relationship between the first camera and the second camera, and the pose relationship between the third camera and the fourth camera. The pose relationship between the first camera and the second camera, and the pose relationship between the third camera and the fourth camera are calculated based on the projection relationship using a dual-camera positioning algorithm.

3. The method as described in claim 2, wherein, The second parameter also includes the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera.

4. The method of claim 3, wherein, Determining the first parameter set and the second parameter set of the at least two cameras based on the projection relationship includes: Based on the multiple images and the projection relationship, the pose relationship between the first camera and the fourth camera is determined; Based on the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera, the pose relationship between the first camera and the third camera is determined; Based on the pose relationship between the first camera and the second camera, and the pose relationship between the first camera and the third camera, determine the pose relationship between the second camera and the third camera; The pose relationship between the second camera and the fourth camera is determined based on the pose relationship between the third camera and the fourth camera, and the pose relationship between the second camera and the third camera.

5. The method of claim 1, wherein, Determining the projection relationship between the pixels of the plurality of images and the calibration reference object includes: Select a first number of target images from the plurality of images; At least one local region is selected in each target image, and a homography transformation matrix between each local region and the calibration reference object is established; The projection relationship between the pixels contained in the local region and the calibration reference object is determined based on the homography transformation matrix. The first number of target images are transformed into a reference coordinate system, and the projection relationship between the pixels of the multiple images and the calibration reference object is determined according to the regions with established projection relationships corresponding to the local regions.

6. The method of claim 1, wherein, Determining the first parameter set and the second parameter set of the at least two cameras based on the projection relationship includes: Based on the projection relationship, an initial first parameter set for the at least two cameras is determined, the initial first parameter set including multiple first parameters that correspond one-to-one with multiple pixels of the image; The initial first parameter set is fitted using a spline surface to obtain the fitted spline surface; The control points of the fitted spline surface are optimized based on the bundle adjustment algorithm to obtain the first parameter set of the at least two cameras.

7. A wearable device, comprising: Binocular display module; Two lens barrels are disposed on the light-emitting side of the binocular display module. At least one of the two lens barrels includes at least two cameras for acquiring human eye images and an optical component disposed inside the lens barrel. The at least two cameras are located between the binocular display module and the optical component.

8. The wearable device as claimed in claim 7, wherein, The binocular display module includes a first display module and a second display module, and the two lens barrels include: A first lens barrel is disposed on the light-emitting side of the first display module, and a first camera and a second camera are disposed inside the first lens barrel for capturing images of the human eye. The second lens barrel is located on the light-emitting side of the second display module, and a third camera and a fourth camera are installed inside the second lens barrel for capturing images of the human eye of the second eye.

9. The wearable device as claimed in claim 8, wherein, The first camera and the second camera are symmetrically arranged inside the first lens barrel with respect to the axis of the first lens barrel; The third camera and the fourth camera are symmetrically arranged inside the second lens barrel with respect to the axis of the second lens barrel.

10. The wearable device of claim 9, wherein, Both the first camera and the second camera are oriented toward the light-emitting side of the first display module, and the angles between the orientations of the first camera and the second camera and the axis of the first lens barrel are equal. Both the third camera and the fourth camera are oriented toward the light-emitting side of the second display module, and the angle between the orientation of the third camera and the fourth camera and the axis of the second lens barrel is equal.

11. The wearable device of claim 9, wherein, Both the first camera and the second camera face the first display module. The first lens barrel is provided with a first reflective structure corresponding to the first camera and a second reflective structure corresponding to the second camera. The first reflective structure is used to reflect light from the first eye into the first camera, and the second reflective structure is used to reflect light from the first eye into the second camera. Both the third camera and the fourth camera face the second display module. The second lens barrel is provided with a third reflection structure corresponding to the third camera and a fourth reflection structure corresponding to the fourth camera. The third reflection structure is used to reflect light from the second eye into the third camera, and the fourth reflection structure is used to reflect light from the second eye into the fourth camera.

12. A camera parameter calibration device, comprising: The receiving module is configured to receive multiple images of a calibration reference object captured by at least two cameras; the wearable device further includes a binocular display module, the lens barrel is disposed on the light-emitting side of the binocular display module, optical components are disposed inside the lens barrel, and the at least two cameras are located between the binocular display module and the optical components. The first determining module is configured to: determine the projection relationship between the pixels of the plurality of images and the calibration reference object; The second determining module is configured to: determine a first parameter set and a second parameter set of the at least two cameras based on the projection relationship; The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

13. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs comprising instructions for performing the method as claimed in any one of claims 1-6.

14. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by one or more processors, causes the processors to perform the method as described in any one of claims 1-6.

15. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images