Calibration method of camera parameters and related device

Abstract

The present disclosure provides a camera parameter calibration method and related equipment. The method comprises: receiving a plurality of images obtained by at least two cameras shooting a calibration reference object, the at least two cameras being arranged in a lens barrel of a wearable device; determining a projection relationship between a pixel point of the plurality of images and the calibration reference object; determining a first parameter set and a second parameter set of the at least two cameras according to the projection relationship; wherein the first parameter set comprises a plurality of first parameters, the first parameters being used to represent a target pixel point in an image shot by the camera to be calibrated and a projection direction corresponding to the target pixel point, and the second parameter set comprises at least one group of second parameters, the at least one group of second parameters being used to indicate a pose relationship between the at least two cameras.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of extended reality, and in particular, to a camera parameter calibration method and related equipment. BACKGROUND

[0002] Extended Reality (XR) refers to combining reality and virtuality through a computer to create a virtual environment that can be interacted with by a human. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), which uses hardware devices and various technical means to combine virtual content and real scenes.

[0003] Generally, an extended reality system provides a wearable device for a user to achieve human-computer interaction, which can be a head-mounted wearable device. In some scenarios, the wearable device can capture human eye images to perform calculations to achieve gaze tracking or interpupillary distance estimation functions.

[0004] However, the present disclosure finds that in the related art, the camera for capturing images is usually arranged outside the lens barrel, which limits the accuracy of the interpupillary distance estimation and gaze tracking algorithm. SUMMARY

[0005] The present disclosure provides a camera parameter calibration method and related equipment to solve or partially solve the above problems.

[0006] In a first aspect, the present disclosure provides a camera parameter calibration method, comprising:

[0007] receiving a plurality of images obtained by at least two cameras capturing a calibration reference object, the at least two cameras being arranged in a lens barrel of a wearable device; the wearable device further comprising a binocular display module, the lens barrel being arranged on an out-light side of the binocular display module, an optical assembly being arranged in the lens barrel, and the at least two cameras being located between the binocular display module and the optical assembly;

[0008] determining a projection relationship between pixel points of the plurality of images and the calibration reference object;

[0009] determining a first parameter set and a second parameter set of the at least two cameras according to the projection relationship;

[0010] The first parameter set includes a plurality of first parameters, and the first parameters are used to represent target pixel points in images captured by the to-be-calibrated cameras and projection directions corresponding to the target pixel points. The second parameter set includes at least one group of second parameters, and the at least one group of second parameters are used to indicate a pose relationship between the at least two cameras.

[0011] In a second aspect, the disclosure provides a wearable device, comprising:

[0012] a binocular display module;

[0013] two barrels disposed at an out-light side of the binocular display module, at least one of the two barrels comprising at least two cameras disposed inside the barrel for capturing images of human eyes and an optical assembly, the at least two cameras being located between the binocular display module and the optical assembly.

[0014] In a third aspect, the disclosure provides a camera parameter calibration device, comprising:

[0015] a receiving module configured to receive a plurality of images captured by at least two cameras of a calibration reference object; the wearable device further comprising a binocular display module, the barrel being disposed at an out-light side of the binocular display module, an optical assembly being disposed inside the barrel, the at least two cameras being located between the binocular display module and the optical assembly;

[0016] a first determining module configured to determine a projection relationship between pixels of the plurality of images and the calibration reference object;

[0017] a second determining module configured to determine a first parameter set and a second parameter set of the at least two cameras according to the projection relationship;

[0018] wherein the first parameter set comprises a plurality of first parameters, the first parameters being used to represent target pixels in an image captured by the camera to be calibrated and a projection direction corresponding to the target pixels, and the second parameter set comprises at least one group of second parameters, the at least one group of second parameters being used to indicate a pose relationship between the at least two cameras.

[0019] In a fourth aspect, the disclosure provides a wearable device, comprising:

[0020] a barrel, the barrel comprising a display module, an optical assembly, and at least two cameras, the at least two cameras and the optical assembly being located at an out-light side of the display module, the at least two cameras being located between the optical assembly and the display module;

[0021] a processing module electrically coupled to the at least two cameras and configured to obtain a first parameter set and a second parameter set obtained by the method of the first aspect and an image captured by the cameras; and solve a position of a target region in the image in space by using the first parameter set, the second parameter set, and the image.

[0022] In a fifth aspect, the present disclosure provides a computer device, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs comprise instructions for performing the method according to the first aspect.

[0023] In a sixth aspect, the present disclosure provides a non-transitory computer readable storage medium containing a computer program which, when executed by one or more processors, causes the processors to perform the method according to the first aspect.

[0024] In a seventh aspect, the present disclosure provides a computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method according to the first aspect.

[0025] The wearable device provided by the embodiments of the present disclosure sets the camera inside the lens barrel, is less affected by the obstruction, and can improve the accuracy of the pupil distance estimation or line-of-sight tracking algorithm. Further, by setting the camera to at least two, more observation information can be provided, and the algorithm accuracy is further improved. The camera parameter calibration method and related device provided by the embodiments of the present disclosure provide a feasible camera parameter calibration scheme for the scene of asymmetric camera imaging distortion in the lens barrel and no unified projection center. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the present disclosure or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0027] FIG. 1A A schematic diagram of an exemplary system provided by the embodiments of the present disclosure is shown.

[0028] FIG. 1B A schematic diagram of an exemplary head-mounted wearable device is shown.

[0029] FIG. 1C And FIG. 1D A schematic diagram of an exemplary human eye image is shown.

[0030] FIG. 2A A schematic diagram of an exemplary wearable device provided by the embodiments of the present disclosure is shown.

[0031] FIG. 2B A schematic diagram of another exemplary wearable device provided by the embodiments of the present disclosure is shown.

[0032] FIG. 2C A schematic diagram of yet another example wearable device provided by an embodiment of the present disclosure is shown.

[0033] FIG. 2D A schematic diagram of yet another example wearable device provided by an embodiment of the present disclosure is shown.

[0034] FIG. 3A A flowchart of an example method provided by an embodiment of the present disclosure is shown.

[0035] FIG. 3B A flowchart of an example method of determining a projection relationship according to an embodiment of the present disclosure is shown.

[0036] FIG. 4A A schematic diagram of an example image acquisition scenario according to an embodiment of the present disclosure is shown.

[0037] FIG. 4B A schematic diagram of another example image acquisition scenario according to an embodiment of the present disclosure is shown.

[0038] FIG. 4C A schematic diagram of three selected target images is shown.

[0039] FIG. 4D A schematic diagram of a projection relationship between a pixel point and a calibration reference object according to an embodiment of the present disclosure is shown.

[0040] FIG. 4E A schematic diagram of an example first parameter according to an embodiment of the present disclosure is shown.

[0041] FIG. 4F A schematic diagram of an example second parameter of an off-axis lens barrel camera according to an embodiment of the present disclosure is shown.

[0042] FIG. 4G A schematic diagram of an image obtained by binarization processing of an image acquired in an embodiment of the present disclosure is shown.

[0043] FIG. 5 A schematic diagram of an example wearable device provided by an embodiment of the present disclosure is shown.

[0044] FIG. 6 A hardware structure schematic diagram of an example computer device provided by an embodiment of the present disclosure is shown.

[0045] FIG. 7 A schematic diagram of an example apparatus provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to specific embodiments and in conjunction with the accompanying drawings.

[0047] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present disclosure should be understood as their common meanings to those skilled in the art to which the present disclosure pertains. The terms "first", "second", and similar terms used in the embodiments of the present disclosure do not denote any order, quantity, or importance, but are merely used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are merely used to indicate relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships can also change accordingly.

[0048] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, and the like of personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0049] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server, or a storage medium, which performs the operation of the technical solutions of the present disclosure, according to the prompt information.

[0050] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0051] It can be understood that the above notification and obtaining of user authorization process is only illustrative, and does not limit the implementation manner of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0052] It can be understood that the data involved in the present technical solutions (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0053] It can be understood that the data involved in the present technical solutions (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.FIG. 1A A schematic diagram of an exemplary extended reality system 100 provided in an embodiment of this disclosure is shown.

[0054] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating an interactive virtual environment. XR technology can further include Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR), utilizing hardware devices and various technologies to merge virtual content with real-world scenes.

[0055] like FIG. 1A As shown, the system 100 may include various types of wearable devices, such as head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMDs)) 104, control handles 108, etc. In some scenarios, a camera / camera 110 may also be provided for taking photos of the operator (user) 102. In some embodiments, when the aforementioned devices do not have processing functions, the system 100 may also include an external control device 112 for providing processing functions. The control device 112 may be, for example, a mobile phone, computer, or other computer device. In some embodiments, when any of the aforementioned devices acts as a control device or a main control device, it can interact with other devices in the system 100 through wired or wireless communication methods to achieve information exchange.

[0056] In system 100, user 102 can interact with extended reality system 100 using head-mounted wearable device 104 and control handle 108. In some scenarios, system 100 can use images captured by camera / video camera 110 to recognize user 102's posture, gestures, etc., and then complete the interaction with user 102 based on the recognized posture and gestures. In some embodiments, user 130 can also perform gesture input using bare hands. Head-mounted wearable device 104 can capture images in front of it in real time using a camera or other camera positioned in front of the head-mounted wearable device 104, and recognize user 130's gestures by recognizing these images.

[0057] In some embodiments, such as FIG. 1A As shown, system 100 can also communicate with server 114 and obtain data from server 114, such as images, audio, and video, and can output this data through head-mounted wearable device 104, for example, displaying images or videos on the display screen of head-mounted wearable device 104, playing audio and video audio through the speaker of head-mounted wearable device 104, etc. In some embodiments, such as FIG. 1AAs shown, the server 114 can retrieve the required data, e.g., pictures, audio, video, etc., from a database server 116 for storing data.

[0058] In some embodiments, the head-mounted wearable device 104 can be provided with a collection unit for collecting information. The type of the collection unit can be various.

[0059] In some embodiments, the collection unit can further include an environment acquisition unit for collecting environment information around (e.g., in front of) the wearable device 104 and a positioning tracking unit for positioning tracking of the wearable device 104. Optionally, the environment acquisition unit can include, but is not limited to, a three-color camera (e.g., an RGB camera), a depth camera, a binocular camera, a laser, etc. light-sensitive element, and the positioning tracking unit can include, but is not limited to, a visual simultaneous localization and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), an ultra-wideband wireless communication technology (UWB), a laser, etc. module.

[0060] In some embodiments, the head-mounted wearable device 104 can also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting speed information or acceleration information of the head-mounted wearable device 104. For example, the operation handle 108 can also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc. for collecting speed information or acceleration information of the operation handle 108. It should be noted that the aforementioned collection unit can be provided on the head-mounted wearable device 104 and the operation handle 108, or can be directly attached to the body part of the interactive user 102 without relying on the hardware device, so as to collect the relevant information of the body part, such as speed or acceleration or angular velocity information, or other information collected by the sensor or collection unit.

[0061] In some embodiments, the head-mounted wearable device 104 can also be provided with a camera or a camera for taking pictures of the operator (user) 102 (e.g., pictures of hands or feet) and environment images.

[0062] In some embodiments, the system 100 can identify the posture, gesture, etc. of the user 102 through the collected information, and then can perform corresponding interaction according to the identified user posture and gesture.

[0063] FIG. 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0064] As FIG. 1BAs shown, the head-mounted wearable device 104 can include a lens barrel 1042, inside which a display screen 1044 for displaying images and an optical assembly 1046 for processing light paths can be arranged. Optionally, the optical assembly 1046 can further include a plurality of lenses (e.g., lenses 1046A and 1046B), which can project light emitted by the display screen 1044 into the human eye 1022 so that the human eye 1022 can view the picture displayed by the display screen 1044. It can be understood that, FIG. 1B Only a single-side structure of the head-mounted wearable device 104 is exemplarily shown in the figures, and to realize binocular display, two lens barrel structures arranged side by side can be included in the head-mounted wearable device 104.

[0065] In some embodiments, as FIG. 1B shown, the head-mounted wearable device 104 can further be provided with a camera 1048 for capturing human eye images, which can be a charge-coupled device (CCD) image sensor, a complementary metal-oxide-semiconductor (CMOS) image sensor, or the like.

[0066] Optionally, the camera 1048 can be an eye tracking (ET) camera, and the human eye images captured thereby can be used to realize functions such as pupil distance estimation and eye tracking.

[0067] As FIG. 1B shown, in the related art, the camera 1048 is usually arranged outside the lens barrel and is usually arranged in correspondence with only one lens barrel. Moreover, to better capture complete human eye images and not affect the human eye's viewing of the picture displayed by the display screen 1044, the common deployment position of the camera 1048 is usually at the outer canthus or the nasal ala. Referring to FIG. 1B shown, if the camera 1048 is close to the outer side of the device, FIG. 1B the camera deployment position shown is the outer canthus, if the camera 1048 is close to the inner side of the device, FIG. 1B the camera deployment position shown is the nasal ala.

[0068] However, the inventors of the present disclosure have found that the way of installing the camera in the related art tends to cause a large installation inclination angle of the camera 1048 relative to the human eye 1022, resulting in a large included angle a between the orientation of the camera 1048 and the forward direction of the human eye 1022, so that the captured human eye images are difficult to reflect the images of the forward viewing angle of the human eye, as FIG. 1C and FIG. 1D shown.

[0069] In some cases, the user can need to wear glasses first and then use the head-mounted wearable device 104. However, since the camera 1048 is arranged outside the lens barrel 1042, the camera 1048 is higher than the lens barrel 1042, which is easy to squeeze the glasses and affect the wearing comfort of the head-mounted wearable device 104. At the same time, the imaging of the camera 1048 is easily affected by the edge of the glasses, and the light passing through the edge of the glasses is refracted, which reduces the image clarity of the camera 1048 and forms many refracted spots in the image, thereby affecting the accuracy of the subsequent algorithm. In particular, when the camera corresponding to the lens barrel is only one, such problems will be further aggravated.

[0070] Therefore, the embodiments of the present disclosure provide a wearable device, at least two cameras are arranged inside the lens barrel, which can solve or partially solve the above problems to some extent.

[0071] FIG. 2A A schematic diagram of an exemplary wearable device 200 provided by the embodiments of the present disclosure is shown.

[0072] As shown in FIG. 2A Similarly, the wearable device 200 can also include a lens barrel 202, and a display module 204 and an optical assembly 206 arranged inside the lens barrel 202. The optical assembly 206 can further include a plurality of lenses (for example, lenses 2062 and 2064), and the combination of the plurality of lenses can project the light emitted by the display module 204 into the human eye 1022, so that the human eye 1022 can watch the picture displayed by the display module 204.

[0073] Unlike the wearable device 104 shown in FIG. 1B The wearable device 200 includes two cameras 208A and 208B, and the two cameras 208A and 208B are arranged inside the lens barrel 202. Since the cameras 208A and 208B are placed inside the lens barrel 202, they will not affect the wearing of glasses, thereby improving the comfort of the wearable device 200. At the same time, as FIG. 2AAs shown, since the cameras 208A and 208B are arranged inside the lens barrel 202, the distance from the cameras 208A and 208B to the human eye 1022 is extended, so that the installation inclination angle of the cameras 208A and 208B relative to the human eye 1022 is smaller, and the angle β between the direction of the cameras 208A and 208B and the normal direction of the human eye 1022 is smaller than the angle α, so that the cameras 208A and 208B have a better observation angle, and the collected human eye image can better reflect the image of the normal viewing angle, and the imaging quality is better. Moreover, since the cameras 208A and 208B are arranged inside the lens barrel 202, the glasses will not interfere with the imaging of the cameras 208A and 208B, further improving the imaging quality. The improvement of the imaging quality improves the accuracy of the pupil distance estimation or the line of sight tracking algorithm. In addition, since two cameras 208A and 208B are used, more observation information (more images collected) can be provided, and the accuracy of the pupil distance estimation or the line of sight tracking algorithm can be improved.

[0074] In some embodiments, in order to improve the accuracy of subsequent pupil distance estimation or line of sight tracking algorithms, as shown in FIG. 2B, the two cameras 208A and 208B can be arranged symmetrically inside the lens barrel 202 relative to the axis (the center dotted line) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2A FIG. 2A In some embodiments, as shown in FIG. 2C, the two cameras 208A and 208B can be arranged symmetrically inside the lens barrel 202 relative to the axis (the center dotted line) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0075] In some embodiments, as shown in FIG. 2D, the two cameras 208A and 208B can be arranged symmetrically inside the lens barrel 202 relative to the axis (the center dotted line) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2A FIG. 2A In some embodiments, as shown in FIG. 2E, the two cameras 208A and 208B can be arranged symmetrically inside the lens barrel 202 relative to the axis (the center dotted line) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0076] FIG. 2B FIG. 2E shows a schematic diagram of another exemplary wearable device 200 provided by the embodiments of the present disclosure.

[0077] As shown in FIG. 2F, the two cameras 208A and 208B can be arranged symmetrically inside the lens barrel 202 relative to the axis (the center dotted line) of the lens barrel 202. In this way, the images collected by the two cameras 208A and 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2B ​​As shown, in some embodiments, the wearable device 200 can further include two reflective structures 210A and 210B, which can be reflective films or mirrors or other structures with reflective surfaces. The reflective structures 210A and 210B can correspond to the cameras 208A and 208B respectively, both of which are directed towards the display module 204, and the reflective structure 210A is used to reflect the light from the human eye 1022 into the camera 208A, and the reflective structure 210B is used to reflect the light from the human eye 1022 into the camera 208B. Through the reflection of the light by the reflective structures 210A and 210B, the cameras 208A and 208B can still capture the human eye images. Moreover, because a reflection process is added in the light path, the observation angle γ is further reduced, and the cameras 208A and 208B can better perform imaging.

[0078] In some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or gaze tracking, as shown in FIG. 2B, the wearable device 200 can further include a light source 212, which can be a light-emitting diode or other light-emitting structure. The light source 212 can be disposed on the mirror barrel 202, and the light emitted by the light source 212 can be reflected by the reflective structures 210A and 210B and then enter the human eye 1022. The light source 212 can be disposed on the mirror barrel 202 in a manner symmetrical to the cameras 208A and 208B, and the light emitted by the light source 212 can be reflected by the reflective structures 210A and 210B and then enter the human eye 1022. In this way, the cameras 208A and 208B can capture images of the human eye 1022 under the same light conditions, and the subsequent algorithm processing efficiency can be further improved. FIG. 2B As shown, the two cameras 208A and 208B can be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202, and the reflective structures 210A and 210B can also be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202. In this way, the images captured by the two cameras 208A and 208B can be symmetrical, and the subsequent algorithm processing efficiency can be further improved. FIG. 2B FIG. 2B As shown, the two cameras 208A and 208B can be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202, and the reflective structures 210A and 210B can also be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202. In this way, the images captured by the two cameras 208A and 208B can be symmetrical, and the subsequent algorithm processing efficiency can be further improved.

[0079] As shown, the two cameras 208A and 208B can be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202, and the reflective structures 210A and 210B can also be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202. In this way, the images captured by the two cameras 208A and 208B can be symmetrical, and the subsequent algorithm processing efficiency can be further improved. FIG. 2B FIG. 2B As shown, the two cameras 208A and 208B can be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202, and the reflective structures 210A and 210B can also be symmetrically disposed in the mirror barrel 202 relative to the axis (the center dotted line) of the mirror barrel 202. In this way, the images captured by the two cameras 208A and 208B can be symmetrical, and the subsequent algorithm processing efficiency can be further improved. FIG. 2B

[0080] The foregoing embodiments are described with two cameras disposed in a single mirror barrel, it can be understood that when the number of cameras is further increased, the observation data can be further increased, thereby further improving the algorithm accuracy, therefore, the embodiments of disposing two or more cameras in a single mirror barrel should all belong to the protection scope of the present disclosure.

[0081] FIG. 2A and FIG. 2B ​​​It is to be understood that the single-side structure of the wearable device 200 is shown only by way of example, and that two barrel structures arranged side by side can be included in the wearable device 200 to realize binocular display.

[0082] FIG. 2C A schematic diagram of another exemplary wearable device 200 is shown.

[0083] As shown in FIG. 2C , the wearable device 200 can include a binocular display module, which can further include a first display module 204A and a second display module 204B. The first display module 204A and the second display module 204B can display images for a first eye 1022A (e.g., a right eye) and a second eye 1022B (e.g., a left eye), respectively.

[0084] In some embodiments, as shown in FIG. 2C , the wearable device 200 can further include two barrels, e.g., a first barrel 202A and a second barrel 202B, arranged on the light-out side of the binocular display module. At least one of the two barrels can further include at least two cameras arranged inside the barrel to collect images of the human eye, thereby providing more observation data and improving the accuracy of subsequent algorithms.

[0085] As an optional embodiment, as shown in FIG. 2C , the first barrel 202A can be arranged on the light-out side of the first display module 204A, and the first barrel 202A can further include a first camera 208A and a second camera 208B arranged inside the first barrel 202A to collect images of the first eye 1022A (e.g., a right eye). In this way, the collected images of the first eye 1022A (e.g., a right eye) can provide more accurate calculation results when used in subsequent calculations of interpupillary distance or line-of-sight tracking. In some embodiments, as shown in FIG. 2C , the first barrel 202A can further include a first optical assembly 206A to project the images displayed by the first display module 204A into the first eye 1022A by optical principles. Optionally, the first optical assembly 206A can further include a first lens 2062A and a second lens 2064A, which can have different or identical parameters and can be designed according to actual needs. It is to be understood that the types and numbers of lenses in the first optical assembly 206A are variable, and the types and numbers of lenses used can be designed according to actual needs.

[0086] Similarly, as another optional embodiment, as shown in FIG. 2CAs shown, the second lens barrel 202B can be disposed at the light exit side of the second display module 204B, and the third camera 208C and the fourth camera 208D for collecting the image of the second eye 1022B can be further disposed in the second lens barrel 202B. In this way, the image of the second eye 1022B (for example, the left eye) collected by the first camera 208A and the second camera 208B can be more accurate when subsequent calculations are performed for calculating the interpupillary distance or tracking the line of sight. Similarly, in some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D can be disposed in the second lens barrel 202B in a symmetrical manner with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2C As shown, the second lens barrel 202B can be disposed at the light exit side of the second display module 204B, and the third camera 208C and the fourth camera 208D for collecting the image of the second eye 1022B can be further disposed in the second lens barrel 202B. In this way, the image of the second eye 1022B (for example, the left eye) collected by the first camera 208A and the second camera 208B can be more accurate when subsequent calculations are performed for calculating the interpupillary distance or tracking the line of sight. Similarly, in some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D can be disposed in the second lens barrel 202B in a symmetrical manner with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0087] In some embodiments, as shown in FIG. 2A, the first camera 208A and the second camera 208B can be disposed in the first lens barrel 202A in a symmetrical manner with respect to the axis of the first lens barrel 202A. In this way, the images of the first eye 1022A collected by the first camera 208A and the second camera 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2C Similarly, in some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D can be disposed in the second lens barrel 202B in a symmetrical manner with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0088] FIG. 2C Similarly, in some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D can be disposed in the second lens barrel 202B in a symmetrical manner with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0089] In some embodiments, as shown in FIG. 2A, the first camera 208A and the second camera 208B can be disposed in the first lens barrel 202A in a symmetrical manner with respect to the axis of the first lens barrel 202A. In this way, the images of the first eye 1022A collected by the first camera 208A and the second camera 208B can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved. FIG. 2C Similarly, in some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D can be disposed in the second lens barrel 202B in a symmetrical manner with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B collected by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of the subsequent algorithm can be further improved.

[0090] FIG. 2C ​​As shown, both the third camera 208C and the fourth camera 208D are oriented towards the light-emitting side of the second display module 204B, and the angles between the orientations of the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, β. In this way, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be strictly symmetrical, further improving the processing efficiency of subsequent algorithms.

[0091] FIG. 2D A schematic diagram of yet another exemplary wearable device 200 provided in this disclosure embodiment is shown.

[0092] In some embodiments, such as FIG. 2D As shown, both the first camera 208A and the second camera 208B face the first display module 204A. The first lens barrel 202A contains a first reflective structure 210A corresponding to the first camera 208A and a second reflective structure 210B corresponding to the second camera 208B. The first reflective structure 210A and the second reflective structure 210B can be structures with reflective surfaces, such as reflective films or mirrors. The first reflective structure 210A and the second reflective structure 210B correspond to the first camera 208A and the second camera 208B, respectively. The first reflective structure 210A reflects light from the first eye 1022A into the first camera 208A, and the second reflective structure 210B reflects light from the first eye 1022A into the second camera 208B. Through the reflection of light by the first reflective structure 210A and the second reflective structure 210B, the first camera 208A and the second camera 208B can still capture the human eye image of the first eye 1022A. Furthermore, because an additional reflection process is added to the optical path, the observation angle γ is further reduced, enabling the first camera 208A and the second camera 208B to better image the first eye 1022A.

[0093] Similarly, in some embodiments, such as FIG. 2DAs shown, the third camera 208C and the fourth camera 208D are both directed towards the second display module 204B, and the third reflection structure 210C corresponding to the third camera 208C and the fourth reflection structure 210D corresponding to the fourth camera 208D are arranged in the second lens barrel 202B. The third reflection structure 210C and the fourth reflection structure 210D can be a structure with a reflecting surface such as a reflecting film or a mirror. The third reflection structure 210C and the fourth reflection structure 210D can correspond to the third camera 208C and the fourth camera 208D respectively. The third reflection structure 210C is used to reflect the light from the second eye 1022B into the third camera 208C, and the fourth reflection structure 210D is used to reflect the light from the second eye 1022B into the fourth camera 208D. The reflection of the light by the third reflection structure 210C and the fourth reflection structure 210D enables the third camera 208C and the fourth camera 208D to still capture the image of the second eye 1022B. Moreover, because the reflection process is added in the light path, the observation angle γ is further reduced, and the third camera 208C and the fourth camera 208D can better image the second eye 1022B.

[0094] In some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or gaze tracking, the first camera 208A and the second camera 208B can be arranged in the first lens barrel 202A to be symmetrical with respect to the axis of the first lens barrel 202A, and the first reflection structure 210A and the second reflection structure 210B can also be arranged in the first lens barrel 202A to be symmetrical with respect to the axis of the first lens barrel 202A. In this way, the images of the first eye 1022A captured by the first camera 208A and the second camera 208B can be symmetrical, and the processing efficiency of subsequent algorithms can be further improved. FIG. 2D

[0095] Similarly, in some embodiments, in order to improve the accuracy of subsequent algorithms such as pupil distance estimation or gaze tracking, the third camera 208C and the fourth camera 208D can be arranged in the second lens barrel 202B to be symmetrical with respect to the axis of the second lens barrel 202B, and the third reflection structure 210C and the fourth reflection structure 210D can also be arranged in the second lens barrel 202B to be symmetrical with respect to the axis of the second lens barrel 202B. In this way, the images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be symmetrical, and the processing efficiency of subsequent algorithms can be further improved. FIG. 2D

[0096] In some embodiments, as shown in FIG. 2, the first camera 208A and the second camera 208B can be arranged in the first lens barrel 202A to be symmetrical with respect to the axis of the first lens barrel 202A, and the first reflection structure 210A and the second reflection structure 210B can also be arranged in the first lens barrel 202A to be symmetrical with respect to the axis of the first lens barrel 202A. In this way, the images of the first eye 1022A captured by the first camera 208A and the second camera 208B can be symmetrical, and the processing efficiency of subsequent algorithms can be further improved. FIG. 2D ​​As shown, the first camera 208A and the second camera 208B are both oriented towards the first display module 204A, and the angles between the orientations of the first camera 208A and the second camera 208B and the axis of the first lens barrel 202A are equal, for example, both angles are γ; meanwhile, the angles between the orientations of the first reflecting structure 210A and the second reflecting structure 210B and the axis of the first lens barrel 202A are also equal. In this way, the human eye images of the first eye 1022A captured by the first camera 208A and the second camera 208B can be strictly symmetrical, and the processing efficiency of subsequent algorithms can be further improved.

[0097] In some embodiments, as shown in FIG. 2B, the third camera 208C and the fourth camera 208D are both oriented towards the second display module 204B, and the angles between the orientations of the third camera 208C and the fourth camera 208D and the axis of the second lens barrel 202B are equal, for example, both angles are γ; meanwhile, the angles between the orientations of the third reflecting structure 210C and the fourth reflecting structure 210D and the axis of the second lens barrel 202B are also equal. In this way, the human eye images of the second eye 1022B captured by the third camera 208C and the fourth camera 208D can be strictly symmetrical, and the processing efficiency of subsequent algorithms can be further improved. FIG. 2D The foregoing embodiments are all described by taking two cameras arranged in each of the two lens barrels, it can be understood that one of the two lens barrels can be arranged with two or more cameras, and the other lens barrel can be arranged with one camera, in this case, the algorithm accuracy can still be improved to a certain extent, which should also belong to the protection scope of the present disclosure. In addition,

[0098] and FIG. 2A The structures shown in FIG. 2B and FIG. 2C can also be matched in the same wearable device 200, for example, the lens barrel corresponding to the first eye has the structure shown in FIG. 2B, and the lens barrel corresponding to the second eye has the structure shown in FIG. 2C. Of course, the structures corresponding to the first eye and the second eye can also be exchanged. FIG. 2B FIG. 2A FIG. 2B

[0099] As can be seen from the foregoing embodiments, the present disclosure arranges at least two cameras 208 in the interior of at least one lens barrel of the wearable device 200, which can obtain better imaging quality and improve the comfort of the wearable device 200. In some embodiments, by arranging two cameras in each of the left and right lens barrels of the wearable device 200, the foregoing adverse effects of cameras outside the lens barrels can be avoided, the two cameras can provide more observation information, and the influence of occlusion is smaller, so the accuracy of algorithms such as pupil distance estimation or gaze tracking can be improved.

[0100] ​​​Furthermore, the inventors of this disclosure have discovered that, compared to placing the camera outside the lens barrel, placing the camera inside the lens barrel results in the camera being affected not only by the camera module itself but also by the optical components within the lens barrel when it is imaging.

[0101] like FIG. 2A and FIG. 2B As shown, relative to FIG. 1B In the optical path from cameras 208A and 208B to the human eye 1022, camera 1048 also contains an optical component 206 or a portion thereof within the lens barrel (this varies depending on the relative positions of the lenses in the optical component 206 and cameras 208A and 208B; for example, besides...). FIG. 2A and FIG. 2B As shown, cameras 208A and 208B can also be positioned between lenses 2062 and 2064, so that when cameras 208A and 208B image the human eye 1022, optical component 206 or a part of optical component 206 affects or changes the light path from cameras 208A and 208B to the human eye 1022, which may cause the distortion of the image obtained by cameras 208A and 208B to no longer be asymmetrical.

[0102] Furthermore, because the light paths of cameras 208A and 208B to the human eye 1022 are altered, cameras 208A and 208B do not have a unified projection center, making it impossible to use a parametric camera model to fit the projection process of the camera inside the lens barrel to process the distortion, thus making it difficult to calibrate the camera parameters.

[0103] In addition, in some cases, such as FIG. 2C and FIG. 2D As shown, since the different cameras are positioned differently within the wearable device 200, it is also necessary to calibrate the pose relationships between the different cameras, which is quite difficult to achieve. In particular, when the different cameras are located within different lens barrels (for example, between the first camera 208A and the third camera 208C or the fourth camera 208D, or between the second camera 208B and the third camera 208C or the fourth camera 208D), such calibration becomes even more complex.

[0104] In view of this, embodiments of this disclosure also provide a method for calibrating camera parameters, which can use a non-parametric camera model to calibrate the parameters of a camera within a lens barrel. In some embodiments, for cameras located in different lens barrels, a camera extrinsic parameter calibration algorithm based on non-common viewpoints is proposed, thereby solving the problem of difficult calibration of cameras in different lens barrels.

[0105] FIG. 3A A flowchart illustrating an exemplary method 300 provided in an embodiment of this disclosure is shown.

[0106] This method 300 can be applied to any computer device with data processing capabilities, and can be used for... FIG. 2A , FIG. 2B , FIG. 2C or FIG. 2D The parameters of cameras 208A, 208B, 208C, or 208D in the wearable device 200 shown are calibrated. For example... FIG. 3A As shown, the method 300 may further include the following steps.

[0107] In step 302, multiple images of the calibration reference object captured by at least two cameras can be received. In this step, the multiple images can be images acquired by different cameras. As an optional embodiment, to calibrate the first parameter (e.g., intrinsic parameter) and the second parameter (e.g., extrinsic parameter) of the cameras, multiple images can be acquired separately for intrinsic parameter calibration and multiple images can be acquired separately for extrinsic parameter calibration.

[0108] FIG. 4A A schematic diagram of an exemplary image acquisition scenario 400 according to an embodiment of the present disclosure is shown.

[0109] like FIG. 4A As shown, in the image acquisition scenario 400, a calibration reference object 402 can be set at a fixed position. In some embodiments, the calibration reference object 402 can be a calibration board or a chart, on which a black and white checkerboard pattern is set. When the camera captures an image of the calibration reference object 402, the corresponding camera parameters can be calculated based on the correspondence between the checkerboard pattern in the image and the checkerboard pattern of the calibration reference object 402.

[0110] When acquiring images of the calibration reference object 402, the camera to be calibrated (e.g., camera 208A) can be controlled to move along a certain trajectory, and the calibration reference object 402 can be continuously photographed during the camera's movement. This results in multiple images acquired by the camera in different poses, which are then used as data for calculating camera parameters. Since the acquired images are obtained by the camera in multiple poses, the calculated camera parameters are more robust and applicable to a wider range.

[0111] In some embodiments, if the at least two cameras disposed in the wearable device are cameras of different types or cameras with different camera parameters, then for each type of camera or each type of parameter camera, a plurality of images need to be captured in the above-described manner for calibrating the first parameters (e.g., intrinsic parameters) of the corresponding camera. If the camera parameters of the at least two cameras disposed in the wearable device are consistent, then one camera can be selected to implement the aforementioned image capturing operation, thereby for calibrating the intrinsic parameters of the camera, and the calibrated intrinsic parameters can be applicable to other cameras in the wearable device.

[0112] FIG. 4B A schematic diagram of another exemplary image capturing scenario 410 is shown.

[0113] As FIG. 4B shown, similar to the image capturing scenario 400, in the image capturing scenario 410, a calibration reference object 402 can also be disposed at a fixed position.

[0114] Unlike the image capturing scenario 400, in order to calculate the pose relationship (second parameter) between the at least two cameras in the wearable device 200, the at least two cameras can be mounted in the lens barrels of the wearable device 200 and the complete assembly of the wearable device 200 is completed, and then the extrinsic parameter image capturing is performed by using the assembled wearable device 200 or its prototype, and then the corresponding extrinsic parameters (e.g., the pose relationship between different cameras) are calculated based on the captured images.

[0115] As FIG. 4B shown, taking an example of disposing two cameras in two lens barrels of the wearable device 200, when capturing the calibration reference object 402 (or image capturing), the wearable device 200 can be controlled to move along a certain motion trajectory, and the four cameras can be used to continuously capture the calibration reference object 402 in the motion process of the wearable device 200, thereby obtaining a plurality of images captured by the four cameras in different poses as image data for subsequent calculation of the second parameters, under the condition that the relative pose relationship of the four cameras is unchanged. Since the captured images are obtained by image capturing of the cameras in multiple poses, the robustness of the camera parameters calculated subsequently is better, and the camera parameters can be applicable to a wider range.

[0116] According to the above description, it can be understood that in the image capturing scenario 410, each camera can capture a plurality of images, and the images can be used for calculating the relative pose relationship of the four cameras, and the images captured by each camera can also be used for calculating the intrinsic parameters of the camera itself. Therefore, in some embodiments, the image capturing scenario 410 can be used to capture images, and the first parameters and the second parameters of the cameras can be calibrated based on the images.

[0117] In step 302, the computer device can receive a plurality of images captured in the aforementioned scene 400 and / or scene 410 for subsequent processing.

[0118] After the plurality of images are captured, the camera can be parameter calibrated based on the plurality of images. The parameter calibration can include calibrating first parameters and second parameters, wherein the first parameters can be intrinsic parameters of the camera, and the second parameters can be extrinsic parameters of the camera.

[0119] In some embodiments, the intrinsic parameters of the camera can be calibrated first. However, as mentioned before, since the camera is arranged in the lens barrel, the camera imaging is not only affected by the camera module itself, but also by the lens of the lens barrel, so that the imaging distortion is no longer symmetric about the principal point of the image, and the camera does not have a unified projection center, which leads to the fact that the asymmetric distortion cannot be processed by using a parameterized camera model to fit the projection process of the camera in the lens barrel. Therefore, in some embodiments, a non-parametric camera model is provided to calibrate the intrinsic parameters of the camera.

[0120] Therefore, in step 304, the projection relationship between the pixel points of the plurality of images and the calibration reference object can be determined first. In this step, the projection relationship between the image and the calibration reference object 402 can be established according to the captured image, so as to establish the corresponding relationship between the pixel points and the pixel-ray.

[0121] It can be understood that the projection relationship between the pixel points of the image and the calibration reference object represents the intrinsic parameters (i.e., intrinsic parameters) of the camera. Therefore, when calculating the projection relationship of a specific camera (e.g., camera 208A), the image captured by the specific camera needs to be used to establish the projection relationship. Therefore, taking the image capturing scene shown in FIG. 4B As an example, in the image capturing scene shown in

[0122] In the following, the first camera 208A is taken as an example to calculate the projection relationship.

[0123] In some embodiments, as shown in FIG. 3B The step 304 of determining the projection relationship between the pixel points of the plurality of images and the calibration reference object can further include the following steps:

[0124] In step 3042, a first number of target images are selected from the plurality of images.

[0125] In this step, a projection relationship can be established by selecting a certain number of target images from the plurality of first images (images captured by the first camera 208A) in the plurality of images, and the remaining first images can be used to supplement the part for which the projection relationship is not established.

[0126] It can be understood that the first number is sufficient to continue the subsequent steps, and the number is not specifically limited. As an optional embodiment, the first number can be 3. FIG. 4C Three target images 412, 414, and 416 are selected as shown. FIG. 4C As shown, the target images 412, 414, and 416 respectively show images of the calibration reference object 402 captured by the first camera 208A in different poses. It can be understood that by calibrating the camera parameters of the images captured in different camera poses, the robustness of the calculated camera parameters can be better.

[0127] In step 3044, at least one local region (for example, a region corresponding to one grid in the chessboard can be considered as a local region) is selected in each of the target images, and a homography transformation matrix of each of the local regions and the calibration reference object is established.

[0128] After selecting at least one local region of each target image, for each local region of each target image, a homography transformation matrix with the calibration reference object 402 can be constructed respectively to establish a corresponding relationship between each pixel point contained in the local region and the coordinate system of the calibration reference object 402.

[0129] It can be known that the coordinate system of the position pair of the calibration reference object 402 is known, and the chessboard on the calibration reference object 402 has a corresponding relationship with the chessboard image in the target image, and the coordinates of the four vertices corresponding to the local region in the camera coordinate system of the target image are also known. According to these known information, the homography transformation matrix H of the local region and the calibration reference object 402 can be determined.

[0130] In this way, the pixel points contained in the local region are corresponded to the calibration reference object 402 or the coordinate system of the calibration reference object 402.

[0131] In step 3046, a projection relationship of the pixel points contained in the local region and the calibration reference object is determined according to the homography transformation matrix.

[0132] FIG. 4D A schematic diagram of the projection relationship of the pixel points and the calibration reference object according to an embodiment of the present disclosure is shown.

[0133] As shown in FIG. 4B, the projection relationship of the pixel points and the calibration reference object is shown. FIG. 4DAs shown, the region corresponding to the four vertices connected by the four dashed lines in the figure is the local region. By constructing the homography transformation matrix, the projection relationship between each pixel in the local region of the target image and the corresponding point on the calibration reference object can be calculated.

[0134] In step 3048, the first number of target images are transformed into the reference coordinate system, and the projection relationship between the pixels of the multiple images and the calibration reference object is determined according to the region with established projection relationship corresponding to the local region.

[0135] like FIG. 4C As shown, the three target images 412, 414, and 416 were acquired under different camera poses. In the previous step of constructing the homography transformation matrix of the local region, it was constructed using the camera coordinate system corresponding to each target image. It can be understood that in order to unify the projection relationship of each target image under the same reference coordinate system, in this step, the three target images 412, 414, and 416 can be transformed to the reference coordinate system.

[0136] After processing the three target images 412, 414, and 416 according to the aforementioned steps, a similar method can be used to select local regions of new target images in the remaining first images to construct homography transformation matrices until the projection relationship between each pixel in the entire image and the calibration reference object is completed.

[0137] As can be seen from the foregoing embodiments, in some embodiments, after all image processing and calibration are completed, some pixels may correspond to multiple projection direction data. Therefore, these data can be averaged to obtain a single projection direction, which can then be used as the projection direction for that pixel to obtain better projection direction data. Furthermore, during subsequent storage, only this averaged projection direction can be stored, thereby saving storage space.

[0138] It is understood that the aforementioned method can be used to establish the projection relationship between image pixels and calibration reference object 402 for the second camera 208B, the third camera 208C, the fourth camera 208D, and possibly more cameras (depending on the number of cameras in the wearable device), which will not be elaborated here.

[0139] After the projection relationship is established, in step 306, a first parameter set and a second parameter set of the at least two cameras can be determined according to the projection relationship. The first parameter set includes a plurality of first parameters, which are used to represent target pixel points in an image captured by the camera to be calibrated and corresponding projection directions of the target pixel points. The second parameter set includes at least one group of second parameters, which are used to indicate a pose relationship between the at least two cameras.

[0140] In some embodiments, the first parameter set of the at least two cameras can be determined according to the projection relationship first, and then the second parameter set is determined.

[0141] FIG. 4E A schematic diagram of an example first parameter according to an embodiment of the present disclosure is shown.

[0142] As FIG. 4E shown, when the projection relationship is constructed, at least some pixel points in the image under the reference coordinate system correspond to some points on the calibration reference object one by one, so that a straight line equation about the pixel points can be obtained. The straight line represented by the straight line equation passes through the corresponding pixel points and has a ray as shown in the figure to represent the projection direction. Such a combination of pixel points and projection directions can be used as the first parameter (which can be considered as the intrinsic parameter of the camera). The camera parameter is used to realize the conversion of the coordinate system of the image captured by the camera to the camera coordinate system, so that it can be used to calculate the position information of some features (for example, the pupil) in the three-dimensional space (the camera coordinate system) in the image captured by the camera. Moreover, since the homography transformation matrix is established by using a plurality of local regions of the first image to respectively establish a homography transformation matrix with the calibration reference object in the calculation process of the camera parameter, when the coordinate system conversion calculation is performed by using the camera parameter, the projection direction corresponding to the pixel point has already included the correction of the distortion of the position image, and additional distortion correction is not needed.

[0143] In some embodiments, the data obtained in the foregoing can be simplified to serve as the first parameter of the camera, so that the space for storing the first parameter can be saved. For example, the initial set of the first parameter obtained by the calibration can be fitted by using a spline surface, and then the control points of the fitted spline surface are optimized by using a bundle adjustment (BA) algorithm. All control points of the optimized spline surface correspond to the first parameter set of the at least two cameras.

[0144] When the first parameter is needed to be used, the complete set of the first parameter can be obtained by using an interpolation algorithm according to the fitted spline surface and the optimized control points.

[0145] It can be seen that the foregoing method only gives the calculation method of the first parameter of a single camera, and the first parameter of each camera in the wearable device 200 can be calculated by using the foregoing method, which will not be described here.

[0146] After obtaining the first parameter set, the second parameter set can be further calibrated.

[0147] Returning to FIG. 2C , the wearable device 200 includes a first camera 208A and a second camera 208B arranged in the first lens barrel 202A for collecting the image of the first eye 1022A and a third camera 208C and a fourth camera 208D arranged in the second lens barrel 202B for collecting the image of the second eye 1022B. Since the positions of the four cameras in the wearable device 200 are different, in order to determine the relative relationship of the images collected by the four cameras, the pose relationship between the four cameras needs to be known as the second parameter, so as to complete the calibration of the camera extrinsic parameters.

[0148] Next, taking the calibration of the four cameras in the wearable device 200 as an example, the calculation of the second parameter is exemplarily described.

[0149] As shown in FIG. 2C , since the two cameras arranged in the same lens barrel of the wearable device 200 need to collect images of the same human eye, generally, the two cameras have a common view point (or light path intersection point), and therefore, a double target calibration algorithm can be used to obtain the pose relationship between the two cameras in the same lens barrel.

[0150] Therefore, in some embodiments, the second parameter can include the pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D, and the pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera are calculated based on a double target calibration algorithm according to the projection relationship.

[0151] Optionally, according to the projection relationship, the step 306 of determining the first parameter set and the second parameter set of the at least two cameras can further include: determining the pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D according to the projection relationship by using a double target calibration algorithm.

[0152] Continuing to refer to FIG. 2CIt can be understood that, since the cameras in different lens barrels capture images of different human eyes, the cameras in different lens barrels can not have a common view point, and the pose relationship calculated by using the two-target positioning algorithm can not be accurate. Therefore, the embodiments of the present disclosure provide a method for calculating the pose relationship between cameras in different lens barrels.

[0153] FIG. 4F A calibration diagram of the second parameters of the cameras in different lens barrels is shown.

[0154] As shown in FIG. 4F , taking the calculation of the pose relationship between the first camera 208A and the fourth camera 208D as an example, the two cameras do not have a common view point, and the relative positions of the two cameras and the calibration reference object 402 when the two cameras move to a certain position are as shown in FIG. 4F The pose relationship of the first camera 208A relative to the calibration reference object 402 is T 1，i , the pose relationship of the fourth camera 208D relative to the calibration reference object 402 is T 4，i , and the pose relationship between the first camera 208A and the fourth camera 208D is T 14 Therefore, by using the observation data (the first image set and the fourth image set obtained in the image capturing step) of the first camera 208A and the fourth camera 208D, a BA optimization problem can be constructed, and the pose relationship T 14 is obtained by solving the problem.

[0155] Therefore, in some embodiments, the second parameters can further include the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera.

[0156] Optionally, the step 306 of determining the first parameter set and the second parameter set of the at least two cameras according to the projection relationship can further include: determining the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera according to the plurality of images and the projection relationship.

[0157] Specifically, an optimization function can be constructed first.

[0158] Optionally, the optimization function can include a first formula for representing an error between a detected point in a first image captured by the first camera 208A and a two-dimensional point in an image coordinate system into which the detected point is projected by a first parameter of the first camera 208A corresponding to the detected point and a spatial position corresponding to the detected point on the calibration reference object 402. Optionally, the first formula is represented as:

[0159] f cam1,1 = π1(T 1，i , P cam1 ) - d cam1

[0160] wherein π1 is a projection function corresponding to the first camera 208A (i.e., the projection relationship between the pixel point and the calibration reference object 402 obtained above), T 1，i is an external parameter of the first camera 208A corresponding to the i-th first image and the calibration reference object 402 (which can be calculated according to the first image and the spatial coordinates of the calibration reference object 402), P cam1 is a 3D point on the calibration reference object 402 corresponding to the i-th first image of the first camera 208A (i.e., the spatial coordinates of the point on the calibration reference object 402 corresponding to each pixel point of the first image), d cam1 is all detected points corresponding to the first camera 208A.

[0161] Optionally, the optimization function can further include a second formula for representing an error between a detected point in a fourth image captured by the fourth camera 208D and a two-dimensional point in an image coordinate system into which the detected point is projected by a first parameter of the fourth camera 208A corresponding to the detected point and a spatial position corresponding to the detected point on the calibration reference object 402. Optionally, the second formula is represented as:

[0162] f cam4,2 = π4(T 4，i , P cam4 ) - d cam4

[0163] wherein π4 is a projection function corresponding to the fourth camera 208D (i.e., the projection relationship between the pixel point and the calibration reference object 402 obtained above), T 4，i is an external parameter of the fourth camera 208D corresponding to the i-th fourth image and the calibration reference object 402 (which can be calculated according to the fourth image and the coordinate information of the calibration reference object 402), P cam4 is a 3D point on the calibration reference object 402 corresponding to the i-th fourth image of the fourth camera 208D (i.e., the spatial coordinates of the point on the calibration reference object 402 corresponding to each pixel point of the fourth image), d cam4 is all detected points corresponding to the fourth camera 208D.

[0164] Optionally, the optimization function can further include a third formula for representing an error between a detection point in a fourth image captured by the fourth camera 208D and a two-dimensional point in the image coordinate system, which is projected by the detection point corresponding first camera 208A first parameter and the pose relationship between the first camera and the fourth camera, to the spatial position corresponding to the detection point (the corresponding position on the calibration reference object 402). Optionally, the third formula is represented as:

[0165] f cam4,1 = π4(T 14 *T 1，i , P cam4 )-d cam4

[0166] Wherein, T 14 is the pose relationship between the first camera 208A and the fourth camera 208D, that is, the extrinsic parameters of the first camera and the fourth camera required to be solved by the embodiment of the present disclosure.

[0167] Optionally, the optimization function can further include a fourth formula for representing an error between a detection point in a fourth image captured by the fourth camera 208D and a two-dimensional point in the image coordinate system, which is projected by the detection point corresponding fourth camera first parameter and the pose relationship between the first camera and the fourth camera, to the spatial position corresponding to the detection point (the corresponding position on the calibration reference object 402). Optionally, the fourth formula is represented as:

[0168]

[0169] Wherein, is the transpose matrix of the pose relationship between the first camera 208A and the fourth camera 208D, which is obtained by transposing T 14 .

[0170] In some embodiments, the detection points d cam1 and d cam4 can be calculated in the following way.

[0171] FIG. 4G A schematic diagram of an image obtained by binarizing the image collected in the embodiment of the present disclosure is shown.

[0172] As shown in FIG. 4G , after binarization processing, the pixel points on the image are either black or white. By feature detection, the vertices of the checkboard grid can be identified, and these identifiable vertices can be used as detection points d. The detection points detected on the ith image can be represented as d i .

[0173] Optionally, the optimization function can be further determined based on the first formula, the second formula, the third formula, and the fourth formula, and the error function is represented as:

[0174] f = f cam1,1 + f cam4,1 + f cam1,2 + f cam4,2

[0175] Finally, the optimization function can be constructed based on the error function to determine the final extrinsic parameters of the first camera 208A and the fourth camera 208D. Optionally, the optimization function is represented as follows:

[0176]

[0177] wherein p is a loss function, f is the error function O i is all information contained in the ith image, and I is the number of input images.

[0178] It can be seen that the loss function res(p, T) is a function of the pose. By solving the loss function using the Levenberg-Marquardt (LM) method, the optimal solution obtained is the final pose relationship T 14 .

[0179] It can be understood that for two cameras in the opposite barrel, the above method can be used to obtain the pose relationship, which will not be described here.

[0180] Considering that there are many ways to arrange and combine the opposite cameras, if the above method is used to calculate the pose relationship for each pair of cameras, the calculation amount will be increased. Therefore, in some embodiments, according to the plurality of images and the projection relationship, the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera, include:

[0181] According to the plurality of images and the projection relationship, the pose relationship between the first camera and the fourth camera is determined.

[0182] According to the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera, the pose relationship between the first camera and the third camera is determined.

[0183] According to the pose relationship between the first camera and the second camera and the pose relationship between the first camera and the third camera, the pose relationship between the second camera and the third camera is determined.

[0184] According to the pose relationship between the third camera and the fourth camera and the pose relationship between the second camera and the third camera, the pose relationship between the second camera and the fourth camera is determined.

[0185] In this way, after the pose relationship between the first camera 208A and the fourth camera 208D is calculated, based on the already calculated pose relationship between the first camera 208A and the second camera 208B and the pose relationship between the third camera 208C and the fourth camera 208D, the pose relationship of other permutations and combinations can be obtained through data conversion, thereby saving the calculation amount.

[0186] It can be understood that the above method provides a calculation method that can be used when there is no common view point between the two cameras in the heterolateral lens barrel, but when there is a common view point between the heterolateral cameras, the pose relationship can still be calculated using the double target determination algorithm.

[0187] It should be noted that the above examples are only used as an example to illustrate that two cameras are arranged in two lens barrels of the wearable device 200 respectively, and it can be understood that the number of cameras in the lens barrel can be less or more according to different actual needs, but no matter how, based on the inventive concept of the embodiments provided by the present disclosure, the corresponding camera parameters can still be calculated, and details are not repeated here.

[0188] In a more specific embodiment, the camera parameter calibration method provided by the embodiments of the present disclosure can include the steps of image acquisition, intrinsic parameter calibration, extrinsic parameter calibration, optimization, and parameter saving, and can obtain better camera parameters for subsequent algorithm calculation.

[0189] As can be seen from the above embodiments, the camera parameter calibration method provided by the embodiments of the present disclosure provides a feasible calibration scheme for the intrinsic and extrinsic parameters of the camera for the scene with asymmetric imaging distortion of the cameras in the lens barrel and no unified projection center.

[0190] It should be noted that the method of the embodiments of the present disclosure can be executed by a single device, such as a computer or a server. The method of the present embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0191] It should be noted that the above description describes some embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0192] This disclosure also provides a wearable device.

[0193] FIG. 5 A schematic diagram of an exemplary wearable device 500 provided in an embodiment of this disclosure is shown.

[0194] like FIG. 5 As shown, with FIG. 2A and FIG. 2B Similarly, the wearable device 500 includes a lens barrel 202, in which a display module 204, cameras 208A and 208B, and an optical component 206 are disposed. The cameras 208A and 208B and the optical component 206 are located on the light-emitting side of the display module 204, and the cameras 208A and 208B are located between the optical component 206 and the display module 204.

[0195] In some embodiments, the shooting direction of the cameras 208A and 208B is directed toward the optical component 206, with reference to... FIG. 2A As shown. In other embodiments, reference is made to... FIG. 2B As shown, reflective structures 210A and 210B are provided between the display module 204 and the cameras 208A and 208B. The reflective surfaces of the reflective structures 210A and 210B face the light-emitting side of the display module 204, and the shooting direction of the cameras 208A and 208B faces the reflective structures 210A and 210B.

[0196] Furthermore, such as FIG. 5 As shown, the wearable device 500 also includes a processing module 502, which is electrically coupled to the camera 208 and the display module 204 respectively, and is configured to: acquire the first parameter set and the second parameter set obtained by the aforementioned method 300, as well as the images captured by the cameras 208A and 208B; and use the first parameter set, the second parameter set, and the images to solve for the spatial position of the target region in the images.

[0197] As mentioned earlier, the first parameter set includes camera parameters (which can be considered intrinsic parameters of the camera) that are combinations of pixel points and projection direction. These camera parameters are used to transform the coordinate system of the image acquired by the camera to the camera coordinate system, thereby enabling the calculation of the positional information of certain features (e.g., pupils) in three-dimensional space (camera coordinate system) within the image acquired by the camera. Furthermore, since the calculation of these camera parameters includes correction for image distortion, distortion can be directly corrected when performing coordinate system transformation calculations using these camera parameters, eliminating the need for additional distortion correction.

[0198] The second parameter set includes a second parameter representing the pose relationship between different cameras. This second parameter can be used to unify the image information acquired by different cameras (e.g., unify them to the same camera coordinate system).

[0199] This disclosure also provides a computer device for implementing the method 300 described above. FIG. 6 A schematic diagram of the hardware structure of an exemplary computer device 600 provided in an embodiment of this disclosure is shown. The computer device 600 can be used to implement... FIG. 1A 104. Head-mounted wearable devices FIG. 2A to FIG. 2D The wearable device 200 can also be used to achieve FIG. 1A External device 112 can also be used to achieve FIG. 1A Server 114. In some scenarios, this computer device 600 can also be used to implement... FIG. 1A Database server 116.

[0200] like FIG. 6 As shown, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608, and a bus 610. The processor 602, memory 604, network module 606, and peripheral interface 608 are interconnected within the computer device 600 via the bus 610.

[0201] Processor 602 may be a central processing unit (CPU), image processor, neural network processor (NPU), microcontroller (MCU), programmable logic device, digital signal processor (DSP), application-specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 can be used to perform functions related to the techniques described in this disclosure. In some embodiments, processor 602 may also include multiple processors integrated as a single logic component. For example, such as... FIG. 6As shown, processor 602 may include multiple processors 602a, 602b and 602c.

[0202] Memory 604 can be configured to store data (e.g., instructions, computer code, etc.). FIG. 6 As shown, the data stored in memory 604 may include program instructions (e.g., program instructions for implementing methods 300 or 500 of embodiments of this disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). Processor 602 may also access the program instructions and data stored in memory 604 and execute the program instructions to operate on the data to be processed. Memory 604 may include volatile or non-volatile storage devices. In some embodiments, memory 604 may include random access memory (RAM), read-only memory (ROM), optical disk, magnetic disk, hard disk, solid-state drive (SSD), flash memory, memory stick, etc.

[0203] Network interface 606 can be configured to provide communication with other external devices to computer device 600 via a network. This network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, Near Field Communication (NFC), etc.), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the specific examples described above.

[0204] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to enable information input and output. For example, peripheral devices may include input devices such as keyboards, mice, touchpads, touch screens, microphones, and various sensors, as well as output devices such as displays, speakers, vibrators, and indicator lights.

[0205] Bus 610 can be configured to transfer information between various components of computer device 600 (e.g., processor 602, memory 604, network interface 606, and peripheral interface 608), such as internal buses (e.g., processor-memory bus), external buses (USB port, PCI-E bus), etc.

[0206] It should be noted that although the architecture of the computer device 600 described above only shows the processor 602, memory 604, network interface 606, peripheral interface 608, and bus 610, in specific implementations, the architecture of the computer device 600 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the architecture of the computer device 600 described above may only include the components necessary for implementing the embodiments of this disclosure, and does not necessarily include all the components shown in the figures.

[0207] The embodiment of the present disclosure further provides a camera parameter calibration device. FIG. 7 A schematic diagram of an exemplary device 700 provided by the embodiment of the present disclosure is shown. As shown, the device 700 can be used to implement the method 300, and can further include the following modules. FIG. 7

[0208] The receiving module 702 is configured to receive a plurality of images obtained by at least two cameras shooting a calibration reference object; the wearable device further includes a binocular display module, the lens barrel is arranged on the light-emitting side of the binocular display module, an optical assembly is arranged in the lens barrel, and the at least two cameras are located between the binocular display module and the optical assembly.

[0209] The first determining module 704 is configured to determine a projection relationship between a pixel point of the plurality of images and the calibration reference object.

[0210] The second determining module 706 is configured to determine a first parameter set and a second parameter set of the at least two cameras according to the projection relationship.

[0211] The first parameter set includes a plurality of first parameters, and the first parameters are used to represent a target pixel point in an image shot by the camera to be calibrated and a projection direction corresponding to the target pixel point; the second parameter set includes at least one group of second parameters, and the at least one group of second parameters are used to indicate a pose relationship between the at least two cameras.

[0212] In some embodiments, the at least two cameras include a first camera and a second camera for collecting a first eye image of a human eye and a third camera and a fourth camera for collecting a second eye image of a human eye, the first camera and the second camera are arranged in a first lens barrel of the wearable device, and the third camera and the fourth camera are arranged in a second lens barrel of the wearable device.

[0213] The second parameters include a pose relationship between the first camera and the second camera and a pose relationship between the third camera and the fourth camera, and the pose relationship between the first camera and the second camera and the pose relationship between the third camera and the fourth camera are calculated based on a double target calibration algorithm according to the projection relationship.

[0214] In some embodiments, the second parameters further include a pose relationship between the first camera and the third camera, a pose relationship between the first camera and the fourth camera, a pose relationship between the second camera and the third camera, and a pose relationship between the second camera and the fourth camera.

[0215] In some embodiments, the second determining module 706 is configured to:​

[0216] determine a pose relationship between the first camera and the fourth camera according to the plurality of images and the projection relationship;

[0217] determine a pose relationship between the first camera and the third camera according to the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera;

[0218] determine a pose relationship between the second camera and the third camera according to the pose relationship between the first camera and the second camera and the pose relationship between the first camera and the third camera;

[0219] determine a pose relationship between the second camera and the fourth camera according to the pose relationship between the third camera and the fourth camera and the pose relationship between the second camera and the third camera.

[0220] In some embodiments, the first determining module 704 is configured to:

[0221] select a first number of target images from the plurality of images;

[0222] select at least one local region in each of the target images, and establish a homographic transformation matrix between each of the local regions and the calibration reference object;

[0223] determine a projection relationship between a pixel point contained in the local region and the calibration reference object according to the homographic transformation matrix;

[0224] convert the first number of target images to a reference coordinate system, and determine a projection relationship between a pixel point of the plurality of images and the calibration reference object according to a region corresponding to the local region for which the projection relationship has been established.

[0225] In some embodiments, the second determining module 706 is configured to:

[0226] determine an initial first parameter set of the at least two cameras according to the projection relationship, the initial first parameter set including a plurality of first parameters corresponding one-to-one to a plurality of pixel points of the images;

[0227] fit the initial first parameter set using a spline surface to obtain a fitted spline surface;

[0228] optimize control points of the fitted spline surface based on a bundle adjustment algorithm to obtain a first parameter set of the at least two cameras.

[0229] For ease of description, the above apparatus is described in various modules in terms of function to describe separately. Of course, in the implementation of the present disclosure, the functions of each module can be implemented in one or more software and / or hardware.

[0230] The apparatus of the above embodiments is used to implement the corresponding method 300 of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0231] Based on the same inventive concept, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method 300 of any of the above embodiments.

[0232] The computer-readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0233] The storage medium of the above embodiments stores computer instructions for causing the computer to perform the method 300 of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0234] Based on the same inventive concept, the present disclosure also provides a computer program product including computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the method 300. Corresponding to the execution subject of each step in each embodiment of the method 300, the processor performing the corresponding step can belong to the corresponding execution subject.

[0235] The computer program product of the above embodiments is used to cause the computer and / or the processor to perform the method 300 of any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0236] Those of ordinary skill in the art will realize that the foregoing discussion of any of the embodiments has been presented for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms described, and that various alternatives, modifications, and variations can be employed without departing from the spirit or scope of the disclosure as set forth in the claims. Examples of such alternate, modified, and varying embodiments have been discussed above in conjunction with the material discussed above.

[0237] In addition, to simplify the description and discussion, and so as not to make the embodiments of the disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. Further, devices can be shown in block diagram form in order to avoid making the embodiments of the disclosure difficult to understand, and this also takes into account the fact that details regarding implementation of these block diagram devices are highly dependent on the platform to which the embodiments of the disclosure are to be implemented (i.e., these details should be well within the understanding of one of ordinary skill in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the disclosure, it should be apparent to one of ordinary skill in the art that the embodiments of the disclosure can be practiced without or with variation of these specific details. Thus, these descriptions should not be construed as limiting, but merely as descriptive of illustrative embodiments of the disclosure.

[0238] While the disclosure has been described in connection with specific embodiments thereof, it will be understood that many modifications, variations and alternatives will be apparent to those skilled in the art as a result of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0239] The embodiments of the disclosure are intended to cover all such alternatives, modifications, and variations as falling within the broad scope of the appended claims. Accordingly, any one of the steps of the embodiments of the disclosure can be performed in any order, and many of the steps can be performed in any order or in parallel, unless otherwise specified.

Claims

1. A method for calibrating camera parameters, comprising: Receive multiple images of a calibration reference object captured by at least two cameras, wherein the at least two cameras are disposed within the lens barrel of the wearable device; The wearable device also includes a binocular display module, the lens barrel is disposed on the light-emitting side of the binocular display module, an optical component is disposed inside the lens barrel, and the at least two cameras are located between the binocular display module and the optical component; Determine the projection relationship between the pixels of the multiple images and the calibration reference object; Based on the projection relationship, determine the first parameter set and the second parameter set of the at least two cameras; The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

2. The method as described in claim 1, wherein, The at least two cameras include a first camera and a second camera for capturing images of the human eye for the first eye, and a third camera and a fourth camera for capturing images of the human eye for the second eye. The first camera and the second camera are disposed in the first lens barrel of the wearable device, and the third camera and the fourth camera are disposed in the second lens barrel of the wearable device. The second parameter includes the pose relationship between the first camera and the second camera, and the pose relationship between the third camera and the fourth camera. The pose relationship between the first camera and the second camera, and the pose relationship between the third camera and the fourth camera are calculated based on the projection relationship using a dual-camera positioning algorithm.

3. The method as described in claim 2, wherein, The second parameter also includes the pose relationship between the first camera and the third camera, the pose relationship between the first camera and the fourth camera, the pose relationship between the second camera and the third camera, and the pose relationship between the second camera and the fourth camera.

4. The method of claim 3, wherein, Determining the first parameter set and the second parameter set of the at least two cameras based on the projection relationship includes: Based on the multiple images and the projection relationship, the pose relationship between the first camera and the fourth camera is determined; Based on the pose relationship between the third camera and the fourth camera and the pose relationship between the first camera and the fourth camera, the pose relationship between the first camera and the third camera is determined; Based on the pose relationship between the first camera and the second camera, and the pose relationship between the first camera and the third camera, determine the pose relationship between the second camera and the third camera; The pose relationship between the second camera and the fourth camera is determined based on the pose relationship between the third camera and the fourth camera, and the pose relationship between the second camera and the third camera.

5. The method of claim 1, wherein, Determining the projection relationship between the pixels of the plurality of images and the calibration reference object includes: Select a first number of target images from the plurality of images; At least one local region is selected in each target image, and a homography transformation matrix between each local region and the calibration reference object is established; The projection relationship between the pixels contained in the local region and the calibration reference object is determined based on the homography transformation matrix. The first number of target images are transformed into a reference coordinate system, and the projection relationship between the pixels of the multiple images and the calibration reference object is determined according to the regions with established projection relationships corresponding to the local regions.

6. The method of claim 1, wherein, Determining the first parameter set and the second parameter set of the at least two cameras based on the projection relationship includes: Based on the projection relationship, an initial first parameter set for the at least two cameras is determined, the initial first parameter set including multiple first parameters that correspond one-to-one with multiple pixels of the image; The initial first parameter set is fitted using a spline surface to obtain the fitted spline surface; The control points of the fitted spline surface are optimized based on the bundle adjustment algorithm to obtain the first parameter set of the at least two cameras.

7. A wearable device, comprising: Binocular display module; Two lens barrels are disposed on the light-emitting side of the binocular display module. At least one of the two lens barrels includes at least two cameras for acquiring human eye images and an optical component disposed inside the lens barrel. The at least two cameras are located between the binocular display module and the optical component. The wearable device further includes a processing module electrically coupled to the at least two cameras and the binocular display module, and is configured to: acquire a first parameter set and a second parameter set as well as images captured by the at least two cameras; and use the first parameter set, the second parameter set, and the images to determine the spatial position of a target region in the images. The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

8. The wearable device as claimed in claim 7, wherein, The binocular display module includes a first display module and a second display module, and the two lens barrels include: A first lens barrel is disposed on the light-emitting side of the first display module, and a first camera and a second camera are disposed inside the first lens barrel for capturing images of the human eye. The second lens barrel is located on the light-emitting side of the second display module, and a third camera and a fourth camera are installed inside the second lens barrel for capturing images of the human eye of the second eye.

9. The wearable device as claimed in claim 8, wherein, The first camera and the second camera are symmetrically arranged inside the first lens barrel with respect to the axis of the first lens barrel; The third camera and the fourth camera are symmetrically arranged inside the second lens barrel with respect to the axis of the second lens barrel.

10. The wearable device of claim 9, wherein, Both the first camera and the second camera are oriented toward the light-emitting side of the first display module, and the angles between the orientations of the first camera and the second camera and the axis of the first lens barrel are equal. Both the third camera and the fourth camera are oriented toward the light-emitting side of the second display module, and the angle between the orientation of the third camera and the fourth camera and the axis of the second lens barrel is equal.

11. The wearable device of claim 9, wherein, Both the first camera and the second camera face the first display module. The first lens barrel is provided with a first reflective structure corresponding to the first camera and a second reflective structure corresponding to the second camera. The first reflective structure is used to reflect light from the first eye into the first camera, and the second reflective structure is used to reflect light from the first eye into the second camera. Both the third camera and the fourth camera face the second display module. The second lens barrel is provided with a third reflection structure corresponding to the third camera and a fourth reflection structure corresponding to the fourth camera. The third reflection structure is used to reflect light from the second eye into the third camera, and the fourth reflection structure is used to reflect light from the second eye into the fourth camera.

12. A camera parameter calibration device, comprising: The receiving module is configured to receive multiple images of a calibration reference object captured by at least two cameras, the at least two cameras being disposed within the lens barrel of a wearable device; the wearable device further includes a binocular display module, the lens barrel being disposed on the light-emitting side of the binocular display module, an optical component being disposed within the lens barrel, and the at least two cameras being located between the binocular display module and the optical component; The first determining module is configured to: determine the projection relationship between the pixels of the plurality of images and the calibration reference object; The second determining module is configured to: determine a first parameter set and a second parameter set of the at least two cameras based on the projection relationship; The first parameter set includes multiple first parameters, which are used to characterize the target pixel in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel. The second parameter set includes at least one set of second parameters, which are used to indicate the pose relationship between the at least two cameras.

13. A computer device comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, the programs comprising instructions for performing the method as claimed in any one of claims 1-6.

14. A non-volatile computer-readable storage medium comprising a computer program, which, when executed by one or more processors, causes the processors to perform the method as described in any one of claims 1-6.

15. A computer program product comprising computer program instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-6.

Images