Camera parameter calibration method and related equipment

Abstract

The invention provides a camera parameter calibration method and related equipment. The camera parameter calibration method comprises the following steps: receiving a plurality of images obtained by shooting a calibration reference object by a camera to be calibrated; determining a projection relationship between pixel points of the plurality of images and the calibration reference object; according to the projection relation, a camera parameter set of the to-be-calibrated camera is determined, the camera parameter set comprises a plurality of camera parameters, and the camera parameters are used for representing pixel points in an image shot by the to-be-calibrated camera and projection directions corresponding to the pixel points.

Description

Technical Field

[0001] The present disclosure relates to the technical field of camera calibration, and in particular to a camera parameter calibration method and related equipment. Background Art

[0002] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging. These geometric model parameters are the camera parameters.

[0003] The inventors of the present disclosure have discovered that, in some scenarios, camera calibration is difficult to achieve using a traditional camera model. Summary of the invention

[0004] The present disclosure proposes a camera parameter calibration method and related devices to solve or partially solve the above problems.

[0005] In a first aspect, the present disclosure provides a method for calibrating camera parameters, comprising:

[0006] Receiving a plurality of images obtained by photographing a calibration reference object by a camera to be calibrated;

[0007] Determining a projection relationship between pixel points of the plurality of images and the calibration reference object;

[0008] According to the projection relationship, a camera parameter set of the camera to be calibrated is determined, where the camera parameter set includes a plurality of camera parameters, and the camera parameters are used to indicate a relationship between a pixel point and a projection direction.

[0009] In a second aspect of the present disclosure, a camera parameter calibration device is provided, comprising:

[0010] The receiving module is configured to: receive a plurality of images obtained by photographing the calibration reference object by the camera to be calibrated;

[0011] A first determination module is configured to: determine a projection relationship between pixel points of the plurality of images and the calibration reference object;

[0012] The second determination module is configured to: determine a camera parameter set of the camera to be calibrated according to the projection relationship, wherein the camera parameter set includes multiple camera parameters, and the camera parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points.

[0013] In a third aspect of the present disclosure, a wearable device is provided, comprising:

[0014] A lens barrel, wherein a display screen, a camera, and an optical component are arranged in the lens barrel, the camera and the optical component are located on one side of the light emitting direction of the display screen, and the camera is located between the optical component and the display screen;

[0015] The processing module is electrically coupled to the camera and is configured to: obtain a camera parameter set obtained by the method of the first aspect and an image taken by the camera; and use the camera parameter set and the image to solve the position of the target area in the image in space.

[0016] In a fourth aspect of the present disclosure, a computer device is provided, comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect.

[0017] According to a fifth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.

[0018] In a sixth aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.

[0019] The wearable device provided by the embodiment of the present disclosure sets the camera inside the lens barrel, which can solve or partially solve the above problems to a certain extent. The camera parameter calibration method and related equipment provided by the embodiment of the present disclosure provide a feasible camera parameter calibration solution for the scene where the camera imaging distortion in the lens barrel is asymmetric and there is no unified projection center. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1A A schematic diagram showing an exemplary system provided by an embodiment of the present disclosure is shown.

[0022] Figure 1B A schematic diagram of an exemplary head-mounted wearable device is shown.

[0023] Figure 1C and Figure 1DA schematic diagram showing an exemplary human eye image.

[0024] Figure 2A A schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure is shown.

[0025] Figure 2B A schematic diagram of another exemplary wearable device provided by an embodiment of the present disclosure is shown.

[0026] Figure 3A A flowchart of an exemplary method provided by an embodiment of the present disclosure is shown.

[0027] Figure 3B A flowchart of an exemplary method for determining a projection relationship according to an embodiment of the present disclosure is shown.

[0028] Figure 3C A schematic flow chart of an exemplary method for converting a coordinate system according to an embodiment of the present disclosure is shown.

[0029] Figure 3D A flowchart of another exemplary method for determining a projection relationship according to an embodiment of the present disclosure is shown.

[0030] Figure 3E A flowchart of an exemplary method for determining a camera parameter set according to an embodiment of the present disclosure is shown.

[0031] Figure 3F A flowchart of an exemplary method for optimizing control points according to an embodiment of the present disclosure is shown.

[0032] Figure 4A A schematic diagram of an exemplary image collection scenario according to an embodiment of the present disclosure is shown.

[0033] Figure 4B A schematic diagram showing three selected first images is shown.

[0034] Figure 4C A schematic diagram showing the projection relationship between pixel points and a calibration reference object according to an embodiment of the present disclosure is shown.

[0035] Figure 4D A schematic diagram showing exemplary camera parameters according to an embodiment of the present disclosure is shown.

[0036] Figure 4E A schematic diagram showing an image obtained after binarization processing of an image collected in an embodiment of the present disclosure is shown.

[0037] Figure 5 A schematic diagram of an exemplary wearable device provided by an embodiment of the present disclosure is shown.

[0038] Figure 6A schematic diagram of the hardware structure of an exemplary computer device provided in an embodiment of the present disclosure is shown.

[0039] Figure 7 A schematic diagram showing an exemplary device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0041] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0042] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0043] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0044] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0045] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0046] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0047] Figure 1A A schematic diagram of an exemplary extended reality system 100 provided by an embodiment of the present disclosure is shown.

[0048] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR (extended reality) technology can further include augmented reality (AR), virtual reality (VR), and mixed reality (MR), using hardware devices combined with a variety of technical means to integrate virtual content and real scenes.

[0049] like Figure 1A As shown, the system 100 may include various types of wearable devices, such as head-mounted wearable devices (e.g., VR / AR glasses or head-mounted displays (HMD)) 104, operating handles 108, etc. In some scenarios, a camera / camera 110 for taking photos of the operator (user) 102 may also be provided. In some embodiments, when the aforementioned device does not have a processing function, the system 100 may also include an external control device 112 for providing a processing function. The control device 112, for example, may be a computer device such as a mobile phone or a computer. In some embodiments, when any of the aforementioned devices is used as a control device or a main control device, it can communicate with other devices in the system 100 through wired or wireless communication methods to achieve information interaction.

[0050] In the system 100, the user 102 can use the head-mounted wearable device 104 and the operating handle 108 to interact with the extended reality system 100. In some scenarios, the system 100 can use the images captured by the camera / camera 110 to recognize the posture and gesture of the user 102, and then complete the interaction with the user 102 based on the recognized posture and gesture. In some embodiments, the user 130 can also implement gesture input through bare hands, and the head-mounted wearable device 104 can collect the front image in real time through the camera or camera set in front of the head-mounted wearable device 104, and recognize the gesture of the user 130 by recognizing the image.

[0051] In some embodiments, Figure 1AAs shown, the system 100 can also communicate with the server 114, and can obtain data from the server 114, such as pictures, audio, video, etc., and can output these data through the head-mounted wearable device 104, for example, displaying pictures or videos on the display screen of the head-mounted wearable device 104, using the speakers of the head-mounted wearable device 104 to play audio and audio carried by the video, etc. In some embodiments, as Figure 1A As shown, the server 114 can retrieve required data, such as pictures, audio, video, etc., from a database server 116 for storing data.

[0052] In some embodiments, a collection unit for collecting information may be provided on the head-mounted wearable device 104. The types of the collection unit may be various.

[0053] In some embodiments, the acquisition unit may further include an environment acquisition unit and a positioning tracking unit, wherein the environment acquisition unit may be used to acquire environment information around (for example, in front of) the wearable device 104, and the positioning tracking unit may be used to position and track the wearable device 104. Optionally, the environment acquisition unit may include but is not limited to a three-color camera (for example, an RGB camera), a depth camera, a binocular camera, a laser and other photosensitive elements, and the positioning tracking unit may include but is not limited to visual simultaneous positioning and mapping (visual SLAM), an inertial measurement unit (IMU), a global positioning system (GPS), an ultra-wideband wireless communication technology (UWB), a laser and other modules.

[0054] In some embodiments, the head-mounted wearable device 104 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc., for collecting speed information or acceleration information of the head-mounted wearable device 104. For another example, the operating handle 108 may also be provided with a speed sensor, an acceleration sensor, an angular velocity sensor (e.g., a gyroscope), etc., for collecting speed information or acceleration information of the operating handle 108. It should be noted that, in addition to being provided on the head-mounted wearable device 104 and the operating handle 108, the aforementioned collection unit may also be provided on a body part of the interactive user 102 directly by attachment without relying on hardware devices, so as to collect relevant information of the body part, such as speed, acceleration, or angular velocity information, or information collected by other sensors or collection units.

[0055] In some embodiments, the head-mounted wearable device 104 may also be provided with a camera or a camera for taking photos of the operator (user) 102 (eg, photos of the hands or feet) and images of the environment.

[0056] In some embodiments, the system 100 can identify the posture and gestures of the user 102 through the collected information, and then can perform corresponding interactions based on the identified user posture and gestures.

[0057] Figure 1B A schematic diagram of an exemplary head-mounted wearable device 104 is shown.

[0058] like Figure 1B As shown, the head-mounted wearable device 104 may include a lens barrel 1042, and a display screen 1044 for displaying images and an optical component 1046 for processing the light path may be arranged inside the lens barrel 1042. Optionally, the optical component 1046 may further include a plurality of lenses (e.g., lenses 1046A and 1046B), and the combination of the plurality of lenses may project the light emitted by the display screen 1044 into the human eye 1022, so that the human eye 1022 may see the image displayed on the display screen 1044. It is understood that Figure 1B Only a single-side structure of the head-mounted wearable device 104 is exemplarily shown. In order to achieve binocular display, the head-mounted wearable device 104 may include two lens barrel structures arranged in parallel.

[0059] In some embodiments, Figure 1B As shown, the head-mounted wearable device 104 may also be provided with a camera 1048 for collecting images of human eyes, and the camera 1048 may be a charge coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, or the like.

[0060] Optionally, the camera 1048 may be an eye tracking (ET) camera, and the human eye images collected by the camera may be used to implement pupil distance estimation and eye tracking functions, etc.

[0061] like Figure 1B As shown, in the related art, the camera 1048 is usually arranged outside the lens barrel. In addition, in order to better capture the complete human eye image and not affect the human eye's observation of the screen 1044, the common deployment position of the camera 1048 is generally at the outer corner of the eye or the nose wing. Figure 1B As shown, if the camera 1048 is close to the outside of the device, then Figure 1B The camera deployment position shown is the outer corner of the eye. If the camera 1048 is close to the inside of the device, then Figure 1B The camera deployment position shown is the nose wing position.

[0062] However, the inventors of the present disclosure have found that the camera installation method in the related art easily makes the camera 1048 installed at a large inclination angle relative to the human eye 1022, resulting in a large angle α between the direction of the camera 1048 and the normal viewing direction of the human eye 1022, making it difficult for the collected human eye image to reflect the image at the normal viewing angle of the human eye, such as Figure 1C and Figure 1D shown.

[0063] In some cases, the user may need to wear glasses before using the head-mounted wearable device 104. However, since the camera 1048 is disposed outside the lens barrel 1042, the camera 1048 is higher than the lens barrel 1042, which may easily squeeze the glasses and affect the wearing comfort of the head-mounted wearable device 104. At the same time, the imaging of the camera 1048 is easily affected by the edge of the glasses. The light refracts through the edge of the glasses, which reduces the clarity of the image of the camera 1048 and forms many refracted light spots in the image, thereby affecting the accuracy of the subsequent algorithm.

[0064] In view of this, an embodiment of the present disclosure provides a wearable device, in which a camera is arranged inside a lens barrel, which can solve or partially solve the above-mentioned problems to a certain extent.

[0065] Figure 2A A schematic diagram of an exemplary wearable device 200 provided by an embodiment of the present disclosure is shown.

[0066] like Figure 2A As shown, similarly, the wearable device 200 may also include a lens barrel 202, a display screen 204 and an optical component 206 disposed inside the lens barrel 202. The optical component 206 may further include a plurality of lenses (e.g., lenses 206A and 206B), and the combination of the plurality of lenses may project light emitted by the display screen 204 into the human eye 1022, so that the human eye 1022 may view the image displayed on the display screen 204.

[0067] and Figure 1B The difference between the wearable device 104 shown in the figure is that the camera 208 of the wearable device 200 is arranged inside the lens barrel 202. Since the camera 208 is placed inside the lens barrel 202, it will not affect the wearing of glasses, thereby improving the comfort of the wearable device 200. Figure 2AAs shown, since the camera 208 is arranged inside the lens barrel 202, the distance between the camera 208 and the human eye 1022 is extended, so that the installation inclination angle of the camera 208 relative to the human eye 1022 becomes smaller, and then the angle β between the direction of the camera 208 and the normal viewing direction of the human eye 1022 is reduced, so that the camera 208 has a better observation angle, and the collected human eye image can better reflect the image of the normal viewing angle of the human eye, and the imaging quality is better. In addition, since the camera 208 is placed inside the lens barrel 202, the glasses will not interfere with the imaging of the camera 208, further improving the imaging quality. The improvement of imaging quality also improves the accuracy of algorithms such as pupil distance estimation or line of sight tracking.

[0068] Figure 2B A schematic diagram of another exemplary wearable device 200 provided by an embodiment of the present disclosure is shown.

[0069] like Figure 2B As shown, in some embodiments, the wearable device 200 may further include a reflective structure 210, which may be a structure with a reflective surface such as a reflective film or a reflective mirror. The camera 208 can still capture human eye images through the reflection of the reflective structure 210. In addition, because a reflection process is added to the optical path, the observation angle γ is further reduced, and the camera 208 can better perform imaging.

[0070] Understandably, Figure 2A and Figure 2B Only a single-side structure of the wearable device 200 is exemplarily shown. In order to achieve binocular display, the wearable device 200 may include two lens barrel structures arranged in parallel.

[0071] It can be seen from the above embodiments that the embodiment of the present disclosure can obtain better imaging quality and improve the comfort of the wearable device 200 by setting the camera 208 inside the wearable device 200.

[0072] Furthermore, the inventors of the present disclosure discovered that, compared with placing the camera outside the lens barrel, placing the camera inside the lens barrel will affect the camera imaging not only by the camera module itself but also by the optical components in the lens barrel.

[0073] like Figure 2A and Figure 2B As shown, relative to Figure 1B In the optical path from the camera 208 to the human eye 1022, there is also an optical assembly 206 or a portion of the optical assembly 206 in the lens barrel (depending on the relative position relationship between the camera 208 and each lens in the optical assembly 206, for example, in addition to Figure 2A and Figure 2BThe camera 208 may also be disposed between the lens 206A and the lens 206B as shown in the figure, so that when the camera 208 is imaging the human eye 1022, the optical component 206 or a portion of the optical component 206 affects or changes the optical path of the camera 208 reaching the human eye 1022, causing the distortion of the image obtained by the camera 208 to no longer be symmetrical.

[0074] Furthermore, since the optical path from the camera 208 to the human eye 1022 is changed, the camera 208 does not have a unified projection center, which makes it impossible to use a parameterized camera model to fit the projection process of the camera in the lens barrel to process the distortion, making it difficult to calibrate the camera parameters.

[0075] In view of this, the embodiments of the present disclosure further provide a camera parameter calibration method to achieve parameter calibration of the camera in the lens barrel.

[0076] Figure 3A A flowchart of an exemplary method 300 provided in an embodiment of the present disclosure is shown.

[0077] The method 300 can be applied to any computer device with data processing capability and can be used to Figure 2A or Figure 2B The parameters of the camera 208 in the wearable device 200 are calibrated. Figure 3A As shown, the method 300 may further include the following steps.

[0078] In step 302, a plurality of images obtained by photographing a calibration reference object by a camera to be calibrated may be received.

[0079] Figure 4A A schematic diagram of an exemplary image collection scene 400 according to an embodiment of the present disclosure is shown.

[0080] like Figure 4A As shown, in the image acquisition scene 400, a calibration reference object 402 may be set at a fixed position. In some embodiments, the calibration reference object 402 may be a calibration plate or a chart, and some black and white checkerboards are set on the calibration reference object 402. When the camera captures the calibration reference object 402 to obtain a corresponding image, the corresponding camera parameters may be calculated according to the correspondence between the checkerboards in the image and the checkerboards of the calibration reference object 402.

[0081] When capturing images (or capturing images) of the calibration reference object 402, the camera to be calibrated (e.g., camera 208) can be controlled to move along a certain motion trajectory, and the calibration reference object 402 can be continuously captured during the movement of the camera, thereby obtaining multiple images captured by the camera in different postures, which are used as data for subsequent calculation of camera parameters. Since the captured images are obtained by the camera performing image capture in multiple postures, the camera parameters calculated subsequently are more robust and can be applied to a wider range.

[0082] In step 302, the computer device may receive a plurality of images captured in the aforementioned scene 400 for subsequent processing.

[0083] In step 304, the projection relationship between the pixels of the plurality of images and the calibration reference object may be determined.

[0084] In this step, a projection relationship between the image and the calibration reference object 402 can be established based on the acquired image, thereby establishing a corresponding relationship between the pixel point and the projection direction (pixel-ray). The projection direction (also referred to as the projection direction) can be a spatial straight line corresponding to a pixel point in the image coordinate system (two-dimensional) in the camera coordinate system (three-dimensional), so that a corresponding relationship between a pixel point in the image and a certain position in the three-dimensional space can be established based on the projection direction.

[0085] In some embodiments, Figure 3B As shown, step 304 of determining the projection relationship between the pixel points of the plurality of images and the calibration reference object may further include the following steps:

[0086] In step 3042, a first number of first images are selected from the plurality of images.

[0087] In this step, a projection relationship may be established first by selecting a certain number of first images from a plurality of images, and the remaining images may be used to supplement the portions for which the projection relationship is not established.

[0088] It can be understood that the first number is not specifically limited as long as it is sufficient to continue the subsequent steps. As an optional embodiment, the first number can be 3. Figure 4B 3 selected first images 412, 414, and 416 are shown. Figure 4B As shown, the first images 412, 414, and 416 respectively show images obtained by photographing the calibration reference object 402 at different camera postures. It can be understood that by calibrating the camera parameters of the images captured at different camera postures, the calculated camera parameters can be made more robust.

[0089] In step 3044, at least one local region is selected in each of the first images, and a homography transformation matrix between each of the local regions and the calibration reference object is established.

[0090] In this step, a suitable local area may be selected from each of the first images 412 , 414 , 416 for processing.

[0091] As an optional embodiment, a region suitable for being the local region may be detected from each first image by means of feature detection.

[0092] It is understandable that the information contained in the image captured by the camera includes pixel coordinates and grayscale values ​​of the corresponding pixels. When performing feature detection, detection can generally be performed based on changes in grayscale values.

[0093] by Figure 4B Taking the first images 412, 414, and 416 in FIG. 1 as an example, since there is a jump in the grayscale value of the adjacent pixel points at the four vertex positions of each grid in the chessboard, by performing feature detection based on the grayscale value, the four vertex positions of each grid in the chessboard can be obtained as feature points.

[0094] However, it is understood that, affected by the imaging quality of the camera itself and the camera shooting angle, this feature detection method may be difficult to detect all the vertices of the chessboard in the image. Therefore, when selecting a local area, a grid that can identify four vertices can be used as a local area. It is understood that at least one local area can be selected from each of the first images, and in some cases, multiple local areas can be selected.

[0095] After selecting at least one local area of ​​each first image, a homography transformation matrix with the calibration reference object 402 can be constructed for each local area of ​​each first image to establish a corresponding relationship (pixel-coordinate) between each pixel point contained in the local area and the coordinate system of the calibration reference object 402.

[0096] Homography, also known as projective transformation, can be used to characterize the mapping relationship between points on a plane in the world coordinate system and pixel points of an image in the camera coordinate system. The homography matrix H maps the plane coordinates in the world coordinate system to the pixel coordinates in the camera coordinate system.

[0097] The calculation formula of the homography transformation matrix H can be abbreviated as p'=Hp, where p' represents the homogeneous coordinates of a pixel point p' in the image, p represents the homogeneous coordinates of a point p corresponding to the point in the world coordinate system, and H is the homography transformation matrix from the calibration reference object 402 to the image.

[0098] Therefore, it can be known that the coordinate system of the location of the calibration reference object 402 is known, and the checkerboard on the calibration reference object 402 corresponds to the checkerboard image in the first image. At the same time, the coordinates of the four vertices corresponding to the local area in the camera coordinate system of the first image are also known. Based on these known information, the homography transformation matrix H between the local area of ​​the first image and the calibration reference object 402 can be determined.

[0099] In this way, a corresponding relationship between the pixel points included in the local area and the calibration reference object 402 or the coordinate system where the calibration reference object 402 is located is established.

[0100] In step 3046, the projection relationship between the pixel points included in the local area and the calibration reference object is determined according to the homography transformation matrix.

[0101] Figure 4C A schematic diagram showing the projection relationship between pixel points and a calibration reference object according to an embodiment of the present disclosure is shown.

[0102] like Figure 4C As shown in the figure, the area corresponding to the four vertices p1', p2', p3', and p4' connected by four dotted lines is the local area (the local area corresponds to the area corresponding to the points p1, p2, p3, and p4 on the calibration reference object 402). By constructing the homography transformation matrix, the homography transformation matrix can be used to calculate the projection relationship between each pixel point contained in the local area in the first image and the corresponding point on the calibration reference object.

[0103] Specifically, the coordinates of each pixel point contained in the local area are known, and by constructing the homography transformation matrix, the coordinates of the pixel point can be converted to the coordinate system of the calibration reference object, thereby establishing a projection relationship between the pixel point and the calibration reference object.

[0104] It can be understood that, for each local area of ​​each first image, the above method can be used to construct a homography transformation matrix, so as to establish a projection relationship between the pixel points in each local area and the calibration reference object.

[0105] In step 3048, the first number of first images are converted into a reference coordinate system, and the projection relationship between the pixel points of the multiple images and the calibration reference object is determined according to the area with the established projection relationship corresponding to the local area.

[0106] like Figure 4BAs shown, the three first images 412, 414, and 416 are acquired under different camera postures. When constructing the homography transformation matrix of the local area, it is constructed based on the camera coordinate system corresponding to each of the first images. It can be understood that in order to unify the projection relationship obtained by each first image under the same reference coordinate system, in this step, the three first images 412, 414, and 416 can be converted to the reference coordinate system.

[0107] In some embodiments, Figure 3C As shown, the step of converting the first number of first images into a reference coordinate system in step 3048 may further include the following steps:

[0108] In step 30482, the first number of feature points in the first image is determined. The feature points may be points that exist in the first images 412, 414, and 416 and have a corresponding relationship.

[0109] Back to Figure 4B , assuming that points a, a', and a" at the upper left corner of the chessboard are detected in the first images 412, 414, and 416 respectively, these three points all indicate the upper left corner vertex of the chessboard. Therefore, points a, a', and a" can be used as feature points to correspond the three first images 412, 414, and 416.

[0110] In some embodiments, in order to achieve more accurate correspondence, this step may select multiple feature points for calculation.

[0111] In step 30484, the relative position relationship between the first number of first images is determined based on the position of the feature point in each of the first images and the corresponding position of the feature point on the calibration reference object.

[0112] In this step, the relative position and posture relationship between the three first images 412 , 414 , and 416 can be calculated based on the position coordinates of the feature points in the respective coordinate systems of the three first images 412 , 414 , and 416 .

[0113] For example, the relative posture relationship may be calculated based on the property that the positions of the same pixel points in the first images 412 , 414 , and 416 on the calibration reference object 402 are located on the same straight line.

[0114] In step 30486, any first image is selected from the first number of first images as a target image.

[0115] In step 30488, the camera coordinate system corresponding to the target image is used as a reference coordinate system, and the first number of first images are converted to the reference coordinate system according to the relative posture relationship.

[0116] According to the above, the relative relationship between the three first images 412, 414, and 416 has been calculated. After determining that one of the first images is the target image, the coordinate system corresponding to the target image can be used as the reference coordinate system, and then the pixel coordinates of the other two first images can be converted to the reference coordinate system. In this way, the image under the same reference coordinate system can include all the pixel points that have been calibrated in each first image by constructing the homography transformation matrix between the local area and the calibration reference object. Because these pixel points can correspond to the calibration reference object through the constructed homography transformation matrix, the projection relationship between these pixel points and the calibration reference object can be known.

[0117] In the aforementioned embodiment, when constructing the homography transformation matrix, a single local area is corresponded to the calibration reference object. As mentioned above, in some embodiments, the local area is determined based on the feature detection result. It can be understood that in some cases, there may be local areas of the three first images 412, 414, and 416 for which the homography transformation matrix has been constructed, which cannot cover all the pixels on the entire image after all are converted to the same reference coordinate system, that is, there are still areas in the image for which the projection relationship has not been established (that is, uncalibrated areas).

[0118] In order to achieve the calibration of all pixels on the entire image, in some embodiments, such as Figure 3D As shown, the step of determining the projection relationship between the pixel points of the plurality of images and the calibration reference object according to the area with established projection relationship corresponding to the local area may further include:

[0119] In step 30490, determine whether there is any area in the first image for which a projection relationship has not been established.

[0120] According to the foregoing, some local areas can be selected from each first image to establish a homography transformation matrix with the calibration reference object, and the first number of first images are further unified into a reference coordinate system by means of coordinate transformation. In this step, when determining whether there is an area for which a projection relationship has not yet been established, for the same local area (for example, the first grid of the first row of the chessboard in each first image), the homography transformation matrix of the local area is constructed in each first image, and it can be determined that the projection relationship has been established for the local area (that is, the local area can be considered to be calibrated). When a local area can only establish a homography transformation matrix through one or two first images (for example, the same local area can only be detected in images 412 and 414 to establish a homography transformation matrix, but cannot be detected from image 416 to establish a homography transformation matrix), even if the local area has established a projection relationship (homography transformation matrix) with the calibration reference object in a single or two first images, the local area on the image is not considered to be an area with an established projection relationship (that is, the local area can be considered to be uncalibrated).

[0121] Through the above processing method, it can be known whether there is an area in the first image for which a projection relationship has not been established (that is, whether there is an image area that has not been calibrated).

[0122] In step 30492, in response to determining that there is no area in the first image for which a projection relationship has not been established, the projection relationship between the pixels of the multiple images and the calibration reference object is determined based on the area with an established projection relationship corresponding to the local area.

[0123] In this case, the calibration of all pixels on the entire image has been completed through the first number of first images, so the projection relationship between the image pixels and the calibration reference object can be determined based on the previously obtained data.

[0124] In step 30494, in response to determining that there is an area in the first image for which a projection relationship has not been established, a projection relationship between pixel points and the calibration reference object is established for the area for which a projection relationship has not been established using remaining images of the multiple images except the first image.

[0125] In this case, since the first number of first images fail to calibrate all the pixels on the entire image, the remaining images except the first images may be selected from the multiple images obtained during image acquisition to continue processing.

[0126] Specifically, a method similar to the above may be used to select a local area in the remaining image to construct a homography transformation matrix, so as to establish a projection relationship between pixel points and the calibration reference object for the area where the projection relationship is not established.

[0127] It can be understood that in some embodiments, in order to ensure the accuracy of subsequent algorithms, if the remaining images still cannot completely cover the areas in the images where no projection relationship has been established, new images can be re-captured to continue the above process until all pixels of the image can establish a projection relationship with the calibration reference object.

[0128] According to the above embodiments, it can be known that in some embodiments, after all image processing is completed and the entire image is calibrated, some pixels may correspond to multiple projection direction data. Therefore, these data can be averaged to obtain a projection direction as the projection direction of the pixel to obtain better projection direction data. In addition, only the projection direction after the average processing can be stored in subsequent storage, thereby saving storage space.

[0129] In step 306, a camera parameter set of the camera to be calibrated may be determined according to the projection relationship. The camera parameter set includes a plurality of camera parameters, and the camera parameters are used to characterize the target pixel points in the image captured by the camera to be calibrated and the projection direction corresponding to the target pixel points. Optionally, the camera parameters may characterize a one-to-one correspondence between the pixel points and the projection direction.

[0130] Figure 4D A schematic diagram showing exemplary camera parameters according to an embodiment of the present disclosure is shown.

[0131] like Figure 4D As shown, when constructing the aforementioned projection relationship, in the reference coordinate system, at least some of the pixels in the image correspond one-to-one with some points on the calibration reference object, so that a straight line equation about the pixel can be obtained, and the straight line represented by the straight line equation passes through the corresponding pixel and has a ray (ray) as shown in the figure to represent its projection direction. Such a combination of pixel points and projection directions can be used as camera parameters (the camera parameters can be considered as the internal parameters of the camera), and the camera parameters are used to realize the conversion from the coordinate system of the image captured by the camera to the camera coordinate system, so that it can be used to calculate the position information of certain features (for example, pupil) in the image captured by the camera in three-dimensional space (camera coordinate system). In addition, since the multiple local areas of the first image are used to respectively establish the homography transformation matrix with the calibration reference object in the process of calculating the camera parameters, when the camera parameters are used to perform the coordinate system conversion calculation, the projection direction corresponding to the pixel point obtained already includes the correction of the image distortion at the position, and no additional distortion correction is required.

[0132] In some embodiments, all pixel point-projection direction data obtained from the aforementioned constructed projection relationship can be saved as camera parameters for subsequent calculation of position information of certain features (e.g., pupil) in the image captured by the camera in three-dimensional space (camera coordinate system).

[0133] According to the above, in order to ensure the accuracy of the subsequent algorithm, it is hoped that a projection relationship is established for at least most of the areas on the image. It is understandable that the number of pixels on an image is relatively large. For example, an image with a resolution of 400×400 has 160,000 pixels. If each pixel stores a set of pixel-projection direction data, then 160,000 sets of data need to be stored when the image is fully calibrated. In addition, the data will increase as the image resolution increases.

[0134] Therefore, in some embodiments, the previously obtained data may be simplified and used as the camera parameters of the camera, thereby saving space for storing the camera parameters.

[0135] Specifically, Figure 3E As shown, step 306 of determining the camera parameter set of the camera to be calibrated according to the projection relationship may further include the following steps:

[0136] In step 3062, an initial camera parameter set is determined according to the projection relationship, where the initial camera parameter set includes a plurality of camera parameters corresponding one-to-one to a plurality of pixel points of the image.

[0137] In this step, the initial camera parameter set may be a data set of all pixel points-projection directions for which a projection relationship has been established. It can be understood that the data volume of the initial camera parameter set is substantially equal to the resolution of the image captured by the camera.

[0138] In step 3064, a spline surface is used to fit the initial camera parameter set to obtain a fitted spline surface.

[0139] A spline surface usually refers to a surface composed of spline curves, and a spline curve can refer to a curve obtained by giving a set of control points, and the general shape of the curve is controlled by these points.

[0140] In this step, the projection direction can be fitted according to the camera parameters in the initial camera parameter set to obtain a fitting spline surface, and the fitting spline surface can be used to represent these camera parameters. Optionally, the fitting spline surface can be a smooth spline surface. In some embodiments, if the projection direction of these camera parameters cannot converge to a point, it is also necessary to fit the starting point corresponding to the direction.

[0141] In step 3066, a camera parameter set of the camera to be calibrated is determined according to the control points of the fitted spline surface.

[0142] In this step, after obtaining the fitted spline surface, the parameters corresponding to the control points of the fitted spline surface can be used as the camera parameters in the camera parameter set. Since the number of control points is much smaller than the number of pixel points, storage space can be saved well. When the camera parameters need to be used for parameter calibration and distortion correction, the complete camera parameters can be obtained through the interpolation algorithm based on the fitted spline surface and the control points.

[0143] In some embodiments, in order to further improve the accuracy of subsequent algorithms and reduce the errors of camera parameters, step 3066 of determining the camera parameter set of the camera to be calibrated according to the control points of the fitting spline surface may further include: optimizing the control points of the fitting spline surface based on a bundle adjustment (BA) algorithm to obtain the camera parameter set of the camera to be calibrated.

[0144] In this way, the corresponding parameter errors of the optimized control points are smaller, which can further improve the accuracy of subsequent algorithms.

[0145] In some embodiments, Figure 3F As shown, the step 3066 of optimizing the control points of the fitted spline surface based on the bundle adjustment (BA) algorithm to obtain the camera parameter set of the camera to be calibrated may further include the following steps:

[0146] In step 30662, an optimization function with respect to the control points is constructed.

[0147] In this step, the control points of the fitting spline surface can be optimized using the information of the image (observation data) obtained by the camera 208 when capturing images. The corresponding optimization function form is as follows:

[0148]

[0149] f i,o =π(T i ,p i )-d i

[0150] Among them, π is the camera parameter, that is, the control point corresponding to the spline surface, ρ is the loss function, T i is the external parameter of the camera and the calibration reference object 402 corresponding to the i-th image (which can be calculated by the corresponding relationship between the image data and the relevant parameters of the calibration reference object 402 and the initialization of the camera center), O i is all the information contained in the i-th image, p iis the 3D point corresponding to the i-th image on the calibration reference object 402 (that is, the spatial coordinates of the points corresponding to each pixel point of the image on the calibration reference object 402), d i are all detection points corresponding to the i-th image, and I is the number of input images (i.e., the number of images obtained in the previous image acquisition step). The initialized camera center can be calculated based on the projection relationship obtained from the first number of first images.

[0151] In some embodiments, the detection point d i The calculation can be performed in the following manner.

[0152] Figure 4E A schematic diagram showing an image obtained after binarization processing of an image collected in an embodiment of the present disclosure is shown.

[0153] like Figure 4E As shown in the figure, after the binarization process, the pixels on the image are either black or white. Through feature detection, the vertices of the checkerboard grid can be identified. These identifiable vertices can be used as detection points d. The detection points detected on the i-th image can be expressed as d i .

[0154] In step 30664, the optimization function is solved using the Levenberg-Marquardt method (LM) to obtain optimized control points.

[0155] In step 30666, the camera parameter set of the camera to be calibrated is determined according to the optimized control points. For example, the camera parameter set can save the information of the optimized control points as camera parameters. When the camera parameters are needed to calculate the position information of certain features (e.g., pupil) in the image captured by the camera in the three-dimensional space (camera coordinate system), the complete camera parameters can be obtained by interpolation algorithm according to the fitted spline surface and the optimized control points.

[0156] In a more specific embodiment, the camera parameter calibration method provided by the embodiment of the present disclosure may include the steps of image acquisition, initialization (pixel-ray calculation) and optimization, which can obtain better camera parameters for subsequent algorithm calculation.

[0157] It can be seen from the above embodiments that the camera parameter calibration method provided by the embodiments of the present disclosure provides a feasible camera parameter calibration solution for scenes where the imaging distortion of the camera in the lens barrel is asymmetric and there is no unified projection center.

[0158] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0159] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] The embodiment of the present disclosure also provides a wearable device.

[0161] Figure 5 A schematic diagram of an exemplary wearable device 500 provided by an embodiment of the present disclosure is shown.

[0162] like Figure 5 As shown, Figure 2A and Figure 2B Similarly, the wearable device 500 includes a lens barrel 202, in which a display screen 204, a camera 208, and an optical component 206 are arranged. The camera 208 and the optical component 206 are located on one side of the light emitting direction of the display screen 204, and the camera 208 is located between the optical component 206 and the display screen 204.

[0163] In some embodiments, the shooting direction of the camera 208 is toward the optical component 206. Figure 2A In some other embodiments, reference Figure 2B As shown, a reflective structure 210 is disposed between the display screen 204 and the camera 208 , the reflective surface of the reflective structure 210 faces the light emitting direction of the display screen 204 , and the shooting direction of the camera 208 faces the reflective structure 210 .

[0164] Furthermore, if Figure 5 As shown, the wearable device 500 also includes a processing module 502, which is electrically coupled to the camera 208 and the display screen 204, respectively, and is configured to: obtain a camera parameter set obtained using the aforementioned method 300 and an image taken by the camera 208; and use the camera parameter set and the image to solve the position of the target area in the image in space.

[0165] As mentioned above, the camera parameter set includes camera parameters based on pixel points and projection directions (the camera parameters can be considered as the internal parameters of the camera), and the camera parameters are used to realize the conversion from the coordinate system of the image captured by the camera to the camera coordinate system, so that it can be used to calculate the position information of certain features (for example, pupil) in the image captured by the camera in the three-dimensional space (camera coordinate system). In addition, when the camera parameters are used for coordinate system conversion calculation, no additional distortion correction is required.

[0166] The embodiment of the present disclosure also provides a computer device for implementing the above method 300. Figure 6 FIG. 6 is a schematic diagram showing the hardware structure of an exemplary computer device 600 provided in an embodiment of the present disclosure. The computer device 600 can be used to implement Figure 1A Head-mounted wearable devices 104, Figure 2A and Figure 2B The wearable device 200 can also be used to implement Figure 1A The external device 112 can also be used to implement Figure 1A In some scenarios, the computer device 600 can also be used to implement Figure 1A The database server 116.

[0167] like Figure 6 As shown, the computer device 600 may include: a processor 602, a memory 604, a network module 606, a peripheral interface 608 and a bus 610. The processor 602, the memory 604, the network module 606 and the peripheral interface 608 are connected to each other in communication within the computer device 600 through the bus 610.

[0168] Processor 602 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 602 may be used to perform functions related to the technology described in the present disclosure. In some embodiments, processor 602 may also include multiple processors integrated into a single logical component. For example, Figure 6 As shown, processor 602 may include a plurality of processors 602a, 602b, and 602c.

[0169] The memory 604 may be configured to store data (eg, instructions, computer code, etc.). Figure 6As shown, the data stored in the memory 604 may include program instructions (e.g., program instructions for implementing the method 300 of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 602 may also access the program instructions and data stored in the memory 604, and execute the program instructions to operate on the data to be processed. The memory 604 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 604 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.

[0170] The network interface 606 can be configured to provide the computer device 600 with communication with other external devices via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC)), a cellular network, the Internet, or a combination thereof. It is understood that the type of network is not limited to the above specific examples.

[0171] The peripheral interface 608 can be configured to connect the computer device 600 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.

[0172] The bus 610 may be configured to transmit information between various components of the computer device 600 (e.g., the processor 602, the memory 604, the network interface 606, and the peripheral interface 608), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.

[0173] It should be noted that, although the architecture of the above-mentioned computer device 600 only shows the processor 602, the memory 604, the network interface 606, the peripheral interface 608 and the bus 610, in the specific implementation process, the architecture of the computer device 600 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the above-mentioned computer device 600 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0174] The disclosed embodiment also provides a camera parameter calibration device. Figure 7 FIG. 7 is a schematic diagram of an exemplary device 700 provided by an embodiment of the present disclosure. Figure 7 As shown, the device 700 can be used to implement the method 300 and can further include the following modules.

[0175] The receiving module 702 is configured to: receive a plurality of images obtained by photographing a calibration reference object by a camera to be calibrated;

[0176] The first determination module 704 is configured to: determine the projection relationship between the pixel points of the multiple images and the calibration reference object;

[0177] The second determination module 706 is configured to: determine a camera parameter set of the camera to be calibrated according to the projection relationship, wherein the camera parameter set includes multiple camera parameters, and the camera parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points.

[0178] In some embodiments, the second determination module 706 is configured to:

[0179] Determine an initial camera parameter set according to the projection relationship, wherein the initial camera parameter set includes a plurality of camera parameters corresponding one-to-one to a plurality of pixel points of the image;

[0180] Fitting the initial camera parameter set using a spline surface to obtain a fitting spline surface;

[0181] A camera parameter set of the camera to be calibrated is determined according to the control points of the fitted spline surface.

[0182] In some embodiments, the second determination module 706 is configured to: optimize the control points of the fitting spline surface based on a bundle adjustment algorithm to obtain a camera parameter set of the camera to be calibrated.

[0183] In some embodiments, the second determination module 706 is configured to:

[0184] constructing an optimization function with respect to the control points;

[0185] The optimization function is solved by using the Levenberg-Marquardt method to obtain optimized control points;

[0186] A camera parameter set of the camera to be calibrated is determined according to the optimized control points.

[0187] In some embodiments, the first determining module 704 is configured to:

[0188] selecting a first number of first images from the plurality of images;

[0189] Selecting at least one local area in each of the first images, and establishing a homography transformation matrix between each of the local areas and the calibration reference object;

[0190] Determining a projection relationship between pixel points included in the local area and the calibration reference object according to the homography transformation matrix;

[0191] The first number of first images are converted into a reference coordinate system, and the projection relationship between the pixel points of the plurality of images and the calibration reference object is determined according to the area with the established projection relationship corresponding to the local area.

[0192] In some embodiments, the first determining module 704 is configured to:

[0193] Determine whether there is an area in the first image for which a projection relationship has not been established;

[0194] In response to determining that there is no area in the first image for which a projection relationship has not been established, determining a projection relationship between pixel points of the multiple images and the calibration reference object according to an area for which a projection relationship has been established corresponding to the local area;

[0195] In response to determining that there is an area in the first image for which a projection relationship has not been established, a projection relationship between pixels and the calibration reference object is established for the area for which a projection relationship has not been established using remaining images of the multiple images except the first image.

[0196] In some embodiments, the first determining module 704 is configured to:

[0197] determining the first number of feature points in the first image;

[0198] Determining a relative position relationship between the first number of first images according to a position of the feature point in each of the first images and a corresponding position of the feature point on the calibration reference object;

[0199] Selecting any first image from the first number of first images as a target image;

[0200] The camera coordinate system corresponding to the target image is used as a reference coordinate system, and the first number of first images are converted to the reference coordinate system according to the relative posture relationship.

[0201] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0202] The device of the above embodiment is used to implement the corresponding method 300 in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. Based on the same inventive concept, corresponding to any of the above embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method 300 described in any of the above embodiments.

[0203] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0204] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method 300 described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0205] Based on the same inventive concept, corresponding to the method 300 of any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the method 300. Corresponding to the execution subject corresponding to each step in each embodiment of the method 300, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0206] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the method 300 described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0207] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0208] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0209] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0210] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A camera parameter calibration method, comprising: Receiving a plurality of images obtained by photographing a calibration reference object by a camera to be calibrated; Determining a projection relationship between pixel points of the plurality of images and the calibration reference object; According to the projection relationship, a camera parameter set of the camera to be calibrated is determined, where the camera parameter set includes a plurality of camera parameters, and the camera parameters are used to characterize target pixels in an image captured by the camera to be calibrated and projection directions corresponding to the target pixels.

2. The method of claim 1, wherein: Determining a camera parameter set of the camera to be calibrated according to the projection relationship includes: Determine an initial camera parameter set according to the projection relationship, wherein the initial camera parameter set includes a plurality of camera parameters corresponding one-to-one to a plurality of pixel points of the image; Fitting the initial camera parameter set using a spline surface to obtain a fitting spline surface; A camera parameter set of the camera to be calibrated is determined according to the control points of the fitted spline surface.

3. The method of claim 2, wherein: Determining a camera parameter set of the camera to be calibrated according to the control points of the fitted spline surface includes: The control points of the fitted spline surface are optimized based on a bundle adjustment algorithm to obtain a camera parameter set of the camera to be calibrated.

4. The method of claim 3, wherein: The control points of the fitted spline surface are optimized based on a bundle adjustment algorithm to obtain a camera parameter set of the camera to be calibrated, including: constructing an optimization function with respect to the control points; The optimization function is solved by using the Levenberg-Marquardt method to obtain optimized control points; A camera parameter set of the camera to be calibrated is determined according to the optimized control points.

5. The method of claim 1, wherein: Determining the projection relationship between the pixel points of the plurality of images and the calibration reference object includes: selecting a first number of first images from the plurality of images; Selecting at least one local area in each of the first images, and establishing a homography transformation matrix between each of the local areas and the calibration reference object; Determining a projection relationship between pixel points included in the local area and the calibration reference object according to the homography transformation matrix; The first number of first images are converted into a reference coordinate system, and the projection relationship between the pixel points of the plurality of images and the calibration reference object is determined according to the area with the established projection relationship corresponding to the local area.

6. The method of claim 5, wherein: Determining the projection relationship between the pixel points of the plurality of images and the calibration reference object according to the area with established projection relationship corresponding to the local area, including: Determine whether there is an area in the first image for which a projection relationship has not been established; In response to determining that there is no area in the first image for which a projection relationship has not been established, determining a projection relationship between pixel points of the multiple images and the calibration reference object according to an area for which a projection relationship has been established corresponding to the local area; In response to determining that there is an area in the first image for which a projection relationship has not been established, a projection relationship between pixels and the calibration reference object is established for the area for which a projection relationship has not been established using remaining images of the multiple images except the first image.

7. The method of claim 5, wherein: Converting the first number of first images into a reference coordinate system comprises: determining the first number of feature points in the first image; Determining a relative position relationship between the first number of first images according to a position of the feature point in each of the first images and a corresponding position of the feature point on the calibration reference object; Selecting any first image from the first number of first images as a target image; The camera coordinate system corresponding to the target image is used as a reference coordinate system, and the first number of first images are converted to the reference coordinate system according to the relative posture relationship.

8. A camera parameter calibration device, comprising: The receiving module is configured to: receive a plurality of images obtained by photographing the calibration reference object by the camera to be calibrated; A first determination module is configured to: determine a projection relationship between pixel points of the plurality of images and the calibration reference object; The second determination module is configured to: determine a camera parameter set of the camera to be calibrated according to the projection relationship, wherein the camera parameter set includes multiple camera parameters, and the camera parameters are used to characterize the target pixel points in the image taken by the camera to be calibrated and the projection direction corresponding to the target pixel points.

9. A wearable device comprising: A lens barrel, wherein a display screen, a camera, and an optical component are arranged in the lens barrel, the camera and the optical component are located on one side of the light emitting direction of the display screen, and the camera is located between the optical component and the display screen; A processing module is electrically coupled to the camera and is configured to: obtain a camera parameter set obtained by any method of claims 1-7 and an image taken by the camera; and use the camera parameter set and the image to solve the position of the target area in the image in space.

10. The wearable device according to claim 9, wherein: The shooting direction of the camera is toward the optical component; or, A reflective structure is arranged between the display screen and the camera, a reflective surface of the reflective structure faces the light emitting direction of the display screen, and a shooting direction of the camera faces the reflective structure.

11. A computer device comprising one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include instructions for executing the method according to any one of claims 1 to 7.

12. The device of claim 11, wherein: The device includes a wearable device.

13. A non-volatile computer-readable storage medium containing a computer program, which, when executed by one or more processors, causes the processors to perform the method according to any one of claims 1 to 7.

14. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 7.

Images