Parameter calibration method, intelligent glasses and readable storage medium
By calibrating external parameters using the coordinate information of the target calibration object and the image recognition algorithm in smart glasses, the problem of inaccurate gaze mapping is solved and higher mapping accuracy is achieved.
Patent Information
- Application Number
- CN202510529430.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-24
- Publication Date
- 2025-08-01
AI Technical Summary
In the prior art, the accuracy of mapping the gaze points to images through smart glasses is low.
By calibrating the coordinate information of the object in the world coordinate system and image plane coordinate system, the external parameters of the image acquisition module are obtained, and combined with the eye movement information acquisition module and image recognition algorithm, the external parameters are calibrated to reduce mapping errors.
Improves the mapping accuracy of gaze points in the image plane coordinate system, and enhances the accuracy of image acquisition and gaze point tracking of smart glasses.
Smart Images
Figure CN120411257A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of smart wearable devices, and more specifically, to a parameter calibration method, smart glasses, and a readable storage medium. Background Art
[0002] With the development of electronic information technology, smart glasses can now capture images and gaze points, and then map the gaze points to the images. However, the accuracy of mapping the gaze points to the images is currently low. Summary of the Invention
[0003] This application proposes a parameter calibration method, smart glasses and a readable storage medium to improve the above-mentioned defects.
[0004] In a first aspect, an embodiment of the present application provides a parameter calibration method, which is applied to a control module of smart glasses, wherein the smart glasses also include an image acquisition module and an eye movement information acquisition module, and the control module is connected to the image acquisition module and the eye movement information acquisition module respectively. The method includes: obtaining an external parameter of the image acquisition module relative to the world coordinate system through first coordinate information of a target calibration object in a world coordinate system and second coordinate information of the target calibration object in an image plane coordinate system, wherein the image plane coordinate system is a two-dimensional coordinate system established based on the image acquisition module; mapping third coordinate information of a gaze point of the target calibration object in the world coordinate system obtained by the eye movement information acquisition module to the image plane coordinate system based on the external parameter to obtain fourth coordinate information; obtaining fifth coordinate information of a reference gaze point of the target calibration object in a target image in the image plane coordinate system based on an image recognition algorithm, wherein the target image is an image including the target calibration object acquired by the image acquisition module; calibrating the external parameter to reduce the difference through the difference between the fifth coordinate information and the fourth coordinate information.
[0005] In a second aspect, an embodiment of the present application further provides a pair of smart glasses, including a control module, an image acquisition module, and an eye movement information acquisition module, wherein the control module is connected to the image acquisition module and the eye movement information acquisition module respectively; the control module is used to calibrate external parameters based on the method described in the first aspect.
[0006] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, in which program code is stored. The program code can be called by a processor to execute the method described in the first aspect.
[0007] The parameter calibration method, smart glasses and readable storage medium provided by this application first obtain the external parameters of the image acquisition module relative to the world coordinate system through the first coordinate information of the target calibration object in the world coordinate system and the second coordinate information of the target calibration object in the image plane coordinate system, where the image plane coordinate system is a two-dimensional coordinate system established based on the image acquisition module; then map the third coordinate information of the fixation point of the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the external parameters to obtain the fourth coordinate information; and then obtain the fifth coordinate information of the reference fixation point of the target calibration object in the image plane coordinate system in the target image based on the image recognition algorithm, where the target image is an image including the target calibration object acquired by the image acquisition module; thereby calibrating the external parameters to reduce the difference through the difference between the fifth coordinate information and the fourth coordinate information. It can be seen that in the solution of this application, the fourth coordinate information is calculated, and the fifth coordinate information is recognized through the image recognition algorithm, and then the external parameters are calibrated through the difference between the fifth coordinate information and the fourth coordinate information to reduce the difference, so that the fixation point can be more accurately mapped to the image plane coordinate system.
[0008] Other features and advantages of the embodiments of this application will be described in the subsequent specification, and, in part, will be obvious from the specification, or will be understood by implementing the embodiments of this application. The objectives and other advantages of the embodiments of this application can be realized and obtained by the structures specifically pointed out in the written specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0010] Figure 1 Shows the structural block diagram of the smart glasses provided by the embodiments of this application;
[0011] Figure 2 Shows the method flow chart of the parameter calibration method provided by the embodiments of this application;
[0012] Figure 3 Shows the application scenario diagram of the parameter calibration method provided by the embodiments of this application;
[0013] Figure 4 Shows the schematic diagram of the target calibration object provided by the embodiments of this application;
[0014] Figure 5 Shows a schematic diagram of a target object provided by an embodiment of the present application;
[0015] Figure 6 Shows a schematic diagram of a target object provided by another embodiment of the present application;
[0016] Figure 7 Shows a schematic diagram of a target object provided by yet another embodiment of the present application;
[0017] Figure 8 Shows a schematic diagram of a target object provided by still another embodiment of the present application;
[0018] Figure 9 Shows a method flow chart of a parameter calibration method provided by another embodiment of the present application;
[0019] Figure 10 Shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Detailed implementation manners
[0020] In order to enable those skilled in the art of the present technology to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Usually, the components of the embodiments of the present application described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application to be protected, but only represents the selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts fall within the scope of protection of the present application.
[0021] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.
[0022] Currently, with the development of electronic information technology, images and gaze points can be collected through smart glasses, and then the gaze points are mapped to the images. However, currently, the accuracy of mapping the gaze points to the images is relatively low. How to improve the accuracy of mapping the target gaze points to the images is an urgent problem to be solved.
[0023] At present, for smart glasses configured with an image acquisition module and an eye movement information acquisition module, during use, generally, the fixation points collected by the eye movement information acquisition module are directly mapped to the images collected by the image acquisition module through preset parameters.
[0024] However, the inventors found in their research that if the mapping parameters are not calibrated, the mapping accuracy may be relatively low.
[0025] Therefore, in order to overcome the above defects, the present application provides a parameter calibration method, a smart glasses and a readable storage medium.
[0026] Please refer to Figure 1 , Figure 1 , which shows the structural block diagram of the smart glasses provided by the embodiment of the present application, specifically showing the smart glasses 100.
[0027] In some embodiments, the smart glasses 100 may be an augmented reality glasses device. In the smart glasses 100 shown in Figure 1 , a control module 110, an eye movement information acquisition module 120, and an image acquisition module 130 are provided. Among them, the control module 110 is respectively connected to the eye movement information acquisition module 120 and the image acquisition module 130.
[0028] The eye movement information acquisition module 120 can be used to collect information on fixation points. Among them, the information on fixation points is the visual attention distribution data recorded by tracking the eye movement, which characterizes the fixation behavior of the user when observing a scene, an image, text, or a dynamic picture.
[0029] The image acquisition module 130 may include at least one camera. The image acquisition module 130 can be used to acquire images. Thus, the smart glasses 100 can combine the acquired images and fixation points to calibrate the external parameters, thereby improving the accuracy of mapping the fixation points to the images. For a detailed introduction, please refer to the subsequent embodiments.
[0030] Please refer to Figure 2 , Figure 2 , which shows the method flowchart of the parameter calibration method provided by the embodiment of the present application. This parameter calibration method can be applied to the control module in the smart glasses shown in Figure 1 . Specifically, this method may include steps S110 to S140.
[0031] Through the foregoing introduction, it can be known that the smart glasses are provided with a control module, an image acquisition module, and an eye movement information acquisition module. The control module is respectively connected to the image acquisition module and the eye movement information acquisition module.
[0032] Exemplarily, please refer to Figure 3 ,Figure 3 FIG. Figure 3 shows an application scenario diagram of the parameter calibration method provided by an embodiment of the present application, that is, a parameter calibration scenario 300. In Figure 3 In the shown parameter calibration scenario 300, there is a user 320, and the user 320 wears smart glasses 310. In the smart glasses 310, an image acquisition module ( Figure 3 not shown in the figure) and an eye movement information acquisition module ( Figure 3 not shown in the figure) can be set. Optionally, in order to balance the weights of the two temple arms of the smart glasses 310, the smart glasses 310 can also be configured with a counterweight module ( Figure 3 not shown in the figure). Thus, the user 320 can execute the parameter calibration method provided by an embodiment of the present application through the worn smart glasses 310.
[0033] Step S110: Obtain the external parameters of the image acquisition module relative to the world coordinate system through the first coordinate information of the target calibration object in the world coordinate system and the second coordinate information of the target calibration object in the image plane coordinate system, where the image plane coordinate system is a two-dimensional coordinate system established based on the image acquisition module.
[0034] Wherein, the target calibration object can be a target object for parameter calibration. For example, a target picture can be displayed through an electronic device, and the electronic device displaying the target picture can be regarded as the target calibration object, where the electronic device can be a smart phone; for another example, a physical target object can be used as the target calibration object, specifically, for example, a plastic target board or a paper target board can be used as the target calibration object.
[0035] It should be noted that the above-mentioned target board can be regarded as a physical object printed with a target image, and the target picture is a picture used to assist the eye movement information acquisition module to obtain the fixation point subsequently.
[0036] Exemplarily, please refer to Figure 4 , Figure 4 FIG. Figure 4 shows a schematic diagram of the target calibration object provided by an embodiment of the present application. In Figure 4 FIG. Figure 4 , a target calibration object 410 is shown, and the target calibration object 410 is a target picture displayed through an electronic device 420. Specifically, in Figure 4 FIG. Figure 4 , the user 320 wearing the smart glasses 310 holds an electronic device displaying the target picture as the target calibration object 410.
[0037] For some embodiments, Figure 4After wearing the smart glasses 310, the user 320 shown in the figure can use a handheld electronic device 420 to display a target picture of a known size as a target calibration object through the electronic device 420. Then, the user 320 can hold the handheld electronic device 420 and place the target calibration object in front of the smart glasses 310 to achieve the calibration of the external parameters.
[0038] It can be understood that the size of the target calibration object and the first coordinate information of the target calibration object in the world coordinate system can be determined in advance. Exemplarily, a world coordinate system can be established first, so that the target calibration object can be placed at a specified position in space according to the indication information, and the coordinate information of the specified position in the world coordinate system is known. Combining with the known size, the first coordinate information of the target calibration object in the world coordinate system can be determined.
[0039] In addition, a two-dimensional coordinate system can be established according to the image acquisition module as an image plane coordinate system. Further, the second coordinate information of the target calibration object in the image plane coordinate system can be obtained.
[0040] Exemplarily, the projection of the target calibration object in the image plane coordinate system can be obtained by detecting the feature points on the target calibration object, and the coordinates corresponding to the projection are the second coordinate information. For some embodiments, the second coordinate information can be represented by (bx, by).
[0041] Further, the external parameters of the image acquisition module relative to the world coordinate system can be calculated according to the first coordinate information and the second coordinate information. Among them, the external parameters can be used to describe the position and orientation of the image acquisition module in the three-dimensional space, and are used to convert the points in the world coordinate system to the coordinate system corresponding to the camera, such as converting to the image plane coordinate system. For some embodiments, the external parameters can include a rotation matrix and a translation vector, and the detailed method for obtaining the external parameters can refer to the introduction of the subsequent embodiments. Exemplarily, the external parameters can be represented by P.
[0042] Optionally, the external parameters can also be obtained through factory calibration. Exemplarily, the smart glasses can be set on a tooling device, where the tooling device can be used to control the smart glasses to be horizontal with the ground, thereby simulating the scenario where a user wears the smart glasses and looks straight ahead. When setting the smart glasses on the tooling device, it can be controlled that the smart glasses are parallel to the plane where the target calibration object is located corresponding to the line connecting the centers of the two eyes' pupils. Thus, the first distance D from the midpoint of the line connecting the centers of the two eyes' pupils corresponding to the smart glasses to the target calibration object can be obtained. Then, an image including the target calibration object is collected through the image acquisition module on the smart glasses, and the first center position of the image and the second center position of the target calibration object in the image are obtained. Thus, the second distance between the first center position and the second center position can be obtained. Then, the angle between the optical axis of the image acquisition module and the perpendicular line of the line connecting the centers of the two eyes' pupils corresponding to the smart glasses can be calculated based on the first distance and the second distance.
[0043] It is also possible to obtain the third center position of the image acquisition module in the world coordinate system and the fourth center position of the center of the line connecting the centers of the two eyes' pupils corresponding to the smart glasses in the world coordinate system. Thus, based on the angle, the third center position, and the fourth center position obtained above, the external parameters can be obtained.
[0044] Step S120: Map the third coordinate information of the fixation point of the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the external parameters to obtain the fourth coordinate information.
[0045] Furthermore, since the eye movement information acquisition module can track eye movements, the third coordinate information of the fixation point of the target calibration object in the world coordinate system can be obtained through the eye movement information acquisition module. However, it can be understood that the third coordinate information of the fixation point collected by the eye movement information acquisition module is in the world coordinate system, rather than in the image plane coordinate system. Therefore, further mapping operations are required to map the fixation point to the plane coordinate system.
[0046] It should be noted that in order to obtain the third coordinate information of the fixation point of the target calibration object in the world coordinate system, the user wearing the smart glasses needs to fixate on the target calibration object.
[0047] Optionally, the third coordinate information of the fixation point of the target calibration object in the world coordinate system can also be obtained through factory calibration. Exemplarily, a human eye model manufactured by machinery can be used to simulate the user's human eye. Specifically, the human eye model can emit coaxial laser light to replace the user's human eye. Thus, the smart glasses can be set on the human eye model, and the human eye model can be controlled to emit coaxial laser light to the target calibration object, so as to obtain the third coordinate information of the fixation point of the target calibration object in the world coordinate system.
[0048] For some embodiments, the third coordinate information can be mapped to the image plane coordinate system based on the external parameters obtained in the foregoing steps, so as to obtain the fourth coordinate information. Exemplarily, the third coordinate information can be first mapped to the camera coordinate system based on the external parameters, and then the corresponding coordinate information in the plane coordinate system can be determined based on the coordinate information mapped in the camera coordinate system, that is, the fourth coordinate information is obtained. Among them, the camera coordinate system is a three-dimensional coordinate system established based on the image acquisition module. And this fourth coordinate information is the position where the fixation point in the world coordinate system obtained by the eye movement information acquisition module is mapped to the image plane coordinate system.
[0049] Step S130: Obtain the fifth coordinate information of the reference fixation point of the target calibration object in the image plane coordinate system based on an image recognition algorithm, where the target image is an image including the target calibration object collected by the image acquisition module.
[0050] It can be understood that the method of determining the external parameters first and then performing coordinate mapping may introduce certain errors, resulting in a low accuracy of the mapped fourth coordinate information. Therefore, in order to further improve the accuracy of the mapped fourth coordinate information, for some embodiments, the fifth coordinate information of the reference fixation point of the target calibration object in the image plane coordinate system can also be obtained through an image recognition algorithm. Among them, the target image is an image including the target calibration object collected by the image acquisition module.
[0051] That is, the target image including the target calibration object can be collected by the image acquisition module first, and then the image recognition algorithm is applied to the target image, so as to determine the fifth coordinate information of the reference fixation point for the target calibration object in the image plane coordinate system through the image recognition algorithm. Among them, the fifth coordinate information of the reference fixation point in the image plane coordinate system can be regarded as the standard position where the fixation point for the target calibration object is mapped to the image plane coordinate system. Therefore, the smaller the difference between the fourth coordinate information obtained through the foregoing steps and the fifth coordinate information, the more accurate the external parameters obtained by the above method, and thus the more accurate the fourth coordinate information calculated based on the external parameters. Therefore, the external parameters can be calibrated based on the difference between the fourth coordinate information and the fifth coordinate information subsequently.
[0052] Exemplarily, please refer to Figure 5 , Figure 5 which shows a schematic diagram of the target object provided by an embodiment of the present application. In Figure 5 a target image 500 is shown, and the target image 500 includes a target calibration object 510.
[0053] Optionally, the user wearing the smart glasses can also keep the head almost stationary and then move the target calibration object, so as to achieve the effect that the target calibration object is in different positions in the target image. Specifically, for example, please refer to Figures 6 to 8 , where Figure 6 shows a schematic diagram of the target object provided by an embodiment of the present application; Figure 7 shows a schematic diagram of the target object provided by an embodiment of the present application; Figure 8 shows a schematic diagram of the target object provided by an embodiment of the present application.
[0054] In Figure 6 the target image 600 shown, it includes a target calibration object 610, and the target calibration object 610 has moved a certain distance to the right compared to the target calibration object 510 in the target image 500. Similarly, in Figure 7 the target image 700 shown, it includes a target calibration object 710, and the target calibration object 710 has moved a certain distance downward compared to the target calibration object 510 in the target image 500. Similarly, in Figure 8 the target image 800 shown, it includes a target calibration object 810, and the target calibration object 810 has moved a certain distance to the right and downward compared to the target calibration object 510 in the target image 500.
[0055] That is to say, for some embodiments, multiple target calibration objects can be set, and each target calibration object can correspond to fifth coordinate information and fourth coordinate information. Subsequently, the external parameters can be calibrated according to the fifth coordinate information and the fourth coordinate information corresponding to each target calibration object, so that the gaze point can be more accurately mapped to the image coordinate system through the calibrated external parameters.
[0056] Step S140: Calibrate the external parameters by the difference between the fifth coordinate information and the fourth coordinate information to reduce the difference.
[0057] For some embodiments, after obtaining the fourth coordinate information and the fifth coordinate information, the difference between the fifth coordinate information and the fourth coordinate information can be further obtained. Exemplarily, the fourth coordinate information and the fifth coordinate information can respectively include a vertical coordinate value and a horizontal coordinate value. Thus, the square value of the difference between the vertical coordinate value of the fifth coordinate information and the vertical coordinate value of the fourth coordinate information can be respectively obtained, and the square value of the difference between the horizontal coordinate value of the fifth coordinate information and the horizontal coordinate value of the fourth coordinate information can be obtained. Then, the sum of the above two square values is used as the difference between the fifth coordinate information and the fourth coordinate information.
[0058] Among them, calibrating the external parameters to reduce the difference can be to adjust the specific parameters in the external parameters to minimize the difference.
[0059] In some embodiments, a threshold difference can be preset. If the difference is less than or equal to the threshold difference, it can be considered that the accuracy of the external parameters obtained at this time is already relatively high, and the fourth coordinate information calculated by the external parameters also has relatively high accuracy. If the difference is greater than the threshold difference, the parameters in the external parameters can be continuously adjusted. For a detailed introduction, reference can be made to the subsequent embodiments.
[0060] The parameter calibration method provided by this application first obtains the external parameters of the image acquisition module relative to the world coordinate system through the first coordinate information of the target calibration object in the world coordinate system and the second coordinate information of the target calibration object in the image plane coordinate system, where the image plane coordinate system is a two-dimensional coordinate system established based on the image acquisition module; then maps the third coordinate information of the fixation point of the target calibration object in the world coordinate system obtained through the eye movement information acquisition module to the image plane coordinate system based on the external parameters to obtain the fourth coordinate information; and then obtains the fifth coordinate information of the reference fixation point of the target calibration object in the image plane coordinate system in the target image based on the image recognition algorithm, where the target image is an image including the target calibration object acquired by the image acquisition module; thereby calibrating the external parameters to reduce the difference through the difference between the fifth coordinate information and the fourth coordinate information. It can be seen that in the solution of this application, the fourth coordinate information is calculated, and the fifth coordinate information is recognized through the image recognition algorithm, and then the external parameters are calibrated through the difference between the fifth coordinate information and the fourth coordinate information to reduce the difference, so that the fixation point can be more accurately mapped to the image plane coordinate system.
[0061] Please refer to Figure 9 , Figure 9 which shows the flowchart of the parameter calibration method provided by the embodiment of this application. This parameter calibration method can be applied to Figure 1 the control module in the smart glasses shown in
[0062] Step S210: Obtain the first coordinate information of the target calibration object in the world coordinate system.
[0063] Step S220: Obtain the second coordinate information of the target calibration object in the image plane coordinate system.
[0064] Step S230: Obtain the internal parameters of the image acquisition module.
[0065] Step S240: Based on the first coordinate information, the second coordinate system information, and the internal parameters, obtain the external parameters of the image acquisition module relative to the world coordinate system.
[0066] In some embodiments, in order to obtain the external parameters, the first coordinate information of the target calibration object in the world coordinate system can be obtained first; then the second coordinate information of the target calibration object in the image plane coordinate system can be obtained. Among them, how to obtain the first coordinate information and the second coordinate information has been introduced in the foregoing embodiments and will not be elaborated here.
[0067] Furthermore, the internal parameters of the image acquisition module can also be obtained. The internal parameters can be used to describe the geometric and optical characteristics inside the image acquisition module and are used to project three-dimensional points in the camera coordinate system onto the two-dimensional image plane.
[0068] The internal parameters of the camera can be obtained in advance and stored in the smart glasses, so that the internal parameters can be directly retrieved from the smart glasses when needed later.
[0069] Optionally, the pre-obtained internal parameters of the camera can also be stored in an electronic device that has a communication connection with the smart glasses. For example, the electronic device is a smartphone. Thus, when the internal parameters are needed, the internal parameters can be obtained from the electronic device through the established communication connection.
[0070] Optionally, the pre-obtained internal parameters of the camera can also be stored in a cloud server. The smart glasses can communicate with the cloud server. Thus, when the internal parameters are needed, the content can be obtained from the cloud server. For example, the smart glasses can be configured with a communication module for accessing the Internet. Specifically, for example, the smart glasses can access the Internet through wireless communication technology Wi-Fi or cellular mobile technology.
[0071] Exemplarily, the internal parameters of the image acquisition module can include the focal length of the image acquisition module and the coordinates of the image center point. In some embodiments, the focal length of the image acquisition module can be represented by (f_x, f_y); the coordinates of the image center point can be represented by (c_x, c_y). Among them, f_x is used to represent the focal length in the x-axis direction, and f_y is used to represent the focal length in the y-axis direction. It should be noted that the above coordinates of the image center point can be the coordinates in the image coordinate system.
[0072] Furthermore, after obtaining the first coordinate information, the second coordinate information, and the internal parameters, the external parameters of the image acquisition module relative to the world coordinate system can be obtained.
[0073] For some embodiments, step S240 may further include steps S241 to S243.
[0074] Step S241: Construct a projection equation that projects the coordinates in the world coordinate system onto the coordinates in the image plane coordinate system based on the internal parameters.
[0075] Step S242: Determine the rotation matrix and the translation vector in the projection equation based on the first coordinate information and the second coordinate system information.
[0076] Step S243: Use the rotation matrix and the translation vector as the external parameters.
[0077] First, a projection equation for projecting coordinates in the world coordinate system onto the coordinates in the image plane coordinate system can be constructed based on the internal parameters. Among them, the projection equation may include a rotation matrix to be determined and a translation vector to be determined. It should be noted that the rotation matrix to be determined is used to represent that the specific parameters of the rotation matrix are to be determined; and the translation vector to be determined is used to represent that the specific parameters of the translation vector are to be determined.
[0078] It can be understood that the coordinate information in the world coordinate system can be mapped to the plane coordinate system through the projection equation. The first coordinate information is located in the world coordinate system, and the second coordinate information is located in the image plane coordinate system. Therefore, by substituting the aforementioned obtained first coordinate information and second coordinate information into the constructed projection equation, the specific parameters of the rotation matrix and the specific parameters of the translation vector can be solved, thereby obtaining the rotation matrix and the translation vector. And the rotation matrix and the translation vector can be used as the external parameters.
[0079] For some embodiments, the above method for solving the external parameters can also be referred to as the Perspective-n-Point (PnP) algorithm. The above rotation matrix can be represented by R; and the translation vector can be represented by T.
[0080] Step S250: Obtain the third coordinate information of the fixation point of the target calibration object in the world coordinate system through the eye movement information acquisition module.
[0081] Among them, the eye movement information acquisition module may correspond to attitude information; in addition, the line of sight of the user wearing the smart glasses also corresponds to direction data. Therefore, the third coordinate information can be obtained according to the attitude information and the direction data.
[0082] Step S250 may further include steps S251 to S253.
[0083] Step S251: Collect the direction data of the fixation point of the target calibration object through the eye movement information acquisition module.
[0084] Step S252: Obtain the attitude information corresponding to the eye movement information acquisition module.
[0085] Step S253: Obtain the third coordinate information of the fixation point in the world coordinate system through the attitude information and the direction data.
[0086] Specifically, the direction data of the fixation point for the target calibration object can be directly collected by the eye movement information collection module. It can be understood that this direction data can be used to represent the direction towards which the line of sight of the user wearing the smart glasses is oriented. In some embodiments, the direction data can be represented by d=(d_x, d_y, d_z).
[0087] Furthermore, even for the same direction data, if the attitude information corresponding to the eye movement information collection module is different, the subsequent obtained coordinate information of the fixation point will also change. Therefore, the attitude information corresponding to the eye movement information collection module can also be obtained.
[0088] It can be understood that the user wears the smart glasses, and the eye movement information collection module is disposed on the smart glasses. Therefore, the head attitude information of the user can be approximately used as the attitude information corresponding to the eye movement information collection module.
[0089] For some embodiments, an Inertial Measurement Unit (IMU) can also be provided in the smart glasses, and the inertial measurement unit is connected to the control module. Thus, the head attitude information of the user can be collected by the inertial measurement unit as the attitude information corresponding to the eye movement information collection module.
[0090] Exemplarily, the attitude information corresponding to the eye movement information collection module can include the rotation matrix corresponding to the user's head and the translation vector corresponding to the user's head.
[0091] Furthermore, based on the attitude information and the direction data, the third coordinate information of the fixation point in the world coordinate system is obtained. Among them, the third coordinate information of the fixation point in the world coordinate system can be obtained by calculation in combination with the attitude information and the direction data.
[0092] Step S260: Map the third coordinate information to the sixth coordinate information in the camera coordinate system based on the external parameters, where the camera coordinate system is a three-dimensional coordinate system established based on the image acquisition module.
[0093] Step S270: Project the sixth coordinate information onto the image plane coordinate system based on the perspective projection principle to obtain the fourth coordinate information in the image plane coordinate system.
[0094] After obtaining the third coordinate information and the extrinsic parameters, in order to obtain the fourth coordinate information in the image plane coordinate system, the third coordinate information can first be mapped to the sixth coordinate information in the camera coordinate system based on the extrinsic parameters. The camera coordinate system is a three-dimensional coordinate system established based on the image acquisition module. Then, based on the principle of perspective projection, the sixth coordinate information is projected onto the image plane coordinate system, thereby obtaining the fourth coordinate information in the image plane coordinate system. Thus, the fixation point is mapped to the image plane coordinate system.
[0095] Step S280: Identify the target image based on the image recognition algorithm, and obtain the region of interest in the target image as the reference fixation point for the target calibration object.
[0096] Step S290: Obtain the fifth coordinate information of the reference fixation point in the image plane coordinate system.
[0097] For some embodiments, when the image recognition algorithm recognizes the target image, it can be through the image recognition algorithm to recognize the region of interest in the target image, and then determine the region of interest as the reference fixation point for the target calibration object.
[0098] It should be noted that the image recognition algorithm can be pre-trained and adjusted to be able to recognize the target image included in the image and recognize the target image as the region of interest. Exemplarily, the recognition algorithm can be implemented through a recognition model, so that the recognition model can be pre-trained to enable the recognition model to recognize the target calibration object as the region of interest.
[0099] Thus, the fifth coordinate information of the reference fixation point in the image plane coordinate system can be further obtained. And this fifth coordinate information can be regarded as the standard position of mapping the fixation point for the target calibration object to the image plane coordinate system.
[0100] Step S2100: Obtain the difference between the fifth coordinate information and the fourth coordinate information.
[0101] Step S2101: Whether the difference is greater than the difference threshold.
[0102] Then, the difference between the fifth coordinate information and the fourth coordinate information can be further obtained, and then this difference is compared with the threshold difference.
[0103] When the difference is greater than the threshold difference, step S2110 can be executed by jumping; when the difference is less than or equal to the difference threshold, step S2120 can be executed by jumping.
[0104] Step S2110: When the difference is greater than the threshold difference, adjust the extrinsic parameters and return to execute mapping the third coordinate information of the fixation point of the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the adjusted extrinsic parameters, to obtain new fourth coordinate information and subsequent steps until the difference is less than or equal to the threshold difference, and complete the calibration of the extrinsic parameters.
[0105] Among them, when the difference is greater than the threshold difference, the extrinsic parameters can be adjusted and returned to execute mapping the third coordinate information of the fixation point of the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the adjusted extrinsic parameters, to obtain new fourth coordinate information and subsequent steps until the difference is less than or equal to the threshold difference, and complete the calibration of the extrinsic parameters.
[0106] It should be noted that in the embodiments provided in the present application, steps S260 and subsequent steps can be returned to execute.
[0107] In addition, adjusting the extrinsic parameters can be to adjust the specific parameters of the rotation matrix and translation vector included in the extrinsic parameters. And the new fourth coordinate information is obtained by mapping the third coordinate information based on the adjusted new extrinsic parameters. Further, after obtaining the new fourth coordinate information, the difference between the new fourth coordinate information and the fifth coordinate information can be obtained, and then the size relationship between this difference and the threshold difference can be judged, so as to judge whether to jump to execute step S2110 or step S2120 subsequently.
[0108] Therefore, for step S2110, steps S260 and subsequent steps will be continuously returned to execute until the difference is less than or equal to the threshold difference and the calibration of the extrinsic parameters is completed, then steps S260 and subsequent steps will not be returned to execute.
[0109] Step S2120: When the difference is less than or equal to the threshold difference, complete the calibration of the extrinsic parameters.
[0110] When the difference is less than or equal to the threshold difference, it can be considered that the difference between the current fourth coordinate information and the fifth coordinate information is already small, and it can be considered that the calibration of the extrinsic parameters has been completed.
[0111] The parameter calibration method provided by the embodiments of the present application compares the difference between the fourth coordinate information calculated according to the external parameters and the fifth coordinate information obtained through the image recognition algorithm. When the difference is greater than the threshold difference, the external parameters are adjusted and the process returns to execute mapping the third coordinate information of the fixation point of the target calibration object obtained by the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the adjusted external parameters to obtain new fourth coordinate information and subsequent steps until the difference is less than or equal to the threshold difference, completing the calibration of the external parameters; and when the difference is less than or equal to the threshold difference, the calibration of the external parameters is completed. Therefore, if the difference between the fourth coordinate information calculated and the fifth coordinate information is large, the external parameters can be further adjusted, thereby improving the accuracy of the fourth coordinate information calculated through the external parameters in the subsequent process. And the fourth coordinate information is the coordinate information obtained by mapping the fixation point to the plane coordinate system through the external parameters. Therefore, the accuracy of the coordinate information obtained by mapping the fixation point to the plane coordinate system can be improved.
[0112] Please continue to refer to Figure 1 , Figure 1 The control module 110 shown in can be used to calibrate the external parameters based on the parameter calibration method provided by the foregoing method embodiments.
[0113] Optionally, Figure 1 An inertial measurement unit 140 can also be provided in the smart glasses 100 shown in. The inertial measurement unit 140 is connected to the control module 110, and the inertial measurement unit 140 is used to collect the attitude information corresponding to the eye movement information acquisition module. For a detailed introduction, please refer to the foregoing method embodiments and will not be elaborated here.
[0114] In some embodiments, the smart glasses may include a front frame and two temple arms connected to the frame. Thus, the eye movement information acquisition module 120 can be disposed on the front frame; the control module 110, the image acquisition module 130 are disposed on the same temple arm; and a counterweight module 150 is disposed on the other temple arm to maintain the balance of the smart glasses 100.
[0115] It can be understood that in order to implement the above method functions of the smart glasses 100, electronic devices such as a circuit board and a battery are also provided in the smart glasses, which are not shown one by one here.
[0116] Optionally, a memory ( Figure 1 not shown in ) can also be provided in the wearable device 100, and the memory is connected to the control module 110.
[0117] Among them, the control module 110 may include one or more processing cores. The control module 110 connects various parts within the entire wearable device 100 using various interfaces and circuits. By running or executing instructions, programs, code sets, or instruction sets stored in the memory, and by calling data stored in the memory, it performs various functions of the wearable device 100 and processes data. Optionally, the control module 110 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The control module 110 may integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, computer programs, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the control module 110 and may be implemented separately through a communication chip. Specifically, the method described in the foregoing embodiment may be executed by one or more control modules 110.
[0118] For some embodiments, the memory may include random access memory (RAM) and may also include read-only memory. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created during the use of the wearable device 100.
[0119] Please refer to Figure 10 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable medium 1000, and the program code can be called by a processor to execute the method described in the above method embodiments.
[0120] The computer-readable storage medium 1000 can be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 1000 has a storage space for program code 1010 that executes any of the method steps in the above-described methods. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed in a suitable form, for example.
[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A parameter calibration method, characterized in that, A control module applied to smart glasses, the smart glasses further comprising an image acquisition module and an eye movement information acquisition module, the control module being respectively connected to the image acquisition module and the eye movement information acquisition module, the method comprising: Obtaining the external parameters of the image acquisition module relative to the world coordinate system through the first coordinate information of the target calibration object in the world coordinate system and the second coordinate information of the target calibration object in the image plane coordinate system, wherein the image plane coordinate system is a two-dimensional coordinate system established based on the image acquisition module; Mapping the third coordinate information of the fixation point on the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the external parameters to obtain fourth coordinate information; Obtaining the fifth coordinate information of the reference fixation point on the target calibration object in the image plane coordinate system in the target image based on an image recognition algorithm, wherein the target image is an image including the target calibration object acquired by the image acquisition module; Calibrating the external parameters through the difference between the fifth coordinate information and the fourth coordinate information to reduce the difference.
2. The method according to claim 1, characterized in that The obtaining the external parameters of the image acquisition module relative to the world coordinate system through the first coordinate information of the target calibration object in the world coordinate system and the second coordinate information of the target calibration object in the image plane coordinate system includes: Obtaining the first coordinate information of the target calibration object in the world coordinate system; Obtaining the second coordinate information of the target calibration object in the image plane coordinate system; Obtaining the internal parameters of the image acquisition module; Obtaining the external parameters of the image acquisition module relative to the world coordinate system based on the first coordinate information, the second coordinate system information and the internal parameters.
3. The method according to claim 2, wherein The obtaining the external parameters of the image acquisition module relative to the world coordinate system based on the first coordinate information, the second coordinate system information and the internal parameters includes: Constructing a projection equation for projecting coordinates in the world coordinate system to coordinates in the image plane coordinate system based on the internal parameters; Determining the rotation matrix and the translation vector in the projection equation based on the first coordinate information and the second coordinate system information; Taking the rotation matrix and the translation vector as the external parameters.
4. The method according to claim 1, wherein The mapping the third coordinate information of the fixation point on the target calibration object obtained through the eye movement information acquisition module in the world coordinate system to the image plane coordinate system based on the external parameters to obtain fourth coordinate information includes: Obtaining the third coordinate information of the fixation point on the target calibration object in the world coordinate system through the eye movement information acquisition module; Mapping the third coordinate information to the sixth coordinate information in the camera coordinate system based on the external parameters, wherein the camera coordinate system is a three-dimensional coordinate system established based on the image acquisition module; Projecting the sixth coordinate information to the fourth coordinate information in the image plane coordinate system based on the perspective projection principle.
5. The method according to claim 4, wherein The obtaining of the third coordinate information of the fixation point of the target calibration object in the world coordinate system through the eye movement information acquisition module includes: Collecting the direction data of the fixation point of the target calibration object through the eye movement information acquisition module; Obtaining the attitude information corresponding to the eye movement information acquisition module; Obtaining the third coordinate information of the fixation point in the world coordinate system through the attitude information and the direction data.
6. The method according to claim 1, wherein The obtaining of the fifth coordinate information of the reference fixation point of the target calibration object in the image plane coordinate system based on the image recognition algorithm includes: Identifying the target image based on the image recognition algorithm to obtain the region of interest in the target image as the reference fixation point of the target calibration object; Obtaining the fifth coordinate information of the reference fixation point in the image plane coordinate system.
7. The method according to claim 1, wherein The calibration of the external parameters by reducing the difference between the fifth coordinate information and the fourth coordinate information includes: Obtaining the difference between the fifth coordinate information and the fourth coordinate information; In the case where the difference is greater than the threshold difference, adjusting the external parameters and returning to execute mapping the third coordinate information of the fixation point of the target calibration object obtained through the eye movement information acquisition module to the image plane coordinate system based on the adjusted external parameters to obtain new fourth coordinate information and subsequent steps until the difference is less than or equal to the threshold difference, and completing the calibration of the external parameters.
8. An intelligent glasses, characterized in that, Including a control module, an image acquisition module, and an eye movement information acquisition module, the control module is respectively connected to the image acquisition module and the eye movement information acquisition module; The control module is used to calibrate the external parameters based on the method according to any one of claims 1-7.
9. The smart glasses according to claim 8, characterized in that, The smart glasses further include an inertial measurement unit, and the inertial measurement unit is connected to the control module; The inertial measurement unit is used to collect the attitude information corresponding to the eye movement information acquisition module.
10. A computer-readable storage medium, characterized in that, Program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the method according to any one of claims 1-7.