Light spot matching method and electronic equipment

Abstract

The invention provides a light spot matching method and electronic equipment. The accuracy and efficiency of light spot matching are improved. Comprising the following steps: performing pupil detection and light spot detection on a human eye image to obtain a position coordinate of a pupil center and a pixel position coordinate of each light spot; determining a to-be-matched LED lamp ring projection table corresponding to the position coordinates of the pupil centers according to the corresponding relation between the position coordinates of the pupil centers and the LED lamp ring projection tables; for any light spot, determining a light spot matching pair corresponding to the light spot based on an LED lamp matched with the light spot obtained by using the pixel position coordinates of the light spot and the to-be-matched LED lamp ring projection table; obtaining each curvature center according to each light spot matching pair; determining a first target curvature center by using the curvature centers obtained by screening the curvature centers; and for the light spot in any light spot matching pair, updating the light spot matching pair by using a target LED lamp which is matched with the light spot and is obtained based on the first target curvature center and the image position coordinate of the light spot.

Description

Background Art

[0002] Virtual reality (VR) allows users to experience immersion and interact with virtual environments, making users feel as if they are actually there. Similarly, mixed reality (MR) or augmented reality (AR) systems combine computer-generated virtual information with the real world or real-world images captured by a camera to provide a visual perspective of information. This device that puts users in the first-person perspective can provide a variety of virtual scenes. For example, virtual environment training, mechanical system interaction, games, and remote meetings.

[0003] The biggest feature of these head-mounted display electronic devices is the immersion of the first-person perspective. Compared with traditional interaction methods, such as control handles, voice, gestures, etc., the interaction method of eye tracking has great advantages. The current vision-based eye tracking algorithms include dark field method, bright field method, corneal reflection method and retinal reflection method. Among them, the corneal reflection method has attracted much attention because it is non-invasive (contactless), suitable for the light field environment of head-mounted devices, and has the advantages of low cost. However, the low matching accuracy of the light spot has always been one of the limitations of the large-scale application of this algorithm.

[0004] like Figure 1 As shown in FIG. 1 , it is a schematic diagram of a head-mounted display device including an eye tracking system, and the head-mounted display device includes a near-eye display screen unit 110, a lens group 120, an infrared / near-infrared LED light ring 130, an infrared camera 140, and a system control unit 150. Among them, the near-eye display screen unit 110 is responsible for providing display images to the user. The system control unit 150 is responsible for obtaining camera images and various sensor data, processing system and algorithm services, rendering images, and sending them to the near-eye display screen for display. The lens group 120 allows the user to see a stereoscopic picture. The eye tracking system unit includes an infrared / near-infrared LED light ring 130 and an infrared camera 140, which are used to detect the user's eye position and gaze point, that is, when the user uses the head-mounted display device, the infrared camera 140 captures a human eye image, and the human eye image includes the human eye and the corresponding light spot displayed by the infrared / near-infrared LED light ring 130. In order to locate the human eye, it is necessary to match the light spot in the human eye image with each LED light in the infrared / near-infrared LED light ring 130.

[0005] In the prior art, the method of spot matching is: projecting the LED light ring in each human eye picture to obtain the projection pixel coordinates of each LED light in the LED light ring, and for any LED light, determining the average value of the projection pixel coordinates of the LED light in each human eye picture as the target pixel coordinate of the LED light, and then determining each distance based on the pixel position coordinates of each LED light and each spot, and performing spot matching based on each distance. However, this method has a high probability of mismatching, resulting in a low accuracy rate of spot matching. Summary of the invention

[0006] The present application provides a light spot matching method and electronic equipment, which are used to improve the accuracy of light spot matching.

[0007] In a first aspect, an embodiment of the present application provides a method for matching a light spot, the method comprising:

[0008] When the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot;

[0009] Using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light in the LED light ring;

[0010] For any one of the light spots, using the pixel position coordinates of the any one light spot and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light matching the any one light spot, and based on the LED light matching the any one light spot, obtain a light spot matching pair corresponding to the any one light spot;

[0011] Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively;

[0012] Using a clustering algorithm to screen the multiple curvature centers to obtain screened curvature centers, and based on the screened curvature centers, obtaining a first target curvature center;

[0013] For any light spot in the light spot matching pair, the light spot is verified by using the first target curvature center and the image position coordinates of the light spot to obtain a target LED light matching the light spot, and the target LED light is used to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

[0014] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0015] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0016] When the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot;

[0017] Using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light in the LED light ring;

[0018] For any one of the light spots, using the pixel position coordinates of the any one light spot and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light matching the any one light spot, and based on the LED light matching the any one light spot, obtain a light spot matching pair corresponding to the any one light spot;

[0019] Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively;

[0020] Using a clustering algorithm to screen the multiple curvature centers to obtain screened curvature centers, and based on the screened curvature centers, obtaining a first target curvature center;

[0021] For any light spot in the light spot matching pair, the light spot is verified by using the first target curvature center and the image position coordinates of the light spot to obtain a target LED light matching the light spot, and the target LED light is used to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

[0022] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0023] In the above-mentioned embodiment of the present application, the corresponding relationship between the preset position coordinates of each pupil center and each LED light ring projection table is used to determine the LED light ring projection table to be matched corresponding to the position coordinates of the user's pupil center, and then each spot matching pair is obtained based on the LED light ring projection table to be matched, and multiple curvature centers are obtained through each spot matching pair, and the curvature centers are screened to obtain the first target curvature center, and the first target curvature center and the image position coordinates of each light spot are used to perform a spot retroreflection line check on each light spot, and each spot matching pair is updated to obtain an updated spot matching pair. Therefore, in the embodiment of the present application, each curvature center corresponding to the initially obtained spot matching pair is screened, and then each spot matching pair is checked based on the screened curvature center to obtain each updated spot matching pair. Therefore, the probability of spot mismatching is reduced and the accuracy of spot matching is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 One of the structural schematic diagrams of the electronic device provided by the embodiment of the present application is exemplarily shown;

[0026] Figure 2 A schematic diagram of an eyeball model provided in an embodiment of the present application is exemplarily shown;

[0027] Figure 3 The following is a schematic diagram showing the process of the user verification stage provided by the embodiment of the present application;

[0028] Figure 4A schematic diagram of a calibration image provided by an embodiment of the present application is exemplarily shown;

[0029] Figure 5 The schematic diagram of the process of retroreflective line verification provided by the embodiment of the present application is exemplarily shown;

[0030] Figure 6 The flowchart for determining multiple centers of curvature provided in an embodiment of the present application is exemplarily shown;

[0031] Figure 7 The schematic diagram of the process of determining the projection pixel position coordinates corresponding to each LED light provided in the embodiment of the present application is exemplarily shown;

[0032] Figure 8 The schematic diagram exemplarily shows the correspondence between the position coordinates of each pupil center and the projection table of each LED light ring provided in the embodiment of the present application;

[0033] Fig. 9 The schematic diagram exemplarily shows the flow chart of the light spot matching method provided in the embodiment of the present application;

[0034] Fig.10 A schematic diagram of spot matching provided by an embodiment of the present application is exemplarily shown;

[0035] Fig.11 The schematic diagram of screening multiple centers of curvature provided in the embodiment of the present application is exemplarily shown;

[0036] Fig.12 The flowchart of the method for matching light spots provided in the embodiment of the present application is exemplarily shown;

[0037] Fig.13 The schematic diagram of the matching device for light spots provided in the embodiment of the present application is exemplarily shown;

[0038] Fig.14 The second schematic diagram of the electronic device provided in the embodiment of the present application is exemplarily shown. DETAILED DESCRIPTION

[0039] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0040] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0041] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0042] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0043] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0044] The idea of ​​the embodiments of the present application is summarized below.

[0045] The light spot matching method in the prior art is: projecting the LED light ring in each human eye picture to obtain the projection pixel coordinates of each LED light in the LED light ring, and for any LED light, determining the average value of the projection pixel coordinates of the LED light in each human eye picture as the target pixel coordinate of the LED light, and then determining each distance based on the pixel position coordinates of each LED light and each light spot, and performing light spot matching based on each distance. However, this method has a high probability of mismatching, resulting in a low accuracy rate of light spot matching.

[0046] Based on the problem of low accuracy of light spot matching in the prior art. The present application proposes a light spot matching method, which determines the LED light ring projection table to be matched corresponding to the position coordinates of the user's pupil center through the preset correspondence between the position coordinates of each pupil center and each LED light ring projection table, and then obtains each light spot matching pair based on the LED light ring projection table to be matched, and obtains multiple curvature centers through each light spot matching pair, and screens the curvature centers to obtain the first target curvature center, and uses the first target curvature center and the image position coordinates of each light spot to perform light spot retroreflection line verification on each light spot, and updates each light spot matching pair to obtain an updated light spot matching pair. Therefore, in the embodiment of the present application, the curvature centers corresponding to the initially obtained light spot matching pairs are screened, and then the light spot matching pairs are verified based on the screened curvature centers to obtain the updated light spot matching pairs. Therefore, the probability of mismatching of light spots is reduced and the accuracy of light spot matching is improved.

[0047] Before introducing the spot matching method in the embodiment of the present application, the eyeball model and line of sight estimation are first described. Figure 2 As shown in the figure, the circular object on the left is the eyeball, point d is the rotation center of the eyeball, point p is the pupil center, point c is the curvature center of the cornea, point o is the optical center of the infrared camera, and the focal plane is the focal plane of the infrared camera in the camera coordinate system. After refraction, the pupil center p passes through point r at the cornea to the optical center o of the camera, and passes through point v on the focal plane.

[0048] li is an LED light in the infrared LED light ring. The light emitted by li hits the cornea and is reflected at point qi before reaching the optical center o of the camera. There is an intersection point ui with the focal plane. The vector formed by the center of curvature c and the center of pupil p is called the optical axis, and the vector formed by the retinal groove fovea and the center of curvature c is called the visual axis, which marks the actual gaze direction of the user.

[0049] The process of a complete eye tracking gaze estimation system includes two parts: the user calibration stage and the user use stage. Among them, the user calibration stage is an auxiliary application of the eye tracking system. After entering the program, the interface will prompt the user to look at the dots in each calibration image displayed in a certain order until the calibration is completed. The function of the user calibration stage is to model the user's eyeballs, calculate and record the necessary user calibration data, such as the projection coordinate information of the LED light ring, etc., to improve the accuracy of the user's gaze estimation during the use process. The user use stage will be entered after the user calibration stage is completed, and the user's gaze point will be output after a series of processes such as pupil detection, corneal reflection spot detection, and spot matching.

[0050] Next, the method of the user verification stage in the embodiment of the present application is first introduced in conjunction with the accompanying drawings. Figure 3 FIG. 1 is a flow chart of the user verification phase, which may specifically include the following steps:

[0051] Step 301: after entering the calibration phase, a plurality of human eye images of the user are obtained, wherein any one of the human eye images is obtained when the user gazes at a calibration point in a calibration image displayed in an electronic device, and the number of the plurality of human eye images is the same as the number of calibration images;

[0052] like Figure 4 As shown in the figure, it is a schematic diagram of the calibration image. Figure 4 It can be seen that the display of each calibration point is pre-set, and is displayed in sequence to allow the user to gaze at the displayed calibration point. Each time the user gazes at a calibration point, the corresponding human eye image is obtained through an electronic device (head-mounted display device, etc.).

[0053] Step 302: for any one of the multiple human eye images, perform pupil detection on the any one of the multiple human eye images to obtain the position coordinates of the pupil center of the any one of the human eye images in the any one of the human eye images; and perform light spot detection on the any one of the human eye images to obtain the pixel position coordinates of each light spot in the any one of the human eye images, and project the pixel position coordinates of each light spot into the camera focal plane to obtain the image position coordinates of each light spot respectively;

[0054] The pupil detection method in the embodiment of the present application is: input any one of the human eye images into the neural network model mobileOne-s1 for target detection, obtain the predicted pupil image coordinate rectangular frame position, perform ellipse fitting on the image of the rectangular frame position, and obtain the position coordinates of the pupil center in any one of the human eye images.

[0055] The method of light spot detection in the embodiment of the present application can be: using the SimpleBlobDetector function of the open source machine vision library opencv to perform light spot detection on any one of the human eye images to obtain the pixel position coordinates of each light spot in the any one of the human eye images.

[0056] In the embodiment of the present application, some key parameter settings of the SimpleBlobDetecor function can be "filterByColor=true, blobColor=255, filterByArea=true, minArea=4, maxArea=30". The specific key parameter settings can be set according to actual conditions, and the embodiment of the present application does not limit the setting of the key parameters of the function.

[0057] In the embodiment of the present application, the pixel position coordinates of each light spot are projected onto the focal plane of the camera by using the projection function projectPoints of the open source library opencv, substituting the preset infrared camera distortion parameters into the pixel position coordinates of each light spot onto the focal plane to obtain the image plane coordinates of each light spot.

[0058] It should be noted that the pupil detection method, light spot detection method and projection method in the embodiments of the present application are only used for illustration, and the pupil detection method, light spot detection method and projection method in the embodiments of the present application are not limited. The pupil detection method, light spot detection method and projection method in the embodiments of the present application can be set according to actual conditions.

[0059] In order to further improve the efficiency of light spot matching, in one embodiment, before executing step 303, for any one of the light spots, based on the pixel position coordinates of the any one light spot and the position coordinates of the pupil center of the user, the distance between the any one light spot and the pupil center is obtained; if the distance between the any one light spot and the pupil center is greater than a preset iris radius, the any one light spot is deleted; or, if the distance between the any one light spot and the pupil center is not greater than a preset iris radius, the any one light spot is retained.

[0060] Since only the light spots within the iris of the human eye are needed in the embodiment of the present application, it is necessary to filter the light spots and only retain the light spots within the iris, thereby improving the efficiency and accuracy of light spot matching.

[0061] Step 303: for any one of the light spots, using the initial curvature center and the image position coordinates of the light spot, perform retroreflection line verification on the light spot to obtain a target LED lamp matching the light spot; wherein the multiple human eye images correspond to an initial curvature center respectively, and the initial curvature center corresponding to any one of the human eye images is obtained based on the image position coordinates of the pupil center of the any one of the human eye images and the calibration image corresponding to the any one of the human eye images;

[0062] Below, first, the method for determining the initial curvature center corresponding to any human eye image is described. Since this method belongs to the prior art, it is only briefly described in the embodiment of the present application: the user's eye parameters are obtained by using the position coordinates of the user's pupil center in the human eye image and the standard visual axis vector corresponding to each human eye image, wherein the eye parameters include the position vector of the eye center, the radius of curvature, the first distance, the second distance and the target angle, the first distance is the distance between the pupil center and the eye center, the second distance is the distance between the curvature center and the eye center, and the target angle includes the horizontal angle and the vertical angle between the visual axis and the optical axis; the current human eye image is obtained at specified intervals, and the current eye rotation angle of the user is obtained based on the user's eye parameters and the position coordinates of the pupil center in the current human eye image; and the user's curvature center is obtained based on the user's current eye rotation angle and the user's eye parameters. The user's curvature center can be obtained by formula (1):

[0063]

[0064] in, is the position vector corresponding to the current curvature center of the user, is the position vector of the eyeball center, D is the second distance, φ eye ′ is the current second eyeball rotation angle, θ eye ' is the current first eye rotation angle.

[0065] Next, the retroreflective line verification in step 303 in the embodiment of the present application is described in detail. Figure 5 The figure is a schematic diagram of the process of retroreflective line verification, which may specifically include the following steps:

[0066] Step 501: for any light spot, based on the image position coordinates of the light spot and the third target curvature center, obtain the position coordinates of the corneal intersection point corresponding to the light spot in the camera coordinate system, wherein the corneal intersection point is the intersection point of the line connecting the image position coordinates of the light spot and the camera optical center and the cornea of ​​the user's human eye, and the third target curvature center is the first target curvature center or the initial curvature center;

[0067] Since the spot retroreflection line calibration is required in both the calibration stage and the use stage in the embodiment of the present application, and the method is the same, the third target curvature center in the embodiment of the present application can be the initial curvature center in the calibration stage or the first target curvature center in the user use stage. If it is the calibration stage, the third target curvature center is the initial curvature center. If it is the user use stage (that is, after the user enters the target scene), the third target curvature center is the first target curvature center. The position coordinates of the corneal intersection point corresponding to any one of the light spots in the camera coordinate system can be obtained by formula (2):

[0068] q i =o+t 1 *(u i -o)......(2);

[0069] Among them, q i is the position coordinate of the corneal intersection point corresponding to any one of the light spots i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, u i is the image position coordinate of any light spot i, t 1 is a first coefficient corresponding to any one of the light spots;

[0070] In the embodiment of the present application, the first coefficient t corresponding to any one of the light spots can be obtained by formula (3): 1 :

[0071] (o x +m*t 1 -c x ) 2 +(o y +n*t 1 -c y ) 2 +(o z +p*t 1 -c z ) 2 =R 2 .......(3);

[0072] Among them, x is the horizontal position coordinate of the optical center of the camera in the camera coordinate system, o y is the vertical position coordinate of the camera optical center in the camera coordinate system, o zis the vertical position coordinate of the camera optical center in the camera coordinate system, m is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, n is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, p is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, c x is the abscissa of the initial curvature center, c y is the ordinate of the initial curvature center, c z is the vertical coordinate of the initial curvature center, and R is the preset curvature radius.

[0073] Step 502: obtaining a first vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the third target curvature center; and obtaining a second vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the camera optical center;

[0074] In one embodiment, the first vector is obtained by subtracting the position coordinates of the corneal intersection point in the camera coordinate system from the third target curvature center to obtain the first vector.

[0075] In one embodiment, the second vector is obtained by: subtracting the position coordinates of the corneal intersection point in the camera coordinate system from the camera optical center to obtain the second vector.

[0076] Step 503: Dividing the first vector by a first target value to obtain a first target vector, and dividing the second vector by a second target value to obtain a second target vector, wherein the first target value is equal to the modulus length of the first vector, and the second target value is equal to the modulus length of the second target vector;

[0077] Step 504: obtaining the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system according to the first target vector and the second target vector;

[0078] The retroreflective line iray in the embodiment of the present application is a ray that is reflected by the second vector and is directed toward the infrared LED light ring, and the retroreflective line is symmetrical with respect to the first vector and the second vector.

[0079] The position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system are obtained by formula (4):

[0080]

[0081] Wherein, iray is the vector corresponding to the position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system, is the second target vector, is the first target vector.

[0082] Step 505: For any LED light in the LED light ring, the distance between the any LED light and the reverse reflection line of the second vector is obtained by using the preset position coordinates of the any LED light in the camera coordinate system and the position coordinates of the reflection point in the camera coordinate system; wherein the distance between the any LED light and the reverse reflection line of the second vector can be obtained by formula (5):

[0083]

[0084] Among them, d n is the distance between the nth LED light and the retroreflected line of the second vector, x iray is the horizontal position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, y iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, z iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, is the horizontal position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, t 2 is the second coefficient.

[0085] Step 506: Determine the LED lamp with the smallest distance from the retroreflected line of the second vector among the LED lamps as the target LED lamp matching the any one light spot.

[0086] In one embodiment, the second coefficient t is obtained by formula (6): 2 :

[0087]

[0088] Among them, iray is the vector corresponding to the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system.

[0089] Step 304: Obtain a light spot matching pair using target LED lights that match the light spots respectively;

[0090] Any light spot matching pair in the embodiments of the present application includes a light spot and a target LED lamp matching the light spot.

[0091] Step 305: obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; and updating the initial curvature center based on the plurality of curvature centers to obtain a second target curvature center;

[0092] In the embodiment of the present application, the method of determining multiple centers of curvature based on the matching pairs of each spot in the calibration stage and the user use stage is the same. The method of determining multiple centers of curvature in step 305 is introduced below. Figure 6 As shown, it is a schematic diagram of a process for determining multiple centers of curvature, which may specifically include the following steps:

[0093] Step 601: randomly selecting two spot matching pairs from the spot matching pairs;

[0094] The set of two spot matching pairs selected each time in the present embodiment is different from the set of two spot matching pairs randomly selected before. The total number of two spot matching pairs selected in the present embodiment is

[0095] Step 602: Determine whether the two spot matching pairs are the same as the two spot matching pairs randomly selected before, if yes, return to step 601, if no, execute step 603;

[0096] Step 603: Based on the two spot matching pairs, the intersection vectors of the planes corresponding to the two spot matching pairs are obtained, wherein the plane corresponding to any one spot matching pair is a plane surrounded by the camera optical center and the spot in any one spot matching pair and the LED light; wherein the intersection vectors of the planes corresponding to the two spot matching pairs can be obtained by formula (7):

[0097]

[0098] in, is the intersection vector, u i is the image position coordinate of spot i in spot matching pair i, l i is the position coordinate of LED light i in the spot matching pair i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, u j are the image position coordinates of the light spot j in the light spot matching pair j, and lj are the position coordinates of the LED light j in the light spot matching pair j in the camera coordinate system.

[0099] Step 604: Based on the intersection vectors of the planes corresponding to the two spot matching pairs, obtain the unit vector of the intersection vector; wherein the unit vector of the intersection vector can be described by formula (8):

[0100]

[0101] in, is the unit vector of the intersection vector.

[0102] Step 605: Obtain the curvature center corresponding to the two light spot matching pairs according to the unit vector of the intersection line vector. The curvature center corresponding to the two light spot matching pairs can be obtained by formula (9):

[0103]

[0104] Where c′ is the center of curvature corresponding to the two matching pairs of light spots, k c is the first proportionality coefficient.

[0105] In the embodiment of the present application, the first proportionality coefficient can be obtained by formula (10):

[0106]

[0107] Wherein, ln is the position coordinate of the LED light in any one of the two spot matching pairs in the camera coordinate system, q n is the position coordinate of the corneal intersection point corresponding to the light spot in any one of the two light spot matching pairs in the camera coordinate system, u n is the image position coordinate of the light spot in any one of the two light spot matching pairs, k q is the second proportional coefficient, R is the preset radius of curvature, and c is the initial curvature center.

[0108] The second proportionality coefficient k in the embodiment of the present application q is also an unknown number, so in the three-row equation matrix corresponding to formula (10) in the embodiment of the present application, there are two unknown numbers k q and k c , the other parameters are known, and the rank of the matrix is ​​3. The equation corresponding to formula (10) is an overdetermined equation, and the least squares method can be used to optimize the number k q and k c The solution.

[0109] However, the embodiment of the present application does not limit the solution method of formula (10), and the specific method can be set according to the specific actual situation.

[0110] Step 306: based on the second target curvature center and the position coordinates of each LED light in the preset LED light ring in the camera coordinate system, obtaining the projection pixel position coordinates corresponding to each LED light, wherein the projection pixel position coordinates of any LED light are used to represent the position coordinates of the projection of any LED light in the pixel plane;

[0111] like Figure 7 As shown, it is a schematic diagram of a process of determining the projection pixel position coordinates corresponding to each LED light, which may specifically include the following steps:

[0112] Step 701: For any LED light in the LED light ring, based on the position coordinates of the any LED light in the camera coordinate system, the second target curvature center, the position coordinates of the camera optical center in the camera coordinate system, and the preset curvature radius, obtain the position coordinates of the corneal intersection point corresponding to the any LED light in the camera coordinate system; wherein the position coordinates of the corneal intersection point corresponding to the LED light in the camera coordinate system can be obtained by formula (11):

[0113]

[0114] Among them, l m is the position coordinate of the mth LED light in the camera coordinate system, q m is the position coordinate of the corneal intersection point corresponding to the mth LED light in the camera coordinate system, c″ is the second target curvature center, o is the position coordinate of the camera optical center in the camera coordinate system, and R is the preset curvature radius.

[0115] Step 702: Project the position coordinates of the corneal intersection point corresponding to any one of the LED lights in the camera coordinate system using a preset projection function to obtain the projection pixel position coordinates corresponding to the any one of the LED lights.

[0116] The projection function in the embodiment of the present application is the projection function projectPoints of the open source library opencv mentioned above, and the embodiment of the present application will not be repeated here.

[0117] Step 307: Obtaining an LED light ring projection table corresponding to any one of the human eye images through the projection pixel position coordinates corresponding to each of the LED lights;

[0118] In the embodiment of the present application, the LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light.

[0119] Step 308: Associating the position coordinates of the pupil center of any one of the human eye images with the LED light ring projection table corresponding to the any one of the human eye images to obtain a sub-correspondence relationship of the any one of the human eye images;

[0120] Step 309: Obtain the correspondence between the position coordinates of each pupil center and the projection table of each LED light ring according to the respective sub-correspondences of the plurality of human eye images.

[0121] Since the embodiment of the present application is explained by taking 9 calibration images as an example, the obtained sub-correspondences are also 9. However, in order to further improve the accuracy of spot matching, it is necessary to interpolate the sub-correspondences to obtain other sub-correspondences so that the obtained correspondences between the position coordinates of each pupil center and each LED light ring projection table are richer. In the embodiment of the present application, the mean of the position coordinates of the pupil center of each of the two adjacent human eye images is determined as the new position coordinates of the pupil center, and based on the mean of the projection pixel position coordinates corresponding to the same LED light in the LED light ring projection table of each of the two adjacent human eye images, a new LED light ring projection table is obtained, and the position coordinates of the new pupil center are corresponded to the new LED light ring projection table to obtain other sub-correspondences; the sub-correspondences of the multiple human eye images and each other sub-correspondence are determined to determine the correspondences between the position coordinates of each pupil center and each LED light ring projection table.

[0122] like Figure 8 As shown in FIG. 1 , the corresponding relationship between the position coordinates of each pupil center and the projection table of each LED light ring is shown in FIG. Figure 8 It can be seen that the position coordinates of each pupil center correspond to an LED light ring projection table.

[0123] After the introduction of the calibration stage, the matching method of the light spot in the embodiment of the present application is introduced below in conjunction with the accompanying drawings. Fig. 9 FIG. 1 is a flow chart of a method for matching light spots, which may specifically include the following steps:

[0124] Step 901: when the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot;

[0125] The target scene in the embodiment of the present application may be virtual environment training, mechanical system interaction, games, remote conferences, etc. The specific setting can be made according to the actual situation, and the embodiment of the present application does not limit the target scene.

[0126] The method of pupil detection and light spot detection in the embodiment of the present application is the same as that in the calibration stage described above, and the embodiment of the present application will not be repeated here.

[0127] In order to improve the accuracy and efficiency of light spot matching, before executing step 902, it is necessary to filter the detected light spots. The filtering method is the same as the filtering method described in the corresponding content of step 302 in the calibration stage described above, and the embodiment of the present application will not be repeated here.

[0128] Step 902: using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light in the LED light ring;

[0129] Step 903: for any one of the light spots, using the pixel position coordinates of the any one light spot and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light matching the any one light spot, and based on the LED light matching the any one light spot, obtain a light spot matching pair corresponding to the any one light spot;

[0130] In an example, step 903 can be specifically implemented as follows: for any LED lamp in the projection table of the LED light ring to be matched, based on the projection pixel position coordinates of the LED lamp and the pixel position coordinates of the any light spot, obtain the distance between the any light spot and the projection spot of the LED lamp; and determine the LED lamp with the smallest value of the distance between the projection spot and the any light spot in the projection table of the LED light ring to be matched as the LED lamp that matches the any light spot.

[0131] like Fig.10 As shown, it is a schematic diagram of spot matching, glint1 and glint2 are two spots after detection and filtering, and the light spot glint1 is taken as an example to illustrate the determination of the LED lamp matching any one of the light spots. Fig.10The two class ellipses in are projections corresponding to the two LED light rings, wherein the dots in any class ellipse represent the LED lights in the LED light ring. Since the projection table of the LED light ring to be matched corresponding to the light spot has been determined based on the corresponding relationship, in the embodiment of the present application, the projection table of the LED light ring to be matched corresponding to the light spot glint1 is taken as LED-id1 as an example, then glint1 needs to determine the distance between the projection pixel position coordinates of each LED light in LED-id1, and if it is determined that the distance between glint1 and p2_id1 in LED-id1 is the smallest, then the LED light matching the light spot glint1 is determined to be p2_id1.

[0132] In the embodiments of the present application, a light spot matching pair corresponding to any light spot includes the light spot and an LED lamp matching the light spot.

[0133] Step 904: obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively;

[0134] The method of determining multiple centers of curvature in the embodiment of the present application is the same as the method of determining multiple centers of curvature in the calibration stage described above, and the details are shown in steps 601 to 605. The embodiment of the present application will not be described in detail here.

[0135] Step 905: Filter the multiple curvature centers using a clustering algorithm to obtain filtered curvature centers, and obtain a first target curvature center based on the filtered curvature centers;

[0136] The clustering algorithm used in the embodiment of the present application is the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, but the embodiment of the present application does not limit the clustering algorithm here. The clustering algorithm in the embodiment of the present application can be set according to actual conditions.

[0137] Below, we take the DBSCAN algorithm as an example to briefly explain how to screen multiple curvature centers:

[0138] For any one of the multiple curvature centers, if the target range corresponding to the any one curvature center includes more than a specified number of curvature centers, then the any one curvature center and other curvature centers within the target range corresponding to the any one curvature center are retained; wherein the target range corresponding to the any one curvature center is a range with the any one curvature center as the origin and the specified threshold as the radius; if the target range corresponding to the any one curvature center does not include more than a specified number of curvature centers, and the any one curvature center is not within the target ranges corresponding to other curvature centers, then the curvature center is deleted.

[0139] like Fig.11 As shown, the curvature center a and the curvature center b are the curvature centers that need to be deleted determined based on the DBSCAN algorithm.

[0140] In one embodiment, the average value of the screened curvature centers is determined as the first target curvature center.

[0141] Step 906: For any light spot in the light spot matching pair, use the first target curvature center and the image position coordinates of the light spot to perform a light spot retroreflection line check on the light spot to obtain a target LED light that matches the light spot, and use the target LED light to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

[0142] The method of retroreflective line verification in the embodiment of the present application is the same as the method described in the calibration stage above, and the details are shown in steps 501 to 506.

[0143] In the embodiment of the present application, the method for updating any one of the light spot matching pairs is: replacing the LED lamp in any one of the light spot matching pairs with the target LED lamp that matches the light spot in the light spot matching pair to obtain an updated light spot matching pair.

[0144] In order to further understand the method for verifying the gaze point in the embodiment of the present application, Fig.12 FIG. 1 is a flow chart of a method for matching light spots, which may specifically include the following steps:

[0145] Step 1201: after entering the calibration phase, a plurality of human eye images of the user are obtained, wherein any one of the human eye images is obtained when the user gazes at a calibration point in a calibration image displayed in the electronic device, and the number of the plurality of human eye images is the same as the number of the calibration images;

[0146] Step 1202: for any one of the multiple human eye images, perform pupil detection on the any one of the multiple human eye images to obtain the position coordinates of the pupil center of the any one of the human eye images in the any one of the human eye images; and perform light spot detection on the any one of the human eye images to obtain the pixel position coordinates of each light spot in the any one of the human eye images, and project the pixel position coordinates of each light spot into the camera focal plane to obtain the image position coordinates of each light spot respectively;

[0147] Step 1203: for any one of the light spots, using the initial curvature center and the image position coordinates of the light spot, perform retroreflection line verification on the light spot to obtain a target LED lamp matching the light spot; wherein the multiple human eye images correspond to an initial curvature center respectively, and the initial curvature center corresponding to any one of the human eye images is obtained based on the image position coordinates of the pupil center of any one of the human eye images and the calibration image corresponding to the any one of the human eye images;

[0148] Step 1204: Obtain a light spot matching pair using target LED lights that match the light spots respectively;

[0149] Step 1205: obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; and updating the initial curvature center based on the plurality of curvature centers to obtain a second target curvature center;

[0150] Step 1206: based on the second target curvature center and the position coordinates of each LED light in the preset LED light ring in the camera coordinate system, obtaining the projection pixel position coordinates corresponding to each LED light, wherein the projection pixel position coordinates of any LED light are used to represent the position coordinates of the projection of any LED light in the pixel plane;

[0151] Step 1207: Obtaining an LED light ring projection table corresponding to any one of the human eye images through the projection pixel position coordinates corresponding to each of the LED lights;

[0152] Step 1208: Associating the position coordinates of the pupil center of any one of the human eye images with the LED light ring projection table corresponding to the any one of the human eye images to obtain a sub-correspondence relationship of the any one of the human eye images;

[0153] Step 1209: Obtaining the correspondence between the position coordinates of each pupil center and the projection table of each LED light ring according to the respective sub-correspondences of the plurality of human eye images;

[0154] Step 1210: when the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot;

[0155] Step 1211: using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any one of the LED light ring projection tables includes the projection pixel position coordinates corresponding to each LED light in the LED light ring;

[0156] Step 1212: for any one of the light spots, using the pixel position coordinates of the any one of the light spots and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light that matches the any one of the light spots, and based on the LED light that matches the any one of the light spots, obtain a light spot matching pair corresponding to the any one of the light spots;

[0157] Step 1213: obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively;

[0158] Step 1214: Filter the multiple curvature centers using a clustering algorithm to obtain filtered curvature centers, and obtain a first target curvature center based on the filtered curvature centers;

[0159] Step 1215: For any light spot in the light spot matching pair, use the first target curvature center and the image position coordinates of the light spot to perform a light spot retroreflection line check on the light spot to obtain a target LED light that matches the light spot, and use the target LED light to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

[0160] Based on the same inventive concept, the light spot matching method disclosed above can also be implemented by a light spot matching device. The effect of the light spot matching device is similar to that of the aforementioned method, and will not be described in detail here.

[0161] Fig.13 It is a schematic diagram of the structure of a light spot matching device according to an embodiment of the present disclosure.

[0162] like Fig.13As shown, the light spot matching device 1300 of the present disclosure may include an acquisition module 1310, a to-be-matched LED light ring projection table determination module 1320, a light spot matching pair determination module 1330, a curvature center determination module 1340, a screening module 1350 and a retroreflection line verification module 1360.

[0163] The acquisition module 1310 is used to perform pupil detection on the acquired human eye image of the user after the user enters the target scene to obtain the position coordinates of the pupil center of the user in the human eye image; and perform light spot detection on the human eye image to obtain the pixel position coordinates of each light spot;

[0164] The module 1320 for determining the projection table of the LED light ring to be matched is used to determine the projection table of the LED light ring to be matched corresponding to the position coordinates of the pupil center of the user by using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and the projection tables of each LED light ring, wherein any one of the projection tables of the LED light ring includes the projection pixel position coordinates corresponding to each LED light in the LED light ring;

[0165] The light spot matching pair determination module 1330 is used to determine, for any one of the light spots, an LED lamp matching the any one of the light spots by using the pixel position coordinates of the any one of the light spots and the projection pixel position coordinates corresponding to each LED lamp in the LED lamp ring projection table to be matched, and obtain a light spot matching pair corresponding to the any one of the light spots based on the LED lamp matching the any one of the light spots;

[0166] A curvature center determination module 1340, configured to obtain a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively;

[0167] A screening module 1350 is used to screen the multiple curvature centers using a clustering algorithm to obtain screened curvature centers, and obtain a first target curvature center based on the screened curvature centers;

[0168] The retroreflection line verification module 1360 is used to perform retroreflection line verification on any light spot in the light spot matching pair by using the first target curvature center and the image position coordinates of the light spot to obtain a target LED light matching the light spot, and use the target LED light to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

[0169] In one embodiment, the apparatus further comprises:

[0170] The calibration module 1370 is used to obtain the corresponding relationship between the position coordinates of each pupil center obtained by the user in the calibration phase and the projection table of each LED light ring in the following manner:

[0171] After entering the calibration phase, a plurality of human eye images of the user are obtained, wherein any one of the human eye images is obtained when the user gazes at a calibration point in a calibration image displayed on the electronic device, and the number of the plurality of human eye images is the same as the number of the calibration images;

[0172] For any one of the multiple human eye images, pupil detection is performed on the any one of the multiple human eye images to obtain the position coordinates of the pupil center of the any one of the human eye images in the any one of the human eye images; and spot detection is performed on the any one of the human eye images to obtain the pixel position coordinates of each light spot in the any one of the human eye images, and the pixel position coordinates of each light spot are projected onto the camera focal plane to obtain the image position coordinates of each light spot respectively; and,

[0173] For any one of the light spots, using the initial curvature center and the image position coordinates of the light spot, the light spot is retroreflected and checked to obtain a target LED light matching the light spot; wherein the multiple human eye images correspond to an initial curvature center respectively, and the initial curvature center corresponding to any one of the human eye images is obtained based on the image position coordinates of the pupil center of any one of the human eye images and the calibration image corresponding to the any one of the human eye images; and,

[0174] Using target LED lamps that match the light spots respectively, to obtain a light spot matching pair; and,

[0175] Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; and updating the initial curvature center based on the plurality of curvature centers to obtain a second target curvature center;

[0176] Based on the second target curvature center and the position coordinates of each LED light in the preset LED light ring in the camera coordinate system, the projection pixel position coordinates corresponding to each LED light are obtained, wherein the projection pixel position coordinates of any LED light are used to represent the position coordinates of the projection of any LED light in the pixel plane; and

[0177] Obtaining an LED light ring projection table corresponding to any one of the human eye images through the projection pixel position coordinates corresponding to each of the LED lights;

[0178] Associating the position coordinates of the pupil center of any one of the human eye images with the LED light ring projection table corresponding to the any one of the human eye images to obtain a sub-correspondence relationship of the any one of the human eye images;

[0179] According to the respective sub-correspondences of the plurality of human eye images, the correspondences between the position coordinates of the centers of the pupils and the projection tables of the LED light rings are obtained.

[0180] In one embodiment, the retroreflective line verification module 1360 is specifically used to:

[0181] For any light spot, based on the image position coordinates of the light spot and the third target curvature center, the position coordinates of the corneal intersection point corresponding to the light spot in the camera coordinate system are obtained, wherein the corneal intersection point is the intersection point of the line connecting the image position coordinates of the light spot and the camera optical center and the cornea of ​​the human eye of the user, and the third target curvature center is the first target curvature center or the initial curvature center;

[0182] Obtaining a first vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the third target curvature center; and obtaining a second vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the camera optical center;

[0183] Dividing the first vector by a first target value to obtain a first target vector, and dividing the second vector by a second target value to obtain a second target vector, wherein the first target value is equal to the modulus length of the first vector, and the second target value is equal to the modulus length of the second target vector;

[0184] According to the first target vector and the second target vector, obtaining the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system;

[0185] For any one of the LED lights in the LED light ring, the distance between the any one LED light and the reverse reflection line of the second vector is obtained by preset position coordinates of the any one LED light in the camera coordinate system and the position coordinates of the reflection point in the camera coordinate system; and,

[0186] The LED lamp with the smallest distance from the retroreflected line of the second vector among the LED lamps is determined as the target LED lamp matching the any one light spot.

[0187] In one embodiment, the retroreflective line verification module 1360 is further used to:

[0188] The position coordinates of the corneal intersection point corresponding to any one of the light spots in the camera coordinate system are obtained by the following formula:

[0189] q i =o+t 1 *(u i-o);

[0190] Wherein, qi is the position coordinate of the corneal intersection point corresponding to any one of the light spots i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, and u i is the image position coordinate of any light spot i, t 1 is a first coefficient corresponding to any one of the light spots;

[0191] The first coefficient t corresponding to any one of the light spots is obtained by the following formula 1 :

[0192] (o x +m*t 1 -c x ) 2 +(o y +n*t 1 -c y ) 2 +(o z +p*t 1 -c z ) 2 =R 2 ;

[0193] Among them, x is the horizontal position coordinate of the optical center of the camera in the camera coordinate system, o y is the vertical position coordinate of the camera optical center in the camera coordinate system, o z is the vertical position coordinate of the camera optical center in the camera coordinate system, m is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, n is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, p is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, c x is the abscissa of the initial curvature center, c y is the ordinate of the initial curvature center, c z is the vertical coordinate of the initial curvature center, and R is the preset curvature radius;

[0194] The position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system are obtained by the following formula:

[0195]

[0196] Wherein, iray is the vector corresponding to the position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system, is the second target vector, is the first target vector;

[0197] The distance between any one of the LED lights and the retroreflected line of the second vector is obtained by the following formula:

[0198]

[0199] Among them, d n is the distance between the nth LED light and the retroreflected line of the second vector, x iray is the horizontal position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, y iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, z iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, is the horizontal position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, t 2 is the second coefficient;

[0200] The second coefficient t is obtained by the following formula 2 :

[0201]

[0202] Among them, iray is the vector corresponding to the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system.

[0203] In one embodiment, the curvature center determination module 1340 is specifically configured to:

[0204] Randomly select two spot matching pairs from the spot matching pairs, determine whether the two spot matching pairs are the same as the two spot matching pairs randomly selected before, and if so, return to the step of randomly selecting two spot matching pairs from the spot matching pairs;

[0205] If not, based on the two spot matching pairs, the intersection vectors of the planes corresponding to the two spot matching pairs are obtained, wherein the plane corresponding to any one spot matching pair is a plane enclosed by the camera optical center and the spot in any one spot matching pair and the LED light; and,

[0206] Based on the intersection vectors of the planes respectively corresponding to the two spot matching pairs, a unit vector of the intersection vector is obtained; and,

[0207] According to the unit vector of the intersection vector, the curvature center corresponding to the two spot matching pairs is obtained.

[0208] In one embodiment, the curvature center determination module 1340 is further configured to:

[0209] The intersection vectors of the planes corresponding to the two spot matching pairs are obtained by the following formula:

[0210]

[0211] in, is the intersection vector, u i is the image position coordinate of spot i in spot matching pair i, l i is the position coordinate of LED light i in the spot matching pair i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, u j is the image position coordinate of the light spot j in the light spot matching pair j, and lj is the position coordinate of the LED light j in the light spot matching pair j in the camera coordinate system;

[0212] The obtaining of the unit vector of the intersection line vector based on the intersection line vectors of the planes respectively corresponding to the two spot matching pairs comprises:

[0213] The unit vector of the intersection vector is obtained by the following formula:

[0214]

[0215] in, is the unit vector of the intersection vector;

[0216] The step of obtaining the curvature center corresponding to the two spot matching pairs according to the unit vector of the intersection line vector comprises:

[0217] The center of curvature corresponding to the two spot matching pairs is obtained by the following formula:

[0218]

[0219] Where c′ is the center of curvature corresponding to the two matching pairs of light spots, k c is the first proportionality coefficient;

[0220] The first proportionality coefficient is obtained by the following formula:

[0221]

[0222] Wherein, ln is the position coordinate of the LED light in any one of the two spot matching pairs in the camera coordinate system, qn is the position coordinate of the corneal intersection point corresponding to the light spot in any one of the two light spot matching pairs in the camera coordinate system, u n is the image position coordinate of the light spot in any one of the two light spot matching pairs, k q is the second proportional coefficient, R is the preset radius of curvature, and c is the initial curvature center.

[0223] In one embodiment, the calibration module 1370 is further configured to:

[0224] For any LED light in the LED light ring, based on the position coordinates of the any LED light in the camera coordinate system, the second target curvature center, the position coordinates of the camera optical center in the camera coordinate system, and a preset curvature radius, obtain the position coordinates of the corneal intersection point corresponding to the any LED light in the camera coordinate system; and,

[0225] The position coordinates of the corneal intersection point corresponding to any one of the LED lights in the camera coordinate system are projected using a preset projection function to obtain the projection pixel position coordinates corresponding to the any one of the LED lights.

[0226] In one embodiment, the calibration module 1370 is further configured to:

[0227] The position coordinates of the corneal intersection point corresponding to the LED light in the camera coordinate system are obtained by the following formula:

[0228]

[0229] Among them, l m is the position coordinate of the mth LED light in the camera coordinate system, q m is the position coordinate of the corneal intersection point corresponding to the mth LED light in the camera coordinate system, c″ is the second target curvature center, o is the position coordinate of the camera optical center in the camera coordinate system, and R is the preset curvature radius.

[0230] In one embodiment, the optical panel matching pair determination module 1330 is specifically configured to:

[0231] For any LED lamp in the projection table of the LED lamp ring to be matched, based on the projection pixel position coordinates of the LED lamp and the pixel position coordinates of the any light spot, obtain the distance between the any light spot and the projection light spot of the LED lamp;

[0232] The LED lamp with the smallest distance between the projection spot and any one of the light spots in the LED lamp ring projection table to be matched is determined as the LED lamp matching any one of the light spots.

[0233] After introducing a method and device for matching light spots according to an exemplary embodiment of the present invention, an electronic device according to another exemplary embodiment of the present invention is introduced next.

[0234] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0235] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the light spot matching method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Fig. 9 Steps 901-906 shown in .

[0236] Refer to the following Fig.14 14 to 15. The electronic device 1400 according to this embodiment of the present invention is described. Fig.14 The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0237] like Fig.14 As shown, the electronic device 1400 is presented in the form of a general AR device. The components of the electronic device 1400 may include but are not limited to: the at least one processor 1401, the at least one computer storage medium 1402, and a bus 1403 connecting different system components (including the computer storage medium 1402 and the processor 1401).

[0238] Bus 1403 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0239] Computer storage media 1402 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1421 and / or cache storage media 1422 , and may further include read-only computer storage media (ROM) 1423 .

[0240] The computer storage medium 1402 may also include a program / utility 1425 having a set (at least one) of program modules 1424, such program modules 1424 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0241] The electronic device 1400 may also communicate with one or more external devices 1404 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 1400, and / or communicate with any device that enables the electronic device 1400 to communicate with one or more other AR devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 1405. In addition, the electronic device 1400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1406. As shown, the network adapter 1406 communicates with other modules for the electronic device 1400 through a bus 1403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0242] In some possible implementations, various aspects of a light spot matching method provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the light spot matching method according to various exemplary embodiments of the present invention described above in this specification.

[0243] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for matching a light spot, characterized in that: The method comprises: When the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot; Using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light in the LED light ring; For any one of the light spots, using the pixel position coordinates of the any one light spot and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light matching the any one light spot, and based on the LED light matching the any one light spot, obtain a light spot matching pair corresponding to the any one light spot; Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; Using a clustering algorithm to screen the multiple curvature centers to obtain screened curvature centers, and based on the screened curvature centers, obtaining a first target curvature center; For any light spot in the light spot matching pair, the light spot is verified by using the first target curvature center and the image position coordinates of the light spot to obtain a target LED light matching the light spot, and the target LED light is used to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

2. The method according to claim 1, characterized in that The corresponding relationship between the position coordinates of each pupil center obtained by the user in the calibration phase and the projection table of each LED light ring is obtained in the following manner: After entering the calibration phase, a plurality of human eye images of the user are obtained, wherein any one of the human eye images is obtained when the user gazes at a calibration point in a calibration image displayed on the electronic device, and the number of the plurality of human eye images is the same as the number of the calibration images; For any one of the multiple human eye images, pupil detection is performed on the any one of the multiple human eye images to obtain the position coordinates of the pupil center of the any one of the human eye images in the any one of the human eye images; and spot detection is performed on the any one of the human eye images to obtain the pixel position coordinates of each light spot in the any one of the human eye images, and the pixel position coordinates of each light spot are projected onto the camera focal plane to obtain the image position coordinates of each light spot respectively; and, For any one of the light spots, using the initial curvature center and the image position coordinates of the light spot, the light spot is retroreflected and checked to obtain a target LED light matching the light spot; wherein the multiple human eye images correspond to an initial curvature center respectively, and the initial curvature center corresponding to any one of the human eye images is obtained based on the image position coordinates of the pupil center of any one of the human eye images and the calibration image corresponding to the any one of the human eye images; and, Using target LED lamps that match the light spots respectively, to obtain a light spot matching pair; and, Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; and updating the initial curvature center based on the plurality of curvature centers to obtain a second target curvature center; Based on the second target curvature center and the position coordinates of each LED light in the preset LED light ring in the camera coordinate system, the projection pixel position coordinates corresponding to each LED light are obtained, wherein the projection pixel position coordinates of any LED light are used to represent the position coordinates of the projection of any LED light in the pixel plane; and Obtaining the LED light ring projection table corresponding to any one of the human eye images through the projection pixel position coordinates corresponding to each of the LED lights; Associating the position coordinates of the pupil center of any one of the human eye images with the LED light ring projection table corresponding to the any one of the human eye images to obtain a sub-correspondence relationship of the any one of the human eye images; According to the respective sub-correspondences of the plurality of human eye images, the correspondences between the position coordinates of the centers of the pupils and the projection tables of the LED light rings are obtained.

3. The method according to claim 1 or 2, characterized in that: Using the third target curvature center and the image position coordinates of the light spot, performing a retroreflection line check on the light spot to obtain a target LED lamp matching the light spot, including: For any light spot, based on the image position coordinates of the light spot and the third target curvature center, the position coordinates of the corneal intersection point corresponding to the light spot in the camera coordinate system are obtained, wherein the corneal intersection point is the intersection point of the line connecting the image position coordinates of the light spot and the camera optical center and the cornea of ​​the human eye of the user, and the third target curvature center is the first target curvature center or the initial curvature center; Obtaining a first vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the third target curvature center; and obtaining a second vector based on the position coordinates of the corneal intersection point in the camera coordinate system and the camera optical center; Dividing the first vector by a first target value to obtain a first target vector, and dividing the second vector by a second target value to obtain a second target vector, wherein the first target value is equal to the modulus length of the first vector, and the second target value is equal to the modulus length of the second target vector; According to the first target vector and the second target vector, obtaining the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system; For any one of the LED lights in the LED light ring, the distance between the any one LED light and the reverse reflection line of the second vector is obtained by preset position coordinates of the any one LED light in the camera coordinate system and the position coordinates of the reflection point in the camera coordinate system; and, The LED lamp with the smallest distance from the retroreflected line of the second vector among the LED lamps is determined as the target LED lamp matching the any one light spot.

4. The method according to claim 3, characterized in that The obtaining, based on the image position coordinates of the light spot and the third target curvature center, the position coordinates of the corneal intersection point corresponding to the light spot in the camera coordinate system comprises: The position coordinates of the corneal intersection point corresponding to any one of the light spots in the camera coordinate system are obtained by the following formula: what i =o+t1*(u i -o); Wherein, qi is the position coordinate of the corneal intersection point corresponding to any one of the light spots i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, and u i is the image position coordinate of any one of the light spots i, and t1 is the first coefficient corresponding to the any one of the light spots; The first coefficient t1 corresponding to any one of the light spots is obtained by the following formula: (or x +m*t1-c x ) 2 +(o y +n*t1-c y ) 2 +(o z +p*t1-c z ) 2 =R 2 ; Among them, x is the horizontal position coordinate of the optical center of the camera in the camera coordinate system, o y is the vertical position coordinate of the camera optical center in the camera coordinate system, o z is the vertical position coordinate of the camera optical center in the camera coordinate system, m is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, n is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, p is the difference between the image position coordinate of any one of the light spots and the position coordinate of the camera optical center in the camera coordinate system, c x is the abscissa of the initial curvature center, c y is the ordinate of the initial curvature center, c z is the vertical coordinate of the initial curvature center, and R is the preset curvature radius; The step of obtaining the position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system according to the first target vector and the second target vector includes: The position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system are obtained by the following formula: Wherein, iray is the vector corresponding to the position coordinates of the reflection point of the reverse reflection line of the second vector in the camera coordinate system, is the second target vector, is the first target vector; The obtaining the distance between any one of the LED lights and the reverse reflection line of the second vector by presetting the position coordinates of any one of the LED lights in the camera coordinate system and the position coordinates of the reflection point in the camera coordinate system comprises: The distance between any one of the LED lights and the retroreflected line of the second vector is obtained by the following formula: Among them, d n is the distance between the nth LED light and the retroreflected line of the second vector, x iray is the horizontal position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, y iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, z iray is the vertical position coordinate of the reflection point of the retroreflected line of the second vector in the camera coordinate system, is the horizontal position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, is the vertical position coordinate of the nth LED light in the camera coordinate system, and t2 is the second coefficient; The second coefficient t2 is obtained by the following formula: Among them, iray is the vector corresponding to the position coordinates of the reflection point of the retroreflected line of the second vector in the camera coordinate system.

5. The method according to claim 1 or 2, characterized in that: The method of obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively includes: Randomly select two spot matching pairs from the spot matching pairs, determine whether the two spot matching pairs are the same as the two spot matching pairs randomly selected before, and if so, return to step 2 of randomly selecting two spot matching pairs from the spot matching pairs. If not, based on the two spot matching pairs, the intersection vectors of the planes corresponding to the two spot matching pairs are obtained, wherein the plane corresponding to any one spot matching pair is a plane enclosed by the camera optical center and the spot in any one spot matching pair and the LED light; and, Based on the intersection vectors of the planes respectively corresponding to the two spot matching pairs, a unit vector of the intersection vector is obtained; and, According to the unit vector of the intersection vector, the curvature center corresponding to the two matching pairs of light spots is obtained.

6. The method according to claim 5, characterized in that The obtaining, based on the two spot matching pairs, the intersection vectors of the planes respectively corresponding to the two spot matching pairs comprises: The intersection vectors of the planes corresponding to the two spot matching pairs are obtained by the following formula: in, is the intersection vector, u i is the image position coordinate of spot i in spot matching pair i, l i is the position coordinate of LED light i in the spot matching pair i in the camera coordinate system, o is the position coordinate of the camera optical center in the camera coordinate system, u j is the image position coordinate of the light spot j in the light spot matching pair j, and lj is the position coordinate of the LED light j in the light spot matching pair j in the camera coordinate system; The obtaining of the unit vector of the intersection line vector based on the intersection line vectors of the planes respectively corresponding to the two spot matching pairs comprises: The unit vector of the intersection vector is obtained by the following formula: in, is the unit vector of the intersection vector; The step of obtaining the curvature center corresponding to the two spot matching pairs according to the unit vector of the intersection line vector comprises: The center of curvature corresponding to the two spot matching pairs is obtained by the following formula: Where c′ is the center of curvature corresponding to the two matching pairs of light spots, k c is the first proportionality coefficient; The first proportionality coefficient is obtained by the following formula: Wherein, ln is the position coordinate of the LED light in any one of the two spot matching pairs in the camera coordinate system, q n is the position coordinate of the corneal intersection point corresponding to the light spot in any one of the two light spot matching pairs in the camera coordinate system, u n is the image position coordinate of the light spot in any one of the two light spot matching pairs, k q is the second proportional coefficient, R is the preset radius of curvature, and c is the initial curvature center.

7. The method according to claim 2, characterized in that The obtaining of the projection pixel position coordinates corresponding to each LED light based on the second target curvature center and the position coordinates of each LED light in the preset LED light ring in the camera coordinate system comprises: For any LED light in the LED light ring, based on the position coordinates of the any LED light in the camera coordinate system, the second target curvature center, the position coordinates of the camera optical center in the camera coordinate system, and a preset curvature radius, obtain the position coordinates of the corneal intersection point corresponding to the any LED light in the camera coordinate system; and, The position coordinates of the corneal intersection point corresponding to any one of the LED lights in the camera coordinate system are projected using a preset projection function to obtain the projection pixel position coordinates corresponding to the any one of the LED lights.

8. The method according to claim 7, characterized in that The obtaining the position coordinates of the corneal intersection point corresponding to the any one LED light in the camera coordinate system based on the position coordinates of the any one LED light in the camera coordinate system, the second target curvature center, the position coordinates of the camera optical center in the camera coordinate system, and the preset curvature radius includes: The position coordinates of the corneal intersection point corresponding to the LED light in the camera coordinate system are obtained by the following formula: Among them, l m is the position coordinate of the mth LED light in the camera coordinate system, q m is the position coordinate of the corneal intersection point corresponding to the mth LED light in the camera coordinate system, c″ is the second target curvature center, o is the position coordinate of the camera optical center in the camera coordinate system, and R is the preset curvature radius.

9. The method according to claim 1, characterized in that: The method of using the pixel position coordinates of the arbitrary light spot and the projection pixel position coordinates corresponding to each LED light in the projection table of the LED light ring to be matched to determine the LED light matching the arbitrary light spot comprises: For any LED lamp in the projection table of the LED lamp ring to be matched, based on the projection pixel position coordinates of the LED lamp and the pixel position coordinates of the any light spot, obtain the distance between the any light spot and the projection light spot of the LED lamp; The LED lamp with the smallest value of the distance between the projection spot and any one of the light spots among the LED lamps in the projection table of the LED lamp ring to be matched is determined as the LED lamp that matches the any one of the light spots.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: When the user enters the target scene, pupil detection is performed on the acquired human eye image of the user to obtain the position coordinates of the center of the pupil of the user in the human eye image; and light spot detection is performed on the human eye image to obtain the pixel position coordinates of each light spot; Using the correspondence between the position coordinates of each pupil center obtained by the user in the calibration phase and each LED light ring projection table, determine the LED light ring projection table to be matched corresponding to the position coordinates of the pupil center of the user, wherein any LED light ring projection table includes the projection pixel position coordinates corresponding to each LED light in the LED light ring; For any one of the light spots, using the pixel position coordinates of the any one light spot and the projection pixel position coordinates corresponding to each LED light in the LED light ring projection table to be matched, determine the LED light matching the any one light spot, and based on the LED light matching the any one light spot, obtain a light spot matching pair corresponding to the any one light spot; Obtaining a plurality of curvature centers according to the light spot matching pairs corresponding to the light spots respectively; Using a clustering algorithm to screen the multiple curvature centers to obtain screened curvature centers, and based on the screened curvature centers, obtaining a first target curvature center; For any light spot in the light spot matching pair, the light spot is verified by using the first target curvature center and the image position coordinates of the light spot to obtain a target LED light matching the light spot, and the target LED light is used to update any light spot matching pair to obtain an updated light spot matching pair, wherein the image position coordinates of the light spot are obtained by projecting the pixel position coordinates of the light spot into the focal plane of the camera.

Images