Lens identification method and device in near-to-eye display equipment and near-to-eye display equipment

Abstract

The invention discloses a lens identification method and device in near-to-eye display equipment and the near-to-eye display equipment. The method comprises the steps of obtaining at least one comparison image based on a near-to-eye display device which is worn by a user and can be provided with a lens; determining a distortion result of the comparison image; determining whether a lens is installed in the near-to-eye display equipment according to the distortion result; wherein the at least one comparison image is acquired by an image acquisition device arranged between the lens mounting position and the near-to-eye display screen. The problem that whether the lens is installed in the near-to-eye display equipment or not cannot be automatically detected is solved, rapid identification of whether the lens is installed or not is achieved, automatic identification can be completed without operation of a user, and the cost of image acquisition through the image acquisition device is low.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, and more particularly to lens recognition methods, devices, and near-eye display devices. Background Technology

[0002] Near-eye display devices are typically worn on a user's head to display images, such as VR devices. Because users have different vision conditions and different refractive errors, different users may need to install different vision correction lenses when wearing near-eye display devices.

[0003] Currently, near-eye display devices typically include eye-tracking functionality. If a vision-correcting lens is added between the user's eye and the camera, the lens's parameters need to be incorporated into the eye-tracking algorithm to correct it and ensure accurate eye-tracking calculations. Generally, after a lens is installed in a near-eye display device, the device needs to obtain the lens information for correcting the user's eye-tracking calculations; therefore, it's essential to ensure that a suitable lens has been installed. Summary of the Invention

[0004] This application provides a lens identification method, apparatus, and near-eye display device to solve the problems of not being able to automatically detect whether a lens is installed in a near-eye display device and the high detection cost.

[0005] According to one aspect of this application, a lens recognition method in a near-eye display device is provided, comprising:

[0006] At least one comparison image is acquired based on a near-eye display device worn by the user with a lens mounted on it;

[0007] Determine the distortion result of the compared images;

[0008] Determine whether a lens is installed in the near-eye display device based on the distortion results;

[0009] At least one of the comparison images is acquired by an image acquisition device positioned between the lens mounting location and the near-eye display screen.

[0010] According to another aspect of this application, a lens recognition device in a near-eye display device is provided, comprising:

[0011] The image acquisition module is used to acquire at least one comparison image;

[0012] A distortion verification module is used to determine the distortion result of the compared images;

[0013] The lens installation judgment module is used to determine whether a lens is installed in the near-eye display device based on the distortion results.

[0014] At least one of the comparison images is acquired by an image acquisition device positioned between the lens mounting location and the near-eye display screen.

[0015] According to another aspect of this application, a near-eye display device is provided, the near-eye display device comprising:

[0016] At least one processor, and a memory communicatively connected to said at least one processor;

[0017] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the lens recognition method in the near-eye display device according to any embodiment of this application.

[0018] The technical solution of this application embodiment is based on a user wearing a near-eye display device on which a lens can be installed, acquiring at least one comparison image; determining the distortion result of the comparison image; and determining whether a lens is installed in the near-eye display device based on the distortion result. This can automatically confirm whether a lens is installed in the near-eye display device, while avoiding the addition of additional sensing and recognition devices, reducing the weight and cost of the near-eye display device. The comparison image is acquired based on an image acquisition device set between the lens installation position and the near-eye display screen. When the lens is installed, the acquisition effect of the comparison image changes. The distortion of the image determines whether a lens is installed in the near-eye display device, achieving rapid identification of whether a lens is installed without the need for active user operation, reducing the user's operational burden and improving the user experience.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a lens recognition method in a near-eye display device according to Embodiment 1 of this application;

[0022] Figure 2 This is a flowchart of a lens recognition method in a near-eye display device according to Embodiment 2 of this application;

[0023] Figure 3aThis is a demonstration example of a first position and a second position provided according to Embodiment 2 of this application;

[0024] Figure 3b This is a demonstration example of another first position and second position provided according to Embodiment 2 of this application;

[0025] Figure 4 This is a flowchart of a lens recognition method in a near-eye display device according to Embodiment 3 of this application;

[0026] Figure 5 This is a flowchart of a lens recognition method in a near-eye display device according to Embodiment 4 of this application;

[0027] Figure 6a This is a demonstration example diagram of a first position provided according to Embodiment 4 of this application;

[0028] Figure 6b This is a demonstration example diagram of another first position provided according to Embodiment 4 of this application;

[0029] Figure 7 This is a schematic diagram of the structure of a lens recognition device in a near-eye display device according to Embodiment 5 of this application;

[0030] Figure 8 This is a schematic diagram of the structure of a near-eye display device that implements the lens recognition method in the near-eye display device according to the embodiments of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart illustrating a lens identification method in a near-eye display device according to Embodiment 1 of this application. This embodiment is applicable to situations where a lens is installed in a near-eye display device. The method can be executed by a lens identification device in the near-eye display device, which can be implemented in hardware and / or software and can be configured within the near-eye display device. Figure 1 As shown, the method includes:

[0035] S10. Based on a near-eye display device with a lens mounted on the user's clothing, acquire at least one comparison image;

[0036] In this embodiment, the comparison image can be understood as an image that may include the user's eye, parts of the near-eye display device, parts of the lens, etc. The comparison image is used to characterize images captured by image acquisition devices at different locations of the near-eye display device, or images captured by image acquisition devices at the same location at one or more times.

[0037] At least one comparison image is acquired by an image acquisition device positioned between the lens mounting location and the near-eye display screen. Specifically, there must be at least one comparison image. If there is only one comparison image, it must be acquired by the image acquisition device positioned between the lens mounting location and the near-eye display screen. If there are two or more comparison images, at least one of them must be acquired by the image acquisition device positioned between the lens mounting location and the near-eye display screen. Whether or not a lens is installed in the near-eye display device can affect the image acquisition effect of the image acquisition device positioned between the lens mounting location and the near-eye display screen, so as to determine the lens installation status based on the comparison image, or to determine some lens characteristics, such as lens power information.

[0038] S20. Determine the distortion results of the compared images;

[0039] In this embodiment, the distortion result can be distortion produced / no distortion produced, or the distortion result can be distortion parameters, etc.

[0040] Optionally, determining the distortion result of the comparison image includes: inputting the comparison image into a deep learning model to determine the distortion result of the comparison image;

[0041] The deep learning model is used to characterize the mapping relationship between the comparison image and the lens features, which include one or more of the following: lens power information, curvature, refractive index, diopter, etc.

[0042] In this embodiment, the deep learning model can be a pre-trained deep learning neural network model. The deep learning neural network model can be obtained by setting lenses with different lens characteristics within the field of view of the image acquisition device in the near-eye display device, such as lenses with diopter information of 100 degrees, 200 degrees, 300 degrees, and 400 degrees. With lenses of different diopter information within the field of view of the image acquisition device, several sets of user eye images are acquired, and the mapping relationship between the eye images and the lens diopter information is obtained. For example, the first 1000 eye images were taken when a 100-degree lens was set within the field of view of the image acquisition device, meaning that the first 1000 eye images correspond to 100-degree lens information; similarly, the first 1001-2000 eye images correspond to 200-degree lens information, and so on. Based on the mapping relationship between multiple sets of eye images and lens information, the deep learning model is trained to obtain the final deep learning model used to determine the distortion results of the comparison images. The deep learning models used for training can be: convolutional neural network models like DenseNet, attention-based models like VisionTransformer and DETR, lightweight models like MobileNet and EfficientNet, etc. No specific restrictions are placed on the models used for training.

[0043] In other words, the trained deep learning model can obtain the lens features of the lens, such as the lens power information, based on the input comparison image. The distortion result is then represented by the lens features. For example, a distortion result of 100 degrees, 200 degrees, etc., indicates that the comparison image has been distorted, confirming that a lens has been added.

[0044] Optionally, determining the distortion result of the comparison image includes: inputting the comparison image into a deep learning model to determine the distortion result of the comparison image; wherein, the deep learning model is used to characterize the mapping relationship between the comparison image and lens features, and the lens features include one or more of lens power information, curvature, refractive index, diopter, etc.

[0045] The comparison images are input into a deep learning model for verification, and the distortion result is determined based on the model's output.

[0046] S30. Determine whether a lens is installed in the near-eye display device based on the distortion results.

[0047] The distortion results are analyzed to determine whether a lens is installed in the near-eye display device. For example, if the distortion result indicates distortion, a lens is installed in the near-eye display device; if the result indicates no distortion, a lens is not installed. If the distortion result is a distortion parameter, it is determined whether the distortion parameter is within the allowable threshold range. If it is, the near-eye display device is determined not to have a lens installed; otherwise, a lens is determined to have a lens installed. If the distortion result is a lens characteristic, such as lens power information, then verifying the relevant lens characteristics proves that a lens has been installed.

[0048] This application provides a lens identification method for near-eye display devices. Based on a user wearing a near-eye display device with a lens mountable, at least one comparison image is acquired; the distortion result of the comparison image is determined; and the presence or absence of a lens in the near-eye display device is determined based on the distortion result. This method automatically confirms whether a lens is installed in the near-eye display device, while avoiding the need for additional sensing and identification devices, thus reducing the weight and cost of the near-eye display device. The comparison image is acquired using an image acquisition device positioned between the lens mounting location and the near-eye display screen. The installation of a lens alters the acquisition effect of the comparison image. The distortion of the image determines whether a lens is installed in the near-eye display device, enabling rapid identification of lens installation without requiring active user operation, reducing user workload, and improving the user experience.

[0049] Example 2

[0050] Figure 2 This is a flowchart illustrating a lens recognition method in a near-eye display device according to Embodiment 2 of this application. This embodiment is a refinement based on the above embodiments. Figure 2 As shown, the method includes:

[0051] S201. Acquire a first image. The first image is acquired by an image acquisition device set at a first position.

[0052] In this embodiment, the first image can be understood as an image that may include the user's eye, a portion of the near-eye display device, a portion of the lens, etc. The first position can be understood as a position or position area set on the near-eye display device; the image acquisition device may be a camera, video recorder, scanner, infrared thermal imaging, etc.

[0053] A first position is predetermined, and an image acquisition device is set at the first position to acquire an image. This image acquisition device can receive signals during the operation of the near-eye display device to trigger the acquisition of the first image, or it can automatically trigger the acquisition of the first image periodically. For example, after the near-eye display device is started, the image acquisition device is triggered to acquire the first image. The user can trigger the image acquisition device to acquire the first image by clicking or sliding a button. The button can be a hardware button or a virtual button on the screen. Trigger conditions are set, and the image acquisition device is triggered to acquire the first image when the trigger conditions are met. The trigger conditions could be, for example, a large calibration result error.

[0054] S202. Acquire a second image. The second image is acquired by an image acquisition device set at a second position.

[0055] In this embodiment, the second image can be understood as an image, which may also include the user's eye, a portion of the near-eye display device, a portion of the lens, etc. The second position can be understood as a position or position area set on the near-eye display device.

[0056] A second location is predetermined, and an image acquisition device is set up at the second location to acquire an image, thus obtaining a second image. The triggering method of the image acquisition device for acquiring the second image is the same as that of the image acquisition device for acquiring the first image, as described above, and will not be repeated here.

[0057] The first image and the second image are acquired simultaneously. The first position is between the lens mounting position and the near-eye display screen, and the second position is between the lens mounting position and the user's eye position.

[0058] To ensure the accuracy of lens recognition, the acquisition time of the first and second images in this embodiment is set to be the same, i.e., the first and second images are acquired simultaneously. The first position is between the lens mounting position and the near-eye display screen, and the second position is between the lens mounting position and the user's eye position. The second position is a position in front of the lens, i.e., between the lens and the user. Regardless of whether a lens is installed, the acquired second image does not pass through the lens. The first position is a position behind the lens, i.e., between the eyepiece and the lens of the near-eye display device, or a position behind the eyepiece, that is, a position close to the side of the near-eye display device's display screen. If a lens is installed, the first image is a distorted image produced by the lens; if no lens is installed, the first image is an undistorted image. Therefore, this embodiment can determine whether a lens is installed by verifying the image distortion.

[0059] For example, Figure 3aAn example diagram of a first position and a second position is provided. After a user wears the near-eye display device 1, the user's eye 2 can watch videos, animations, etc. through the eyepiece 11 of the near-eye display device 1. Image acquisition devices are respectively set on the first position 12 and the second position 13 of the near-eye display device 1. The lens 3 can be set between the first position 12 and the second position 13. The first position 12 is between the eyepiece 11 and the near-eye display screen 14. The near-eye display device 1 may also include a light source 15.

[0060] For example, Figure 3b Another example diagram of the first and second positions is provided. After the user wears the near-eye display device 1, the user's eye 2 can watch videos, animations, etc. through the eyepiece 11 of the near-eye display device 1. Image acquisition devices are respectively set on the first position 12 and the second position 13 of the near-eye display device 1. The first position 12 is the position between the lens 3 mounting position and the near-eye display screen 14, or the first position 12 is the position between the lens 3 mounting position and the eyepiece 11. The second position 13 is the position between the lens 3 mounting position and the user's eye 2. The near-eye display device 1 may also include a light source 15.

[0061] Image acquisition devices and at least one set of light sources are installed both in front of and behind the lens. The two sets of image acquisition devices for judging image distortion can be shared with the image acquisition device used for eye tracking. Normally, the image acquisition device for eye tracking is hidden behind the eyepiece, and a corrective lens is installed in front of the eyepiece. This solution uses at least two image acquisition devices. One image acquisition device can still be located behind the eyepiece (i.e., behind the lens), which can be used to acquire eye images for eye tracking calculations, or to acquire distorted images through the lens. The other image acquisition device cannot be hidden behind the lens; that is, when the corrective lens is installed on the near-eye display device, this image acquisition device will not be obstructed by the lens. In this case, the image acquisition device can capture eye images without looking through the lens, i.e., undistorted eye images. Of course, this image acquisition device can also be used to acquire eye images for eye tracking calculations (on some eye tracking calculation devices, the eye image for each eye is acquired using two image acquisition devices). Two sets of image acquisition devices are positioned one in front of the other. When lenses are added, one set of image acquisition devices (the rear image acquisition device) is covered by the lens, while the other set (the front image acquisition device) is not covered by the lens. When the image acquisition device is placed behind the eyepiece, the first image acquired is pre-corrected to correct the distortion effect of the eyepiece on the first image.

[0062] S203. Verify the distortion result of the first image based on the second image.

[0063] In this embodiment, the distortion result can be either distortion occurring or no distortion occurring, or the distortion result can be distortion parameters, etc. The second image is used as a reference image, and the distortion of the first image is verified using a distortion model (e.g., the Brown model), image registration, or other methods. For example, the first and second images are input into a distortion model for verification, and the distortion result is determined based on the model's output; or, the first and second images are registered to determine whether the first image is distorted, etc.

[0064] S204. Determine whether a lens is installed in the near-eye display device based on the distortion results.

[0065] The distortion results are analyzed to determine whether a lens is installed in the near-eye display device. For example, if the distortion result indicates that distortion has occurred, it is determined that a lens is installed in the near-eye display device; if the distortion result indicates that no distortion has occurred, it is determined that no lens is installed in the near-eye display device. The distortion result is a distortion parameter. It is determined whether the distortion parameter is within the allowable threshold range. If it is, it is determined that no lens is installed in the near-eye display device; otherwise, it is determined that a lens is installed in the near-eye display device.

[0066] The lens identification method for near-eye display devices provided in this application solves the problems of not being able to automatically detect whether a lens is installed in a near-eye display device and the high detection cost. It simultaneously acquires a first image and a second image at two different positions before and after the location where the lens is installed. The distortion result of the first image is verified by the second image. The presence or absence of distortion in the first image determines whether a lens is installed in the near-eye display device, achieving rapid identification of lens installation. Automatic identification can be completed without user intervention, and no additional image acquisition device is needed; the image acquisition device in the eye-tracking device can be used directly, eliminating the need for additional sensing equipment and avoiding increased weight and cost of the near-eye display device. Images used for comparing image distortion are captured by different image acquisition devices behind and in front of the lens, obtaining images without lens distortion and images with lens distortion. Comparison of these simultaneously acquired original and distorted images makes the distortion comparison results more accurate. The lens identification method provided in this application does not require additional lens identification equipment when identifying whether a lens is installed; it uses an algorithm to compare whether a lens is installed, reducing the weight of the device and the complexity of its structural design.

[0067] Example 3

[0068] Figure 4 This is a flowchart illustrating a lens recognition method in a near-eye display device according to Embodiment 3 of this application. This embodiment is a refinement based on the above embodiments. Figure 4 As shown, the method includes:

[0069] S301. Acquire a first image. The first image is acquired by an image acquisition device set at a first position.

[0070] S302. Acquire a second image. The second image is acquired by an image acquisition device set at a second position.

[0071] The first image and the second image are acquired simultaneously. The first position is between the lens mounting position and the near-eye display screen, and the second position is between the lens mounting position and the user's eye position.

[0072] S303. Adjust at least one of the first image and the second image to obtain a standard image and an image to be verified, wherein the standard image is determined based on the second image and the image to be verified is determined based on the first image.

[0073] In this embodiment, the standard image can be understood as an image used as a reference standard to determine whether other images are distorted; the image to be verified can be understood as an image that needs to be verified to determine whether it is distorted based on the lens.

[0074] At least one of the first and second images is adjusted in terms of image parameters. These parameters include at least one of the following: image size, position of feature points in the image, and correction of eyepiece effects. Correction of eyepiece effects refers to pre-correcting distortions in the first image caused by the eyepiece when the first image is acquired after it is viewed through the eyepiece. For example, the first image can be enlarged, the second image reduced, or the first image can be cropped to change the relative coordinates of feature points within the image. When only the first image is adjusted, the adjusted first image is used as the image to be verified, and the second image is used directly as the standard image. When only the second image is adjusted, the adjusted second image is used as the standard image, and the first image is used directly as the image to be verified. When both the first and second images are adjusted simultaneously, the adjusted second image is used as the standard image, and the adjusted first image is used as the image to be verified. Whether to adjust the first image, and what kind of adjustment to make, can be determined in advance based on information such as the position and parameters of the two image acquisition devices.

[0075] This application embodiment adjusts at least one of the first image and the second image to make them the same size, position, and angle. That is, the adjusted first image and the second image can be considered as images of the same content captured by the same image acquisition device at the same time and position, thereby eliminating verification errors and improving the accuracy of verification results.

[0076] As an optional embodiment, this optional embodiment further adjusts at least one of the first image and the second image to obtain a standard image and an image to be verified, including: adjusting at least one of the first image and the second image based on a difference parameter to obtain a standard image and an image to be verified.

[0077] In this embodiment, the difference parameter can be understood as a parameter describing the differences between two image acquisition devices or the differences in parameters of the same image acquisition device at different times, such as camera intrinsic parameters, angle, etc. In this embodiment, two image acquisition devices acquire images of the same location at the same time. Since the positions of the two image acquisition devices are different, their camera intrinsic parameters, angles, and other parameters may also differ. Therefore, the difference parameter between the image acquisition device at the first location and the image acquisition device at the second location is determined. Based on the difference parameter, the size, relative position of feature points, and other relationships between the first and second images are determined. Based on these relationships, at least one of the first and second images is adjusted. A standard image is obtained based on the second image, and an image to be verified is obtained based on the first image.

[0078] As an optional embodiment, this optional embodiment further adjusts at least one of the first image and the second image to obtain a standard image and an image to be verified, including:

[0079] A1. Identify the feature locations on the first and second images.

[0080] In this embodiment, feature location can be understood as the location of a specific feature in the image, such as coordinates, pixel value, etc. Feature location includes the location of eye features and / or non-eye features, and eye features include the pupil and / or light spots.

[0081] Image recognition algorithms and models are used to identify at least one type of eye feature and non-eye feature in the first image, determining the location corresponding to the eye feature and at least one location corresponding to the non-eye feature. For example, the pupil in the first image is identified, and its location is used as a feature location; or, the pupil and light spot in the first image are identified, and the midpoint between the pupil and light spot is calculated based on their coordinates and used as a feature location; or, the brow bone in the first image is identified as a non-eye feature location; or, a specific location of the near-eye display device in the first image is identified as a non-eye feature location, and so on. Similarly, the second image is identified to determine the feature locations on it. It is important to note that the first and second images identify the same or the same type of feature when identifying feature locations; for example, both may identify the pupil.

[0082] A2. Obtain the first feature image based on the feature positions on the first image.

[0083] In this embodiment, the first feature image can be understood as an image including specific features. The first feature image is determined based on the first image and can be a part of the first image or the entire first image. The first image is cropped according to the feature position on the first image. For example, a quadrilateral of a set length and width is cropped with the feature position as the center point to obtain the first feature image.

[0084] A3. Obtain the second feature image based on the feature positions on the second image.

[0085] In this embodiment, the second feature image can be understood as an image including specific features. The second feature image is determined based on the second image and can be a portion of the second image or the entire second image. The second image is cropped according to the feature positions on the second image. For example, a quadrilateral of a set length and width is cropped with the feature position as the center point to obtain the second feature image.

[0086] This application embodiment filters out useless data in the first and second images by cropping and other processing, and retains the data required for distortion verification. For example, the first image is an image that includes eyebrows, forehead and eyes, but the data of forehead, eyebrows and other parts are not needed during distortion verification. Through the above processing, a first feature image that only includes eyes can be obtained.

[0087] A4. Based on the difference parameter, adjust at least one of the first feature image and the second feature image to obtain a standard image and an image to be verified.

[0088] The size, relative position of feature points, and other relationships between the first feature image and the second feature image are determined based on the difference parameters. At least one of the first feature image and the second feature image is adjusted based on the above relationships. A standard image is obtained based on the second feature image, and an image to be verified is obtained based on the first feature image.

[0089] Optionally, when the first image is acquired by an image acquisition device located at a first position and the second image is acquired by an image acquisition device located at a second position, the difference parameters include at least one of the following: position difference parameters, intrinsic parameter difference parameters of the image acquisition device, and angle difference parameters of the image acquisition device.

[0090] Because the two image acquisition devices are positioned at different locations, under the same conditions, the images captured will differ in distance from the target due to the different distances between the two devices. Consequently, the same target in the final image may appear to be different sizes. Furthermore, differences in the intrinsic parameters and angles of the image acquisition devices will also affect the image. Therefore, the difference parameters in this embodiment include at least one of the following: positional difference parameters, differences in the intrinsic parameters of the image acquisition devices, and differences in the angles of the image acquisition devices. Different settings for these various types of difference parameters may also cancel out the effects of other difference parameters; for example, adjusting the focal length of the image acquisition device can offset positional differences.

[0091] S304. Verify the distortion results of the image to be verified based on the standard image.

[0092] Using a standard image as a reference image, the image to be verified is examined for distortion using methods such as distortion models (e.g., Brown models) and image registration. For example, the standard image and the image to be verified are input into the distortion model for verification, and the distortion result is determined based on the model's output. Alternatively, the standard image and the image to be verified are registered to determine whether the image to be verified is distorted, and so on.

[0093] As an optional embodiment, this optional embodiment further optimizes the verification of distortion results of the image to be verified based on a standard image as follows:

[0094] B1. Register the standard image and the image to be verified to determine the transformation matrix.

[0095] The standard image and the image to be verified are registered using a pre-defined image registration algorithm to obtain the optimal transformation matrix. The image registration algorithm can be based on grayscale and templates, feature-based matching methods, domain transformation-based methods, and so on.

[0096] Image registration refers to the process of matching and overlaying two or more images acquired at different times, using different sensors (imaging devices), or under different conditions (camera position and angle, etc.). The registration process is as follows: First, feature points are extracted from the two images; then, matching feature point pairs are found through similarity measurement; next, the image space coordinate transformation matrix is ​​obtained from the matching feature point pairs; finally, image registration is performed using the coordinate transformation matrix.

[0097] B2. Determine the first distortion parameter based on the transformation matrix.

[0098] In this embodiment, the first distortion parameter is a type of distortion parameter used to describe the distortion of the image. The first distortion parameter can be determined by analyzing the transformation matrix and considering the relationship between different transformation matrices and distortion parameters; alternatively, the transformation matrix can be calculated (e.g., its rank can be calculated), and the first distortion parameter can be determined based on the calculation result. For example, the calculation result can be used as the first distortion parameter, or the corresponding relationship table can be searched based on the calculation result to determine the first distortion parameter corresponding to the calculation result; alternatively, the maximum element in the transformation matrix can be determined, and the distortion parameter can be determined based on the maximum element. For example, the maximum element can be used as the first distortion parameter, or the corresponding relationship table can be searched based on the maximum element to determine the first distortion parameter corresponding to the maximum element; alternatively, the number of elements in the transformation matrix greater than a set threshold can be counted, and the distortion parameter can be determined based on this number. For example, this number can be used as the first distortion parameter, or the corresponding relationship table can be searched based on this number to determine the first distortion parameter corresponding to this number, and so on.

[0099] B3. Determine the distortion result based on the first distortion parameter.

[0100] The first distortion parameter is compared with a set reasonable threshold range. If it is within the reasonable threshold range, the distortion result is determined to be no distortion; otherwise, the distortion result is determined to be distorted. For example, if the first distortion parameter is 0, it is determined that no distortion has occurred; otherwise, it is determined that distortion has occurred.

[0101] As another optional embodiment, this optional embodiment further optimizes the distortion result of the image to be verified based on the standard image verification as follows:

[0102] C1. Identify the first eye feature in the standard image and the second eye feature in the image to be verified.

[0103] In this embodiment, the first eye feature may include, but is not limited to, one or more of iris size, pupil size, spot size, and corneal radius of curvature; the second eye feature may include, but is not limited to, one or more of iris size, pupil size, spot size, and corneal radius of curvature. Iris size can be expressed as the diameter of the iris or the radius of the iris. Pupil size can be expressed as the diameter of the pupil or the radius of the pupil. Spot size refers to the size of the reflected spot (also called the Pulcim spot) formed on the cornea by the infrared light source; spot size can be expressed as the diameter of the spot or the radius of the spot. The spot appears on the user's eye image acquired by the eye-tracking device, which uses the pupil-corneal reflection method. Corneal radius of curvature refers to the user's corneal radius of curvature calculated based on the eye-tracking calculation model after the user's eye image is acquired by the eye-tracking device.

[0104] In this application embodiment, eye features in a standard image are identified as first eye features, and eye features in an image to be verified are identified as second eye features.

[0105] C2. Determine the difference value between the first eye feature and the second eye feature, where the distortion result is the difference value.

[0106] In this embodiment, the difference value represents the change in eye features. For example, if both the first and second eye features are represented by the diameter of the pupil, and the first eye feature in the standard image is identified as having a first pupil diameter of 3mm, and the second eye feature in the image to be verified is identified as having a second pupil diameter of 2.7mm, then the difference between the first and second eye features is 0.3mm, so the distortion result is 0.3mm. Of course, the change in eye features is not limited to changes in the size of the eye features; other parameters that measure the degree of deformation of the eye features can also be used.

[0107] S305. Determine whether a lens is installed in the near-eye display device based on the distortion results.

[0108] As an optional embodiment, this optional embodiment further includes: determining first lens information based on a user wearing a near-eye display device with a detachable lens, according to a first distortion parameter and / or a first user identity information, and correcting the eye-tracking calculation of the near-eye display device based on the first lens information.

[0109] In this embodiment, the first user identity information can be understood as information used to uniquely identify the user; the first lens information can be understood as information used to describe the lens, such as spherical power, cylindrical power, axial power, etc.

[0110] As an optional embodiment, this optional embodiment further determines whether a lens is installed in the near-eye display device based on the distortion result, including:

[0111] D1. Determine whether the difference value is within the preset distortion range;

[0112] In this embodiment, the preset distortion range represents the criterion for determining whether the image to be verified is distorted. If the image to be verified is not distorted, the difference between the first eye feature and the second eye feature in the standard image and the image to be verified should be within the preset distortion range. If the image to be verified is distorted, the difference between the first eye feature and the second eye feature in the standard image and the image to be verified should be outside the preset distortion range.

[0113] D2. If the difference value is within the preset distortion range, it is determined that the near-eye display device is not equipped with a lens;

[0114] D3. If the difference value is outside the preset distortion range, determine that a lens is installed in the near-eye display device;

[0115] In this embodiment of the application, if the difference value is determined to be within a preset distortion range, it indicates that the image to be verified is not distorted and the near-eye display device does not have a lens installed. If the difference value is determined to be outside the preset distortion range, it indicates that the image to be verified is distorted and the near-eye display device has a lens installed.

[0116] Optionally, after determining whether a lens should be installed in the near-eye display device based on the distortion results, the method further includes:

[0117] E1. If the difference value is outside the preset distortion range, the difference value is matched to the lens feature range.

[0118] E2. Based on the lens characteristic range where the difference value is located, determine the lens characteristics of the installed lens, where the lens characteristics include the lens power.

[0119] In this embodiment, the lens feature interval represents the range of variation determined based on different lens feature differences. That is, installing lenses with different parameters in a near-eye display device will result in varying degrees of variation in the difference value. For example, if the lens feature is the lens power, such as 100 degrees, 200 degrees, 300 degrees, etc., then the lens feature interval can be expressed as: the range of variation for 200 degrees of myopia is [0.1mm-0.15mm], and the range of variation for 300 degrees of myopia is […].

[0120] The difference range for a myopia of 400 degrees is [0.2mm-0.25mm]. The eye feature is the user's pupil size. The pupil size in the standard image is identified as 3mm, and the pupil size in the image to be verified is identified as 2.88mm. The difference between the pupil size in the standard image and the pupil size in the image to be verified is 0.12mm. Since the difference of 0.12mm is within the range of [0.1mm-0.15mm], it proves that the lens installed in the near-eye display device has a power of 200 degrees.

[0121] This application embodiment can determine partial lens features, such as the lens power, of a lens installed in a near-eye display device based on the difference between a first eye feature in a standard image and a second eye feature in an image to be verified. Based on the partial lens features determined by the near-eye display device, the user can be prompted about the lens features, for example, by indicating that the installed lens has a power of 200 degrees.

[0122] Based on a near-eye display device worn by a user and equipped with a detachable lens, the system identifies the user to obtain first user identity information. The identification method for this user identity information can be at least one of the following: identifying the user's iris information; identifying the user's fingerprint information; identifying the user's voice information; identifying the user's corneal information; identifying the user's capillary information; establishing a connection with a smart device to obtain identity information from the smart device, and using this identity information as the user's identity information, which includes at least an identity identifier. Based on the first distortion parameter and / or the first user identity information, the system queries local storage or a service device to obtain first lens information that matches the first distortion parameter and / or the first user identity information. This first lens information is used to correct the eye-tracking calculations of the near-eye display device.

[0123] The service equipment can be mobile devices or the cloud, etc.

[0124] The following provides methods for matching lens information in different scenarios:

[0125] Scenario 1: Near-eye display devices do not store distortion parameters, while service devices only store user identity information and lens information;

[0126] The decision to install a lens is based on the image distortion results;

[0127] Identify the first user's identity information (iris recognition, manual input, fingerprint, voiceprint, etc.);

[0128] The user is prompted to confirm whether they wish to install their own lens.

[0129] Based on user confirmation, the lens information is matched in the storage of the near-eye display device;

[0130] If a match is found, the information of the first matched lens is used as an auxiliary parameter to correct the eye-tracking calculation of the VR device.

[0131] If no match is found, the first user's identity information will be sent to the service device to match the lens information;

[0132] The near-eye display device uses the first lens information matched by the service device as an auxiliary parameter to correct the eye-tracking calculation of the near-eye display device;

[0133] The first lens information matched by the service device is associated and bound with the first user identity information and stored in the storage terminal of the near-eye display device;

[0134] Scenario 2: Near-eye display devices store distortion parameters, while service devices only store user identity information and lens information;

[0135] Determine the lens to be installed and the first distortion parameter of the lens based on the image distortion results;

[0136] Identify the first user's identity information (iris recognition, manual input, fingerprint, voiceprint, etc.);

[0137] Based on the first user identity information and the first distortion parameter of the lens, the lens information is matched in the storage terminal of the near-eye display device;

[0138] If the first user identity information and the first distortion parameter are both matched and are within the same set of associated information, then the first lens information matched in the storage terminal of the near-eye display device is used as an auxiliary parameter to correct the eye tracking calculation of the near-eye display device.

[0139] If the first user identity information and the first distortion parameter are both matched, but the first user identity information and the first distortion parameter are not in the same set of associated information, it may be that the user installed the wrong lens. The user is prompted to confirm whether the lens installed is correct.

[0140] If the first user identity information is matched but the first distortion parameter is not matched, it may be that the user has added a lens, the lens information has not been recorded, or the lens has been installed incorrectly. The user is prompted to confirm whether a new lens has been purchased or whether the lens is installed incorrectly. Based on the user's confirmation of purchasing a new lens, the service device matches the newly added first lens information based on the first user identity information. The newly added first lens information, the first user identity information, and the first distortion parameter are associated and stored in the near-eye display device's storage. The obtained new first lens information can be used as an auxiliary parameter to correct the eye-tracking calculation of the near-eye display device.

[0141] If the first user identity information is not matched, but the first distortion parameter is matched, it may be that the user has not removed someone else's lens. The user will be prompted to remove the lens.

[0142] If neither the first user identity information nor the first distortion parameter is matched, the user is prompted to confirm whether their lens has been installed and whether initial matching information is required. Based on the user's confirmation, the first user identity information is sent to the service device to match the relevant first lens information. The newly added first lens information, user identity information, and first distortion parameter are associated and stored in the near-eye display device's storage. The obtained new first lens information can be used as an auxiliary parameter to correct the eye-tracking calculation of the near-eye display device.

[0143] Scenario 3: Near-eye display devices store distortion parameters, while service devices store user identity information, lens information, and lens distortion parameters.

[0144] The lens to be installed and its distortion parameters are determined based on the image distortion results.

[0145] Identify the first user's identity information (iris recognition, manual input, fingerprint, voiceprint, etc.);

[0146] Based on the calculated first distortion parameter and the first user identity information, the service device matches the lens information.

[0147] Among them, a successful match is achieved if the first distortion parameter calculated is within the error range of the lens distortion parameters stored by the service device for the user.

[0148] In other words, the conditions for a successful match are that the first user's identity information is successfully matched and the calculated first distortion parameter is within the error range of the lens distortion parameter stored by the service device for that user. Then the first lens information of the associated information can be matched.

[0149] Of course, the local saving function of near-eye display devices can also be retained in this scenario.

[0150] First, the calculated first distortion parameter and the first user identity information are matched to the storage end of the near-eye display device;

[0151] If the calculated first distortion parameter and the first user identity information can both match, then the matched first lens information is used as an auxiliary parameter to correct the eye tracking calculation of the near-eye display device.

[0152] If neither the calculated first distortion parameter nor the first user identity information can be matched, then the service device will match the first lens information.

[0153] If the calculated first distortion parameter and other cases of matching the first user identity information are as described in scenario 2, then the following can be used as a reference.

[0154] The lens identification method for near-eye display devices provided in this application solves the problems of not being able to automatically detect whether a lens is installed in a near-eye display device and the high detection cost. It simultaneously acquires a first image and a second image at two different positions before and after the location where the lens is installed. By processing at least one of the second and first images, a standard image and a verification image are obtained. Distortion verification is performed on the verification image based on the standard image, eliminating verification errors and improving the accuracy of the verification results. The distortion result determines whether a lens is installed in the near-eye display device, achieving rapid identification of whether a lens is installed. Automatic identification can be completed without user intervention, and the cost of image acquisition using an image acquisition device is low. The images used for comparing image distortion are captured by different image acquisition devices behind and in front of the lens, obtaining images without lens distortion and images with lens distortion. Comparing these simultaneously acquired original and distorted images of the eye makes the distortion comparison results more accurate. The lens identification method provided in this application does not require additional lens identification equipment when identifying whether a lens is installed. It uses an algorithm to compare whether a lens is installed, reducing the weight of the device and the complexity of its structural design.

[0155] Example 4

[0156] Figure 5 This is a flowchart illustrating a lens recognition method in a near-eye display device according to Embodiment 4 of this application. This embodiment is a refinement based on the above embodiments. Figure 5 As shown, the method includes:

[0157] S401, Obtain the first image and the second image.

[0158] In this embodiment, the first image and the second image can be understood as a single image. Both the first image and the second image can include parts of the near-eye display device, parts of the lens, user eye features, etc.

[0159] The first and second images can be acquired using an image acquisition device, which can be pre-set on the near-eye display device. The image acquisition device can receive a trigger to acquire the first image during the operation of the near-eye display device, or it can be automatically triggered periodically. For example, after the near-eye display device is started, the image acquisition device is triggered to acquire the first image. The user can trigger the image acquisition device to acquire the first image by clicking or sliding a button (which can be a hardware button or a virtual button on the screen). Trigger conditions can be set, and the image acquisition device will acquire the first image when these conditions are met. Trigger conditions could include a large calibration result error, etc.

[0160] The second image is a pre-captured image. Both the first and second images were acquired by an image acquisition device positioned at a first location. The second image is the image captured by the image acquisition device at the first location when the near-eye display device is not equipped with a lens. The first location is between the lens mounting position and the near-eye display screen.

[0161] S402. Verify the distortion result of the first image based on the second image.

[0162] In this embodiment, the second image and the first image are acquired at different times. The first position is between the lens mounting position and the near-eye display screen; that is, the lens is installed between the third position and the user's eye position. The second image is pre-acquired when the lens is not installed. The first position is the position after the lens; that is, if the lens is installed, the first image at this time is a distorted image produced by the lens; if the lens is not installed, the first image is an undistorted image.

[0163] For example, Figure 6aAn example diagram of a first position is provided. After a user wears the near-eye display device 1, the user's eye 2 can view videos, animations, etc. through the eyepiece 11 of the near-eye display device 1. An image acquisition device is set on the first position 12 of the near-eye display device 1. The lens 3 can be set in front of the first position 12 and behind the eyepiece 11. In other words, the first position 12 is located between the installation position of the lens 3 and the near-eye display screen 14. The near-eye display device 1 may also include a light source 15.

[0164] For example, Figure 6b Another example diagram of the first position is provided, in which the first position 12 is in front of the eyepiece 11 and behind the lens 3.

[0165] An image acquisition device and at least one set of light sources are set behind the lens. The image acquisition device can be placed directly behind the eyepiece (that is, the camera will be covered by the lens after the lens is installed). It can be used to acquire distorted images as well as eye images for calculating eye tracking.

[0166] The same image acquisition device can acquire a second image on the near-eye display device before the lens is installed, and acquire a first image while the near-eye display device is in operation. When the image acquisition device is placed behind the eyepiece, the acquired first image is pre-corrected to correct the distortion effect of the eyepiece on the first image.

[0167] The image acquisition device must remain in a fixed position while acquiring the first and second images, or move within a movable range. If it exceeds this range, the position is considered to have changed. If the position changes, the second image must be re-acquired to improve the accuracy of the distortion verification results.

[0168] Using the second image as a reference image, the distortion of the first image is verified through distortion models (e.g., Brown model), image registration, etc. For example, the second image and the first image are input into the distortion model for verification, and the distortion result is determined based on the model output; or, the second image and the first image are registered to determine whether the first image is distorted, and so on.

[0169] Optionally, verifying the distortion result of the first image based on the second image includes:

[0170] At least one of the first image and the second image is adjusted to obtain a standard image and an image to be verified, wherein the standard image is determined based on the second image and the image to be verified is determined based on the first image;

[0171] The distortion results of the image to be verified are based on standard images.

[0172] Optionally, at least one of the first image and the second image is adjusted to obtain a standard image and an image to be verified, including:

[0173] Identify feature locations on the first and second images;

[0174] The first feature image is obtained based on the feature locations on the first image;

[0175] The second feature image is obtained based on the feature locations on the second image;

[0176] Based on the difference parameter, at least one of the first feature image and the second feature image is adjusted to obtain a standard image and an image to be verified.

[0177] The feature locations include locations of eye features and / or non-eye features, with eye features including the pupil and / or light spots.

[0178] Optionally, when both the first image and the second image are acquired by an image acquisition device located at the first position, the difference parameters include at least one of the intrinsic difference parameters of the image acquisition device, the angle difference parameters of the image acquisition device, and the size parameter difference between the first image and the second image.

[0179] S403. Determine whether a lens is installed in the near-eye display device based on the distortion results.

[0180] The distortion results of the first image are analyzed to determine whether a lens is installed in the near-eye display device. For example, if the distortion result indicates that distortion has occurred, it is determined that a lens is installed in the near-eye display device; if the distortion result indicates that no distortion has occurred, it is determined that no lens is installed in the near-eye display device. The distortion result is a distortion parameter. It is determined whether the distortion parameter is within the allowable threshold range. If it is, it is determined that no lens is installed in the near-eye display device; otherwise, it is determined that a lens is installed in the near-eye display device.

[0181] The lens identification method for near-eye display devices provided in this application solves the problems of not being able to automatically detect whether a lens is installed in a near-eye display device and the high detection cost. It uses the same image acquisition device located at the same position to acquire a first image and a second image without distortion at two different times. The distortion result of the first image is verified by the second image. The presence or absence of distortion in the first image determines whether a lens is installed in the near-eye display device, achieving rapid identification of whether a lens is installed. Automatic identification can be completed without user intervention, and the image acquisition cost is low. The images used for comparing image distortion are captured by the same image acquisition device at different times, obtaining a second image without lens distortion and a first image that may have undergone lens distortion. Only one image acquisition device is needed. The lens identification method provided in this application does not require an additional lens identification device when identifying whether a lens is installed. It uses an algorithm to compare whether a lens is installed, reducing the weight of the device and the complexity of its structural design.

[0182] As an optional embodiment, this optional embodiment further optimizes the acquisition of the first image and the second image as follows:

[0183] F1. When the user wears a near-eye display device without a lens installed, a prompt point will be displayed;

[0184] In this embodiment, the second image requires user cooperation. The user actively wears a near-eye display device without lenses, and the display device shows a prompt point to guide the user's gaze direction. Controlling the display prompt point can be based on user-initiated triggering of the second image acquisition, such as pressing a physical button, using a preset posture, or voice prompting. Alternatively, controlling the display prompt point can be a step in the initial setup process of the near-eye display device. For example, when a user first wears a near-eye display device without lenses, the initial setup process includes a step to display a prompt point to guide the user's gaze, thereby acquiring a second image when the user gazes at the prompt point.

[0185] F2. Based on the user's gaze cue point, obtain a second image, wherein the second image includes the user's eye features.

[0186] In this embodiment, the second image acquired by the image acquisition device at the first position, where the user is gazing at the prompt point, is an image of the user's eyes when gazing at the prompt point. The second image contains the features exhibited when the user's eyes are gazing at the prompt point. The near-eye display device uses the second image acquired when the user's eyes are gazing at the prompt point to verify the distortion results of the first image. Eye features may include pupil features, corneal features, etc.

[0187] F3. When the user wears the near-eye display device again, a prompt will be displayed.

[0188] In this embodiment, after the user completes the second image acquisition process, the near-eye display device stores the second image with the user's gaze prompt point. When the user wears the near-eye display device again, the near-eye display device displays the prompt point again. The prompt point displayed by the near-eye display device for acquiring the second image is at the same position as the prompt point displayed by the near-eye display device for acquiring the first image. This ensures that the image acquisition device at the first position can acquire images with the same or similar user eye features at different acquisition times, facilitating image distortion verification of the second and first images acquired at different times.

[0189] F4. Based on the user's gaze cue, acquire the first image.

[0190] In this embodiment, the user gazes at the prompt point again, and the image acquisition device at the first position acquires a first image while the user is gazing at the prompt point. The distortion of the first image is verified using a second image acquired when the user is wearing a near-eye display device without lenses while gazing at the prompt point. As an optional embodiment, this optional embodiment further acquires the first and second images, including:

[0191] G1. Based on a near-eye display device without a lens installed, acquire a second image, wherein the second image includes device features and lens features of the near-eye display device.

[0192] In this embodiment, the second image is obtained without user cooperation. The features included in the second image are the device features and lens features of the near-eye display device. The second image can be automatically acquired in advance based on the image acquisition device at the first position and pre-stored within the near-eye display device. Alternatively, the second image can be formed by combining the device features and lens features of the near-eye display device, the position of the image acquisition device at the first position, and internal parameter data, to ensure that the image acquired at the first position inside the near-eye display device can obtain an image with parameters consistent with the pre-stored second image.

[0193] G2. Acquire the first image when the user is wearing a near-eye display device.

[0194] In this embodiment, when a user wears a near-eye display device, the image acquisition device at the first position acquires a first image, and the distortion of the first image is verified using a second image pre-stored in the near-eye display device.

[0195] As an optional embodiment, this optional embodiment further verifies the distortion result of the first image based on the second image, including:

[0196] H1. Perform feature point recognition on the first image to determine the feature points at a set location in the first image.

[0197] Feature points in the first image are identified according to a preset feature point recognition algorithm. During recognition, all feature points in the first image can be identified, and then filtered based on a set location to determine the feature points at that location. Alternatively, the set location can be determined first, and only the image within that location can be identified to obtain the feature points at that location. Once the position, angle, and other parameters of the image acquisition device are fixed, its acquisition range is determined. The range that the image acquisition device can acquire after being set up is predetermined, and features that will definitely appear within this range are identified, such as the border of a near-eye display device at a specified location, etc. The position of this feature in the image is determined, and this position is used as the set location. In the subsequent lens recognition process, only the feature points at the set location need to be identified and compared.

[0198] H2. Match the feature points corresponding to the first image with the feature points at the set positions in the second image to determine the second distortion parameter.

[0199] In this embodiment, the second distortion parameter is a distortion parameter used to describe the distortion of the image. The feature points corresponding to the first image and the feature points at a predetermined location in the second image should be invariant features within the same location region. Therefore, theoretically, if no lens is installed, the feature points corresponding to the two images should be the same or matched. The feature points at the predetermined location in the second image can be pre-identified and stored, and directly retrieved during subsequent lens recognition processes. It is not necessary to perform recognition every time, or they can be obtained during each lens recognition process without storage. A pre-set matching algorithm is used to match the feature points corresponding to the two images, and the second distortion parameter is determined based on the degree of matching. For example, a similarity score is calculated, with similarity values ​​between 0 and 1. The difference between 1 and the similarity score is used as the second distortion parameter.

[0200] H3. Determine the distortion result of the first image based on the second distortion parameter.

[0201] The second distortion parameter is compared with a set reasonable threshold range. If it is within the reasonable threshold range, the distortion result is determined to be no distortion; otherwise, the distortion result is determined to be distorted. For example, if the second distortion parameter is 0, it is determined that no distortion has occurred; otherwise, it is determined that distortion has occurred.

[0202] Optionally, the feature point at the set position in the second image is at least one of the device features, lens features, and eye features of the near-eye display device.

[0203] Since a second image is pre-acquired, to ensure the accuracy of recognition, it is necessary to reduce the influence of the external environment on the matching results, that is, to select features that are not affected by the external environment for feature matching. Therefore, in this embodiment of the application, at least one of the device features, lens features, and eye features of the near-eye display device is used as a feature point at a predetermined position in the second image.

[0204] When the feature point at the designated location in the second image is a device feature of the near-eye display device, the image should display the peripheral features of the near-eye display device. In other words, during distortion comparison, the peripheral features of the near-eye display device in the image are compared. These peripheral features could be specific colors or shapes around the user's eyes when wearing the near-eye display device, ensuring that the image acquisition device can stably capture these features when taking images of the eyes. If the feature point at the designated location in the second image is a lens feature of the near-eye display device, such as a groove, identification point, or barcode on the edge of the lens, capturing this feature can also prove that a lens has been installed.

[0205] As an optional embodiment, this optional embodiment further verifies the distortion result of the first image based on the second image, including:

[0206] I1. Identify the first marker features in the second image and the second marker features in the first image.

[0207] In this embodiment, the marking features may include, but are not limited to, device features, lens features, and eye features of the eye display device. Marking features refer to pre-defined features used for identification. Eye features may include, but are not limited to, iris size, pupil size, and light spot size.

[0208] In this application embodiment, the marker features in the second image are identified as the first marker features, and the marker features in the first image are identified as the second marker features.

[0209] I2. Determine the difference value between the first and second labeled features, where the distortion result is the difference value.

[0210] In this embodiment, the difference value represents the change in the marked feature. For example, if both the first and second marked features represent the border length of the near-eye display device, and the first standard feature in the second image is identified as having a first border length of 20cm, and the second marked feature in the first image is identified as having a second border length of 18cm, then the difference between the first and second marked features is 2cm, so the distortion result is 2cm. Of course, the change in the marked feature is not limited to the change in the size of the marked feature; other parameters that measure the degree of deformation of the marked feature can also be used.

[0211] As an optional embodiment, this optional embodiment further determines whether a lens is installed in the near-eye display device based on the distortion result of the first image, including:

[0212] J1. Determine whether the difference value is within the preset distortion range;

[0213] In this embodiment, the preset distortion range represents the criterion for determining whether the image to be verified is distorted. If the image to be verified is not distorted, the difference between the first eye feature and the second eye feature in the standard image and the image to be verified should be within the preset distortion range. If the image to be verified is distorted, the difference between the first eye feature and the second eye feature in the standard image and the image to be verified should be outside the preset distortion range.

[0214] J2. If the difference value is within the preset distortion range, it is determined that the near-eye display device is not equipped with a lens;

[0215] J3. If the difference value is outside the preset distortion range, determine that a lens is installed in the near-eye display device;

[0216] In this embodiment of the application, if the difference value is determined to be within a preset distortion range, it indicates that the image to be verified is not distorted and the near-eye display device does not have a lens installed. If the difference value is determined to be outside the preset distortion range, it indicates that the image to be verified is distorted and the near-eye display device has a lens installed.

[0217] Optionally, after determining whether a lens is installed in the near-eye display device based on the distortion results, the process may also include:

[0218] K4. If the difference value is outside the preset distortion range, the difference value is matched to the lens feature range.

[0219] K5. Based on the lens characteristic range where the difference value is located, determine the lens characteristics of the installed lens, where the lens characteristics include the lens power.

[0220] In this embodiment, the lens feature interval represents the range of variation based on different lens feature differences. That is, installing lenses with different parameters in a near-eye display device will result in varying degrees of variation in the difference value. For example, if the lens feature is the lens power, such as 100 degrees, 200 degrees, 300 degrees, etc., then the lens feature interval can be expressed as: the range of variation for 200 degrees of myopia is [0.5cm-1cm], the range for 300 degrees of myopia is [1cm-1.5cm], and the range for 400 degrees of myopia is [1.5cm-2cm]. The standard feature is the border length of the near-eye display device. If the border length in the second image is identified as 20cm, and the border length in the first image is identified as 19.2cm, then the difference between the border length in the second image and the border length in the first image is 0.8cm. Since the difference of 0.8cm is within the range of [0.5cm-1cm], it proves that the lens power installed in the near-eye display device is 200 degrees.

[0221] This application embodiment can determine partial lens features, such as the lens power, of a lens installed in a near-eye display device based on the difference between a first marker feature in a second image and a second marker feature in an image to be verified. Based on the partial lens features determined by the near-eye display device, the user can be prompted about the lens features, for example, by indicating that the installed lens has a power of 200 degrees.

[0222] As an optional embodiment, this optional embodiment further includes: determining second lens information based on a second distortion parameter and / or second user identity information, based on a user wearing a near-eye display device with a detachable lens, and correcting the eye-tracking calculation of the near-eye display device using the second lens information.

[0223] In this embodiment, the second user identity information can be understood as information used to uniquely identify the user; the second lens information can be understood as information used to describe the lens, such as spherical power, cylindrical power, axial power, etc.

[0224] Based on a near-eye display device worn by a user and equipped with a detachable lens, the system identifies the user and obtains second user identity information. The user identity information can be identified through at least one of the following methods: identifying the user's iris information; identifying the user's fingerprint information; identifying the user's voice information; identifying the user's corneal information; identifying the user's capillary information; establishing a connection with a smart device to obtain identity information from the smart device, and using this identity information as the user's identity information, which includes at least an identity identifier. Based on the second distortion parameters and / or the second user identity information, the system queries local storage or a service device to obtain second lens information that matches the second distortion parameters and / or the second user identity information. This second lens information is used to correct the eye-tracking calculations of the near-eye display device.

[0225] In this embodiment, different scenarios and implementation methods for determining the second lens information based on the second distortion parameter and / or the second user identity information can refer to the multiple implementation methods for determining the first lens information based on the first distortion parameter and / or the first user identity information described in the above embodiments. Their implementation principles are the same, and this embodiment will not elaborate further here.

[0226] The lens identification method for near-eye display devices provided in this application solves the problems of not being able to automatically detect whether a lens is installed in a near-eye display device and the high detection cost. It acquires a first image and a second image without distortion at two different times using the same image acquisition device located at the same position. Distortion verification is performed using feature points at designated locations in the second and first images to determine whether a lens is installed in the near-eye display device. This achieves rapid identification of whether a lens is installed, without user intervention. Furthermore, the image acquisition device is a component of the eye-tracking device, eliminating the need for additional sensing equipment and avoiding increased manufacturing costs. Images used for comparing image distortion are captured by the same image acquisition device at different times, resulting in a second image without lens distortion and a first image that may have undergone lens distortion. Only one image acquisition device is required, making the process simpler and more convenient. The lens identification method provided in this application does not require additional lens identification equipment; it uses an algorithm to compare whether a lens is installed, reducing the weight of the device and the complexity of its structural design.

[0227] Example 5

[0228] Figure 7 This is a schematic diagram of the structure of a lens recognition device in a near-eye display device provided in Embodiment 5 of this application. Figure 7 As shown, the device includes: an image acquisition module 51, a distortion verification module 52, and a lens installation judgment module 53.

[0229] The image acquisition module 51 is used to acquire at least one comparison image;

[0230] The distortion verification module 52 is used to determine the distortion result of the compared images;

[0231] The lens installation judgment module 53 is used to determine whether a lens is installed in the near-eye display device based on the distortion result.

[0232] At least one of the comparison images is acquired by an image acquisition device positioned between the lens mounting location and the near-eye display screen.

[0233] This application provides a lens recognition device for near-eye display devices. Based on a user wearing a near-eye display device with a lens installed, at least one comparison image is acquired; the distortion result of the comparison image is determined; and the presence or absence of a lens in the near-eye display device is determined based on the distortion result. This device can automatically confirm whether a lens is installed in the near-eye display device, while avoiding the need for additional sensing and recognition devices, thus reducing the weight and cost of the near-eye display device. The comparison image is acquired by an image acquisition device positioned between the lens installation location and the near-eye display screen. The installation of a lens changes the acquisition effect of the comparison image. The distortion of the image determines whether a lens is installed in the near-eye display device, achieving rapid identification of whether a lens is installed without requiring active user operation, reducing the user's operational burden and improving the user experience.

[0234] Optionally, the image acquisition module 51 includes:

[0235] The first image acquisition unit is used to acquire a first image, which is acquired by an image acquisition device set at a first position.

[0236] The second image acquisition unit is used to acquire a second image, which is acquired by an image acquisition device set at a second position.

[0237] The first image and the second image are acquired simultaneously. The first position is between the lens mounting position and the near-eye display screen, and the second position is between the lens mounting position and the user's eye position.

[0238] Optionally, the image acquisition module 51 includes:

[0239] The third image acquisition unit is used to acquire a first image and a second image, both of which are acquired by an image acquisition device set at a first position.

[0240] The first image and the second image were acquired at different times. The first position is between the lens installation position and the near-eye display screen, and the second image was acquired in advance when the lens was not installed.

[0241] Optional, distortion verification module 52 includes:

[0242] The first verification unit is used to verify the distortion result of the first image based on the second image.

[0243] Optional, distortion verification module 52 includes:

[0244] The second verification unit is used to input the comparison image into the deep learning model to determine the distortion result of the comparison image;

[0245] The deep learning model is used to characterize the mapping relationship between the comparison image and the lens features, which include one or more of the following: lens power information, curvature, refractive index, diopter, etc.

[0246] Optionally, the first verification unit is specifically used to: adjust at least one of the first image and the second image to obtain a standard image and an image to be verified, wherein the standard image is determined based on the second image and the image to be verified is determined based on the first image; and verify the distortion result of the image to be verified based on the standard image.

[0247] Optionally, adjusting at least one of the first image and the second image to obtain a standard image and an image to be verified includes:

[0248] Identify feature locations on the first image and the second image;

[0249] The first feature image is obtained based on the feature positions on the first image;

[0250] The second feature image is obtained based on the feature positions on the second image;

[0251] Based on the difference parameters between the image acquisition device at the first position and the image acquisition device at the second position, at least one of the first feature image and the second feature image is adjusted to obtain a standard image and an image to be verified.

[0252] The feature locations include locations of eye features and / or non-eye features, and the eye features include the pupil and / or light spots.

[0253] Optionally, the difference parameters include at least one of the following: position difference parameters, intrinsic parameter difference parameters of the image acquisition device, and angle difference parameters of the image acquisition device.

[0254] Optionally, verifying the distortion result of the image to be verified based on the standard image includes:

[0255] Identify a first eye feature in the standard image and a second eye feature in the image to be verified;

[0256] Determine the difference value between the first eye feature and the second eye feature, wherein the distortion result is the difference value;

[0257] The first eye feature includes at least one of iris size, pupil size, spot size, and corneal curvature radius; the second eye feature includes at least one of iris size, pupil size, spot size, and corneal curvature radius; and the difference value represents the variation of the eye feature.

[0258] Optionally, the lens installation judgment module 53 is specifically used to: determine whether the difference value is within a preset distortion range; if the difference value is within the preset distortion range, determine that the near-eye display device has not installed a lens; if the difference value is outside the preset distortion range, determine that the near-eye display device has installed a lens.

[0259] Optionally, the device may also include:

[0260] The feature interval determination module is used to match a lens feature interval to the difference value if the difference value is outside the preset distortion range, wherein the lens feature interval represents the range of variation of different lens feature difference values;

[0261] The lens feature determination module is used to determine the lens features of the installed lens based on the lens feature range where the difference value is located, wherein the lens features include the lens power.

[0262] Optionally, the third image acquisition unit is specifically configured to: display a prompt point when the user wears a near-eye display device without a lens; acquire a second image based on the user's gaze at the prompt point, wherein the second image includes the user's eye features; display the prompt point again when the user wears the near-eye display device; acquire a first image based on the user's gaze at the prompt point; or, acquire a second image based on a near-eye display device without a lens, wherein the second image includes device features or lens features of the near-eye display device; and acquire a first image when the user wears the near-eye display device.

[0263] Optionally, the first verification unit is specifically used for: performing feature point recognition on the first image to determine feature points at a set position in the first image; matching the feature points corresponding to the first image with the feature points at the set position in the second image to determine a second distortion parameter; and determining the distortion result of the first image based on the second distortion parameter.

[0264] Optionally, the feature points at the set positions in the second image include at least one of the device features of the near-eye display device, lens features, and eye features.

[0265] The lens recognition device in the near-eye display device provided in the embodiments of this application can execute the lens recognition method in the near-eye display device provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of executing the method.

[0266] Example 8

[0267] Figure 8 A schematic diagram of the structure of a near-eye display device 60 that can be used to implement embodiments of this application is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0268] like Figure 8 As shown, the near-eye display device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the near-eye display device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0269] Multiple components in the near-eye display device 60 are connected to the I / O interface 65, including: an input unit 66, such as a keyboard, mouse, etc.; an output unit 67, such as various types of displays, speakers, etc.; a storage unit 68, such as a disk, optical disk, etc.; and a communication unit 69, such as a network card, modem, wireless transceiver, etc. The communication unit 69 allows the near-eye display device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0270] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as lens recognition methods in near-eye display devices.

[0271] In some embodiments, the lens identification method in the near-eye display device may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or mounted onto the near-eye display device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the lens identification method in the near-eye display device described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to perform the lens identification method in the near-eye display device by any other suitable means (e.g., by means of firmware).

[0272] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0273] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0274] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0275] To provide user interaction, the systems and techniques described herein can be implemented on a near-eye display device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the near-eye display device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0276] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0277] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0278] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0279] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A lens identification method in a near-eye display device, characterized by, The method comprises: obtaining at least one comparison image based on a user wearing a mountable lens of a near-eye display device; determining a distortion result of the comparison image; determining whether the lens is mounted in the near-eye display device according to the distortion result; wherein the at least one comparison image is captured by an image capturing device arranged between the lens mounting position and the near-eye display screen.

2. The method of claim 1, wherein, The obtaining of the at least one comparison image comprises: obtaining a first image captured by an image capturing device arranged at a first position; obtaining a second image captured by an image capturing device arranged at a second position; wherein the first image and the second image are captured at the same time, the first position is between the lens mounting position and the near-eye display screen, and the second position is between the lens mounting position and the user's eye position.

3. The method of claim 1, wherein, The obtaining of the at least one comparison image comprises: obtaining a first image and a second image, both of which are captured by an image capturing device arranged at a first position; wherein the first image and the second image are captured at different times, the first position is between the lens mounting position and the near-eye display screen, and the second image is pre-captured when the lens is not mounted.

4. The method according to claim 2 or 3, characterized in that, The determination of the distortion result of the comparison image comprises: verifying the distortion result of the first image based on the second image.

5. The method of claim 1, wherein, The determination of the distortion result of the comparison image comprises: inputting the comparison image into a deep learning model to determine the distortion result of the comparison image; wherein the deep learning model is used to represent the mapping relationship between the comparison image and the lens features, and the lens features include one or more of lens power information, curvature, refractive index, and diopter.

6. The method of claim 4, wherein, The verification of the distortion result of the first image based on the second image comprises: adjusting at least one of the first image and the second image to obtain a standard image and a to-be-verified image, the standard image being determined according to the second image, and the to-be-verified image being determined according to the first image; verifying the distortion result of the to-be-verified image based on the standard image.

7. The method of claim 6, wherein, The adjustment of at least one of the first image and the second image to obtain a standard image and a to-be-verified image comprises: identifying feature positions on the first image and the second image; obtaining a first feature image according to the feature positions on the first image; obtaining a second feature image according to the feature positions on the second image; adjusting at least one of the first feature image and the second feature image based on a difference parameter to obtain a standard image and a to-be-verified image; wherein the feature positions include positions of eye features and / or non-eye features, and the eye features include pupils and / or light spots.

8. The method of claim 7, wherein, The difference parameter includes at least one of a position difference parameter, an internal parameter difference parameter of the image capturing device, and an angle difference parameter of the image capturing device.

9. The method of claim 6, wherein, The verification of the distortion result of the to-be-verified image based on the standard image comprises: identifying a first eye feature in the standard image and a second eye feature in the to-be-verified image; determining a difference value between the first eye feature and the second eye feature, wherein the distortion result is the difference value. The first eye feature includes at least one of an iris size, a pupil size, a light spot size, and a corneal curvature radius, the second eye feature includes at least one of an iris size, a pupil size, a light spot size, and a corneal curvature radius, and the difference value represents a change of the eye feature.

10. The method of claim 5, wherein, The method further includes: determining whether the difference value is within a preset distortion range; if the difference value is within the preset distortion range, determining that the near-eye display device is not installed with a lens; if the difference value is outside the preset distortion range, determining that the near-eye display device is installed with a lens.

11. The method of claim 6, wherein, The method further includes: if the difference value is outside the preset distortion range, matching the difference value with a lens feature interval, wherein the lens feature interval represents a change range based on different lens feature difference values; based on the lens feature interval in which the difference value is located, determining a lens feature of the installed lens, wherein the lens feature includes a lens power.

12. The method of claim 3, wherein, The method further includes: displaying a prompt point when a user wears the near-eye display device without a lens; based on the user gazing at the prompt point, acquiring a second image, wherein the second image includes a user eye feature; displaying the prompt point when the user wears the near-eye display device again; based on the user gazing at the prompt point, acquiring a first image; or based on the near-eye display device without a lens, acquiring a second image, wherein the second image includes a device feature or a lens feature of the near-eye display device; acquiring a first image when the user wears the near-eye display device.

13. The method of claim 4, wherein, The method further includes: performing feature point recognition on the first image to determine feature points at a set position in the first image; matching the feature points corresponding to the first image with feature points at the set position in the second image to determine second distortion parameters; determining a distortion result of the first image based on the second distortion parameters.

14. The method of claim 13, wherein, The feature points at the set position in the second image include at least one of a device feature, a lens feature, and an eye feature of the near-eye display device.

15. A lens identification apparatus in a near-eye display device, comprising: The method further includes: an image acquisition module configured to acquire at least one comparison image; a distortion verification module configured to determine a distortion result of the comparison image; a lens installation judgment module configured to determine whether a lens is installed in a near-eye display device based on the distortion result; wherein the at least one comparison image is acquired by an image acquisition device arranged between a lens installation position and a near-eye display screen.

16. A near-eye display device, comprising: The near-eye display device includes: at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the lens identification method of any one of claims 1-14.

Images