Method and device for adjusting image combination distance of intelligent wearable equipment, equipment and medium

Abstract

According to the image combination distance adjusting method and device of the intelligent wearable device, the equipment and the medium, the first image combination distance is calculated through the first interpupillary distance of the target user, customized image combination distance calculation is achieved for the target user, and the applicability of the intelligent wearable device is improved. According to the current scene depth, the current parallax of the target user and the second pupil distance, the image combination distance adjustment amount is calculated, so that the image combination distance can be dynamically changed according to the actual use environment, and the adaptability and flexibility of equipment display are enhanced; by adjusting the image combination distance of the intelligent wearable device, the display position of the target display object is matched with the sight distance of the target user, so that the positioning accuracy of the target display object is improved, the visual fatigue of the user is reduced, and the visual experience comfort of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent display technology, and in particular to a method, device, equipment and medium for adjusting the combined image distance of an intelligent wearable device. Background Art

[0002] Augmented reality (AR) technology is a technology that superimposes virtual images on the real world and is commonly used in smart wearable devices, such as AR devices, VR devices, etc. At present, augmented reality technology has been widely used in games, education, medical care, construction and other fields.

[0003] In current smart wearable devices, the image-joining distance is usually fixed. However, it is difficult to adapt to the differences in pupil distances of different users, and in actual use, the image-joining distance cannot be dynamically adjusted according to the scene. This fixed image-joining distance method is prone to cause user visual fatigue, inaccurate positioning of virtual objects, and other problems, affecting the user experience.

[0004] Therefore, how to improve the adjustment flexibility of the image distance of smart wearable devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a method, device, equipment and medium for adjusting the image distance of a smart wearable device, aiming to improve the adjustment flexibility of the image distance of the smart wearable device.

[0006] In a first aspect, the present application provides a method for adjusting the image distance of a smart wearable device, the method comprising:

[0007] Calculate a first image combination distance of the smart wearable device based on a first pupil distance of the target user;

[0008] Calculating a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user;

[0009] Calculating a target combined image distance based on the first combined image distance and the first combined image distance adjustment amount;

[0010] Based on the target image distance, the image distance of the smart wearable device is adjusted so that the display position of the target display object matches the viewing distance of the target user.

[0011] In a second aspect, the present application further provides a device for adjusting the image distance of a smart wearable device, the device for adjusting the image distance of a smart wearable device comprising:

[0012] A first image combination distance calculation module, used to calculate a first image combination distance of the smart wearable device based on a first pupil distance of the target user;

[0013] A first adjustment amount calculation module, configured to calculate a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user;

[0014] A target image combination distance module, configured to calculate a target image combination distance based on the first image combination distance and the first adjustment amount of the image combination distance;

[0015] The image combination distance adjustment module is used to adjust the image combination distance of the smart wearable device based on the target image combination distance so that the display position of the target display object matches the viewing distance of the target user.

[0016] In a third aspect, the present application also provides a smart wearable device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for adjusting the combined image distance of the smart wearable device as described above are implemented.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method for adjusting the combined image distance of the smart wearable device as described above are implemented.

[0018] The present application provides a method, device, equipment and medium for adjusting the image-joint distance of a smart wearable device. The method of the present application calculates the first image-joint distance through the first pupil distance of the target user, realizes customized image-joint distance calculation for the target user, and improves the applicability of the smart wearable device. The image-joint distance adjustment amount is calculated according to the current scene depth, the current parallax of the target user and the second pupil distance; according to the image-joint distance adjustment amount, the image-joint distance of the smart wearable device is adjusted so that the display position of the target display object matches the viewing distance of the target user, so that the image-joint distance can change dynamically according to the actual use environment, improves the adjustment flexibility of the image-joint distance of the smart wearable device and the positioning accuracy of the target display object, enhances the adaptability and flexibility of the device display, thereby reducing the user's visual fatigue and improving the user's visual experience comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a first embodiment of a method for adjusting the combined image distance of a smart wearable device provided in an embodiment of the present application;

[0021] Figure 2 A schematic flow chart of a second embodiment of a method for adjusting the combined image distance of a smart wearable device provided in an embodiment of the present application;

[0022] Figure 3 It is a structural schematic diagram of a first embodiment of a combined image distance adjustment device for a smart wearable device provided by the present application;

[0023] Figure 4 It is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application.

[0024] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a method for adjusting the combined image distance of a smart wearable device provided in an embodiment of the present application.

[0029] like Figure 1 As shown, the method for adjusting the combined image distance of the smart wearable device includes steps S101 to S104.

[0030] S101, calculating a first image combination distance of the smart wearable device based on a first pupil distance of the target user;

[0031] In one embodiment, the first pupil distance of the target user can be collected by a sensor device built into the smart wearable device, or the target user can actively input the first pupil distance data into the smart wearable device (such as AR glasses).

[0032] In one embodiment, the sensor device may be an image sensor, such as a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like.

[0033] In one embodiment, the first pupil distance may be pupil distance data measured when the target user wears the smart wearable device for the first time; the first pupil distance may also be pupil distance data collected for the last time the target user wore the smart wearable device for the last time; the first pupil distance may also be pupil distance data collected for the first time when the target user is currently wearing the smart wearable device.

[0034] In one embodiment, the target user may be a user who wears the smart wearable device for the first time; the target user may also be a user who wore the smart wearable device last time; the target user may also be a user who currently wears the smart wearable device.

[0035] Generally, inter-pupillary distance (IPD) refers to the distance between the center points of the two eyes. Accurate inter-pupillary distance data is crucial for adjusting the image display posture of smart wearable devices, which can avoid image distortion and visual fatigue and improve the user experience of wearing smart wearable devices.

[0036] In one embodiment, a standard pupil distance data is preset before the smart wearable device leaves the factory, and then a preset image-joining distance is set according to the preset standard pupil distance data. For example, the pupil distance of an average adult male is between 60 mm and 73 mm, and that of a female is between 53 mm and 68 mm. The standard pupil distance data of the smart wearable device can be determined based on empirical values ​​or by collecting pupil distance detection big data.

[0037] In one embodiment, when the target user wears the smart wearable device, he or she can actively input the first pupil distance, or collect the first pupil distance of the target user through the image sensor built into the smart wearable device. According to the data difference between the first pupil distance and the preset pupil distance of the smart wearable device, the image distance suitable for the target user is calculated, and then the preset image distance is adjusted, so that the display content of the smart wearable device is displayed at the adjusted image distance, thereby improving the display quality of the smart wearable device and improving the user experience of the target user.

[0038] Exemplarily, the second adjustment amount of the combined image distance is determined according to the pupil distance difference between the first pupil distance and the preset pupil distance.

[0039] In one embodiment, when determining the second adjustment amount of the combined image distance according to the first pupil distance and the preset pupil distance, the following steps may be used for calculation:

[0040] Determine the difference: First calculate the pupil distance difference between the first pupil distance and the preset pupil distance:

[0041] Difference = preset pupil distance - first pupil distance

[0042] Calculate the adjustment amount: Calculate the second adjustment amount of the combined image distance according to the pupil distance difference and specific parameters of the optical system (such as magnification, focal length, etc.).

[0043] Exemplarily, the second adjustment amount of the combined image distance may be calculated as follows:

[0044]

[0045] Among them, the magnification refers to the proportional relationship between the size of the image formed by the imaging system and the actual size of the object, that is, the magnification of the display content of the smart wearable device.

[0046] Then, based on the preset image combination distance and the second adjustment amount of the image combination distance, the first image combination distance is calculated.

[0047] The calculation formula of the first combined image distance may be:

[0048] D1=D0+ΔD1

[0049] Wherein, D1 represents the first combined image distance, ΔD1 represents the second adjustment amount of the combined image distance, and D0 represents the preset combined image distance.

[0050] In one embodiment, ΔD1 may be determined by an empirical formula or a calibration process, typically based on the pupil distance difference between the first pupil distance and a preset pupil distance.

[0051] Furthermore, a preset pupil distance and a preset combined image distance of the smart wearable device are obtained; a second adjustment amount of the combined image distance is determined based on the pupil distance difference between the first pupil distance and the preset pupil distance; and the first combined image distance is determined based on the preset combined image distance and the second adjustment amount of the combined image distance.

[0052] Exemplarily, the correspondence between pupil distance and combined image distance can be collected through big data, or the correspondence between the pupil difference between the first pupil distance and the preset pupil distance and the combined image distance adjustment amount can be learned based on the preset pupil distance and its corresponding preset combined image distance, and an empirical formula can be constructed based on the correspondence as the basis for calculating the second adjustment amount ΔD1 of the combined image distance.

[0053] Exemplarily, the empirical formula of the second adjustment amount ΔD1 of the combined image distance can be expressed as:

[0054] ΔD1=λ0·(d1-d0)

[0055] Wherein, d1 represents the first pupil distance of the target user, d0 represents the preset pupil distance of the smart wearable device, and λ0 represents the conversion coefficient between the pupil distance difference and the combined image distance adjustment amount.

[0056] It is understandable that the empirical formula provided in the embodiment of the present application is only an example and does not represent actual use. The corresponding relationship between the pupil distance difference and the image-joint distance adjustment amount (i.e., the calculation method of the image-joint distance adjustment amount) can be determined according to the actual application of the smart wearable device.

[0057] In one embodiment, when a user wears a smart wearable device, as the user's pupil distance, viewing angle, and scene depth change, the image distance also needs to be dynamically adjusted to reduce visual fatigue caused by long-term use, improve the positioning accuracy of the virtual object display position, and improve the user's experience.

[0058] Furthermore, an eye image of the target user is collected; pupil positioning detection is performed on the eye image based on a feature point detection model to obtain left eye pupil coordinates and right eye pupil coordinates of the target user; and a second pupil distance of the target user is calculated based on the left eye pupil coordinates and the right eye pupil coordinates.

[0059] In one embodiment, when the target user wears the smart wearable device, the target user's eye image can be collected through a sensor device (such as an image sensor) built into the smart wearable device. It can be understood that the eye image needs to clearly observe the pupil of the target user.

[0060] In one embodiment, a binocular vision system may be used to capture eye images of the target user to obtain precise coordinates of the pupil of the target user.

[0061] In one embodiment, the collected eye image may be preprocessed, including grayscale, denoising, enhancement and other operations, to improve the image quality and prepare for feature point detection. A feature point detection model, such as a method based on deep learning, is used to perform pupil location detection on the eye image to determine the precise position of the pupil in the image, thereby obtaining the left eye pupil coordinates and the right eye pupil coordinates. Based on the detected left and right eye pupil coordinates, the distance between the two, i.e., the second pupil distance, is calculated.

[0062] For example, if the coordinates of the left pupil are (100, 150) and the coordinates of the right pupil are (120, 150), the second pupil distance can be calculated by the following formula:

[0063]

[0064] In one embodiment, the unit of the pupil distance can be converted according to the resolution of the image and the actual measurement unit.

[0065] Furthermore, based on the feature point detection model, pupil contour features in the eye image are detected; based on the pupil positioning algorithm, the pupil center point coordinates are positioned and calculated to obtain the left eye pupil coordinates and the right eye pupil coordinates of the target user.

[0066] The feature point detection model can use feature point detection algorithms such as Harris corner detection, SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented FAST and Rotated BRIEF) to identify key points in eye images.

[0067] In one embodiment, based on the feature point detection, a pupil localization algorithm is further used, such as a cascade classifier based on Haar features or a pupil detection model based on deep learning, to identify the pupil area. After the pupil area is detected, the pupil localization algorithm, such as morphological operations, edge detection, threshold processing, etc., is used to extract the precise outline of the pupil.

[0068] In one embodiment, the pupil contour feature can be fitted with an ellipse by using a least square method or a sub-pixel edge detection technique, and then the variance of the distance between the ellipse center and the edge is calculated, wherein the point with the smallest variance is used as the pupil center. Then, the coordinate point of the pupil center in the camera coordinate system where the image sensor is located is used as the pupil coordinate.

[0069] In one embodiment, the pupil center can also be located based on gradient, corneal reflection, etc., and the pupil coordinates can be calculated. For example, the gradient-based method is to calculate the gradient of each pixel in the image, and find the point where the straight lines in the gradient direction intersect the most as the pupil center; the corneal reflection-based method is to assist in locating the pupil center by detecting the reflected light spot on the cornea, which can reduce the image processing area to a certain extent, improve the real-time performance and the recognition rate of the reflected light spot. In the actual application process, the pupil coordinates of the target user (including the left eye pupil coordinates and the right eye pupil coordinates) can be collected using a suitable pupil positioning algorithm according to the actual application requirements.

[0070] It is understandable that the above pupil positioning algorithm is a related technology that has been disclosed in the art and will not be introduced in detail in the embodiments of the present application.

[0071] S102, calculating a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user;

[0072] In one embodiment, the current disparity refers to the disparity between the left eye and the right eye of the target user, that is, the direction difference produced by the target user's binocular observation of the same target display object. The angle between two points viewed from the target display object is used as the disparity angle of the current disparity.

[0073] In one embodiment, the current scene depth refers to depth information of the scene observed by the target user.

[0074] In one embodiment, the calculation formula of the first adjustment amount of the combined image distance can be expressed as:

[0075] ΔD2 = f(IPD, current disparity, scene depth)

[0076] Wherein, f is a function used to dynamically adjust the combined image distance by combining the inter-pupillary distance (IPD) of the target user, the parallax, and the scene depth of the smart wearable device.

[0077] In one embodiment, the first adjustment amount of the image combination distance can be a real-time adjustment, that is, real-time collection of data such as the current scene depth, the current parallax of the target user, and the second pupil distance of the target user, calculates the first adjustment amount of the image combination distance, and adjusts the image combination distance of the smart wearable device accordingly.

[0078] In another embodiment, real-time adjustment of the combined image distance may result in excessive data calculation, so the triggering condition for adjusting the combined image distance may also be set according to the actual needs of the user. By detecting specific data, it is determined whether the combined image distance needs to be adjusted, such as a large change in scene depth, a significant change in user pupil distance, etc., and by monitoring these data that will significantly affect the current display effect of the smart wearable device, it is used as a basis for determining whether the combined image distance needs to be adjusted, thereby reducing the amount of calculation.

[0079] In one embodiment, at least one set of target monitoring data is obtained, wherein the target monitoring data includes one or more of scene depth, user parallax, and user pupil distance; a data difference between the target monitoring data and corresponding preset benchmark data is calculated; when the data difference between any of the target monitoring data and the corresponding preset benchmark data is greater than or equal to a preset difference threshold, based on a preset data collection method, the current scene depth, the current parallax of the target user, and the second pupil distance of the target user are collected.

[0080] In one embodiment, when the data differences between all target monitoring data and the corresponding preset reference data are less than the preset difference threshold, no coincidence distance adjustment calculation and operation is performed, and the target monitoring data continues to be collected and detected.

[0081] In one embodiment, the target monitoring data may also be other data that may affect the above data, such as user identity information, user age or years of use, user exercise status, etc.

[0082] Among them, the pupil distance of different users will be different. When there are multiple different users wearing the same smart wearable device at different times, it is necessary to collect their pupil distances based on the user's identity information, and then adjust the image distance. For adolescents with faster physical development, their pupil distance may change significantly within a certain period of time. Therefore, for rapidly developing users, the user's pupil distance data can be updated every once in a while, and the image distance of the smart wearable device can be adjusted accordingly. When the user is in motion, the scene depth may change significantly. Therefore, the image distance needs to be adjusted according to the scene depth so that the virtual object can adapt to the display requirements in different scenes.

[0083] In one embodiment, a trigger condition for adjusting the combined image distance may be set. When the smart wearable device detects that certain data (such as scene depth, user parallax, etc.) meets the trigger condition, the current scene depth, the current parallax of the target user, and the second pupil distance of the target user are collected, and the first adjustment amount of the combined image distance is calculated.

[0084] In one embodiment, the second pupil distance of the target user can be tracked and calculated in real time during the pupil distance calculation process, and a pupil distance change threshold can be set. When the pupil distance data changes significantly, such as when the pupil distance difference between the second pupil distance and the first pupil distance (or the preset pupil distance) is greater than the pupil distance change threshold, the calculation of the first adjustment amount of the combined image distance is triggered, and the current combined image distance of the smart wearable device (such as the first combined image distance or the preset combined image distance) is adjusted according to the calculation result to adapt to the optimal viewing distance corresponding to the second pupil distance.

[0085] In one embodiment, the scene depth of the smart wearable device can also be monitored in real time. When the current scene depth of the smart wearable device changes significantly, such as when the scene depth change is greater than a preset scene depth change threshold, the second pupil distance of the target user is measured once, and then the first adjustment amount of the combined image distance is calculated based on the current scene depth, the second pupil distance and the current parallax, and the combined image distance of the smart wearable device is adjusted.

[0086] The scene depth corresponding to the preset combined image distance or the scene depth at other known combined image distances may be used as a reference to calculate the scene depth difference between the currently acquired scene depth and the reference scene depth as the scene depth variation.

[0087] Exemplarily, the first pupil distance corresponding to the first combined image distance (or the pupil distance corresponding to the last combined image distance adjustment or the preset pupil distance, the first pupil distance is used as an example here) can be used as a reference. The smart wearable device collects the second pupil distance of the target user in real time through tools such as image sensors and pupil positioning algorithms. When the pupil distance difference between the second pupil distance and the first pupil distance is greater than the preset pupil distance difference threshold, the current parallax of the target user and the current scene depth of the smart wearable device are collected, and the adjustment amount of the optimal combined image distance corresponding to the second pupil distance relative to the first combined image distance is calculated, that is, the first adjustment amount of the combined image distance.

[0088] In another embodiment, the current scene depth of the smart wearable device, the current parallax of the target user, and the second pupil distance of the target user may also be periodically collected, the first adjustment amount of the combined image distance corresponding to each collection period is calculated, and the combined image distance of the smart wearable device is adjusted. For example, the first adjustment amount of the combined image distance is calculated every one minute, and the combined image distance of the display object of the smart wearable device is adjusted according to the first adjustment amount of the combined image distance.

[0089] For example, the collection period can be set according to the usage scenario. For example, when the target user is in a sitting state and the scene changes less frequently, the collection period can be set longer, such as collecting relevant data every 15 minutes and calculating the first adjustment amount of the combined image distance, and adjusting the combined image distance once; and when the target user is in motion (such as walking, running, cycling, etc.), the collection period can be set shorter according to the motion state. For example, the combined image distance is adjusted every 10 seconds when walking, every 5 seconds when running, and every 3 seconds when cycling.

[0090] In a specific embodiment, the calculation formula of the first adjustment amount of the combined image distance can be expressed as:

[0091] ΔD2 = f(IPD, parallax, scene depth) = k1·(d2-d1)+k2·μ0+k3·ω0

[0092] Among them, ΔD2 represents the first adjustment amount of the combined image distance, d2 represents the second pupil distance, d1 represents the first pupil distance, μ0 represents the current parallax of the target user, ω0 represents the current scene depth of the smart wearable device, k1, k2, and k3 represent weight coefficients, which can be pre-set with fixed values ​​and adjusted according to actual needs.

[0093] S103, calculating a target image convergence distance based on the first image convergence distance and the first adjustment amount of the image convergence distance;

[0094] In one embodiment, the first image convergence distance is added to the first adjustment amount of the image convergence distance to obtain the target image convergence distance.

[0095] In one embodiment, when calculating the first adjustment amount of the combined image distance, the direction needs to be considered.

[0096] For example, in the display space of the smart wearable device, movement toward the user can be set to be represented as negative, that is, the combined image distance decreases; and movement away from the user can be set to be represented as positive, that is, the combined image distance increases.

[0097] S104: Based on the target image distance, adjust the image distance of the smart wearable device so that the display position of the target display object matches the viewing distance of the target user.

[0098] In one embodiment, the target display object is rendered according to the target image distance, and the rendering engine resets the display position of the target display object, that is, the target image distance, and then updates the display position and display effect of the target display object according to the adjusted target image distance, so that the display position of the target display object matches the viewing distance of the target user.

[0099] The adjustment of the display effect may be determined according to the display position and the characteristics of the target display object.

[0100] For example, if the combined image distance is reduced, the overall display volume of the target display object can be reduced, or the display font can be reduced, etc. If the combined image distance is increased, the overall display volume of the target display object can be enlarged, or the display font can be enlarged, etc.

[0101] This embodiment dynamically adjusts the image-joint distance to adapt to the pupil distance and viewing angle of different users, reducing visual fatigue caused by long-term use. Through intelligent scene adaptation technology, combined with scene depth information, the image-joint distance is dynamically adjusted to improve the positioning accuracy of virtual objects in the real environment. According to the user's personalized needs (such as pupil distance), the image-joint distance is dynamically adjusted to provide a better user experience.

[0102] This embodiment provides a method for adjusting the image-joint distance of a smart wearable device, which calculates the first image-joint distance through the first pupil distance of the target user, implements customized image-joint distance calculation for the target user, and improves the applicability of the smart wearable device. The image-joint distance adjustment amount is calculated according to the current scene depth, the current parallax of the target user, and the second pupil distance, so that the image-joint distance can change dynamically according to the actual use environment, thereby enhancing the adaptability and flexibility of the device display; by adjusting the image-joint distance of the smart wearable device so that the display position of the target display object matches the viewing distance of the target user, the positioning accuracy of the target display object is improved, thereby reducing the user's visual fatigue and improving the user's visual experience comfort.

[0103] Please refer to Figure 2 , Figure 2A schematic flow chart of a second embodiment of a method for adjusting the combined image distance of a smart wearable device provided in an embodiment of the present application.

[0104] like Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, after step S103, the following steps are further included:

[0105] S201, calculating the left eye matrix and the right eye matrix of the target user respectively based on a preset reference matrix and the second pupil distance;

[0106] In one embodiment, a reference matrix of the smart wearable device may be set.

[0107] For example, it is assumed that the two lenses of the smart wearable device are plane lenses and are located in the same plane. The midpoint of the line connecting the centers of the two lenses of the smart wearable device is used as the reference point, the line connecting the centers of the two lenses is used as the horizontal axis X, the reference point is used as the intersection point and the direction axis perpendicular to the plane where the two planes are located is used as the depth axis Z, and the reference point is used as the intersection point and the direction axis perpendicular to both the horizontal axis X and the depth axis Z is used as the vertical axis Y to construct a reference coordinate system.

[0108] For example, a reference matrix can be constructed with the reference point as the starting point and the Z axis as the direction. The reference matrix can be expressed as a 3×3 unit matrix, but only has non-zero elements in the Z axis direction, that is:

[0109]

[0110] The reference matrix constructed in this way represents the unit vector along the Z-axis direction.

[0111] It is understandable that the construction of the reference matrix can be constructed according to actual application requirements. The above-mentioned reference matrix constructed in the embodiment of the present application is only used as an example for explanation and does not constitute a specific limitation on the reference matrix.

[0112] Furthermore, current viewing angle data of the target user is obtained, wherein the current viewing angle data includes left-eye viewing angle data and right-eye viewing angle data; based on the current viewing angle data and the second pupil distance, a left-eye rotation matrix of the left-eye viewing angle data relative to the reference matrix and a right-eye rotation matrix of the right-eye viewing angle data relative to the reference matrix are respectively calculated; based on the reference matrix, the left-eye rotation matrix and the right-eye rotation matrix, a left-eye matrix and a right-eye matrix of the target user are calculated.

[0113] In one embodiment, the current viewing angle data may include pupil coordinates and sight line direction, including left eye pupil coordinates and left eye sight line direction, right eye pupil coordinates and right eye sight line direction.

[0114] The pupil coordinates may be obtained through a pupil positioning algorithm, and the second pupil distance between the left eye pupil and the right eye pupil may be calculated according to the left eye pupil coordinates and the right eye pupil coordinates.

[0115] In one embodiment, the line of sight direction can be realized by using eye tracking technology, such as pupil corneal reflection method, retinal image positioning, structured light tracking, optical waveguide eye tracking, and eye tracking sensor based on laser scanning.

[0116] For example, the pupil-corneal reflection method determines the direction of sight by tracking the light spots reflected by the cornea. When an infrared light source is irradiated onto the eye, the cornea reflects the light, and the reflected light spots are captured by an infrared camera to calculate the direction of sight.

[0117] Exemplarily, optical waveguide eye tracking uses optical waveguide technology combined with eye tracking to capture image data of the eyeball through multiple infrared LEDs and cameras, supporting real-time eye tracking and gaze point rendering.

[0118] In one embodiment, the rotation angle of the left / right eye matrix relative to the reference matrix is ​​calculated according to the second pupil distance of the target user and the device parameters of the smart wearable device.

[0119] In one embodiment, the rotation angle can be calculated by trigonometric functions. Assuming that the pupil distance offset is d, the rotation angle θ of the left eye and the right eye can be calculated by the following formula:

[0120]

[0121] Where f is the focal length, which is usually determined by the optical design of the smart wearable device. d represents the pupil distance offset, θ left Indicates the left eye rotation angle, θ right Indicates the right eye rotation angle.

[0122] Among them, the calculation formula of the pupil distance offset of the left and right eyes can be expressed as:

[0123]

[0124] Wherein, d represents the pupil distance offset, d2 represents the second pupil distance, and d0 represents the preset pupil distance of the smart wearable device.

[0125] In one embodiment, a rotation matrix is ​​constructed according to the rotation angle. The rotation matrix of the left and right eyes is constructed using the calculated rotation angle:

[0126]

[0127]

[0128] Among them, R leftRepresents the left rotation matrix, R right Represents a right rotation matrix.

[0129] In one embodiment, it is assumed that the reference matrix is ​​represented as M base , multiply the left / right eye rotation matrix by the reference matrix to get the final left / right eye matrix:

[0130] M left =M base ·R left

[0131] M right =M base ·R right

[0132] Thus, the rotation angle of the left and right eyes relative to the reference matrix can be calculated, thereby providing the necessary perspective information for the visual rendering of the smart wearable device and enabling users to obtain the best visual experience when using the smart wearable device, reducing visual fatigue and discomfort.

[0133] This embodiment can provide customized viewing angle adjustment for each user by acquiring the current viewing angle data of the target user, including left-eye viewing angle data and right-eye viewing angle data, which helps to improve the personalization and comfort of the augmented reality experience.

[0134] S202, determining a target display posture of the target display object at the target image distance based on the left eye matrix and the right eye matrix;

[0135] In one embodiment, in a smart wearable device, a rendering engine uses a right eye matrix to determine the correct position of a target display object in a right eye view; similarly, a left eye matrix is ​​used to determine the correct position of a target display object in a left eye view.

[0136] In one embodiment, in a smart wearable device, the rendering engine can determine the correct position of the target display object in the left eye view and the right eye view according to the left eye matrix and the right eye matrix, respectively, so that the left eye view and the right eye view can achieve view overlap at the target image distance, avoiding obvious parallax and improving the user's visual experience.

[0137] It is understandable that the rendering technology related to the view by the rendering engine is a known technology in the art, and this application does not make any specific limitation or description on this.

[0138] This embodiment uses the current viewing angle data and the second pupil distance to calculate the rotation matrix of the left eye and the right eye, thereby ensuring the accuracy of the view conversion, so that the virtual objects are displayed in the user's field of view with precise posture, thereby improving the realism and immersion of the augmented reality image.

[0139] S203: Based on the target display posture, adjust the posture of the target display object so that the display posture of the target display object matches the viewing distance of the target user.

[0140] In one embodiment, the target display object is adjusted according to the target display posture, including translation, rotation and scaling operations, to ensure that the object is properly aligned in the user's field of view. For example, if the user moves his head, the position and direction of the object need to be updated in real time to keep it consistent with the relative position of the user's line of sight.

[0141] The user's viewing distance refers to the straight-line distance from the user's eyes to the target object.

[0142] In one embodiment, the adjusted posture information is used in a rendering engine to ensure that the target display object is correctly displayed in the field of view of the target user.

[0143] In one embodiment, the target image distance and posture of the target display object can be adjusted separately. Because the user's line of sight and head position may change continuously, the change of head posture may not necessarily affect the scene depth, or change the user's pupil distance and parallax, but it will inevitably cause the relative posture of the target display object and the user's head to change. Therefore, the smart wearable device can update the posture of the target display object in real time, but the image distance of the target display object can be adjusted according to the actual application scenario.

[0144] Generally, motion sensors (such as gyroscopes, image sensors, etc.) can be used to track the user's line of sight and head position, and adjust the object's posture in real time, so that the target display object always maintains the target display posture relative to the target user, thereby matching the target user's viewing distance.

[0145] This embodiment calculates the left eye matrix and the right eye matrix of the target user, and determines the display posture of the target display object at the target image distance based on these matrices, so as to optimize the image distance, make the display position of the virtual object more consistent with the user's natural visual habits, and reduce visual fatigue. The posture of the target display object is adjusted based on the target display posture, so that its display posture matches the user's viewing distance, so that the augmented reality system dynamically adapts to the user's perspective changes, improves the flexibility and responsiveness of the system, and at the same time, ensures that the user can obtain clear and accurate virtual object display at different perspectives and distances, and enhances the user's augmented reality experience.

[0146] See also Figure 3 , Figure 3 It is a structural schematic diagram of a first embodiment of a combined image distance adjustment device for a smart wearable device provided in the present application, and the combined image distance adjustment device for the smart wearable device is used to execute the combined image distance adjustment method for the smart wearable device described above.

[0147] like Figure 3 As shown, the combined image distance adjustment device 300 of the smart wearable device includes: a first combined image distance calculation module 301, a first adjustment amount calculation module 302, a target combined image distance module 303 and a combined image distance adjustment module 304.

[0148] A first image combination distance calculation module 301 is used to determine a first image combination distance of the smart wearable device based on a first pupil distance of the target user;

[0149] A first adjustment amount calculation module 302 is used to determine a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user;

[0150] A target image coincidence distance module 303 is used to determine a target image coincidence distance based on the first image coincidence distance and the first adjustment amount of the image coincidence distance;

[0151] The image combination distance adjustment module 304 is used to adjust the image combination distance of the smart wearable device based on the target image combination distance so that the display position of the target display object in the display space of the smart wearable device matches the viewing distance of the target user.

[0152] In one embodiment, the combined image distance adjustment device 300 of the smart wearable device further includes a posture adjustment module, including:

[0153] A matrix calculation unit, used to calculate the left eye matrix and the right eye matrix of the target user respectively based on a preset reference matrix and the second pupil distance;

[0154] A position determination unit, configured to determine a target display posture of the target display object at the target combined image distance based on the left eye matrix and the right eye matrix;

[0155] The posture adjustment unit is used to adjust the posture of the target display object based on the target display posture, so that the display posture of the target display object matches the viewing distance of the target user.

[0156] In one embodiment, the matrix calculation unit includes:

[0157] A viewing angle data acquisition subunit, configured to acquire current viewing angle data of the target user, wherein the current viewing angle data includes left-eye viewing angle data and right-eye viewing angle data;

[0158] a rotation matrix calculation subunit, configured to calculate, based on the current viewing angle data and the second pupil distance, a left-eye rotation matrix of the left-eye viewing angle data relative to the reference matrix, and a right-eye rotation matrix of the right-eye viewing angle data relative to the reference matrix;

[0159] A left-eye matrix calculation subunit, configured to calculate the left-eye matrix based on the reference matrix and the left-eye rotation matrix;

[0160] The right-eye matrix calculation subunit is used to calculate the right-eye matrix based on the reference matrix and the right-eye rotation matrix.

[0161] In one embodiment, the image combination distance adjustment device 300 of the smart wearable device further includes a second pupil distance acquisition module, including:

[0162] An eye image acquisition unit, used for acquiring an eye image of the target user;

[0163] A pupil coordinate acquisition unit, configured to perform pupil location detection on the eye image based on a feature point detection model to obtain left eye pupil coordinates and right eye pupil coordinates of the target user;

[0164] The second pupil distance calculation unit is used to calculate the second pupil distance of the target user based on the left eye pupil coordinates and the right eye pupil coordinates.

[0165] In one embodiment, the pupil coordinate acquisition unit includes:

[0166] A feature detection subunit, configured to detect pupil contour features in the eye image based on the feature point detection model;

[0167] The pupil positioning subunit is used to perform positioning calculation on the pupil center point coordinates based on the pupil positioning algorithm to obtain the left eye pupil coordinates and the right eye pupil coordinates of the target user.

[0168] In one embodiment, the first combined image distance calculation module 301 includes:

[0169] A preset data acquisition unit, used to acquire a preset pupil distance and a preset image combination distance of the smart wearable device;

[0170] A second adjustment amount calculation unit, configured to determine a second adjustment amount of the combined image distance based on a pupil distance difference between the first pupil distance and the preset pupil distance;

[0171] The first combined image distance calculation unit is used to determine the first combined image distance based on the preset combined image distance and the second combined image distance adjustment amount.

[0172] In one embodiment, the combined image distance adjustment device 300 of the smart wearable device further includes a combined image distance adjustment judgment module, including:

[0173] A monitoring data acquisition unit, used to acquire at least one set of target monitoring data;

[0174] A data difference calculation unit, used to calculate the data difference between the target monitoring data and the corresponding preset reference data;

[0175] The combined image distance adjustment judgment unit is used to collect the current scene depth, the current parallax of the target user and the second pupil distance of the target user based on a preset data collection method when the data difference is greater than or equal to a preset difference threshold.

[0176] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the embodiment of the combined image distance adjustment method of the aforementioned smart wearable device, and will not be repeated here.

[0177] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 The system runs on the smart wearable device shown.

[0178] See also Figure 4 , Figure 4 1 is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application. The smart wearable device may be a server.

[0179] See also Figure 4 The smart wearable device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0180] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for adjusting the combined image distance of a smart wearable device.

[0181] The processor is used to provide computing and control capabilities to support the operation of the entire smart wearable device.

[0182] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for adjusting the combined image distance of the smart wearable device.

[0183] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the smart wearable device to which the scheme of the present application is applied. The specific smart wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0184] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0185] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0186] Determine a first image combination distance of the smart wearable device based on a first pupil distance of the target user;

[0187] Determining a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user;

[0188] Determining a target combined image distance based on the first combined image distance and the first adjustment amount of the combined image distance;

[0189] Based on the target image-joint distance, the image-joint distance of the smart wearable device is adjusted so that the display position of the target display object in the display space of the smart wearable device matches the viewing distance of the target user.

[0190] In one embodiment, after implementing the adjustment of the image convergence distance of the smart wearable device based on the target image convergence distance, the processor is further configured to implement:

[0191] Calculating a left-eye matrix and a right-eye matrix of the target user based on a preset reference matrix and the second pupil distance;

[0192] Determining a target display posture of the target display object at the target combined image distance based on the left eye matrix and the right eye matrix;

[0193] Based on the target display posture, the target display object is adjusted in posture so that the display posture of the target display object matches the viewing distance of the target user.

[0194] In one embodiment, when the processor calculates the left eye matrix and the right eye matrix of the target user based on the preset reference matrix and the second pupil distance, it is used to implement:

[0195] Acquire current viewing angle data of the target user, wherein the current viewing angle data includes left-eye viewing angle data and right-eye viewing angle data;

[0196] Based on the current viewing angle data and the second pupil distance, calculating a left-eye rotation matrix of the left-eye viewing angle data relative to the reference matrix, and a right-eye rotation matrix of the right-eye viewing angle data relative to the reference matrix;

[0197] Calculating the left-eye matrix based on the reference matrix and the left-eye rotation matrix;

[0198] The right-eye matrix is ​​calculated based on the reference matrix and the right-eye rotation matrix.

[0199] In one embodiment, before calculating the first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user, the processor is further configured to implement:

[0200] Acquiring an eye image of the target user;

[0201] Based on the feature point detection model, pupil location detection is performed on the eye image to obtain left eye pupil coordinates and right eye pupil coordinates of the target user;

[0202] Based on the left eye pupil coordinates and the right eye pupil coordinates, a second pupil distance of the target user is calculated.

[0203] In one embodiment, when the processor implements the pupil location detection based on the feature point detection model to the eye image and obtains the left eye pupil coordinates and the right eye pupil coordinates of the target user, it is used to implement:

[0204] Based on the feature point detection model, detecting pupil contour features in the eye image;

[0205] Based on the pupil positioning algorithm, the pupil center point coordinates are positioned and calculated to obtain the left eye pupil coordinates and the right eye pupil coordinates of the target user.

[0206] In one embodiment, when the processor calculates the first image combination distance of the smart wearable device based on the first pupil distance of the target user, it is used to implement:

[0207] Obtaining a preset pupil distance and a preset image combination distance of the smart wearable device;

[0208] Determining a second adjustment amount of the combined image distance based on a pupil distance difference between the first pupil distance and the preset pupil distance;

[0209] The first combined image distance is determined based on the preset combined image distance and the second adjustment amount of the combined image distance.

[0210] In one embodiment, before calculating the first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user, the processor is further configured to implement:

[0211] Acquire at least one set of target monitoring data;

[0212] Calculating the data difference between the target monitoring data and the corresponding preset reference data;

[0213] When the data difference is greater than or equal to a preset difference threshold, the current scene depth, the current parallax of the target user, and the second pupil distance of the target user are collected based on a preset data collection method.

[0214] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement a method for adjusting the combined image distance of any smart wearable device provided in an embodiment of the present application.

[0215] The computer-readable storage medium may be an internal storage unit of the smart wearable device described in the foregoing embodiment, such as a hard disk or memory of the smart wearable device. The computer-readable storage medium may also be an external storage device of the smart wearable device, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc., equipped on the smart wearable device.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for adjusting the image distance of a smart wearable device, characterized in that: The method comprises: Determine a first image combination distance of the smart wearable device based on a first pupil distance of the target user; Determining a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user; Determining a target combined image distance based on the first combined image distance and the first adjustment amount of the combined image distance; Based on the target image-joint distance, the image-joint distance of the smart wearable device is adjusted so that the display position of the target display object in the display space of the smart wearable device matches the viewing distance of the target user.

2. The method for adjusting the combined image distance of a smart wearable device according to claim 1, characterized in that: After adjusting the image distance of the smart wearable device based on the target image distance, the method further includes: Calculating a left-eye matrix and a right-eye matrix of the target user based on a preset reference matrix and the second pupil distance; Determining a target display posture of the target display object at the target combined image distance based on the left eye matrix and the right eye matrix; Based on the target display posture, the target display object is adjusted in posture so that the display posture of the target display object matches the viewing distance of the target user.

3. The method for adjusting the combined image distance of a smart wearable device according to claim 2, characterized in that: The calculating the left eye matrix and the right eye matrix of the target user based on the preset reference matrix and the second pupil distance includes: Acquire current viewing angle data of the target user, wherein the current viewing angle data includes left-eye viewing angle data and right-eye viewing angle data; Based on the current viewing angle data and the second pupil distance, respectively calculating a left-eye rotation matrix of the left-eye viewing angle data relative to the reference matrix, and a right-eye rotation matrix of the right-eye viewing angle data relative to the reference matrix; Calculating the left-eye matrix based on the reference matrix and the left-eye rotation matrix; The right-eye matrix is ​​calculated based on the reference matrix and the right-eye rotation matrix.

4. The method for adjusting the combined image distance of a smart wearable device according to claim 1, characterized in that: Before calculating the first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user and the second pupil distance of the target user, the method further includes: Acquiring an eye image of the target user; Based on the feature point detection model, pupil location detection is performed on the eye image to obtain left eye pupil coordinates and right eye pupil coordinates of the target user; Based on the left eye pupil coordinates and the right eye pupil coordinates, a second pupil distance of the target user is calculated.

5. The method for adjusting the combined image distance of a smart wearable device according to claim 4, characterized in that: The method of performing pupil location detection on the eye image based on the feature point detection model to obtain left eye pupil coordinates and right eye pupil coordinates of the target user includes: Based on the feature point detection model, detecting pupil contour features in the eye image; Based on the pupil positioning algorithm, the pupil center point coordinates are positioned and calculated to obtain the left eye pupil coordinates and the right eye pupil coordinates of the target user.

6. The method for adjusting the combined image distance of a smart wearable device according to claim 1, characterized in that: The step of calculating a first image combination distance of the smart wearable device based on a first pupil distance of the target user includes: Obtaining a preset pupil distance and a preset image combination distance of the smart wearable device; Determining a second adjustment amount of the combined image distance based on a pupil distance difference between the first pupil distance and the preset pupil distance; The first combined image distance is determined based on the preset combined image distance and the second adjustment amount of the combined image distance.

7. The method for adjusting the combined image distance of a smart wearable device according to claim 1, characterized in that: Before calculating the first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user and the second pupil distance of the target user, the method further includes: Acquire at least one set of target monitoring data, wherein the target monitoring data includes one or more of scene depth, user parallax, and user pupil distance; Calculating the data difference between the target monitoring data and the corresponding preset reference data; When the data difference between any of the target monitoring data and the corresponding preset reference data is greater than or equal to the preset difference threshold, the current scene depth, the current parallax of the target user and the second pupil distance of the target user are collected based on a preset data collection method.

8. A device for adjusting the combined image distance of a smart wearable device, characterized in that: The image combination distance adjustment device of the smart wearable device comprises: A first image combination distance calculation module, used to calculate a first image combination distance of the smart wearable device based on a first pupil distance of the target user; A first adjustment amount calculation module, configured to calculate a first adjustment amount of the combined image distance based on the current scene depth, the current parallax of the target user, and the second pupil distance of the target user; A target image-joint distance module, configured to calculate a target image-joint distance based on the first image-joint distance and the first adjustment amount of the image-joint distance; The image combination distance adjustment module is used to adjust the image combination distance of the smart wearable device based on the target image combination distance so that the display position of the target display object matches the viewing distance of the target user.

9. A smart wearable device, characterized in that: The smart wearable device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the method for adjusting the combined image distance of the smart wearable device according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for adjusting the combined image distance of the smart wearable device according to any one of claims 1 to 7 are implemented.

Images