Wearing deviation measurement method and apparatus, and electronic device and storage medium

Abstract

Provided in the embodiments of the present disclosure are a wearing deviation measurement method and apparatus, and an electronic device and a storage medium. The method comprises: collecting an eye image; obtaining at least two eye feature points according to the eye image, wherein the eye feature points are distributed on the contour of an eye area in the eye image; and obtaining deviation information according to location features of the at least two eye feature points, wherein the deviation information represents a deviation type and / or a deviation amount of the current wearing pose of a terminal device relative to a standard wearing pose. An eye image is collected, and a deviation type and / or a deviation amount of the current wearing pose of a terminal device relative to a standard wearing pose are / is obtained on the basis of eye feature points on the contour of an eye area in the eye image, so as to realize early detection and prompt when a wearing pose is abnormal, thereby avoiding the problem of the accuracy of gaze point detection being reduced due to the fact that the wearing pose of the terminal device is abnormal, and improving the image display effect and the human-computer interaction efficiency.

Description

Wearing deviation detection method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Wearing Deviation Detection Method, Device, Electronic Device and Storage Medium” filed on June 26, 2023, with application number 2023107649197. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of virtual reality technology, and in particular to a wearing deviation detection method, device, electronic device, and storage medium. Background Art

[0003] Currently, in application scenarios based on virtual reality (VR), the terminal devices worn by users will perform eye tracking on the user's eyes to achieve gaze point detection, thereby obtaining better image display effects and human-computer interaction purposes.

[0004] In the existing technology, eye tracking technology is usually achieved by processing eye images captured by a terminal device worn by the user. However, when the wearing posture of the terminal device is abnormal, the accuracy of gaze point detection will be reduced, thereby affecting the image display effect and human-computer interaction efficiency.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a wearing deviation detection method, device, electronic device and storage medium to overcome the problem of reduced accuracy of gaze point detection when the wearing posture of a terminal device is abnormal.

[0007] In a first aspect, an embodiment of the present disclosure provides a wearing deviation detection method, comprising: collecting an eye image; obtaining at least two eye feature points based on the eye image, wherein the eye feature points are distributed on the contour of the eye area in the eye image; obtaining deviation information based on the position characteristics of at least two of the eye feature points, wherein the deviation information characterizes the deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture.

[0008] In a second aspect, an embodiment of the present disclosure provides a wearing deviation detection device, comprising: an acquisition module for acquiring an eye image; a processing module for obtaining at least two eye feature points based on the eye image, wherein the eye feature points are distributed on the contour of the eye area in the eye image; a detection module for obtaining deviation information based on positional characteristics of at least two of the eye feature points, wherein the deviation information represents the deviation direction and / or deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture.

[0009] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the wearing deviation detection method described in the first aspect and various possible designs of the first aspect.

[0010] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the wearing deviation detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0011] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the wearing deviation detection method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0013] FIG1 is a diagram illustrating an application scenario of a wearing deviation detection method provided by an embodiment of the present disclosure;

[0014] FIG2 is a flow chart of a first embodiment of a wearing deviation detection method according to an embodiment of the present disclosure;

[0015] FIG3 is a flowchart of a specific implementation method of step S102 in the embodiment shown in FIG2 ;

[0016] FIG4 is a schematic diagram of eye feature points provided by an embodiment of the present disclosure;

[0017] FIG5 is a flowchart of a specific implementation method of step S1023 in the embodiment shown in FIG3 ;

[0018] FIG6 is a schematic diagram of a process for determining a position extreme point according to an embodiment of the present disclosure;

[0019] FIG7 is a flowchart of a specific implementation of step S103 in the embodiment shown in FIG2 ;

[0020] FIG8 is a flowchart of a specific implementation method of step S1031 in the embodiment shown in FIG5 ;

[0021] FIG9 is a schematic diagram of longitudinal center point coordinates provided by an embodiment of the present disclosure;

[0022] FIG10 is a second flow chart of a wearing deviation detection method provided by an embodiment of the present disclosure;

[0023] FIG11 is a flowchart of a possible specific implementation of step S205 in the embodiment shown in FIG10 ;

[0024] FIG12 is a flowchart of another possible specific implementation of step S205 in the embodiment shown in FIG10 ;

[0025] FIG13 is a schematic diagram of second deviation information provided by an embodiment of the present disclosure;

[0026] FIG14 is a structural block diagram of a wearing deviation detection device provided by an embodiment of the present disclosure;

[0027] FIG15 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0028] FIG16 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] The following explains the application scenarios of the embodiments of the present disclosure:

[0032] FIG1 is a diagram illustrating an application scenario of the wearing deviation detection method provided by an embodiment of the present disclosure. The wearing deviation detection method provided by an embodiment of the present disclosure can be applied to products based on virtual reality (VR) and augmented reality (AR). More specifically, it can be applied to pre-use detection scenarios for wearable terminal devices with eye tracking capabilities. As shown in FIG1 , the method provided by an embodiment of the present disclosure can be applied to a wearable terminal device, such as the virtual reality glasses 11 shown in the figure. After the user wears the virtual reality glasses 11, the virtual reality glasses 11 detect the current wearing posture (position, angle) of the virtual reality glasses 11 by executing the wearing deviation detection method provided by this embodiment, and feedback the detection results to the user in the form of images and sounds. For example, as shown in the figure, the virtual reality glasses 11 play a voice message "The wearing position is 10 mm to the left," allowing the user to adjust the wearing posture of the virtual reality glasses based on the above detection results, thereby achieving a more accurate eye tracking effect.

[0033] In the prior art, the eye tracking function implemented by a wearable terminal device is usually achieved by using a camera built into the terminal device that is facing the user's eyes to capture the user's eye image, and then by processing the eye image to identify the position of the light spot formed by corneal reflection in the eye image, thereby achieving gaze point detection based on the light spot position and realizing the eye tracking function. However, when the wearing posture of the terminal device is abnormal, the quality of the light spot in the captured eye image will be reduced, which in turn will reduce the accuracy of gaze point detection, affecting the effect of eye tracking. In the prior art, for the above-mentioned problem of abnormal wearing posture, the user will only be notified when a serious or large posture abnormality occurs. In the case of a minor posture abnormality, software correction is usually used to handle it. This results in a greater probability of gaze point jumping and shaking during subsequent use due to the deviation in the wearing posture of the terminal device itself, affecting the image display effect and human-computer interaction efficiency. The embodiment of the present disclosure provides a wearing deviation detection method to solve the above-mentioned problem.

[0034] Referring to FIG2 , FIG2 is a flow chart of a wearing deviation detection method provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device, and the wearing deviation detection method includes:

[0035] Step S101: collecting eye images.

[0036] Step S102: obtaining at least two eye feature points according to the eye image, where the eye feature points are distributed on the contour of the eye area in the eye image.

[0037] By way of example, referring to the application scenario schematic diagram shown in FIG1 , the method provided in this embodiment can be implemented by a wearable terminal device, such as smart glasses or a virtual reality helmet. The terminal device is provided with an image acquisition unit (e.g., a camera) facing the user's eyes. Upon receiving a user instruction or detecting that the terminal device is being worn by a user, the image acquisition unit acquires an image of the user's eyes, thereby obtaining an eye image. More specifically, the image includes an eye area. The eye image in this embodiment can be an image that includes the eye areas corresponding to both eyes of the user in the same frame, or an image that only includes the eye areas corresponding to one eye of the user. That is, the eye image can be a binocular image that includes both left and right eye areas, or two monocular images (i.e., a left eye image and a right eye image). Furthermore, by way of example, multiple image acquisition units can be provided in the terminal device to acquire images of the user's eyes from different angles at the same time, thereby obtaining eye images acquired synchronously at the same time. The specific implementation method can be configured as needed and will not be described in detail here.

[0038] For example, after acquiring the eye image, feature recognition is performed on the eye image to obtain at least two eye feature points located on the contour of the eye area in the eye image. In one possible implementation, as shown in FIG3 , the specific implementation of step S101 includes:

[0039] Step S1021: extracting features from the eye image to obtain a first feature map, where the first feature map represents pixel features of the eye image;

[0040] Step S1022: processing the first feature map based on the pre-trained prediction model to determine the contour line of the eye area in the eye image;

[0041] Step S1023: Obtain at least two eye feature points based on the contour line of the eye area.

[0042] FIG4 is a schematic diagram of an eye feature point provided by an embodiment of the present disclosure. As shown in FIG4 , taking the eye image as a monocular image as an example, after obtaining the eye image, feature extraction is performed on the eye image to obtain a first feature map. The first feature map represents the arrangement characteristics of the pixels in the eye image, that is, the pixel features. Afterwards, the first feature map is used as input and processed using a pre-trained prediction model to obtain the pixel points corresponding to the outline of the eye area in the eye image, that is, the contour line of the eye area, which is formed by connecting multiple contour nodes. Furthermore, based on the shape of the contour line of the above-mentioned eye area, at least two eye feature points can be determined, such as the eye feature point P1 and the eye feature point P2 shown in the image (shown as P1 and P2 in the figure). The above-mentioned two eye feature points are respectively located at the left end point and the right end point of the eye area, corresponding to the left corner of the eye and the right corner of the eye, respectively. Among them, exemplarily, the method for determining the above-mentioned eye feature point P1 and eye feature point P2 can be achieved by obtaining the coordinates of each contour node of the contour line corresponding to the eye area, and determining the leftmost contour node as the eye feature point P1 and the rightmost contour node as the eye feature point P2 according to the coordinates.

[0043] Furthermore, in one possible implementation, in the process of determining the eye feature points, at least two extreme position points in each of the horizontal and vertical directions (or two other perpendicular directions) can be determined based on the circumscribed rectangle of the contour line, and at least four extreme position points can be used as the eye feature points. The coordinates of the at least four extreme position points can be determined based on the shape of the contour line. In one possible implementation, as shown in FIG5 , the specific implementation of step S1023 includes:

[0044] Step S1023A: Obtain at least four position extreme points of the contour line of the eye area according to the tangent direction of each contour node in the contour line.

[0045] Step S1023B: Obtain at least four eye feature points based on at least four position extreme points.

[0046] Exemplarily, for the contour line obtained in the previous step, based on the tangent direction of each contour node in the contour line and according to the change of the tangent direction, at least four position extreme points of the contour line of the eye area can be determined, thereby obtaining the corresponding at least four eye feature points. Specifically, after being processed by the prediction model, a contour line composed of contour line nodes is obtained, and the implementation method of the contour line can be: an array composed of the coordinates of multiple contour line nodes. Afterwards, based on the coordinates of the contour line nodes, the direction vectors of each contour line node and the adjacent contour line nodes are calculated to obtain the tangent direction of each contour node (i.e., the vector node of the direction vector). The specific implementation method will not be repeated here. Afterwards, based on the tangent direction of each contour node, the contour node with a specific value of the tangent direction (for example, 0) is determined as the position extreme point.

[0047] For example, FIG6 is a schematic diagram of a process for determining position extreme points provided by an embodiment of the present disclosure. As shown in FIG6 , after obtaining the contour line data, the leftmost position extreme point P1 and the rightmost position extreme point P2 are first determined based on the coordinates of the contour nodes. The specific implementation method will not be repeated here. Afterwards, the target direction vectors of the position extreme point P1 and the position extreme point P2, as well as the direction Seta_1 of the target direction vector, are calculated. Afterwards, based on the above steps, the tangent direction of each contour node is calculated, and the contour nodes P3 and P4 corresponding to the same tangent direction as the direction Seta_1 of the target direction vector vec_1 are determined as position extreme points. In this embodiment, by calculating the tangent of the contour node, the contour nodes whose tangent direction is parallel to the eye area are obtained as position extreme points, so that the distribution of the position extreme points on the eye contour is more uniform and reasonable, thereby improving the accuracy of the subsequent calculation of the deviation information based on the eye feature points.

[0048] Furthermore, based on the eye feature points obtained based on the position extreme points, contour nodes on the eye region's contour line can be further selected to increase the number of eye feature points. For example, based on the eye feature points (hereinafter referred to as base points) generated based on the position extreme points, a number of eye feature points can be evenly added between adjacent base points to improve the accuracy of subsequent deviation information calculation. A first limiting distance exists between any two adjacent eye feature points, and this first limiting distance is determined based on the resolution of the eye image. That is, the higher the eye image resolution, the more eye feature points are inserted between the base points. Conversely, the lower the eye image resolution, the fewer eye feature points are inserted between the base points. This fully utilizes the content information in the eye image, reduces the positional error of the eye contour, improves the accuracy of the eye region, and further improves the accuracy of subsequent deviation information calculation based on the eye region's contour line.

[0049] Step S103: obtaining deviation information based on the positional features of at least two eye feature points, where the deviation information represents a deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture.

[0050] Exemplarily, after obtaining the position of the eye feature points based on the previous steps, the deviation information corresponding to the terminal device is determined based on the positional features formed by the positional relationship between the eye feature points, that is, the deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture. Specifically, the deviation types include: wearing to the left, wearing to the right, wearing to the upper left, wearing tilted (left high and right low, right high and left low), etc. The deviation amounts include: wearing 10 mm to the left, wearing 5 mm to the upper right, wearing tilted 10 degrees, etc. In one possible implementation, the position features include the position coordinates of the eye feature points in the eye image, as shown in FIG7 , and the specific implementation of step S103 includes:

[0051] Step S1031: Obtain eye center coordinates based on the position coordinates of at least two eye feature points.

[0052] Step S1032: Obtain a deviation vector according to the eye center coordinates and the reference coordinates, where the reference coordinates are the standard eye center coordinates corresponding to the standard wearing posture.

[0053] Step S1033: Obtain deviation information according to the deviation vector.

[0054] For example, first, based on the previous steps, the position coordinates of the eye feature points can be obtained, such as the coordinates of the leftmost position extreme point and the rightmost position extreme point, i.e., the coordinates of the reference points referred to in the above embodiments. Then, based on the shape characteristics of the eye region, the coordinates of the leftmost position extreme point and the rightmost position extreme point are averaged to obtain the approximate coordinates of the center point of the eye region, i.e., the eye center coordinates. Furthermore, in one possible implementation, the eye feature points used in the above steps, in addition to the reference points, may also include the topmost position extreme point, the bottommost position extreme point, and other eye feature points located between the reference points. The coordinates of all of these evenly distributed eye feature points are averaged to obtain the eye center coordinates, thereby improving the accuracy of the eye center coordinates. The specific implementation method can be set as needed and will not be further described here.

[0055] Afterwards, the preset reference coordinates, that is, the standard eye center coordinates corresponding to the standard wearing posture, are obtained, and the eye center coordinates and the reference coordinates are calculated to obtain the deviation vector between the two, and then the corresponding direction and distance, that is, the deviation information, are obtained based on the deviation vector. The deviation information can be a value in the form of a key-value pair [direction, distance], or it can be data in other forms, which is not specifically limited here. In this embodiment, the eye center coordinates are calibrated by pre-calibrated reference coordinates to obtain deviation information that characterizes the type and / or amount of deviation of the wearing posture, thereby realizing the inspection and calibration of the wearing position without using the light spot reflected by the iris, thereby improving the accuracy and stability of subsequent gaze point detection.

[0056] Furthermore, in a possible implementation, the position coordinates include a longitudinal coordinate. As shown in FIG8 , a specific implementation of step S1031 includes:

[0057] Step S1031A: Acquire shooting angle information, where the shooting angle information represents the shooting angle when the image acquisition unit captures the eye image.

[0058] Step S1031B: Determine the longitudinal weighting coefficient according to the shooting angle information.

[0059] Step S1031C: Perform weighted averaging on the longitudinal coordinates according to the longitudinal weighting coefficient to obtain the longitudinal center point coordinates.

[0060] Step S1031D: Obtain the eye center coordinates based on the longitudinal center coordinates.

[0061] Exemplarily, the shooting angle information refers to the shooting angle of an image acquisition unit, such as a camera built into a terminal device, when capturing an eye image. Different terminal devices may have different positions of their corresponding built-in image acquisition units, resulting in corresponding pitch angles for the captured eye images. Exemplarily, the shooting angle information corresponds one-to-one to the model of the terminal device. The shooting angle information is preset in the terminal device or a fixed value obtained by the terminal device from a server. The shooting angle information can be represented by an angle value or a vector value, without specific limitation.

[0062] Furthermore, after obtaining the shooting angle information, the vertical coordinates of the visual feature points are weighted using the shooting angle information to obtain more accurate vertical center point coordinates that are closer to the actual situation. Figure 9 is a schematic diagram of a vertical center point coordinate provided by an embodiment of the present disclosure. As shown in Figure 9, based on the position of the image acquisition unit D1 (shown as D1 in the figure) inside the terminal device, the corresponding shooting angle information is Info_1, and the corresponding shooting angle is Theta_1. When the image acquisition unit D1 calculates the eye center point coordinates, without considering the shooting angle information, the coordinates of each eye feature point are averaged, and the eye center point coordinate obtained is P1. Based on the shooting angle information Info_1, the corresponding weighting coefficient is obtained, and after weighting the vertical coordinates of each eye feature point, the eye center point coordinate obtained is P2 based on the coordinates of the eye feature points. The eye center point coordinate P2, which is weighted by the shooting angle information Info_1, is closer to the actual eye center point position. In this embodiment, the longitudinal weighting coefficient is obtained through the shooting angle information, and the longitudinal coordinates are weighted averaged to make the generated longitudinal center point coordinates more accurate, that is, the eye center point coordinates are more accurate, thereby improving the accuracy of the subsequent deviation information obtained based on the eye center point coordinates.

[0063] In this embodiment, an eye image is captured; based on the eye image, at least two eye feature points are obtained, and the eye feature points are distributed on the contour of the eye area in the eye image; based on the position characteristics of the at least two eye feature points, deviation information is obtained, and the deviation information characterizes the deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture. By capturing an eye image and, based on the eye feature points on the contour of the eye area in the eye image, obtaining the deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture, early detection and prompting of abnormal wearing posture are achieved, avoiding the problem of reduced accuracy of gaze point detection due to abnormal wearing posture of the terminal device, and improving image display effect and human-computer interaction efficiency.

[0064] Referring to FIG10 , FIG10 is a second flow chart of the wearing deviation detection method provided by the embodiment of the present disclosure. Based on the embodiment shown in FIG2 , this embodiment further refines step S103 , and the wearing deviation detection method includes:

[0065] Step S201: collecting eye images, which include left eye images and right eye images.

[0066] Step S202: obtaining at least two eye feature points according to the eye image, where the eye feature points include a first feature point corresponding to the left eye image and a second feature point corresponding to the right eye image.

[0067] For example, in this embodiment, an image acquisition unit captures a left image corresponding to the left eye and a right image corresponding to the right eye. The left and right images are then processed to obtain first feature points corresponding to the left image and second feature points corresponding to the right image. The first feature points are located on the outline of the eye region in the left image, and the second feature points are located on the outline of the eye region in the right image. The specific method for obtaining the first feature points corresponding to the left image and the second feature points corresponding to the right image can be found in the description of the specific implementation method for obtaining eye feature points from a monocular image in the embodiment shown in FIG2 , and will not be further elaborated here.

[0068] For example, the left eye image and the right eye image can be acquired by different image acquisition units. After obtaining the left eye image and the right eye image, the left eye image and the right eye image can be aligned first, that is, the two images are converted to the same image coordinate system. The alignment can be performed by referring to the positional relationship between the image acquisition units. The details will not be repeated here. Afterwards, the aligned left eye image and the right eye image are processed to obtain the corresponding first feature points and second feature points. The obtained first feature points and second feature points are located in the same image coordinate system, which facilitates subsequent distance measurement. In another possible implementation, the left eye image and the right eye image can be captured by the same image acquisition unit, and the left eye image and the right eye image can be obtained by (logically) dividing the acquired eye image. The left eye image and the right eye image can be respectively represented as an image area or pixel set of the eye image. In this case, the left eye image and the right eye image are always in the same image coordinate system. In the subsequent processing, the eye image can be directly processed.

[0069] Step S203: obtaining the first center point coordinates corresponding to the left eye area in the left eye image according to the position feature of the first feature point.

[0070] Step S204: Obtain the coordinates of the second center point corresponding to the right eye area in the left eye image according to the positional features of the second feature point.

[0071] For example, after obtaining the first and second feature points, based on their respective positional characteristics, the coordinates of the first center point of the left eye region in the left image and the coordinates of the second center point corresponding to the right eye region in the right image can be obtained. Specifically, using the left image as an example, the coordinates of the first center point can be obtained by calculating the weighted average of the coordinates of each first feature point in the left image. For a specific implementation, refer to the method for calculating the eye center coordinates in the embodiment shown in FIG2 , and will not be further described here.

[0072] Step S205: Obtain deviation information according to the positional relationship between the first center point coordinates and the second center point coordinates.

[0073] For example, after obtaining the first center point coordinates and the second center point coordinates, deviation information is obtained based on the positional relationship between the first and second center point coordinates, i.e., the type and / or amount of deviation of the current wearing posture of the terminal device relative to the standard wearing posture. Specifically, for example, the horizontality of the line formed by the first center point coordinates and the second center point coordinates can be used to obtain deviation information indicating whether the terminal device is worn horizontally.

[0074] In a possible implementation, as shown in FIG11 , a specific implementation of step S205 includes:

[0075] Step S2051: Calculating a weighted sum based on the first center point coordinate and the second center point coordinate to obtain a weighted center point coordinate;

[0076] Step S2052: Obtain first deviation information based on the positional relationship between the weighted center point coordinates and the reference coordinates. The first deviation information represents the deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture in the horizontal direction or the vertical direction.

[0077] For example, first, the center point coordinates of the first center point coordinate and the second center point coordinate are calculated, that is, the center point coordinates are obtained by calculating the average value of the first center point coordinate and the second center point coordinate. On this basis, based on the shooting angle information of the image acquisition unit, the horizontal weighting coefficient and the vertical weighting coefficient corresponding to the first center point coordinate and the second center point coordinate are obtained, and then based on the horizontal weighting coefficient and the vertical weighting coefficient, the first center point coordinate and the second center point coordinate are weighted respectively, and the weighted average value is calculated to obtain the weighted center point coordinate. Among them, the specific method for obtaining the horizontal weighting coefficient and the vertical weighting coefficient can refer to the relevant introduction of obtaining the vertical weighting coefficient in the embodiment shown in Figure 2, and will not be repeated here.

[0078] Furthermore, after obtaining the weighted center point coordinates, the first deviation information is obtained based on the positional relationship between the weighted center point coordinates and the reference coordinates. The first deviation information represents the deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture in the horizontal direction and / or vertical direction, that is, the first deviation information represents whether the current wearing position of the terminal device is offset in the horizontal and vertical directions, and the specific offset distance. Specifically, for example, the first deviation information Info_1 = [10, -20], which indicates that the current wearing posture of the terminal device is 10 mm to the left in the horizontal direction and 20 mm below in the vertical direction relative to the standard posture.

[0079] In this embodiment, the corresponding weighted center point coordinates are obtained through the first center point coordinates and the second center point, and the first deviation information characterizing the wearing posture deviation of the terminal device in the horizontal or vertical direction is obtained based on the weighted center point coordinates. Compared with the scheme of using monocular images for judgment, the accuracy and stability are better, and the accuracy of gaze point detection is further improved.

[0080] In another possible implementation, as shown in FIG12 , the specific implementation of step S205 includes:

[0081] Step S2053: Generate an inclination vector according to the first center point coordinates and the second center point coordinates.

[0082] Step S2054: obtaining second deviation information according to the direction of the inclination vector, where the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture.

[0083] For example, in another possible implementation, after obtaining the first center point coordinates and the second center point coordinates, an inclination vector is generated based on the first center point coordinates and the second center point coordinates. Then, based on the direction of the inclination vector, second deviation information is obtained, wherein the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture. Figure 13 is a schematic diagram of a second deviation information provided by an embodiment of the present disclosure. As shown in Figure 13, for example, after obtaining the first center point coordinates P1 and the second center point coordinates P2 through the above steps, the first center point coordinates P1 and the second center point coordinates P2 are connected to obtain the inclination vector Vp (shown as Vp in the figure). Then, the inclination angle of the inclination vector Vp in the image coordinate system, that is, the angle Theta relative to the horizontal direction (0 degrees), is obtained to generate the second deviation information. For example, when Theta is greater than zero, it represents that the wearing posture of the terminal device is tilted to the left relative to the standard wearing posture (the terminal device is worn horizontally), that is, the left is lower and the right is higher; and when Theta is less than zero, it represents that the wearing posture of the terminal device is tilted to the right relative to the standard wearing posture, that is, the left is higher and the right is lower.

[0084] In this embodiment, an inclination vector is generated through the coordinates of the first center point and the second center point, and second deviation information representing the wearing inclination angle of the terminal device is generated based on the inclination vector, thereby realizing the detection of the tilt posture of the terminal device. The step of obtaining the deviation information (second deviation information) provided in the steps of this embodiment can be combined with the step of obtaining the deviation information (first deviation information) provided in the embodiment shown in Figure 11, that is, the terminal device obtains the first deviation information and the second deviation information synchronously or asynchronously (without limiting the order), and then uses the set of the first deviation information and the second deviation information as the deviation information for subsequent steps, thereby realizing posture detection in multiple dimensions of horizontal, vertical, and tilt, and further improving the accuracy of gaze point detection.

[0085] Optionally, after step S205, the method further includes:

[0086] Step S206: If the deviation information is greater than or equal to the deviation threshold, first indication information generated based on the deviation information is played in the terminal device, and the process returns to step S201, wherein the first indication information is used to indicate the action of adjusting the terminal device;

[0087] Step S207: If the deviation information is less than the deviation threshold, a second indication information is played in the terminal device, where the second indication information indicates that the current wearing posture of the terminal device is a standard wearing posture.

[0088] Exemplarily, after obtaining the deviation information, if the deviation between the current wearing posture of the terminal device represented by the deviation information and the standard wearing position is greater than the deviation threshold, a first prompt message is generated based on the deviation information, and the first indication message is played to the user through the human-computer interaction unit of the terminal device to prompt the user. For example, the deviation information is Info_1, and the content represented is: the wearing posture is biased to the left (deviation type), then the corresponding first prompt message generated is an icon of "wearing position is biased to the left", or a voice or text of "please move the terminal device to the right". For another example, the deviation information is Info_2, and the content represented is: the wearing posture is tilted downward by 10 degrees on the left (deviation type and deviation amount), then the corresponding first prompt message generated is an icon of "wearing position is tilted downward by 10 degrees on the left", or a voice or text of "please rotate the left side of the terminal device upward by 10 degrees".

[0089] Furthermore, after displaying the first indication information, the process returns to step S201, recaptures the real-time eye image, and repeats the above steps to continuously detect and prompt the wearing posture. After the deviation information is less than the deviation threshold, the second indication information is displayed to inform the user terminal device that the current wearing posture is the standard posture and that subsequent gaze point detection can be executed and initiated. The first indication information and the second indication information may include at least one of the following: prompt text, logo, or voice message, which can be set as needed and are not specifically limited here.

[0090] In this embodiment, by continuously displaying the indication information generated by the deviation information, automatic guidance of the user in manually adjusting the wearing posture of the terminal device is achieved, so that the user can adjust the terminal device to the standard wearing posture faster and more accurately, thereby improving the efficiency and accuracy of the wearing posture adjustment, and further improving the accuracy of subsequent gaze point detection.

[0091] Corresponding to the wearing deviation detection method of the above embodiment, FIG14 is a block diagram of the wearing deviation detection device provided by the embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Referring to FIG14, the wearing deviation detection device 3 includes:

[0092] An acquisition module 31 is used to acquire eye images;

[0093] a processing module 32, configured to obtain at least two eye feature points based on the eye image, where the eye feature points are distributed on the contour of the eye region in the eye image;

[0094] The detection module 33 is used to obtain deviation information based on the position characteristics of at least two eye feature points, where the deviation information represents the deviation direction and / or deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture.

[0095] In one embodiment of the present disclosure, processing module 32 is specifically configured to: perform feature extraction on an eye image to obtain a first feature map, wherein the first feature map represents pixel features of the eye image; process the first feature map based on a pre-trained prediction model to determine a contour line of the eye region in the eye image; and obtain at least two eye feature points based on the contour line of the eye region.

[0096] In one embodiment of the present disclosure, when obtaining at least two eye feature points based on the contour line of the eye region, the processing module 32 is specifically configured to obtain at least four position extreme value points of the contour line of the eye region based on the tangent direction of each contour node in the contour line. At least four eye feature points are obtained based on the at least four position extreme value points.

[0097] In one embodiment of the present disclosure, there is a first limiting distance between any two adjacent eye feature points, and the first limiting distance is determined based on the resolution of the eye image.

[0098] In one embodiment of the present disclosure, the positional features include the positional coordinates of eye feature points in the eye image. Detection module 33 is specifically configured to: obtain eye center coordinates based on the positional coordinates of at least two eye feature points; obtain a deviation vector based on the eye center coordinates and reference coordinates, where the reference coordinates are the standard eye center coordinates corresponding to the standard wearing posture; and obtain deviation information based on the deviation vector.

[0099] In one embodiment of the present disclosure, the position coordinates include longitudinal coordinates. When the detection module 33 obtains the eye center coordinates based on the position coordinates of at least two eye feature points, it is specifically configured to: obtain shooting angle information, where the shooting angle information represents the shooting angle at which the image acquisition unit captured the eye image; determine a longitudinal weighting coefficient based on the shooting angle information; perform a weighted average of the longitudinal coordinates based on the longitudinal weighting coefficient to obtain the longitudinal center coordinates; and obtain the eye center coordinates based on the longitudinal center coordinates.

[0100] In one embodiment of the present disclosure, the eye image includes a left eye image and a right eye image, and the eye feature points include a first feature point corresponding to the left eye image and a second feature point corresponding to the left eye image. The detection module 33 is specifically configured to: obtain the coordinates of a first center point corresponding to the left eye region in the left eye image based on the positional characteristics of the first feature point; obtain the coordinates of a second center point corresponding to the right eye region in the left eye image based on the positional characteristics of the second feature point; and obtain deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates.

[0101] In one embodiment of the present disclosure, when the detection module 33 obtains the deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates, it is specifically configured to calculate a weighted sum based on the first center point coordinates and the second center point coordinates to obtain the weighted center point coordinates. Based on the positional relationship between the weighted center point coordinates and the reference coordinates, first deviation information is obtained, where the first deviation information represents the deviation distance of the current wearing posture of the terminal device from the standard wearing posture in the horizontal direction or the vertical direction.

[0102] In one embodiment of the present disclosure, when the detection module 33 obtains deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates, it is specifically configured to generate an inclination vector based on the first center point coordinates and the second center point coordinates. Based on the direction of the inclination vector, second deviation information is obtained, where the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture.

[0103] In one embodiment of the present disclosure, after obtaining the deviation information based on the position characteristics of at least two eye feature points, the detection module 33 is further used to: play the indication information generated based on the deviation information in the terminal device until the deviation information is less than the deviation threshold, and the indication information is used to represent the action of adjusting the terminal device, wherein the indication information includes at least one of the following: prompt text, logo, voice message.

[0104] The acquisition module 31, processing module 32 and detection module 33 are connected in sequence. The wearing deviation detection device 3 provided in this embodiment can implement the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0105] FIG15 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG15 , the electronic device 4 includes:

[0106] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0107] Memory 42 stores computer-executable instructions;

[0108] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the wearing deviation detection method in the embodiments shown in Figures 2 to 13.

[0109] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0110] The relevant explanations can be understood by referring to the relevant descriptions and effects corresponding to the steps in the embodiments corresponding to Figures 2 to 13, and no further details will be given here.

[0111] An embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the wearing deviation detection method provided in any of the embodiments corresponding to Figures 2 to 13 of the present disclosure.

[0112] The present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the wearing deviation detection method in the embodiments shown in FIG. 2 to FIG. 13 .

[0113] Referring to FIG16 , a schematic diagram of the structure of an electronic device 900 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 900 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG16 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0114] As shown in FIG16 , the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the electronic device 900 are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0115] Typically, the following devices can be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 can allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although FIG16 shows an electronic device 900 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0116] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0117] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0118] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0119] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0120] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0123] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0124] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] In a first aspect, according to one or more embodiments of the present disclosure, a wearing deviation detection method is provided, comprising:

[0126] Capture an eye image; obtain at least two eye feature points based on the eye image, wherein the eye feature points are distributed on the contour of the eye area in the eye image; obtain deviation information based on the position characteristics of the at least two eye feature points, wherein the deviation information represents the deviation type and / or deviation amount of the current wearing posture of the terminal device relative to the standard wearing posture.

[0127] According to one or more embodiments of the present disclosure, obtaining at least two eye feature points based on the eye image includes: performing feature extraction on the eye image to obtain a first feature map, wherein the first feature map represents pixel features of the eye image; processing the first feature map based on a pre-trained prediction model to determine a contour line of the eye area in the eye image; and obtaining at least two eye feature points based on the contour line of the eye area.

[0128] According to one or more embodiments of the present disclosure, at least two eye feature points are obtained based on the contour line of the eye area, including: obtaining at least four position extreme points of the contour line of the eye area according to the tangent direction of each contour node in the contour line; and obtaining at least four eye feature points according to the at least four position extreme points.

[0129] According to one or more embodiments of the present disclosure, there is a first limiting distance between any two adjacent eye feature points, and the first limiting distance is determined based on the resolution of the eye image.

[0130] According to one or more embodiments of the present disclosure, the position features include the position coordinates of the eye feature points in the eye image; the deviation information is obtained based on the position features of at least two of the eye feature points, including: obtaining the eye center point coordinates based on the position coordinates of at least two of the eye feature points; obtaining a deviation vector based on the eye center point coordinates and reference coordinates, the reference coordinates being the standard eye center point coordinates corresponding to the standard wearing posture; and obtaining the deviation information based on the deviation vector.

[0131] According to one or more embodiments of the present disclosure, the position coordinates include longitudinal coordinates; obtaining eye center point coordinates based on the position coordinates of the at least two eye feature points includes: acquiring shooting angle information, the shooting angle information representing the shooting angle when the image acquisition unit captures the eye image; determining a longitudinal weighting coefficient based on the shooting angle information; performing weighted averaging on the longitudinal coordinates based on the longitudinal weighting coefficient to obtain longitudinal center point coordinates; and obtaining the eye center point coordinates based on the longitudinal center point coordinates.

[0132] According to one or more embodiments of the present disclosure, the eye image includes a left-eye image and a right-eye image, and the eye feature points include a first feature point corresponding to the left-eye image and a second feature point corresponding to the right-eye image; the deviation information is obtained based on the position characteristics of at least two of the eye feature points, including: obtaining the first center point coordinates corresponding to the left-eye area in the left-eye image based on the position characteristics of the first feature point; obtaining the second center point coordinates corresponding to the right-eye area in the left-eye image based on the position characteristics of the second feature point; and obtaining the deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates.

[0133] According to one or more embodiments of the present disclosure, the deviation information is obtained based on the positional relationship between the first center point coordinates and the second center point coordinates, including: calculating a weighted sum based on the first center point coordinates and the second center point coordinates to obtain weighted center point coordinates; and obtaining first deviation information based on the positional relationship between the weighted center point coordinates and the reference coordinates, the first deviation information representing the deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture in the horizontal direction or the vertical direction.

[0134] According to one or more embodiments of the present disclosure, the deviation information is obtained based on the positional relationship between the first center point coordinates and the second center point coordinates, including: generating an inclination vector based on the first center point coordinates and the second center point coordinates; and obtaining second deviation information based on the direction of the inclination vector, wherein the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture.

[0135] According to one or more embodiments of the present disclosure, after obtaining the deviation information based on the position characteristics of at least two of the eye feature points, it also includes: playing indication information generated based on the deviation information in the terminal device until the deviation information is less than the deviation threshold, the indication information is used to represent the action of adjusting the terminal device, wherein the indication information includes at least one of the following: prompt text, logo, voice message.

[0136] In a second aspect, according to one or more embodiments of the present disclosure, a wearing deviation detection device is provided, comprising:

[0137] An acquisition module, for acquiring eye images;

[0138] a processing module, configured to obtain at least two eye feature points according to the eye image, wherein the eye feature points are distributed on the contour of the eye region in the eye image;

[0139] The detection module is used to obtain deviation information based on the position characteristics of at least two of the eye feature points, where the deviation information represents the deviation direction and / or deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture.

[0140] According to one or more embodiments of the present disclosure, the processing module is specifically used to: perform feature extraction on the eye image to obtain a first feature map, where the first feature map represents pixel features of the eye image; based on a pre-trained prediction model, process the first feature map to determine the contour line of the eye area in the eye image; and obtain at least two eye feature points based on the contour line of the eye area.

[0141] According to one or more embodiments of the present disclosure, when the processing module obtains at least two eye feature points based on the contour line of the eye area, it is specifically used to: obtain at least four position extreme points of the contour line of the eye area according to the tangent direction of each contour node in the contour line; and obtain at least four eye feature points according to the at least four position extreme points.

[0142] According to one or more embodiments of the present disclosure, there is a first limiting distance between any two adjacent eye feature points, and the first limiting distance is determined based on the resolution of the eye image.

[0143] According to one or more embodiments of the present disclosure, the position features include the position coordinates of the eye feature points in the eye image; the detection module is specifically used to: obtain the eye center point coordinates based on the position coordinates of at least two of the eye feature points; obtain the deviation vector based on the eye center point coordinates and the reference coordinates, and the reference coordinates are the standard eye center point coordinates corresponding to the standard wearing posture; and obtain the deviation information based on the deviation vector.

[0144] According to one or more embodiments of the present disclosure, the position coordinates include longitudinal coordinates; when the detection module 33 obtains the eye center point coordinates based on the position coordinates of the at least two eye feature points, it is specifically used to: obtain shooting angle information, the shooting angle information represents the shooting angle when the image acquisition unit captures the eye image; determine the longitudinal weighting coefficient based on the shooting angle information; perform weighted averaging on the longitudinal coordinates according to the longitudinal weighting coefficient to obtain the longitudinal center point coordinates; and obtain the eye center point coordinates based on the longitudinal center point coordinates.

[0145] According to one or more embodiments of the present disclosure, the eye image includes a left-eye image and a right-eye image, and the eye feature points include a first feature point corresponding to the left-eye image and a second feature point corresponding to the left-eye image; the detection module is specifically used to: obtain the first center point coordinates corresponding to the left-eye area in the left-eye image according to the position characteristics of the first feature point; obtain the second center point coordinates corresponding to the right-eye area in the left-eye image according to the position characteristics of the second feature point; and obtain the deviation information according to the positional relationship between the first center point coordinates and the second center point coordinates.

[0146] According to one or more embodiments of the present disclosure, when the detection module obtains the deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates, it is specifically used to: calculate a weighted sum based on the first center point coordinates and the second center point coordinates to obtain weighted center point coordinates; obtain first deviation information based on the positional relationship between the weighted center point coordinates and the reference coordinates, the first deviation information representing the deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture in the horizontal direction or vertical direction.

[0147] According to one or more embodiments of the present disclosure, when the detection module obtains the deviation information based on the positional relationship between the first center point coordinates and the second center point coordinates, it is specifically used to: generate an inclination vector based on the first center point coordinates and the second center point coordinates; obtain second deviation information based on the direction of the inclination vector, and the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture.

[0148] According to one or more embodiments of the present disclosure, after obtaining the deviation information based on the positional features of at least two of the eye feature points, the detection module is further used to: play indication information generated based on the deviation information in the terminal device until the deviation information is less than a deviation threshold, the indication information being used to represent the action of adjusting the terminal device, wherein the indication information includes at least one of the following: prompt text, logo, and voice message.

[0149] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0150] The memory stores computer-executable instructions;

[0151] The processor executes the computer-executable instructions stored in the memory to implement the wearing deviation detection method as described in the first aspect and various possible designs of the first aspect.

[0152] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the wearing deviation detection method described in the first aspect and various possible designs of the first aspect is implemented.

[0153] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the wearing deviation detection method described in the first aspect and various possible designs of the first aspect.

[0154] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0155] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0156] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A wearing deviation detection method, comprising: Collecting eye images; According to the eye image, at least two eye feature points are obtained, where the eye feature points are distributed on the contour of the eye area in the eye image; Deviation information is obtained according to the positional features of at least two of the eye feature points, where the deviation information represents a deviation type and / or deviation amount of a current wearing posture of the terminal device relative to a standard wearing posture.

2. The method according to claim 1, wherein obtaining at least two eye feature points according to the eye image comprises: Performing feature extraction on the eye image to obtain a first feature map, where the first feature map represents pixel features of the eye image; Based on the pre-trained prediction model, the first feature map is processed to determine the contour line of the eye area in the eye image; At least two eye feature points are obtained based on the contour line of the eye region.

3. The method according to claim 2, wherein obtaining at least two eye feature points based on the contour line of the eye region comprises: Obtaining at least four position extreme value points of the contour line of the eye area according to the tangent direction of each contour node in the contour line; At least four eye feature points are obtained according to the at least four position extreme value points. 4 . The method according to claim 1 , wherein there is a first limiting distance between any two adjacent eye feature points, and the first limiting distance is determined based on the resolution of the eye image.

5. The method according to claim 1, wherein the position feature comprises the position coordinates of the eye feature point in the eye image; The obtaining of deviation information according to the position features of at least two eye feature points includes: Obtaining eye center point coordinates according to the position coordinates of at least two eye feature points; Obtaining a deviation vector according to the eye center point coordinates and the reference coordinates, wherein the reference coordinates are the standard eye center point coordinates corresponding to the standard wearing posture; The deviation information is obtained according to the deviation vector.

6. The method according to claim 5, wherein the position coordinates include longitudinal coordinates; obtaining eye center coordinates according to the position coordinates of the at least two eye feature points comprises: Acquire shooting angle information, where the shooting angle information represents a shooting angle when an image acquisition unit captures the eye image; Determining a longitudinal weighting coefficient according to the shooting angle information; Performing weighted averaging on the longitudinal coordinates according to the longitudinal weighting coefficient to obtain the longitudinal center point coordinates; The eye center point coordinates are obtained according to the longitudinal center point coordinates.

7. The method according to claim 1, wherein the eye image comprises a left eye image and a right eye image, and the eye feature points comprise a first feature point corresponding to the left eye image and a second feature point corresponding to the right eye image; The obtaining of deviation information according to the position features of at least two eye feature points includes: Obtaining, according to the positional features of the first feature points, the coordinates of a first center point corresponding to a left-eye region in the left-eye image; Obtaining the coordinates of a second center point corresponding to the right eye area in the left eye image according to the positional features of the second feature point; The deviation information is obtained according to the positional relationship between the first center point coordinates and the second center point coordinates.

8. The method according to claim 7, wherein obtaining the deviation information according to the positional relationship between the first center point coordinates and the second center point coordinates comprises: Calculating a weighted sum based on the first center point coordinates and the second center point coordinates to obtain weighted center point coordinates; According to the positional relationship between the weighted center point coordinates and the reference coordinates, first deviation information is obtained, where the first deviation information represents the deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture in the horizontal direction or the vertical direction.

9. The method according to claim 7, wherein obtaining the deviation information according to the positional relationship between the first center point coordinates and the second center point coordinates comprises: Generate an inclination vector according to the first center point coordinates and the second center point coordinates; According to the direction of the inclination vector, second deviation information is obtained, where the second deviation information represents the inclination angle of the current wearing posture of the terminal device relative to the standard wearing posture.

10. The method according to claim 1, wherein after obtaining the deviation information according to the position features of at least two of the eye feature points, the method further comprises: Playing indication information generated based on the deviation information in the terminal device until the deviation information is less than a deviation threshold, the indication information being used to represent an action of adjusting the terminal device, wherein the indication information includes at least one of the following: Prompt text, logo, voice message.

11. A wearing deviation detection device, comprising: An acquisition module, used for acquiring eye images; A processing module, configured to obtain at least two eye feature points according to the eye image, wherein the eye feature points are distributed on the contour of the eye region in the eye image; The detection module is used to obtain deviation information based on the position characteristics of at least two of the eye feature points, wherein the deviation information represents the deviation direction and / or deviation distance of the current wearing posture of the terminal device relative to the standard wearing posture.

12. An electronic device comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the wearing deviation detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the wearing deviation detection method according to any one of claims 1 to 10 is implemented.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the wearing deviation detection method according to any one of claims 1 to 10 is implemented.

Images