Intelligent control method and system for AR equipment

Abstract

The invention discloses an intelligent control method and system for AR equipment, and the method specifically comprises the steps: determining a pupil center position coordinate and a cornea reflection point position coordinate of a user according to an eye image of the user, and inputting the pupil center position coordinate and the cornea reflection point position coordinate into a geometric mapping model for matching, obtaining user sight direction information; acquiring image brightness value distribution of an area near a user fixation point, determining a plurality of candidate saliency areas according to the image brightness value distribution, and comparing the candidate saliency areas with a preset saliency area template to determine a user attention focus; and acquiring head posture information of the user in real time by adopting a head posture tracking algorithm, and dynamically controlling the shooting direction of a camera of the AR equipment according to the head posture information, the sight direction information of the user and the attention focus of the user. According to the method and the device, accurate alignment of the AR content and the visual focus of the user is realized, and the immersion and interaction experience of the AR application are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of AR devices, and in particular to an intelligent control method and system for AR devices. Background Art

[0002] With the rapid development of augmented reality (AR) technology, accurate matching between the user's line of sight and the camera of the AR device is crucial to improving the user experience. However, in practical applications, this technical problem has never been perfectly solved. When users wear AR devices, due to individual differences in human facial structure, such as pupil distance, eyeball size, and nose bridge height, there is a significant deviation between the image captured by the device camera and the user's actual line of sight. This deviation is enough to affect the user experience in a static environment, and in a dynamic scene, the user's head movement and eye movement exacerbate this problem.

[0003] Traditional AR devices usually rely on fixed camera positions and directions to capture images and present AR content. However, this fixed method cannot track and compensate for the deviation between the user's line of sight and the camera in real time. As the user's head and eyeballs move, the matching accuracy between the AR content and the user's line of sight gradually decreases, resulting in inaccurate overlap between the user's visual focus and content when observing AR content. This not only affects the user's perception and understanding of AR content, but may also cause discomfort symptoms such as dizziness and headaches, greatly reducing the user's comfort and the usability of the device.

[0004] In addition, the complexity of the use environment also poses a higher challenge to the accuracy of sight matching. Under different lighting conditions, the quality and contrast of the camera image will be affected, thereby affecting the accuracy of sight matching. For example, in a low-light environment, the camera may not be able to capture a clear image, causing the sight matching algorithm to fail; and in the case of direct strong light, the reduction in image contrast will also increase the difficulty of sight matching. In complex environments such as outdoors, the rapid changes in the background and the appearance of interference pose a severe test to the stability and robustness of the sight matching algorithm.

[0005] In order to solve the above problems, existing AR devices and technologies try to use various methods to improve the accuracy of line of sight matching. However, these methods often have limitations and cannot fully adapt to the needs of different users and usage environments. Therefore, how to achieve accurate matching between the user's line of sight and the AR device camera, and improve the presentation accuracy of AR content and user experience, is still a key issue that needs to be solved in the current development of AR technology. Summary of the invention

[0006] The purpose of the present invention is to provide an intelligent control method and system for an AR device, which achieves precise alignment of AR content with the user's visual focus, greatly improves the immersiveness and interactive experience of AR applications, and solves at least one of the above-mentioned problems of the prior art.

[0007] In a first aspect, the present invention provides an intelligent control method for an AR device, the method specifically comprising: Obtaining facial biometric data of at least one user and camera data of an AR device, and establishing a geometric mapping model between a user's line of sight and a device camera according to the facial biometric data and the camera data; Acquire at least one user eye image, determine the user's pupil center position coordinates and corneal reflection point position coordinates according to the user eye image, input the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtain user line of sight direction information, wherein the user line of sight direction information includes the user line of sight movement trajectory and the user gaze point; Obtaining a distribution of image brightness values ​​in an area near the user's gaze point, determining a plurality of candidate salient regions according to the image brightness value distribution, and determining the user's attention focus by comparing the candidate salient regions with a preset salient region template; A head posture tracking algorithm is used to obtain the user's head posture information in real time, and the camera shooting direction of the AR device is dynamically controlled according to the head posture information, the user's line of sight direction information and the user's attention focus.

[0008] In a second aspect, the present invention provides an intelligent control system for an AR device, the system specifically comprising: A first control module, configured to obtain facial biometric data of at least one user and camera data of an AR device, and to establish a geometric mapping model between a user's line of sight and a device camera according to the facial biometric data and the camera data; a second control module, configured to obtain at least one user eye image, determine the user's pupil center position coordinates and corneal reflection point position coordinates according to the user eye image, input the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtain user line of sight direction information, wherein the user line of sight direction information includes the user line of sight movement trajectory and the user gaze point; a third control module, configured to obtain a distribution of image brightness values ​​in an area near the user's gaze point, determine a plurality of candidate salient regions according to the image brightness value distribution, and determine the user's attention focus by comparing the candidate salient regions with a preset salient region template; The fourth control module is used to adopt a head posture tracking algorithm to obtain the user's head posture information in real time, and dynamically control the camera shooting direction of the AR device according to the head posture information, the user's line of sight direction information and the user's attention focus.

[0009] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor and a computer program stored in the memory, and when the computer program is executed on the processor, an intelligent control method for an AR device as described in any one of the above methods is implemented.

[0010] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the intelligent control method of an AR device as described in any one of the above methods is implemented.

[0011] Compared with the prior art, the present invention has at least one of the following technical effects: 1. The present invention realizes the precise alignment of AR content with the user's visual focus, greatly improving the immersiveness and interactive experience of AR applications.

[0012] 2. The present invention not only improves the matching accuracy between AR content and user's line of sight, but also enhances the user's immersion and interactive experience when using AR devices. It can effectively solve the problems existing in traditional AR devices in line of sight matching and promote the further development of AR technology.

[0013] 3. By acquiring the user's facial biometric data and the camera data of the AR device, the present invention can establish an accurate geometric mapping model between the user's line of sight and the device camera, thereby achieving real-time and accurate matching between the user's line of sight and the AR content, and improving the user's immersion and interactive experience.

[0014] 4. The present invention can determine candidate salient areas by analyzing the distribution of image brightness values ​​in the area near the user's gaze point, and compare them with preset salient area templates to determine the user's focus of attention, which helps the AR device to present content of interest to the user more intelligently.

[0015] 5. The present invention obtains the user's head posture information in real time, and combined with the user's line of sight direction information and attention focus, can dynamically control the camera shooting direction of the AR device, further enhancing the presentation effect of the AR content and improving the user experience.

[0016] 6. The present invention can construct a high-precision geometric mapping model between the user's line of sight and the device camera by establishing the mapping relationship data between the line of sight direction and the camera imaging plane coordinates, and adopting the support vector machine algorithm for modeling training, thereby improving the accuracy and stability of line of sight tracking.

[0017] 7. The present invention performs grayscale processing and histogram equalization on the user's eye image and adopts the Hough circle detection algorithm to quickly and accurately determine the user's pupil center position coordinates and corneal reflection point position coordinates, thereby obtaining a clearer and more standardized user eye image, which helps to improve the determination accuracy of the pupil center position coordinates and corneal reflection point position coordinates.

[0018] 8. The present invention obtains the ambient light intensity data near the AR device and the user's pupil size change data in real time, and performs correlation analysis, so as to construct a pupil size change prediction model, which helps the AR device to adjust the retinal imaging data in real time according to environmental changes and user pupil changes, thereby improving imaging quality and user experience.

[0019] 9. The present invention uses a Gaussian mixture model and a K-means algorithm to screen and cluster gaze areas, and can construct a gaze category tree model with a hierarchical relationship, which helps to more accurately predict the user's future attention transfer trend.

[0020] 10. The present invention calculates the probability of user attention transfer between different gaze categories based on the gaze category tree model and the hidden Markov model, and can generate an attention prediction list, which provides strong support for the intelligent rendering and presentation of AR content and helps to improve the user's interactive experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of an intelligent control method of an AR device provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of an intelligent control system of an AR device provided by one embodiment of the present invention; Figure 3 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0027] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A schematic flow chart of an intelligent control method for an AR device disclosed in an embodiment of the present invention is shown, and the details are as follows: S101, acquiring facial biometric data of at least one user and camera data of an AR device, and establishing a geometric mapping model between a user's line of sight and a device camera according to the facial biometric data and the camera data.

[0030] In this embodiment, a high-resolution facial scanning device, such as a 3D facial scanner, is used to scan the faces of several users. These devices can capture detailed geometric information of the face, including the three-dimensional coordinates of key feature points such as the facial contour, eyes, nose, and mouth. In addition, deep learning algorithms can be used to process facial images to extract facial feature descriptors, such as Histogram of Oriented Gradients (HOG), Local Binary Patterns (LBP), etc.

[0031] Use the camera on the AR device (such as AR glasses or AR helmet) to capture the user's real-time video stream. These cameras have high frame rate and high resolution, and can capture the user's facial details and slight eye movements. By processing and analyzing the video stream, the user's eye features can be extracted, such as the position and size of the pupil and the movement trajectory of the eyeball.

[0032] Preprocess the acquired facial biometric data and camera data, including denoising, image enhancement, feature extraction and other steps. Use machine learning or deep learning algorithms, such as convolutional neural networks (CNNs), to train a geometric mapping model. The model takes the user's facial biometric data and camera data as input and outputs the geometric mapping relationship between the user's line of sight and the device camera. During the training process, a large amount of annotated data can be used to optimize the parameters of the model and improve its generalization ability. The model is optimized through methods such as cross-validation and regularization to reduce the risk of overfitting and underfitting. At the same time, data enhancement techniques such as rotation, scaling, and translation can also be used to increase the diversity of the data and further improve the accuracy of the model.

[0033] The trained geometric mapping model can be used to track the user's line of sight in real time. When the user wears an AR device, the camera captures the user's facial image and calculates the user's line of sight through the model. This information can be used for interactive control in AR applications, such as selecting menu items and moving virtual objects. The geometric mapping model can provide users with a more personalized AR experience. For example, according to the user's line of sight, the display position, size, angle and other parameters of the virtual object can be adjusted to make it more in line with the user's visual habits and needs.

[0034] In this embodiment, by combining facial biometric data and camera data, the user's line of sight can be tracked more accurately, reducing errors and interference. Using the geometric mapping model, a more natural and smooth AR interactive control can be achieved, improving the user's immersion and participation.

[0035] S102, obtaining at least one user eye image, determining the user's pupil center position coordinates and corneal reflection point position coordinates based on the user eye image, inputting the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtaining the user's line of sight direction information, wherein the user's line of sight direction information includes the user's line of sight movement trajectory and the user's gaze point.

[0036] In this embodiment, a high-precision camera (such as an infrared camera or a high-resolution visible light camera) is used to capture at least one user's eye image. The camera should be placed in a position where it can clearly capture the details of the user's eyes, such as a built-in camera on AR glasses. The acquired eye image is preprocessed, including steps such as denoising, contrast enhancement, and grayscale conversion to improve the accuracy of subsequent processing. In addition, image processing techniques (such as edge detection, morphological processing, etc.) are used to extract the preliminary outlines of the pupil and corneal reflection point.

[0037] The pupil area in the preprocessed image is accurately fitted by using ellipse fitting algorithm or Hough transform, so as to determine the position coordinates of the pupil center. The reflection points on the cornea generated by the light source (such as infrared light source) are used to locate the position coordinates of the corneal reflection points through image processing technology (such as template matching, feature point detection, etc.). The corneal reflection point is usually used to assist in determining the direction of sight, because its relative position relationship with the pupil center can reflect the tilt angle of sight.

[0038] A geometric mapping model is trained in advance, which can calculate the user's line of sight direction based on the pupil center position coordinates and corneal reflection point position coordinates. The model can be constructed using machine learning algorithms (such as support vector machines, neural networks, etc.) or geometric optics principles. The pupil center position coordinates and corneal reflection point position coordinates acquired in real time are input into the geometric mapping model, and the model calculates the user's line of sight direction information based on these input data, including the line of sight movement trajectory (i.e., the line of sight changes over time) and the gaze point (i.e., the spatial position to which the line of sight points at a certain moment).

[0039] In this embodiment, by accurately locating the pupil center and the corneal reflection point and combining the geometric mapping model, the accuracy and stability of line of sight tracking can be significantly improved.

[0040] S103, obtaining the image brightness value distribution of the area near the user's gaze point, determining a number of candidate salient areas according to the image brightness value distribution, and determining the user's attention focus by comparing the candidate salient areas with a preset salient area template.

[0041] In this embodiment, first, ensure that the user is in an environment with stable light, and use a high-precision camera (such as a camera on AR glasses) to capture images of the area near the user's gaze point. The camera should have high resolution and fast response capabilities to ensure that the captured images are clear and real-time. The captured images are preprocessed, including steps such as denoising and contrast enhancement to improve the accuracy of subsequent processing. Then, the image is converted into a grayscale image for brightness value analysis. Using image processing software or algorithms, the brightness value of each pixel in the grayscale image is calculated, and a brightness value distribution map is generated. This figure can intuitively show the brightness changes in the area near the user's gaze point.

[0042] According to the brightness value distribution map, one or more brightness thresholds are set. These thresholds are used to distinguish between salient areas and non-salient areas. Generally, areas with higher brightness values ​​are more likely to attract the user's attention. Areas with brightness values ​​exceeding the set thresholds are extracted as candidate salient areas. These areas contain important information or objects that the user is currently paying attention to.

[0043] According to the application scenarios and user needs, a series of salient area templates are designed and stored in advance. These templates can be areas of specific shapes, colors or textures, which are used to compare with candidate salient areas near the user's gaze point. The extracted candidate salient areas are compared with the preset salient area templates to find the best matching template. The comparison process can be implemented using algorithms such as similarity calculation and feature matching. Based on the comparison results, the salient area template that best matches the area near the user's gaze point is determined and used as the user's current focus of attention. The focus area contains the key information or object that the user is paying attention to.

[0044] In this embodiment, by accurately calculating the image brightness value distribution and extracting candidate salient areas, and then comparing them with the preset template, the user's current focus of attention can be more accurately identified. In applications such as AR, adjusting the interaction method or display content according to the changes in the user's focus of attention can provide users with a more personalized and smooth experience.

[0045] S104, using a head posture tracking algorithm to obtain the user's head posture information in real time, and dynamically controlling the camera shooting direction of the AR device according to the head posture information, the user's line of sight direction information and the user's attention focus.

[0046] In this embodiment, the AR device includes: a head posture tracking module, which uses technologies such as an optical barcode tracking system, a sensor array, an infrared camera or a laser scanner to obtain the user's head posture information in real time, including the rotation angle and tilt degree of the head; a line of sight direction detection module, which captures the user's eye image, determines the pupil center position coordinates and the corneal reflection point position coordinates, and inputs them into a geometric mapping model to obtain the user's line of sight direction information in real time; an attention focus recognition module, which determines the candidate salient area based on the image brightness value distribution of the area near the user's gaze point, and compares it with the preset salient area template to determine the user's attention focus.

[0047] When a user wears an AR device, the head posture tracking module captures the user's head movement in real time and converts it into digital signals. After processing, these signals can obtain the precise posture information of the user's head. At the same time, the sight direction detection module captures the user's eye image and calculates the user's sight direction in real time through image processing algorithms and geometric mapping models. The attention focus identification module determines the user's attention focus based on the image brightness value distribution of the area near the user's gaze point. This is usually achieved by comparing the similarity of the candidate salient area with the preset template. The control algorithm built into the AR device dynamically adjusts the shooting direction of the AR device camera based on the head posture information, sight direction information, and attention focus. For example, if the user's head is tilted to the right, the sight direction points to the right, and the attention focus is also on the right, the camera will adjust the shooting direction to the right accordingly to ensure that the user can clearly see the content they are paying attention to.

[0048] In this embodiment, the camera shooting direction is dynamically adjusted to ensure that users can clearly see the content they are concerned about, thereby enhancing the interactive experience and immersion of the AR device. Combining head posture tracking and line of sight direction detection, users can control the camera shooting direction through natural head movement and line of sight shifting, enhancing the naturalness and intuitiveness of the interaction.

[0049] In some embodiments, in the above step S101, establishing a geometric mapping model between the user's line of sight and the device camera according to the facial biometric data and the camera data specifically includes: Establishing mapping relationship data between sight direction and camera imaging plane coordinates according to the spatial position relationship between the facial biometric data and the camera data; The mapping relationship data is used as input, and a support vector machine algorithm is used for modeling training to construct a geometric mapping model between the user's line of sight and the device camera.

[0050] In this embodiment, facial biometric data, including facial key point coordinates, facial posture angles and other data, are obtained; coordinate system parameters of the camera imaging plane are obtained to establish a camera coordinate system; the three-dimensional coordinates of the facial feature points relative to the camera are calculated based on the spatial position of the facial biometric data in the camera coordinate system; the vector representation of the line of sight direction in the camera coordinate system is calculated based on the facial posture angle and the three-dimensional coordinates of the feature points; the line of sight direction vector is intersected with the camera imaging plane to obtain the two-dimensional coordinates of the line of sight on the imaging plane; mathematical methods such as the least squares method are used to fit the mapping relationship between the line of sight direction vector and the imaging plane coordinates to obtain mapping relationship data; the mapping relationship data is used as a training sample and input into a support vector machine algorithm for modeling training to obtain a geometric mapping model between the user's line of sight and the camera.

[0051] Exemplarily, the spatial position relationship between facial biometric data and camera data is the basis for building a user gaze tracking system. This relationship can be represented by the two-dimensional coordinates of facial feature points (such as eye corners, nose tip, etc.) on the camera imaging plane. For example, assuming that the coordinates of the outer corner of the left eye are detected to be (100,150) pixels, and the coordinates of the outer corner of the right eye are (300,150) pixels, these data constitute the spatial position information of the facial features. Establishing a mapping relationship between the gaze direction and the coordinates of the camera imaging plane is the next key step. This mapping relationship can be modeled using methods such as polynomial functions or neural networks. For example, a quadratic polynomial function can be used to represent the relationship between the coordinates (x, y) of the pupil center point on the camera imaging plane and the gaze direction angle (θ, φ): x=a 1 θ²+b 1 θ+c 1 φ²+d 1 φ+e 1 , y = a 2 θ²+b 2 θ+c 2 φ²+d 2 φ+e 2 . Among them a 1 , b 1 , c 1 , d 1 , e 1 , a 2 , b 2 , c 2 , d 2 , e 2The support vector machine (SVM) algorithm is a powerful machine learning method that is suitable for building a geometric mapping model between the user's gaze and the device camera. SVM can process high-dimensional feature spaces and achieve nonlinear mapping through kernel function techniques. In this case, the radial basis function (RBF) kernel can be used, which can capture complex nonlinear relationships. The input features can include the coordinates of facial feature points, head posture angles, etc., and the output is the angle of the gaze direction or the coordinates of the gaze point on the screen. During the training process, a large amount of calibration data needs to be collected. For example, let the user look at different positions on the screen (such as a grid of 9 points), and record the facial feature data and the actual gaze position at the same time. Assume that 1000 sets of sample data are collected, each set contains the coordinates of 10 facial feature points and the corresponding gaze direction angle. These data are used to train the SVM model, and the model performance is optimized by adjusting the kernel function parameters (such as the γ value of the RBF kernel) and the regularization parameter C. The constructed geometric mapping model can achieve real-time gaze tracking. When a new facial image is input, the model can quickly predict the user's gaze direction or the gaze point on the screen. This machine learning-based gaze tracking method has significant advantages over traditional geometric methods. It can adapt to the differences in facial features of different users and is more robust to changes in lighting and slight head movements. Through continuous learning and model updates, the system can continuously improve its accuracy and adaptability, providing users with a more natural and intuitive interactive experience.

[0052] In some embodiments, in the above step S102, determining the pupil center position coordinates and corneal reflection point position coordinates of the user according to the user eye image, inputting the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtaining the user's line of sight direction information specifically includes: Performing grayscale processing and histogram equalization on the user eye image to obtain a standard user eye image; The standard user eye image is analyzed using a Hough circle detection algorithm to determine the coordinates of the user's pupil center and the corneal reflection point. Calculate the offset between the user's pupil center position coordinates and the corneal reflection point position coordinates to obtain an offset vector; The offset vector is matched with the geometric mapping model to obtain user sight direction information.

[0053] In this embodiment, the user's eye image is obtained, the image is grayed, and the color image is converted into a gray image to reduce the interference of color information in the image. The grayed eye image is subjected to histogram equalization processing to enhance the contrast of the image, make the details in the image clearer, and facilitate subsequent feature extraction and analysis. The pre-processed standardized eye image is input into the Hough circle detection algorithm, and the position of the pupil is determined by detecting the circular features in the image. According to the pupil position obtained by the Hough circle detection algorithm, a coordinate system is established with the center of the pupil as the origin to determine the coordinates of the center of the pupil. In the standardized eye image, the features of the corneal reflection point are extracted, and the position of the corneal reflection point is determined by analyzing the highlight reflection points on the corneal surface. The position coordinates of the corneal reflection point are converted into a coordinate system with the center of the pupil as the origin to obtain the coordinates of the corneal reflection point relative to the center of the pupil.

[0054] The offset between the pupil center coordinates and the corneal reflection point coordinates is calculated to obtain an offset vector, which includes two components: horizontal offset and vertical offset. A machine learning algorithm is used to establish a geometric mapping model, and the corresponding relationship between eye feature parameters and line of sight direction is modeled to obtain a mapping function. Eye feature parameters include pupil center coordinates, corneal reflection point coordinates, offset vector, etc. The calculated offset vector is input into the established geometric mapping model, and the corresponding user line of sight direction information is obtained through the mapping function calculation.

[0055] Exemplarily, first, the user's eye image is grayed to convert the color image into a gray image, which can reduce the amount of calculation and highlight the eye structure features. Histogram equalization further enhances the image contrast and makes the pupil and iris boundaries clearer. Taking an eye image with a resolution of 640x480 pixels as an example, the brightness value of each pixel after graying ranges from 0 to 255. Histogram equalization may stretch the brightness value of dark areas (such as pupils) from the original 10-30 to 5-50, and the bright areas (such as sclera) from 200-220 to 180-250, thereby increasing the contrast between the pupil and the surrounding area. Next, the Hough circle detection algorithm is used to locate the pupil, which detects circular structures by accumulating votes in the parameter space. Assume that the detected pupil radius is 20 pixels and the center coordinates are (320,240). At the same time, the corneal reflection point usually appears as a small area of ​​high brightness, which may be located at (325,235). Calculating the offset vector between the center of the pupil and the corneal reflection point is a key step. In this example, the offset vector is (5,-5) pixels. This vector contains rich information about the direction of the gaze. For example, when the user looks to the right, the corneal reflection point moves to the left relative to the center of the pupil, resulting in an offset vector like (-8,0). Finally, this offset vector is input into a pre-trained geometric mapping model. The model may be a polynomial function or a neural network that learns the complex nonlinear relationship between the offset vector and the actual gaze direction. For example, an offset of (5,-5) may correspond to a direction in which the user's gaze is slightly offset to the upper right by about 10 degrees relative to the center of the screen. The advantage of this method lies in its robustness and adaptability. Through grayscale and histogram equalization, the system can adapt to different lighting conditions. The Hough circle detection algorithm can handle partial occlusion, such as drooping eyelids or eyelash interference. The machine learning-based geometric mapping model can adapt to the differences in eye structure of different users and provide a personalized gaze tracking experience. In practical applications, this technology can achieve gaze tracking accuracy of 1-2 degrees.

[0056] In some embodiments, in the above step S102, the step of obtaining the image brightness value distribution of the area near the user's gaze point and determining a plurality of candidate salient areas according to the image brightness value distribution specifically includes: Acquire a first image where the user's gaze point is located, and perform Gaussian filtering and denoising processing on the first image to obtain a second image; The second image is processed into blocks, and a sliding window of a fixed size is used to slide on the second image with a certain step length to obtain a plurality of image blocks; For each image block, calculate the arithmetic mean of the brightness values ​​of all pixels in the image block to obtain the average brightness value of the image block; According to the average brightness value of each image block, a Sobel operator is used to calculate the brightness gradient between two adjacent image blocks to obtain a gradient distribution map of the second image; Binarizing the gradient distribution map, setting pixel points whose gradient values ​​are greater than or equal to a preset gradient value threshold to 1, and setting pixel points whose gradient values ​​are less than the preset gradient value threshold to 0, to obtain a binary gradient distribution map; The connected domain analysis is performed on the binary gradient distribution map to extract the connected areas with pixel value 1, and the geometric center of each connected area is calculated to obtain the candidate salient areas.

[0057] In this embodiment, a first image including a user's gaze point is obtained; a Gaussian filter denoising process is performed on the first image to obtain a second image; a sliding window of a preset size is used to slide on the second image with a preset step length to obtain multiple image blocks; for each of the image blocks, the arithmetic mean of the brightness values ​​of all pixels in the image block is calculated to obtain the average brightness value of the image block; based on the average brightness values ​​of two adjacent image blocks, the brightness gradient between them is calculated using the Sobel operator to obtain the gradient distribution map of the second image. According to the preset gradient threshold, the gradient distribution map is binarized, and the pixel points with gradient values ​​greater than or equal to the threshold are set to 1, and the pixel points with gradient values ​​less than the threshold are set to 0, to obtain a binarized gradient map. A connected domain analysis algorithm is used to extract the connected domain of the binarized gradient map to obtain the connected areas in which all pixel values ​​in the image are 1. For each extracted connected area, the coordinates of the geometric center point of the connected area are calculated, and the coordinates of the geometric center point are used as the representative point of the connected area. According to the size, shape and other characteristics of the connected area, it is judged whether each connected area meets the conditions of the candidate salient area. If it does, the connected area is determined as the candidate salient area. All candidate salient regions are merged, and adjacent candidate regions that meet certain conditions are merged to obtain merged candidate salient regions. The saliency metric value of each merged candidate salient region is calculated, and the candidate regions are sorted according to the saliency metric value to obtain a sorted candidate region list. The first N regions with the largest saliency metric value in the sorted candidate region list are determined as the final salient regions, where N is the preset threshold value of the number of salient regions.

[0058] Exemplarily, the first image is first subjected to Gaussian filtering denoising to obtain a second image. Gaussian filtering can effectively remove high-frequency noise in an image while retaining important edge information. For example, for an eye image of 640x480 pixels, a 5x5 Gaussian kernel with a standard deviation of 1 may be used, which can smooth out small noise points but will not overly blur the edge of the pupil. Next, the second image is processed in blocks. The purpose of this step is to divide the image into smaller units for subsequent local feature analysis. Assuming that a 32x32 pixel sliding window is used and slid across the image with a step size of 16 pixels, approximately 600 overlapping image blocks can be obtained. The average brightness value is calculated for each image block as the brightness distribution feature of the block. This method can effectively capture local brightness changes, which is particularly important for identifying pupil and iris boundaries. Then, the Sobel operator is used to calculate the brightness gradient between adjacent image blocks to generate a gradient distribution map. The Sobel operator is a commonly used edge detection operator that can effectively identify brightness mutation areas in an image. In eye images, the boundaries between the pupil and iris, and between the iris and sclera usually appear as large brightness gradients. For example, at the edge of the pupil, the gradient value may suddenly increase from 10 to 100. The purpose of binarization of the gradient distribution map is to further highlight the salient area. Setting a suitable gradient value threshold, such as 50, can effectively separate potential regions of interest. Pixels above this threshold are set to 1, and those below the threshold are set to 0, forming a binary gradient distribution map. This step can effectively remove background noise and retain the contours of key structures such as the pupil and iris. Finally, the binary gradient distribution map is subjected to connected domain analysis to extract connected areas with a pixel value of 1. These areas are likely to correspond to important features such as the pupil, iris edge, or corneal reflection point. By calculating the geometric center of each connected area, the candidate salient area can be obtained. For example, if a nearly circular connected area is detected with an area of ​​about 1000 pixels and a geometric center near the center of the image, this is likely to be the pupil area. The advantage of this method lies in its robustness and adaptability. Through multi-step image processing and feature extraction, key eye structures can be accurately located under different lighting conditions and individual differences. For example, gradient analysis and connected domain analysis can effectively identify the pupil position even in low-contrast images.

[0059] In some embodiments, in the above step S103, determining the user's attention focus by comparing the candidate salient region with a preset salient region template specifically includes: Collecting massive image data, marking significant areas of the massive image data, and forming an image training data set; Taking the image training data set as input, a convolutional neural network model is used for modeling training to obtain a salient region detection model; Inputting the candidate salient regions into the salient region detection model for matching, and obtaining a salient probability value of each candidate salient region; The candidate salient region with the largest salient probability value is taken as the user's attention focus region, and the geometric center position coordinates of the user's attention focus region are calculated, and the geometric center position coordinates of the user's attention focus region are taken as the user's attention focus.

[0060] In this embodiment, a large amount of image data is obtained, and the image data is preprocessed, including operations such as image size normalization and image noise removal, to obtain preprocessed image data. For the preprocessed image data, the salient areas in the image are marked by manual annotation or semi-automatic annotation to obtain image data with salient area annotations. The image data with salient area annotations are divided into a training set and a test set, wherein the training set is used for model training and the test set is used for model evaluation. According to the characteristics of the salient area detection task, the architecture of the convolutional neural network model is designed, including a convolutional layer, a pooling layer, a fully connected layer, etc., and the hyperparameters of the model are determined. The training set is input into the convolutional neural network model, and the parameters of the model are continuously updated through the iterative process of forward propagation and back propagation until the loss function of the model on the training set converges or reaches a preset number of iterations. The trained convolutional neural network model is used to predict the test set to obtain the salient area detection results of the test set image, and the evaluation indicators of the model on the test set, such as precision and recall, are calculated. If the performance of the model on the test set meets the expected requirements, the trained convolutional neural network model is saved as a salient region detection model; otherwise, the model architecture or hyperparameters are adjusted, and retraining and testing are performed until a detection model that meets the requirements is obtained.

[0061] A set of candidate salient regions is obtained, and the set of candidate regions is input into a pre-trained salient region detection model; the model extracts and matches features of each candidate region, and calculates the salient probability value of each candidate region; according to the salient probability value, the candidate region with the largest probability value is selected, and it is determined as the user's attention focus area; for the determined user's attention focus area, a geometric calculation method is used to obtain the geometric center position coordinates of the area; it is determined whether the geometric center position coordinates are located in the central area of ​​the image, and if so, it is used as the user's attention focus, otherwise, proceed to the next step; according to the relative position relationship between the salient region and the image center, the geometric center position coordinates are adjusted by weighted averaging to obtain the corrected position coordinates; the corrected geometric center position coordinates are determined as the user's attention focus, and the focus position coordinates are output as input for subsequent business processing.

[0062] For example, first, a large amount of image data is collected and salient regions are marked. These images may include eye images in various scenarios, such as different lighting conditions, wearing glasses, etc. The marking process is usually completed by professionals, who will mark key areas such as pupils and iris edges in the image. For example, in a 640x480 pixel eye image, the pupil area may be marked as a circular area with a diameter of about 100 pixels. Next, the convolutional neural network model is trained using the labeled image training data set. For example, classic network architectures such as VGG16 or ResNet50 can be used as a basis, and then fine-tuned according to specific tasks. During the training process, the network learns how to extract key features from the input image and finally outputs a probability map of the salient area. After the training is completed, the candidate salient area is input into the model for matching. The model outputs a saliency probability value for each candidate area, which usually ranges from 0 to 1. For example, the real pupil area may get a high probability value of 0.95, while the background area may only have a low probability value of about 0.1. Finally, the area with the highest saliency probability value is selected as the user's focus area. This area is likely to correspond to the pupil or other important eye structures. By calculating the geometric center of the area, the precise coordinates of the user's focus of attention can be obtained. For example, if the detected high-probability area is a 100x100 pixel square with the coordinates of its upper left corner at (200,150), then the coordinates of the focus of attention are (250,200).

[0063] In some embodiments, in the above steps S101 to S104, the method further includes: Acquire the ambient light intensity data near the AR device and the user's pupil size change data in real time; Performing correlation analysis on the ambient light intensity data and the pupil size change data to form a first training data set; Using the first training data set as input, using a machine learning algorithm to perform modeling training to obtain a pupil size change prediction model; The user's pupil size change prediction data is determined according to the pupil size change prediction model, and the retinal imaging data of the AR device is updated in real time using the pupil size change prediction data.

[0064] In this embodiment, the real-time data of the illumination intensity of the environment around the AR device and the real-time change data of the user's pupil size are obtained; the obtained ambient illumination intensity data and pupil size change data are preprocessed, including data cleaning, data normalization and other operations; according to the preprocessed ambient illumination intensity data and pupil size change data, the degree of correlation between the two is determined by correlation analysis; if the correlation analysis result shows that the correlation between the two is higher than the preset threshold, the ambient illumination intensity data and pupil size change data are combined to form a training data set; a supervised learning algorithm such as a support vector machine or a neural network is used, and the training data set is used as input to obtain a pupil size change prediction model through algorithm training. The user's current pupil image data is obtained, the pupil size feature parameters are extracted, and the pupil size change prediction model is input for prediction processing to obtain the user's pupil size change prediction data in the future. According to the pupil size change prediction data, the AR device retinal imaging parameters that need to be adjusted are determined, including the focal length, resolution and other parameters of the retinal imaging, and the retinal imaging adjustment parameters are generated. The retinal imaging adjustment parameters are transmitted to the display module of the AR device and presented in the user's field of view in real time to complete the dynamic adjustment of the AR retinal imaging.

[0065] For example, the light intensity in an indoor office environment is about 500 lux, while it can reach 100,000 lux under direct sunlight. Changes in pupil size can be captured by a built-in eye tracking camera, recording real-time changes in pupil diameter. The pupil diameter of a normal person varies between 2 and 8 mm. Correlating these two sets of data to form a training data set can reveal the relationship between light intensity and pupil size. Generally, the stronger the light, the smaller the pupil to adjust the amount of light entering the eye. For example, in an indoor environment of 500 lux, the pupil diameter may be 5 mm; while in outdoor sunlight of 10,000 lux, the pupil may shrink to 3 mm. Using machine learning algorithms to model and train these data can obtain a more accurate pupil size change prediction model. Commonly used algorithms include support vector machines (SVMs), random forests, etc. The input features of the model may include current light intensity, light change speed, user age, etc., and the output is the predicted pupil diameter. For example, for a 30-year-old user, when the light increases rapidly from 500 lux to 5000 lux, the model may predict that the pupil diameter will shrink from 5 mm to 3.5 mm in 2 seconds. The advantage of this prediction model is that it can predict pupil changes in advance, so that the retinal imaging parameters of the AR device can be adjusted more quickly. Traditional methods need to wait for the pupil to actually change before making adjustments, which may cause short-term visual discomfort. The prediction model can start adjusting the image at the moment of light change, greatly improving the user experience. The retinal imaging data of the AR device is updated in real time based on the predicted pupil size change data, which mainly involves adjusting the brightness, contrast and sharpness of the image. For example, when it is predicted that the pupil is about to shrink, the system can reduce the image brightness and increase the contrast in advance to ensure that the image is still clearly visible under strong light. Conversely, when it is predicted that the pupil is about to enlarge, the system can increase the brightness and reduce the contrast to avoid the image being too glaring in the dark. This prediction-based real-time adjustment not only improves visual comfort, but also reduces eye fatigue, which is particularly important for users who use AR devices for a long time. At the same time, it also provides the basis for personalized visual experience. The system can provide image adjustment strategies that are more in line with personal needs based on the unique response pattern of each user's pupil to light.

[0066] In some embodiments, in the above steps S101 to S104, the method further includes: Acquire historical interaction behavior data of the user, and extract eye movement data of the user when browsing according to the historical interaction behavior data; A Gaussian mixture model is used to calculate the distribution point probability value of the eye movement data, and a set of gaze regions whose distribution point probability values ​​are higher than a preset distribution point probability threshold are screened out; Using the K-means algorithm to cluster all gaze areas in the gaze area set, setting the Euclidean distance as the centroid distance calculation standard, if the centroid distance between two gaze areas is lower than a preset centroid distance threshold, they are classified as gaze areas of the same gaze category, and constructing a gaze category tree model with a hierarchical relationship; Based on the gaze category tree model, a hidden Markov model is used to calculate the probability of user attention transfer between different gaze categories, and an attention prediction list is generated; The user's real-time interactive behavior data is obtained, and the current AR content of the AR device is visually rendered according to the attention prediction list and the real-time interactive behavior data.

[0067] In this embodiment, the user's historical interaction behavior data is obtained, and the eye movement data of the user when browsing is extracted based on the historical interaction behavior data. A convolutional neural network is used to extract features of the eye movement data to obtain a feature vector of the eye movement data. According to the feature vector of the eye movement data, a Gaussian mixture model is used to calculate the distribution point probability value of the eye movement data. According to a preset distribution point probability threshold, eye movement data points whose distribution point probability values ​​are higher than the preset threshold are screened out, and the screened eye movement data points are divided into gaze areas. The divided gaze areas are clustered to obtain multiple gaze area sets, each of which contains multiple gaze areas.

[0068] Obtain a set of gaze regions, and extract the centroid coordinates of each gaze region in the set. According to the centroid coordinates, use the Euclidean distance formula to calculate the centroid distance between any two gaze regions. Set a preset centroid distance threshold, and determine whether they belong to the same gaze category by comparing the centroid distance between gaze regions with the threshold. If the centroid distance is less than the preset threshold, the corresponding two gaze regions are classified into the same gaze category; if the centroid distance is greater than or equal to the preset threshold, the two gaze regions are classified into different gaze categories. In the clustering process, dynamically adjust the centroid coordinates of each gaze category, and use the mean of the centroid coordinates of all gaze regions in the same category as the new centroid coordinates of the category until the clustering results converge. According to the clustering results of the gaze regions, construct a gaze category tree model, and determine the structure and depth of the category tree by analyzing the hierarchical relationship and inclusion relationship between different gaze categories. In the gaze category tree model, each gaze category obtained by clustering is used as a tree node, and according to the hierarchical relationship between categories, a gaze category tree with a hierarchical structure is formed through the linking of parent and child nodes.

[0069] For the nodes in the gaze category tree model, the hidden Markov model is used to calculate the state transition probability matrix between adjacent nodes to obtain the probability of the user's attention shifting between different gaze categories. Based on the calculated state transition probability matrix, an attention prediction list is generated, which contains the gaze categories and their probabilities that the user may focus on in the future. The real-time user interaction behavior data collected by the sensor of the AR device is obtained, including the user's head movement data, gesture interaction data, etc. The acquired real-time user interaction behavior data is matched with the attention prediction list to determine the gaze category that the user is most likely to focus on at present, and determine the AR content that needs to be rendered in focus. According to the determined focus AR content, the rendering parameters of the AR device are dynamically adjusted, and the relevant AR content is visually enhanced and rendered to improve the rendering quality and frame rate. The user's real-time interaction behavior data changes are continuously tracked, and the rendering strategy of the AR content is dynamically adjusted in real time according to the update of the attention prediction list to keep the AR content synchronized with the user's attention.

[0070] Exemplarily, first obtain the user's historical interactive behavior data, which includes the user's browsing trajectory, dwell time, interactive operations, etc. in the AR environment. For example, in an AR museum tour application, the system can record the time the user views each exhibit, the path of sight movement, etc. The Gaussian mixture model is a commonly used probability density estimation method, which is suitable for analyzing the distribution characteristics of eye movement data. In the AR scene, it can help identify the area that the user is most interested in. Assume that in the AR tour, the eye movement data of a user shows that the probability of his gaze point distribution in the "Mona Lisa" painting area is the highest, exceeding the preset threshold of 0.8, and this area will be screened as an important gaze area. The K-means clustering algorithm can further classify these gaze areas. In the AR museum example, if the user frequently looks back and forth between the "Mona Lisa" and the nearby "Last Supper", the centroid distance between the two areas may be less than the preset threshold of 50 pixels, so they are classified into the same gaze category, forming the "Da Vinci Works" level. This hierarchical gaze category tree model helps to understand the user's interest structure. The hidden Markov model is used to predict the transfer of user attention. In the AR tour, it can calculate the probability that the user will switch from the "Da Vinci works" category to the "Impressionist works" category. For example, if this probability is 0.6, which is higher than other categories, the system can predict that the user may be interested in Impressionist works next. Finally, visual rendering is performed based on real-time interactive behavior and attention prediction lists. For example, when a user is watching "Mona Lisa" and the system predicts that he may be interested in Impressionist works, the AR device can slightly highlight nearby Monet works at the edge of the user's field of view to guide the user's attention to shift naturally. This intelligent rendering based on user behavior and interests can greatly improve the personalization and fluency of the AR experience, making the information presentation more in line with the user's cognitive habits and interest preferences. The comprehensive application of this series of technologies not only improves the presentation effect of AR content, but also helps content creators better understand user behavior, thereby optimizing the AR experience design. For example, by analyzing the gaze category tree model of a large number of users, a museum may find that there is a strong association between attention transfer between the works of Da Vinci and Monet, based on which the exhibit layout can be adjusted or a new thematic tour route can be designed. This data-driven approach opens up a broad space for innovation in AR applications, making it not only a display technology but also an intelligent platform that deeply understands and meets user needs.

[0071] Reference Figure 2 An embodiment of the present invention provides an intelligent control system 2 for an AR device, and the system 2 specifically includes: A first control module 201 is used to obtain facial biometric data of at least one user and camera data of an AR device, and establish a geometric mapping model between the user's line of sight and the device camera according to the facial biometric data and the camera data; The second control module 202 is used to obtain at least one user eye image, determine the user's pupil center position coordinates and corneal reflection point position coordinates according to the user eye image, input the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtain the user's line of sight direction information, wherein the user's line of sight direction information includes the user's line of sight movement trajectory and the user's gaze point; The third control module 203 is used to obtain the image brightness value distribution of the area near the user's gaze point, determine a number of candidate salient areas according to the image brightness value distribution, and determine the user's attention focus by comparing the candidate salient areas with a preset salient area template; The fourth control module 204 is used to use a head posture tracking algorithm to obtain the user's head posture information in real time, and dynamically control the camera shooting direction of the AR device according to the head posture information, the user's line of sight direction information and the user's attention focus.

[0072] It is understandable that if Figure 1 The contents of the embodiment of the intelligent control method of the AR device shown in the figure are applicable to the embodiment of the intelligent control system of the AR device. The functions specifically implemented by the embodiment of the intelligent control system of the AR device are similar to those of the embodiment of the intelligent control method of the AR device shown in the figure. Figure 1 The embodiment of the intelligent control method of the AR device shown in FIG. Figure 1 The beneficial effects achieved by the embodiment of the intelligent control method for AR equipment shown are also the same.

[0073] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0074] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0075] Reference Figure 3 The embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, an intelligent control method for an AR device as described in any one of the above methods is implemented.

[0076] The computer device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation on computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0077] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 3. Further, the memory 302 may also include both an internal storage unit and an external storage device of the computer device 3. The memory 302 is used to store an operating system, an application program, a boot loader, data, and other programs, such as the program code of the computer program. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0079] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the intelligent control method of an AR device as described in any one of the above methods is implemented.

[0080] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. An intelligent control method for an AR device, characterized in that: The method specifically comprises: Obtaining facial biometric data of at least one user and camera data of an AR device, and establishing a geometric mapping model between a user's line of sight and a device camera according to the facial biometric data and the camera data; Acquire at least one user eye image, determine the user's pupil center position coordinates and corneal reflection point position coordinates according to the user eye image, input the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtain user line of sight direction information, wherein the user line of sight direction information includes the user line of sight movement trajectory and the user gaze point; Obtaining a distribution of image brightness values ​​in an area near the user's gaze point, determining a plurality of candidate salient regions according to the image brightness value distribution, and determining the user's attention focus by comparing the candidate salient regions with a preset salient region template; A head posture tracking algorithm is used to obtain the user's head posture information in real time, and the camera shooting direction of the AR device is dynamically controlled according to the head posture information, the user's line of sight direction information and the user's attention focus.

2. The method according to claim 1, characterized in that The step of establishing a geometric mapping model between the user's sight line and the device camera according to the facial biometric data and the camera data specifically includes: Establishing mapping relationship data between sight direction and camera imaging plane coordinates according to the spatial position relationship between the facial biometric data and the camera data; The mapping relationship data is used as input, and a support vector machine algorithm is used for modeling training to construct a geometric mapping model between the user's line of sight and the device camera.

3. The method according to claim 1, characterized in that The determining of the pupil center position coordinates and the corneal reflection point position coordinates of the user according to the user eye image, inputting the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtaining the user line of sight direction information specifically includes: Performing grayscale processing and histogram equalization on the user eye image to obtain a standard user eye image; The standard user eye image is analyzed using a Hough circle detection algorithm to determine the coordinates of the user's pupil center and the corneal reflection point. Calculate the offset between the user's pupil center position coordinates and the corneal reflection point position coordinates to obtain an offset vector; The offset vector is matched with the geometric mapping model to obtain user sight direction information.

4. The method according to claim 1, characterized in that: The obtaining of the image brightness value distribution of the area near the user's gaze point and determining a plurality of candidate salient areas according to the image brightness value distribution specifically includes: Acquire a first image where the user's gaze point is located, and perform Gaussian filtering and denoising processing on the first image to obtain a second image; The second image is processed into blocks, and a sliding window of a fixed size is used to slide on the second image with a certain step length to obtain a plurality of image blocks; For each image block, calculate the arithmetic mean of the brightness values ​​of all pixels in the image block to obtain the average brightness value of the image block; According to the average brightness value of each image block, a Sobel operator is used to calculate the brightness gradient between two adjacent image blocks to obtain a gradient distribution map of the second image; Binarizing the gradient distribution map, setting pixel points whose gradient values ​​are greater than or equal to a preset gradient value threshold to 1, and setting pixel points whose gradient values ​​are less than the preset gradient value threshold to 0, to obtain a binary gradient distribution map; The connected domain analysis is performed on the binary gradient distribution map to extract the connected areas with pixel value 1, and the geometric center of each connected area is calculated to obtain the candidate salient areas.

5. The method according to claim 4, characterized in that The step of comparing the candidate salient region with a preset salient region template to determine the user's attention focus specifically includes: Collecting massive image data, marking significant areas of the massive image data, and forming an image training data set; Taking the image training data set as input, using a convolutional neural network model for modeling training to obtain a salient region detection model; Inputting the candidate salient regions into the salient region detection model for matching, and obtaining a salient probability value of each candidate salient region; The candidate salient region with the largest salient probability value is taken as the user's attention focus region, and the geometric center position coordinates of the user's attention focus region are calculated, and the geometric center position coordinates of the user's attention focus region are taken as the user's attention focus.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire the ambient light intensity data near the AR device and the user's pupil size change data in real time; Performing correlation analysis on the ambient light intensity data and the pupil size change data to form a first training data set; Using the first training data set as input, using a machine learning algorithm to perform modeling training to obtain a pupil size change prediction model; The user's pupil size change prediction data is determined according to the pupil size change prediction model, and the retinal imaging data of the AR device is updated in real time using the pupil size change prediction data.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire historical interaction behavior data of the user, and extract eye movement data of the user when browsing according to the historical interaction behavior data; A Gaussian mixture model is used to calculate the distribution point probability value of the eye movement data, and a set of gaze regions whose distribution point probability values ​​are higher than a preset distribution point probability threshold are screened out; Using the K-means algorithm to cluster all gaze areas in the gaze area set, setting the Euclidean distance as the centroid distance calculation standard, if the centroid distance between two gaze areas is lower than a preset centroid distance threshold, they are classified as gaze areas of the same gaze category, and constructing a gaze category tree model with a hierarchical relationship; Based on the gaze category tree model, a hidden Markov model is used to calculate the probability of user attention transfer between different gaze categories, and an attention prediction list is generated; The user's real-time interactive behavior data is obtained, and the current AR content of the AR device is visually rendered according to the attention prediction list and the real-time interactive behavior data.

8. An intelligent control system for an AR device, characterized in that: The system specifically comprises: A first control module, configured to obtain facial biometric data of at least one user and camera data of an AR device, and to establish a geometric mapping model between a user's line of sight and a device camera according to the facial biometric data and the camera data; a second control module, configured to obtain at least one user eye image, determine the user's pupil center position coordinates and corneal reflection point position coordinates according to the user eye image, input the pupil center position coordinates and the corneal reflection point position coordinates into the geometric mapping model for matching, and obtain user line of sight direction information, wherein the user line of sight direction information includes the user line of sight movement trajectory and the user gaze point; a third control module, configured to obtain a distribution of image brightness values ​​in an area near the user's gaze point, determine a plurality of candidate salient regions according to the image brightness value distribution, and determine the user's attention focus by comparing the candidate salient regions with a preset salient region template; The fourth control module is used to adopt a head posture tracking algorithm to obtain the user's head posture information in real time, and dynamically control the camera shooting direction of the AR device according to the head posture information, the user's line of sight direction information and the user's attention focus.

9. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory. When the computer program is executed on the processor, the intelligent control method of the AR device according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the intelligent control method of the AR device according to any one of claims 1 to 7 is implemented.

Images