AI glasses real-time cooperation system and method based on user intention prediction

Abstract

The invention discloses an AI glasses real-time cooperation system and method based on user intention prediction, relates to the technical field of education scene computer vision, and proposes the following scheme: obtaining a classroom panoramic image, and collecting a user head steering angle and an eyeball fixation coordinate sequence at the same time; the method comprises the following steps: segmenting a classroom panoramic image into different functional areas through semantic segmentation, distributing differentiated illumination weights according to optical characteristics of the functional areas, calculating an eyeball motion vector based on an eyeball fixation coordinate sequence, and superposing the eyeball motion vector and a head steering angle to generate a future continuous multi-frame prediction view area. And dividing all the functional areas into uniform grids. According to the method, the functional area is divided into the uniform grids, the theoretical illumination value of the predicted visual field area is weighted and calculated based on the illumination weight of the functional area, the brightness weight distribution difference of the light environment area is quantified in the spatial dimension, the nonlinear illumination sequence conforming to the actual optical gradient is generated, and distortion mapping of a linear model to the transition gradient of the functional area is eliminated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision technology in educational scenarios, and more particularly, the present application relates to an AI glasses real-time collaboration system based on user intention prediction and a method thereof. BACKGROUND

[0002] The current educational scenario has deeply integrated the multimedia teaching paradigm, and heterogeneous light sources such as blackboards, light-transmitting windows, and electronic teaching screens coexist in the classroom space, forming a dynamic light environment with a gradient difference in illumination distribution. Students usually complete an average of 25-30 cross-zone visual line switches within a class hour, and each switch requires the pupil to quickly adapt to different illumination environments. Frequent light-dark alternation in a short period of time easily leads to an increased visual accommodation burden, especially in fast switching scenarios, students often experience temporary visual blurring, eye acid swelling, and other discomforts, and long-term accumulation may also affect visual health. Therefore, it is urgent to develop an intelligent visual assistance technology that can dynamically adapt to dramatic changes in the light environment.

[0003] Existing AI glasses have basic environmental response capabilities, but usually use a single-point prediction linear adjustment mechanism to gradually adjust the transmittance based on the brightness of the field of view region between the start and end points, ignoring the discontinuous brightness mutation caused by rapid visual line switching, which leads to the accumulation of visual fatigue and the risk of retinal light damage due to the lagging adjustment of the pupil, thereby increasing the risk of myopia in students. Therefore, an AI glasses real-time collaboration system based on user intention prediction and a method thereof are proposed to solve this problem. SUMMARY

[0004] To solve the above technical problems, the AI glasses real-time collaboration system based on user intention prediction and the method thereof are provided, which solve the problems raised in the background technology.

[0005] To achieve the above purpose, the technical scheme of the present application is as follows:

[0006] In a first aspect, the present application provides an AI glasses real-time collaboration method based on user intention prediction, which comprises:

[0007] Obtaining a classroom panoramic image, and simultaneously collecting a user head turning angle and an eye gaze coordinate sequence;

[0008] Segmenting the classroom panoramic image into different functional areas through semantic segmentation, and assigning different illumination weights according to the optical properties of the functional areas;

[0009] Calculating an eye movement vector based on the eye gaze coordinate sequence, and generating a predicted field of view area for future continuous multiple frames by superimposing the eye movement vector and the head turning angle;

[0010] The all functional areas are divided into uniform grids, and for the grid set covered by each frame of the predicted visual field area, a weighted average value of the grid pixel brightness value and the illumination weight of the functional area is calculated as a theoretical illumination value of the frame of the predicted visual field area;

[0011] Based on the theoretical illumination values of the continuous multiple frames, corresponding light transmittances are independently generated to form a light transmittance sequence.

[0012] According to the increasing or decreasing direction and the change amplitude of the light transmittance values of the continuous multiple frames in the light transmittance sequence, a turning point is identified, and a gradual change instruction containing all the turning points is generated.

[0013] In a second aspect, the present application provides an AI glasses real-time collaboration system based on user intention prediction, which is used to implement the AI glasses real-time collaboration method based on user intention prediction described in any of the above aspects, and includes:

[0014] A multi-source data acquisition module is configured to acquire a classroom panoramic image, and simultaneously acquire a user head turning angle and an eye gaze coordinate sequence;

[0015] An illumination weight distribution module is configured to segment the classroom panoramic image into different functional areas through semantic segmentation, and distribute differentiated illumination weights according to the optical properties of the functional areas;

[0016] A future visual field prediction module is configured to calculate an eye movement vector based on the eye gaze coordinate sequence, and generate a predicted visual field area of future continuous multiple frames after superimposing the eye movement vector and the head turning angle;

[0017] A light transmittance sequence generation module is configured to divide all the functional areas into uniform grids, calculate a weighted average value of a grid pixel brightness value and an illumination weight of the functional area for a grid set covered by each frame of the predicted visual field area as a theoretical illumination value of the frame of the predicted visual field area, and independently generate corresponding light transmittances based on the theoretical illumination values of the continuous multiple frames to form a light transmittance sequence.

[0018] A gradual change instruction generation module is configured to identify a turning point according to the increasing or decreasing direction and the change amplitude of the light transmittance values of the continuous multiple frames in the light transmittance sequence, and generate a gradual change instruction containing all the turning points.

[0019] In a third aspect, the present application provides a computer device, which includes a memory and a processor, the memory stores a code, and the processor is configured to acquire the code and execute the feedback adjustment method based on somatosensory interaction described above.

[0020] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the feedback adjustment method based on somatosensory interaction described above.

[0021] Compared with the prior art, the application has the beneficial effects that:

[0022] The application generates a continuous multi-frame predicted visual field area by superimposing the eye movement speed and the head turning angle, extends the adjustment unit from the start and end points to the continuous frame sequence on the visual line track, realizes dynamic capture of non-continuous brightness mutation in the rapid switching process, and avoids adjustment lag caused by insufficient predicted visual field range;

[0023] The application divides the functional area into a uniform grid, calculates the theoretical illumination value of the predicted visual field area based on the illumination weight of the functional area, quantizes the brightness weight distribution difference of the space dimension of the light environment area, generates a nonlinear illumination sequence that fits the actual optical gradient, and eliminates the distortion mapping of the transition gradient of the functional area by the linear model;

[0024] The application generates gradual change instructions by dynamically identifying the numerical turning point of the light transmittance sequence, locates the brightness jump critical frame on the time axis, inserts a Sigmoid response curve compensation frame between adjacent mutation frames, matches the light transmittance change with the physiological delay characteristics of pupil contraction, and inhibits the instantaneous light damage of the retina. BRIEF DESCRIPTION OF DRAWINGS

[0025] The disclosure of the application will be described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes, and are not intended to limit the scope of protection of the application. Among them:

[0026] Figure 1 The flowchart of the AI glasses real-time collaboration method based on user intention prediction proposed by the application;

[0027] Figure 2 The method flowchart of environmental mutation compensation in the application;

[0028] Figure 3 The structural block diagram of the AI glasses real-time collaboration system based on user intention prediction proposed by the application. DETAILED DESCRIPTION

[0029] It is easy to understand that according to the technical scheme of the application, those skilled in the art can propose a plurality of structure modes and implementation modes which can be replaced with each other without changing the essential spirit of the application. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical scheme of the application, and should not be regarded as the whole or as a limitation or restriction on the technical scheme of the application.

[0030] In the prior art, heterogeneous light sources such as blackboards, light-transmitting windows, and electronic teaching screens coexist in an educational scene, forming a dynamic light environment. Frequent switching of students' visual lines leads to repeated pupil adjustment. The existing AI glasses use a single-point prediction linear adjustment mechanism, which only adjusts the light transmittance based on the start and end point luminance gradient, and is not easy to cope with the non-continuity luminance mutation caused by rapid switching, leading to pupil adjustment lag, causing visual fatigue accumulation and retinal damage risk.

[0031] To solve the above problems, the existing technology light transmittance adjustment lag problem is caused by not considering the prediction of the visual line movement trend to the light environment change. By analyzing the correlation between the user's head turning angle and the eye movement trajectory, it is found that the superposition of the eye movement vector and the head turning angle can predict the future visual field area. Based on this, a continuous light transmittance sequence generation method combining multiple frame prediction visual field area is proposed, which realizes gradual adjustment by identifying the turning point in the light transmittance change trend.

[0032] Reference Figures 1-2 As shown in the figure, an AI glasses real-time collaboration method based on user intention prediction is proposed, which includes:

[0033] Obtain the panoramic image of the classroom, and simultaneously collect the user's head turning angle and eye gaze coordinate sequence;

[0034] The panoramic image of the classroom is segmented into different functional areas by semantic segmentation, and different illumination weights are assigned according to the optical characteristics of the functional areas;

[0035] Exemplarily, the classroom panoramic image is identified and divided into the following multi-class functional areas by a semantic segmentation model such as DeepLabv3+, and the semantic segmentation identification features are explained:

[0036] Blackboard area: dark rectangular area (RGB mean ≤ 50), located in the center of the front wall of the classroom, with high contrast (gradient value > 100) between the edge and the white wall, and possibly containing local highlight pixel clusters; Light-transmitting window: high-luminance area (RGB mean ≥ 200), geometric shape is a regular rectangular array, containing vertical line features inside, and connected to the sky or landscape outline outside; Electronic teaching screen: self-luminous uniform bright area (RGB variance < 15), aspect ratio 16:9 or 4:3, often attached to the side of the blackboard, with V channel value > 90% in HSV space when turned on; Wall: large area of light color continuous area (RGB mean 150-180), texture is flat (LBP feature value < 0.1), and has clear boundaries with light-transmitting window, blackboard area, etc.; Desk area: dark rectangular array (desk surface RGB mean 70-90), surface may be covered with books / devices (irregular convex outline), height range 0.7-0.8m; Mixed personnel area: unstructured dynamic area (passage / lectern), containing human body outline (HOG feature), moving target (optical flow vector > 5px / frame), and high texture complexity;

[0037] On the basis of dividing the functional area, the illuminance weight is allocated according to the optical characteristics (basic brightness, visual importance, glare risk) of the functional area, and the allocation model is:

[0038] ;

[0039] In the formula, is the illuminance weight, is the visual importance coefficient, is the measured average brightness of the functional area, is the maximum brightness reference value of the classroom, is the glare risk coefficient, is the glare risk level (1 for light-transmitting window, 0.8 for electronic screen teaching, and 0.3 for others);

[0040] Taking 10000 as the maximum brightness reference value of the classroom, the higher the visual importance and the greater the glare risk, the higher the illuminance weight:

[0041] Blackboard area: , , Then the calculation is ;

[0042] Light-transmitting window: , , Then the calculation is ;

[0043] Electronic teaching screen: , , Then the calculation is ;

[0044] Wall: , , Then the calculation is ;

[0045] Desk area: , , Then the calculation is ;

[0046] Mixed personnel area: , , Then the calculation is ;

[0047] The eye movement vector is calculated based on the eye gaze coordinate sequence, and the predicted visual field area of the future continuous multiple frames is generated by superimposing the eye movement vector and the head turning angle.

[0048] Specifically, the acquisition method of the predicted visual field area is as follows: based on the eye gaze coordinate sequence, the displacement change of the eye in a unit time is calculated to output an eye movement vector; the head turning angle is converted into a spatial rotation vector to establish a dynamic reference coordinate system with the head as the center; the eye velocity vector and the head turning vector are superimposed in the head coordinate system to form a composite motion vector representing the actual moving direction of the visual line; based on the direction and rate of the composite vector, the landing point of the visual line in the continuous time frame is calculated, and each frame of the predicted visual field area is a rectangular window (the size is determined by the FOV of the AI glasses) with the landing point as the center; when a sudden change (such as acceleration exceeding a threshold) in the eye or head movement is detected, the prediction time window is reduced, and the prediction trajectory is calibrated by the next frame of measured data; a plurality of frames of the predicted visual field area are generated by superimposing the eye movement vector and the head turning angle, the adjustment unit is expanded from the start and end points to the entire visual line trajectory, and the non-continuous brightness mutation in the rapid switching of the visual line can be captured.

[0049] All the functional areas are divided into uniform grids, and for the grid set covered by each frame of the predicted visual field area, the weighted average value of the grid pixel brightness value and the functional area illuminance weight is calculated as the theoretical illuminance value of the frame of the predicted visual field area;

[0050] Based on the theoretical illuminance values of the continuous multiple frames, corresponding light transmittances are independently generated to form a light transmittance sequence;

[0051] According to the increasing or decreasing direction and the change amplitude of the light transmittance values in the light transmittance sequence, a turning point is identified, and a gradual change instruction containing all the turning points is generated;

[0052] It should be noted that the weighted calculation of the theoretical illuminance value is used to integrate the functional area illuminance weight and the grid pixel brightness to quantize the light environment difference in the spatial dimension and generate a non-linear illuminance sequence; then the turning point is identified to generate the gradual change instruction, which matches the physiological delay characteristics of pupil contraction and reduces the adjustment lag.

[0053] Through the above technical solutions, the three-layer mechanisms of the predicted visual field area generation, the non-linear illuminance sequence and the gradual change instruction solve the technical problems of prediction deficiency and adjustment lag, so that the non-monotonic light transmittance change instruction can be realized, and the traditional linear gradual adjustment mode based on the start and end points is replaced.

[0054] In an optional embodiment, the division process of the uniform grid specifically includes:

[0055] Each functional area is divided into grids with a preset area;

[0056] It should be noted that the preset area is dynamically determined by the resolution of the classroom panoramic image, and the ratio of the total image area to the number of functional areas multiplied by the grid density coefficient is taken as the preset area, the grid density coefficient is in the range of 100-200, and can be adjusted according to the resolution; for example, the total image area is 1920*1080 pixels, the number of functional areas is 6, and the grid density coefficient is 150, then the preset area is about 50*50 pixels;

[0057] Calculate the adjacent functional area illumination weight difference, and generate the transition width n according to the comparison of the difference and the preset segmentation interval;

[0058] For example, if the illumination weight difference , then ; if the illumination weight difference , then ; if the illumination weight difference , then ; the greater the illumination weight difference, the steeper the light gradient, and the wider the transition zone is needed to smooth the brightness change;

[0059] Along the normal direction of the boundary of the adjacent functional area, n rows / columns of grids are taken on both sides of the boundary to form a transition zone;

[0060] For the grids in the transition zone, the illumination weight is corrected according to the optical weight gradual change formula;

[0061] When the boundary is inclined relative to the grid coordinate system, the relative position sequence number of the grid to the boundary is re-designated based on the shortest Euclidean distance from the center of the grid to the boundary, and the illumination weight value is corrected; the grid position relationship of the inclined boundary is corrected by using the Euclidean distance calculation, so that smooth transition can be realized in the boundary area of any shape;

[0062] Compared with the prior art, the prior art directly divides the functional area, and the illumination weight jumps at the boundary (such as the blackboard 0.897 directly transitions to the wall 0.305), which makes it difficult for the pupil to adapt; the present scheme generates a continuous light gradient through the dynamic transition zone and the weight gradual change formula, and the pupil can adjust along the optical gradient, eliminating the visual blur caused by instantaneous light and dark switching;

[0063] The optical weight gradual change formula is:

[0064] ;

[0065] In the formula, is the corrected illumination weight, is the relative position sequence number of the current grid to the boundary, , is the illumination weight of the adjacent functional area, and ;

[0066] Through the above technical solution, this application effectively solves the problem of visual adaptation delay caused by sudden changes in illuminance at the boundary of adjacent functional areas. By dynamically generating a transition zone, the brightness change conforms to the human eye's adaptation law, reducing the pupil adjustment burden caused by rapid gaze switching and improving the user's visual comfort in complex lighting environments.

[0067] In one optional implementation, the transmittance is independently generated based on the theoretical illuminance values ​​of multiple consecutive frames, specifically including:

[0068] For the theoretical illuminance values ​​of each frame, the transmittance is calculated according to the mapping formula;

[0069] The mapping formula is:

[0070] ;

[0071] In the formula, For light transmittance, This is the theoretical illuminance value. This is the pupil constriction sensitivity coefficient. This is the glare suppression constant. Illuminance in response to pupil constriction;

[0072] This includes proposing a transmittance mapping formula based on the Sigmoid function to simulate the physiological response characteristics of the pupil; introducing a glare suppression constant to dynamically suppress the risk of light damage in high-brightness areas; and achieving personalized transmittance adjustment through the user's pupil constriction baseline parameter. For example, when... hour, ;when hour, ;when hour, ;

[0073] The pupil constriction sensitivity coefficient and pupil constriction response illuminance were obtained experimentally: the user gazed at no fewer than 5 sets of stepped brightness targets (200~5000 lux), and an infrared camera recorded the rate of change in pupil diameter. The illuminance gradient corresponding to the maximum pupil constriction acceleration was recorded as the pupil constriction sensitivity coefficient. The critical illuminance value at which the pupil diameter begins to shrink is taken as the pupil constriction response illuminance. Typical values ​​for adolescents It is 0.02-0.03. 800-1200 lux;

[0074] Compared with the prior art, the existing scheme usually directly correlates luminance and light transmittance by using a linear mapping function, without considering the nonlinear perception characteristics of the human eye to light intensity changes, and lacks a glare interference suppression mechanism. The present scheme simulates the nonlinear response of the pupil to light intensity by introducing a Sigmoid function, compensates for the high-frequency interference components in the ambient light using a glare suppression constant, and simultaneously achieves personalized adaptation through user physiological parameters, so that the light transmittance adjustment process is closer to the actual visual physiological needs.

[0075] Through the above technical solution, the present application can effectively suppress glare interference caused by sudden changes in environmental brightness, so that the light transmittance curve and the pupil constriction speed form a dynamic match, avoiding retinal light intensity overload caused by adjustment lag and reducing the risk of visual fatigue accumulation.

[0076] In an optional implementation, after the light transmittance sequence is generated, the environmental mutation compensation specifically includes:

[0077] The light transmittance intensity value of each frame of the predicted field of view is obtained, and the light transmittance measured value is calculated. The light transmittance intensity value of the lens is collected in real time by a photoelectric sensor (such as a photodiode) built in the AI glasses, and the light transmittance measured value is calculated by using an inverse mapping model;

[0078] The deviation rate of the light transmittance measured value and the light transmittance of the predicted field of view is calculated. The relative error (in percentage form) of the measured light transmittance and the theoretical light transmittance is quantified, which is used to detect sudden environmental changes.

[0079] If the deviation rate is greater than a preset deviation rate threshold, the grid pixel luminance values of subsequent frames are compensated according to a luminance compensation formula. Exemplarily, the preset deviation rate threshold can be 10%-15%, and a typical value is 12%. The deviation rate is obtained based on a stepped luminance target, and the deviation rate exceeding the preset deviation rate threshold indicates that the environmental mutation has obviously affected the visual comfort.

[0080] The theoretical illuminance value is recalculated based on the compensated grid pixel luminance value, and the light transmittance sequence is corrected.

[0081] Compared with the prior art, the traditional method only generates a light transmittance sequence based on a prediction model in one direction, and cannot cope with sudden changes in light such as blackboard reflection or strong light outside the window. The present scheme dynamically corrects the luminance input parameters of subsequent frames through a closed-loop feedback mechanism of the measured light transmittance and the theoretical value, so that the light transmittance adjustment process has real-time correction ability.

[0082] The luminance compensation formula is:

[0083] ;

[0084] In the formula, is the compensated grid pixel luminance value, a grid pixel luminance value of a current frame, a compensation intensity coefficient, a light transmittance measured value, a theoretical illumination value; exemplary, the compensation intensity coefficient is in the range of [0.1, 0.3], if , indicating that the environment is too bright, then , the compensation formula reduces the luminance value of the subsequent frame (to prevent overexposure);

[0085] Through the above technical solution, the present application effectively solves the light transmittance adjustment lag problem caused by sudden changes in environmental light. Through a closed-loop feedback mechanism, the prediction model output is dynamically corrected, so that the lens light transmittance can quickly respond to sudden changes in light such as blackboard reflection, window strong light, etc., reducing the risk of retinal light damage caused by delayed adjustment, while improving the visual comfort during cross-functional area visual line switching.

[0086] In an optional implementation, the identification process of the turning point specifically includes:

[0087] For each light transmittance value in the light transmittance sequence, calculate the increase / decrease direction identifier of the previous frame;

[0088] In the direction identifier sequence of the three consecutive frames, if the first frame increase / decrease direction is opposite to the second frame increase / decrease direction and the second frame increase / decrease direction is opposite to the third frame increase / decrease direction, the second frame is marked as a candidate turning point;

[0089] Take the absolute value of the difference between the candidate turning point and the light transmittance value of the previous frame as the first change amplitude, and the absolute value of the difference between the candidate turning point and the light transmittance value of the next frame as the second change amplitude. The first change amplitude minus the second change amplitude to get the change amplitude difference value;

[0090] If the absolute value of the change amplitude difference value is greater than the preset amplitude threshold, it is confirmed as an effective turning point;

[0091] It should be noted that when the predicted visual field crosses the functional area boundary, the system automatically triggers the threshold relaxation mechanism: due to the existence of the transition zone at the boundary, the luminance change should be gentle. If the change amplitude difference value is greater than the preset amplitude threshold at this time, it indicates that a real mutation has occurred, and the turning point needs to be captured. Accurate identification of the turning point ensures that the Sigmoid compensation is triggered at the real luminance jump position by the gradual instruction, avoiding the pupil adjustment lag caused by linear transition;

[0092] When the predicted visual field corresponding to the candidate turning point crosses the functional area boundary, multiply the preset amplitude threshold by the preset coefficient and recompare;

[0093] Exemplarily, the preset amplitude threshold is set based on a proportion of the transmittance change range, needs to cover the brightness jump critical value that can be perceived by human eyes, if the transmittance range is 0-100%, the preset amplitude threshold is taken as 10%; the preset coefficient is obtained by formula , wherein, is a proportional coefficient and is fixedly taken as 0.05, and n is the transition width;

[0094] Compared with the prior art, the single threshold or derivative extremum method is easy to be disturbed by noise, leading to misjudgment of the turning point (such as identifying a small fluctuation as a sudden change), causing frequent adjustment of the transmittance, and aggravating pupil fatigue, while the scheme distinguishes the real environment mutation from noise through the three mechanisms of continuous three-frame direction reversal detection, change amplitude difference significance judgment, and boundary adaptive threshold adjustment, and ensures the generation of accurate progressive instructions;

[0095] Through the above technical solution, the present application can accurately capture the transmittance mutation node caused by the rapid switching of the user's visual line, adapt to the complex light changes at the junction of different functional areas through the dynamic threshold adjustment mechanism, ensure that the generation time of the progressive change instruction is synchronized with the real environment change, thereby significantly reducing the pupil regulation lag phenomenon caused by the delay of turning point recognition, and effectively alleviating the visual fatigue problem caused by frequent light and dark alternation.

[0096] In an optional implementation, the progressive change instruction containing all the turning points is generated, specifically including:

[0097] Taking the transmittance value of the adjacent turning point as the starting point and the end point ;

[0098] The Sigmoid response curve is used to generate the compensation frame transmittance sequence, and the Sigmoid response curve is:

[0099] ;

[0100] In the formula, is the transmittance of the i-th frame, is a curve steepness coefficient, is the total number of interpolation frames, ;

[0101] Specifically, the generated compensation frame sequence is inserted into the original transmittance sequence to form the final progressive change instruction to drive the smooth transition of the lens transmittance;

[0102] ​Compared with the prior art, the traditional method generates the light transmittance transition sequence by linear interpolation, resulting in constant light transmittance change rate, which cannot adapt to the nonlinear response of the pupil to the sudden change in brightness, and aggravates the adjustment lag. The present scheme uses the nonlinear characteristics of the Sigmoid curve and the dynamic parameter adjustment mechanism to make the light transmittance change curve quickly converge in the case of sudden brightness switching and gently transition in the case of slight changes, so that the light transmittance change trajectory is synchronized with the actual response of the pupil, reducing the risk of transient overload of the retina.

[0103] The curve steepness coefficient is:

[0104]

[0105] In the formula, is the curve steepness coefficient, is the change amplitude difference, is a preset amplitude threshold;

[0106] The total interpolation frame number is dynamically calculated based on the eye movement vector, and the calculation formula is:

[0107]

[0108] In the formula, is the total interpolation frame number, is the eye movement vector;

[0109] According to the above technical scheme, the present application can dynamically generate a light transmittance transition sequence that conforms to the physiological regulation of the human eye according to the actual light environment change amplitude and the eye movement speed, effectively reducing the pupil lag adjustment problem caused by sudden changes in light transmittance, and reducing the risk of visual fatigue accumulation.

[0110] In an optional embodiment, after generating the gradual change instruction, the illumination weight is dynamically corrected, specifically including:

[0111] Real-time collection of user frown intensity and pupil contraction speed;

[0112] According to the preset illumination value change rate-pupil contraction speed mapping table, the theoretical contraction speed is obtained, and the ratio of the difference between the pupil contraction speed and the theoretical contraction speed to the theoretical contraction speed is calculated as the pupil contraction lag rate. The mapping table is constructed by recording the pupil response speed of healthy subjects under different brightness gradients through step brightness experiments;

[0113] Based on the lens light transmittance value and the corresponding predicted visual field area light transmittance, the measured illumination value is calculated by inversely solving the mapping formula of the theoretical illumination value and the light transmittance;

[0114] ​If the frown intensity is greater than a preset intensity threshold or the pupil contraction lag rate is greater than a preset lag threshold, the illumination weight of the corresponding functional area is updated according to a fixed learning formula; wherein the pupil contraction lag rate is the absolute value of the pupil contraction speed minus the theoretical contraction speed in the mapping table, compared with the theoretical contraction speed; for example, the preset intensity threshold can be taken as 0.1 , and the preset lag threshold can be taken as 15%;

[0115] The fixed learning formula is:

[0116] ;

[0117] In the formula, is the updated illumination weight, is the functional area illumination weight, is the fixed learning rate, is the difference between the measured illumination value and the theoretical illumination value; for example, the fixed learning rate is taken as 0.03;

[0118] Compared with the prior art, the traditional method only adjusts the light transmittance based on the preset weight, and cannot be dynamically optimized according to the real-time physiological feedback of the user. The present scheme constructs a quantitative evaluation system of human visual comfort by fusing the dual biological characteristics of frown intensity and pupil contraction speed, and realizes online correction of weight parameters by combining a fixed learning formula, effectively solving the adaptability problem of the static weight model in a dynamic light environment.

[0119] Through the above technical scheme, the present application can dynamically optimize the functional area weight parameters according to the real-time physiological response of the user, so that the light transmittance adjustment process is more in line with the actual visual comfort needs of the user. By iterative updating, the deviation between the theoretical model and the actual perception is eliminated, thereby reducing the accumulation of visual fatigue caused by inaccurate light transmittance adjustment, and improving the visual adaptation efficiency in a complex light environment.

[0120] Referring to Figure 3 , the present scheme proposes an AI glasses real-time collaboration system based on user intention prediction, which is used to realize the above-mentioned AI glasses real-time collaboration method based on user intention prediction, comprising:

[0121] A multi-source data acquisition module is used to acquire a classroom panoramic image, and simultaneously acquire a user head turning angle and an eye gaze coordinate sequence;

[0122] An illumination weight distribution module is used to segment the classroom panoramic image into different functional areas by semantic segmentation, and distribute differentiated illumination weights according to the optical characteristics of the functional areas;

[0123] A future view prediction module is configured to calculate an eye movement vector based on the eye gaze coordinate sequence, and generate a predicted view area of a plurality of continuous frames in the future by superimposing the eye movement vector and a head turning angle.

[0124] A transmittance sequence generation module is configured to divide all functional areas into uniform grids, calculate a weighted average of grid pixel brightness values and the functional area illumination weight for a grid set covered by each frame of the predicted view area, and take the weighted average as a theoretical illumination value of the frame of the predicted view area, independently generate corresponding transmittances based on the theoretical illumination values of the plurality of continuous frames, and form a transmittance sequence.

[0125] A gradual change instruction generation module is configured to identify turning points according to the increasing or decreasing direction and the change amplitude of the transmittance values of the plurality of continuous frames in the transmittance sequence, and generate a gradual change instruction containing all the turning points.

[0126] In addition, in an embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above embodiments when executing the computer program.

[0127] In an embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above embodiments.

[0128] In an embodiment, a computer program product or a computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the steps in the above embodiments.

[0129] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should all be within the protection scope of the present application.

Claims

1. A real-time collaborative method for AI glasses based on user intent prediction, characterized in that, The method includes: Acquire panoramic images of the classroom, and simultaneously collect user head turning angle and eye gaze coordinate sequences; The panoramic image of the classroom is segmented into different functional areas through semantic segmentation, and differentiated illuminance weights are assigned according to the optical characteristics of the functional areas. The eye movement vector is calculated based on the eye gaze coordinate sequence, and the eye movement vector is superimposed with the head turning angle to generate the predicted visual field area for multiple consecutive frames in the future. All functional areas are divided into uniform grids. For each frame, the set of grids covered by the predicted field of view is calculated. The weighted average of the grid pixel brightness value and the illuminance weight of the functional area is used as the theoretical illuminance value of the predicted field of view for that frame. The transmittance is generated independently based on the theoretical illuminance values ​​of multiple consecutive frames, forming a transmittance sequence. Based on the direction and magnitude of the increase or decrease in transmittance values ​​across multiple consecutive frames in the transmittance sequence, inflection points are identified, and a gradual change instruction containing all inflection points is generated.

2. The method according to claim 1, characterized in that, The process of dividing the uniform grid specifically includes: Divide each functional area into grids of equal, pre-defined size; Calculate the difference in illuminance weight between adjacent functional areas, and generate a transition width n based on the comparison of this difference with the preset segmented interval; Along the boundary normal of the adjacent functional area, n rows / columns of grid are taken on each side of the boundary to form a transition zone; For the grid within the transition zone, the illuminance weight is corrected according to the optical weight gradient formula; When the boundary is tilted relative to the grid coordinate system, the relative position index of the grid to the boundary is recalibrated based on the shortest Euclidean distance from the grid center to the boundary, and the illuminance weight value is corrected. The optical weight gradient formula is: ； In the formula, The corrected illuminance weights, This represents the relative position number of the current grid cell from the boundary. , The illuminance weights of adjacent functional areas and .

3. The method according to claim 1, characterized in that, The process of independently generating corresponding transmittance based on theoretical illuminance values ​​from multiple consecutive frames specifically includes: For the theoretical illuminance values ​​of each frame, the transmittance is calculated according to the mapping formula; The mapping formula is: ； In the formula, For light transmittance, This is the theoretical illuminance value. This is the pupil constriction sensitivity coefficient. This is the glare suppression constant. Illuminance is the pupil constriction response.

4. The method according to claim 1, characterized in that, After generating the transmittance sequence, environmental mutation compensation is performed, specifically including: Obtain the lens transmittance value corresponding to the predicted field of view in each frame, and then retrieve the measured transmittance value. Calculate the deviation rate between the measured transmittance value and the transmittance of the predicted field of view area; If the deviation rate is greater than the preset deviation rate threshold, the brightness value of the grid pixels in subsequent frames will be compensated according to the brightness compensation formula. The theoretical illuminance value is recalculated based on the compensated grid pixel brightness value, and the transmittance sequence is corrected. The brightness compensation formula is as follows: ； In the formula, The compensated grid pixel brightness value, This represents the brightness value of the grid pixels in the current frame. To compensate for the strength coefficient, This is the measured transmittance value. This is the theoretical illuminance value.

5. The method according to claim 1, characterized in that, The process of identifying the turning point specifically includes: For each frame of transmittance value in the transmittance sequence, calculate its increase or decrease direction indicator relative to the previous frame; In a sequence of three consecutive frames of directional markings, if the direction of increase or decrease in the first frame is opposite to that in the second frame and the direction of increase or decrease in the second frame is opposite to that in the third frame, then the second frame is marked as a candidate turning point. The absolute value of the difference between the transmittance value of the candidate inflection point and the previous frame is taken as the first change range, and the absolute value of the difference between the transmittance value of the candidate inflection point and the next frame is taken as the second change range. The first change range is subtracted from the second change range to obtain the change range difference. If the absolute value of the difference in the magnitude of the change is greater than the preset magnitude threshold, it is confirmed as a valid turning point; When the predicted field of view corresponding to a candidate inflection point crosses the functional area boundary, the preset amplitude threshold is multiplied by a preset coefficient and then re-compared.

6. The method according to claim 5, characterized in that, The generation of progressive change instructions containing all inflection points specifically includes: Starting from the transmittance values ​​of adjacent inflection points and termination point ; The transmittance sequence of compensated frames is generated using a Sigmoid response curve, wherein the Sigmoid response curve is as follows: ； In the formula, For the first The transmittance of the frame, This is the curve steepness coefficient. This represents the total number of interpolated frames. .

7. The method according to claim 4, characterized in that, After generating the gradual change instruction, dynamic correction of illuminance weights is performed, specifically including: Real-time data collection of user frown intensity and pupil constriction speed; The theoretical contraction speed is obtained based on the preset illuminance value change rate-pupil contraction speed mapping table. The ratio of the difference between the pupil contraction speed and the theoretical contraction speed to the theoretical contraction speed is calculated as the pupil contraction hysteresis rate. Calculate the measured illuminance value based on the lens transmittance value and the corresponding predicted transmittance of the visual field area. If the frown intensity is greater than the preset intensity threshold or the pupil contraction hysteresis rate is greater than the preset hysteresis threshold, the illuminance weight of the corresponding functional area is updated according to the fixed learning formula. The fixed learning formula is: ； In the formula, For the updated illuminance weights, As functional zone illuminance weights, For a fixed learning rate, It represents the difference between the measured illuminance value and the theoretical illuminance value.

8. A real-time collaborative system for AI glasses based on user intent prediction, characterized in that, A method for implementing real-time collaborative AI glasses based on user intent prediction as described in any one of claims 1-7 includes: The multi-source data acquisition module is used to acquire panoramic images of the classroom, while simultaneously acquiring the user's head turning angle and eye gaze coordinate sequence; The illuminance weight allocation module is used to segment the panoramic image of the classroom into different functional areas through semantic segmentation, and to allocate differentiated illuminance weights according to the optical characteristics of the functional areas. The future vision prediction module is used to calculate the eye movement vector based on the eye gaze coordinate sequence, and then superimpose the eye movement vector with the head turning angle to generate the predicted vision area for multiple consecutive future frames. The transmittance sequence generation module is used to divide all functional areas into uniform grids. For each frame, the module calculates the weighted average of the brightness value of the grid pixels and the illuminance weight of the functional area for the grid set covered by the predicted field of view. This average is used as the theoretical illuminance value of the predicted field of view for that frame. Based on the theoretical illuminance values ​​of multiple consecutive frames, the module independently generates the corresponding transmittance to form a transmittance sequence. The gradual change instruction generation module is used to identify turning points based on the direction and magnitude of increase or decrease in transmittance values ​​across multiple consecutive frames in the transmittance sequence, and to generate gradual change instructions containing all turning points.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images