Education robot behavior evaluation method and system based on reinforcement learning

Abstract

The invention relates to the technical field of reinforcement learning, and discloses an education robot behavior evaluation method and system based on reinforcement learning, and the method comprises the steps: extracting a facial feature measuring scale and a book region through collecting a video sequence, and calculating a relative eye-book distance; generating relative adjustment demand intensity according to the statistical base line of the distance; constructing continuous short-distance intensity and identifying a continuous short-distance interval to form a short-distance cluster, and obtaining a cluster duration sample set; calculating a risk quantitative index according to a specific quantile of the sample set; recording the duration of the current short-distance cluster as the current cluster age; the risk quantitative index and the current cluster age form a reinforcement learning state, voice reminding and non-reminding are taken as actions, risk quantitative index change, intervention overhead and potential energy adjustment form return, and a reinforcement learning strategy is updated through a strategy gradient method; and outputting a voice prompt or non-prompt decision according to the updated strategy, thereby realizing intelligent evaluation and dynamic intervention of the eye use risk.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of reinforcement learning, more particularly, it relates to a method and system for evaluating educational robot behavior based on reinforcement learning. BACKGROUND

[0002] In the current learning environment, students often unconsciously shorten the reading distance when writing or reading for a long time. Prolonged close reading distance is a key risk factor for inducing visual accommodation fatigue and accelerating myopia progression. However, visual health risk depends not only on instantaneous eye-book distance, but more importantly on the duration pattern and time distribution structure of close reading behavior. For example, short and multiple close reading behaviors have much lower risk than single long and uninterrupted close reading behavior.

[0003] Existing educational robot visual monitoring technologies mostly focus on measuring and threshold alarming of instantaneous eye-book distance. Although this method can capture the state at a single moment, it has limitations: It is difficult to extract persistent risk from time series behavior: it is difficult to identify and quantify long-term unchanged high-risk eye behavior clusters from continuous image sequences, and it is also difficult to represent statistical features such as extreme duration distribution, so it cannot accurately assess the cumulative risk level of behavior.

[0004] It is difficult to achieve adaptive and forward-looking intervention: due to the lack of dynamic description of risk accumulation process, the robot cannot determine whether the current behavior is forming a high-risk continuous close reading, resulting in rigid and lagging reminder strategies, which are prone to ineffective reminders or interruptions and poor user experience.

[0005] When reinforcement learning is introduced into educational robot behavior evaluation, the traditional reinforcement learning framework faces serious challenges in dealing with duration-based risks: delay and sparsity of reward signals. For duration-based risks, the true cost (reward) can only be calculated completely after the behavior ends, which leads to the learning system being unable to obtain timely feedback during the critical process of risk accumulation, resulting in problems such as delay of strategy update, dependence of learning path, and instability of strategy. Ultimately, the robot cannot reasonably allocate rewards and punishments, and cannot achieve a stable and efficient adaptive reminder strategy. SUMMARY

[0006] The present application provides a method and system for evaluating educational robot behavior based on reinforcement learning, which solves the technical problem of how to identify and quantify the risk intensity changes of students' continuous close reading behavior in real time only relying on image data, so that the educational robot can realize dynamic, delay-sensitive, and strategy self-adjusting voice intervention decisions based on reinforcement learning algorithms.

[0007] In a first aspect, the method for evaluating the behavior of an educational robot based on reinforcement learning comprises: collecting a video sequence, calculating a relative eye-book distance based on a facial feature scale in the video sequence and a book region, and generating a relative adjustment demand intensity based on a statistical baseline of the relative eye-book distance; constructing a continuous near-distance intensity from the relative adjustment demand intensity, defining a near-distance cluster as a continuous interval where the continuous near-distance intensity is greater than a preset threshold, and obtaining a cluster duration sample set; calculating a risk quantification index according to a specific quantile of the cluster duration sample set, and recording a current near-distance cluster duration as a current cluster age; using the risk quantification index and the current cluster age as the state of reinforcement learning, using voice reminders and no reminders as actions, using changes in the risk quantification index, intervention costs, and potential energy adjustments based on the current cluster age as rewards, and updating the reinforcement learning strategy through a policy gradient method; outputting a decision of voice reminders or no reminders according to the updated reinforcement learning strategy.

[0008] In a second aspect, the system for evaluating the behavior of an educational robot based on reinforcement learning is applied to any of the methods for evaluating the behavior of an educational robot based on reinforcement learning, and comprises: a baseline module that collects a video sequence, calculates a relative eye-book distance based on a facial feature scale in the video sequence and a book region, and generates a relative adjustment demand intensity based on a statistical baseline of the relative eye-book distance; a cluster set construction module that constructs a continuous near-distance intensity from the relative adjustment demand intensity, defines a near-distance cluster as a continuous interval where the continuous near-distance intensity is greater than a preset threshold, and obtains a cluster duration sample set; a cluster set construction module that calculates a risk quantification index according to a specific quantile of the cluster duration sample set, and records a current near-distance cluster duration as a current cluster age; a strategy update module that uses the risk quantification index and the current cluster age as the state of reinforcement learning, uses voice reminders and no reminders as actions, uses changes in the risk quantification index, intervention costs, and potential energy adjustments based on the current cluster age as rewards, and updates the reinforcement learning strategy through a policy gradient method; an intervention decision module that outputs a decision of voice reminders or no reminders according to the updated reinforcement learning strategy.

[0009] The beneficial effects of this invention include: by introducing a cluster duration distribution quantile ratio, it enables real-time, quantitative, and learnable assessment of students' continuous near-field eye use behavior without relying on individual physiological parameters. By automatically extracting eye-book distance changes from video sequences to reflect the heavy tails of sustained near-field activity, and transforming these heavy tails into state variables and reward signals for reinforcement learning, the educational robot can adaptively learn the optimal timing for voice reminders through policy gradients without human intervention. This effectively reduces excessively long near-field eye use behaviors and improves the targeting and stability of behavioral interventions. This invention enhances the educational robot's ability to intelligently assess and dynamically manage students' eye-use risks. Attached Figure Description

[0010] Figure 1 This is a flowchart of the reinforcement learning-based behavior evaluation method for educational robots according to the present invention; Figure 2 This is a block diagram of the reinforcement learning-based educational robot behavior evaluation system of the present invention. Detailed Implementation

[0011] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0012] Example 1: As Figure 1 As shown, the reinforcement learning-based method for evaluating the behavior of educational robots includes: Acquire video sequences, calculate the relative eye-to-book distance based on facial feature scales and book areas in the video sequences; generate the relative accommodation demand intensity based on the statistical baseline of the relative eye-to-book distance. A continuous near-range intensity is constructed based on the relative adjustment demand intensity. The continuous intervals where the continuous near-range intensity is greater than a preset threshold are defined as near-range clusters, and a sample set of cluster durations is obtained. Based on specific quantiles of the cluster duration sample set, a risk quantification index is calculated; the duration of the current nearby cluster is recorded as the current cluster age. The state of reinforcement learning is defined by risk quantification indicators and the current cluster age. Voice prompts and no prompts are used as actions. The reward is defined by changes in risk quantification indicators, intervention costs, and potential energy adjustments based on the current cluster age. The reinforcement learning strategy is updated through a policy gradient method. Based on the updated reinforcement learning strategy, output a decision on whether to provide a voice prompt or not.

[0013] In one embodiment of the present invention, a video sequence is acquired, and the relative eye-to-book distance is calculated based on a facial feature scale and a book area in the video sequence, including: Extract the pixel coordinates of the center of the left pupil and the center of the right pupil from each frame of the video sequence; Calculate the straight-line distance between the center of the left pupil and the center of the right pupil as the pupillary distance in pixels; The book region is obtained by performing region segmentation on the book in the same frame, and the set of boundary pixels of the book region is determined. Calculate the pixel coordinates of the midpoint between the two eyes based on the pixel coordinates of the center of the left pupil and the center of the right pupil; Calculate the straight-line distance from the pixel coordinates of the midpoint of both eyes to each pixel in the set of pixels at the boundary of the book area, and select the smallest straight-line distance as the target pixel distance; The ratio of the target pixel distance to the interpupillary distance pixel is used as the relative eye-book distance.

[0014] The pixel coordinates of the center of the left pupil are the two-dimensional positioning coordinates of the geometric center of the left pupil in the video frame. They consist of horizontal and vertical positions, with the horizontal values ​​increasing from left to right and the vertical values ​​increasing from top to bottom. Specifically, the eye region image is processed using grayscale thresholding combined with morphological closing operations. First, a grayscale threshold of 10 to 30 is set to adapt to indoor classroom lighting. Dark areas below this threshold are filtered out. Then, the pupil region is determined by a standard of circularity greater than 0.7. Finally, the geometric center coordinates of the pupil region are calculated, and the result is the pixel coordinates of the center of the left pupil.

[0015] The pixel coordinates of the center of the right pupil are two-dimensional positioning coordinates of the geometric center of the right pupil in the video frame. They consist of horizontal and vertical positions, with the horizontal values ​​increasing from left to right and the vertical values ​​increasing from top to bottom. The structure is consistent with the pixel coordinates of the center of the left pupil. Specifically, the image of the eye region containing the right eye is segmented by grayscale thresholding, using a grayscale threshold of 10 to 30. Then, morphological closing operations are performed. The right pupil region is selected by using a circularity greater than 0.7 as a criterion, and the geometric center coordinates of this region are calculated. The result is the pixel coordinates of the center of the right pupil.

[0016] Interpupillary distance (IPD) pixels are quantized values ​​representing the spatial distance between the centers of the pupils of both eyes in a video frame. They serve as a facial feature measure for subsequent distance normalization processing. Specifically, the difference between the horizontal coordinate of the left pupil center and the horizontal coordinate of the right pupil center is calculated first, and then squared. Next, the difference between the vertical coordinate of the left pupil center and the vertical coordinate of the right pupil center is calculated, and then squared. The results of the two square operations are added together, and the square root of the sum is taken. The result is the IPD pixels.

[0017] The book region is the set of all pixels occupied by the book in the video frame. It needs to be clearly distinguished from the background areas such as the desktop and hands to provide a basis for extracting the boundary pixel set of the book region. Specifically, an algorithm based on color space conversion and edge detection is used. First, the RGB color space of the image is converted to the HSV color space. An HSV range suitable for common white paper books is set, with hue from 0 to 30, saturation from 0 to 100, and lightness from 20 to 255. Pixel areas that conform to this HSV range are selected. Then, the outline of the region is extracted by the Canny edge detection algorithm, and the inside of the outline is filled. The filled area is the book region.

[0018] The set of boundary pixels of the book region is the set of all pixels that constitute the outline of the book region. Each pixel in the set is the boundary point between the book region and the background region. Specifically, the Sobel operator is used to extract the outline of the obtained book region and all pixels on the extracted outline are selected. The set of these pixels is the set of boundary pixels of the book region.

[0019] The pixel coordinates of the midpoint between the two eyes are the two-dimensional coordinates of the midpoint of the line connecting the centers of the pupils of both eyes, used to represent the overall horizontal and vertical positions of the eyes. Specifically, the horizontal coordinate of the midpoint between the two eyes is equal to the sum of the horizontal coordinates of the center of the left pupil and the center of the right pupil, divided by two; the vertical coordinate of the midpoint between the two eyes is equal to the sum of the vertical coordinates of the center of the left pupil and the center of the right pupil, divided by two. Combining the calculated horizontal and vertical coordinates of the midpoint between the two eyes gives the pixel coordinates of the midpoint between the two eyes.

[0020] The target pixel distance is the shortest pixel interval from the midpoint of the eyes to the edge of the book, reflecting the pixel-level proximity between the eyes and the book. Specifically, it iterates through each pixel in the set of pixels at the boundary of the book region. For a single pixel, first calculate the difference between the horizontal coordinate of the pixel and the horizontal coordinate of the midpoint of the eyes, then square this difference. Next, calculate the difference between the vertical coordinate of the pixel and the vertical coordinate of the midpoint of the eyes, then square this difference. Add the results of the two square operations and take the square root to obtain the straight-line distance from the pixel to the midpoint of the eyes. After the iteration is complete, select the minimum distance value from all the calculated straight-line distances; this value is the target pixel distance.

[0021] Relative eye-to-book distance is a dimensionless relative distance indicator that eliminates the influence of differences in camera focal length, shooting angle, and individual head size. It is used to characterize the relative size of the actual eye-to-book distance. Specifically, the target pixel distance is divided by the interpupillary distance in pixels, and the result is rounded to two decimal places. This design achieves normalization through the ratio of the target pixel distance to the interpupillary distance in pixels, which can adapt to differences in interpupillary distance among children and changes in camera position under different shooting conditions.

[0022] In one embodiment of the present invention, a relative accommodation demand intensity is generated based on a statistical baseline of relative eye-to-book distance, including: The original value of the adjustment demand intensity is obtained by taking the reciprocal of the relative eye-to-book distance; Determine the statistical baseline time length and define a time interval with the current time as the endpoint and a duration equal to the statistical baseline time length; Extract all raw quantities of adjustment demand intensity within the time interval, and calculate the median of all raw quantities of adjustment demand intensity as the statistical baseline; The relative adjustment demand intensity is obtained by calculating the ratio between the original quantity of adjustment demand intensity and the statistical baseline.

[0023] The raw quantity of accommodation demand intensity is a basic quantitative value that reflects the accommodation burden of the eye. The larger the value, the higher the accommodation demand. Specifically, divide the value 1 by the relative eye-book distance to get the raw quantity of accommodation demand intensity, and keep the result to three decimal places.

[0024] The statistical baseline time length is the duration of the time window used to calculate the statistical baseline, and it needs to be adapted to the fluctuation characteristics of eye behavior in classroom reading and writing scenarios. Specifically, the statistical baseline time length is set to 60 seconds. This duration can cover a sufficient number of eye behavior samples and reflect recent changes in eye habits in a timely manner, adapting to the distance fluctuation characteristics of students reading and writing in classroom scenarios.

[0025] A time interval is a specific period of time used to extract the raw quantity of adjustment demand intensity to calculate the statistical baseline. The end point of the interval is the current time, and the total duration of the interval is equal to the duration of the statistical baseline. Specifically, starting from the current time, the continuous time period formed by tracing back to the duration equal to the duration of the statistical baseline is the time interval.

[0026] The statistical baseline is the median of the raw quantities of adjustment demand intensity within a time interval. It serves as a benchmark for measuring the current adjustment demand intensity, is unaffected by extreme values, and has greater stability. Specifically, the first step is to extract all raw quantities of adjustment demand intensity within the time interval; the second step is to sort all extracted raw quantities in ascending order of value; the third step is to take the median if the number of raw quantities is odd, and the median is the value of the middle position after sorting; if the number is even, the median is the average of the two middle values ​​after sorting. This median is the statistical baseline.

[0027] The relative accommodation demand intensity is used to characterize the degree of deviation of the current accommodation demand from the recent baseline level. A value greater than 1 indicates that the current accommodation demand is higher than the recent average level. Specifically, the relative accommodation demand intensity is obtained by dividing the original amount of accommodation demand intensity at the current moment by the statistical baseline and the result is rounded to two decimal places. This design avoids dependence on individual physiological thresholds by self-normalizing the median baseline and adapts to the differences in eye habits of different students.

[0028] In one embodiment of the present invention, a continuous proximity intensity is constructed based on the relative adjustment demand intensity, and a continuous interval where the continuous proximity intensity is greater than a preset threshold is defined as a proximity cluster. A cluster duration sample set is obtained, including: The continuous short-range intensity is constructed by taking the difference between the relative adjustment demand intensity and the unit constant and taking the result that is not less than zero. Set a preset threshold; Determine whether the continuous near-field intensity is greater than a preset threshold, and mark all moments when the continuous near-field intensity is greater than the preset threshold; Determine the continuous time interval between two consecutive moments when the intensity of the proximity is no greater than a preset threshold, define this continuous time interval as a proximity cluster, and record the start and end times of each proximity cluster. The cluster duration of the corresponding close cluster is obtained by calculating the difference between the end time and the start time of each close cluster. Collect the cluster duration of all nearby clusters to form a cluster duration sample set.

[0029] The unit constant is used as a benchmark threshold for the relative intensity of adjustment demand, distinguishing between adjustment demand that is higher than the recent benchmark and that that is lower than the recent benchmark. Specifically, the value of the unit constant is fixed at one, and it is directly used for the difference calculation with the relative intensity of adjustment demand.

[0030] Continuous near intensity is a continuous quantitative indicator that characterizes the current state of relatively closer eye use. The larger the value, the higher the relative nearness. A value of zero indicates that there is no deviation from the recent baseline. Specifically, the relative accommodation demand intensity is subtracted from the unit constant one. If the difference is greater than zero, the difference is taken as the result. If the difference is less than or equal to zero, zero is taken as the result. The result is the continuous near intensity. This preserves the gradient information of nearness rather than simple binarization, and is adapted to dynamic eye use scenarios that are relatively closer.

[0031] The preset threshold is a critical value used to filter noise and define the effective near distance state. It needs to be adapted to the fluctuation range of continuous near distance intensity in the classroom scenario. Specifically, the preset threshold is set to 0.05 to 0.1. This range can filter out the small noise generated by image recognition and distance calculation, and accurately capture meaningful near distance deviations, adapting to the fluctuation characteristics of eye use in classroom reading and writing scenarios.

[0032] The moment when the continuous close-range intensity is greater than the preset threshold refers to the specific time point when the continuous close-range intensity value exceeds the preset threshold, which is used to locate the time when the effective close-range state occurs; specifically, the continuous close-range intensity at each moment is traversed, and if the continuous close-range intensity value at that moment is greater than the preset threshold, then that moment is marked as the moment when the continuous close-range intensity is greater than the preset threshold.

[0033] The moment when two consecutive close-range intensities are not greater than a preset threshold refers to two adjacent time points where the consecutive close-range intensities are less than or equal to the preset threshold. This is used to define the start and end boundaries of close-range clusters. Specifically, among all the marked moments, moments when the consecutive close-range intensities are not greater than the preset threshold are selected. After sorting them in chronological order, every two adjacent moments of this type are considered to be two consecutive close-range intensities that are not greater than the preset threshold.

[0034] A near cluster refers to a continuous time period between two consecutive moments when the near intensity is no greater than a preset threshold. It is used to characterize a complete and continuous near eye use behavior. Specifically, the continuous time period between two consecutive moments when the near intensity is no greater than the preset threshold is the near cluster. If there is only a single consecutive moment when the near intensity is no greater than the preset threshold, then the continuous effective near time period from the start time to that moment or from that moment to the end time is the near cluster.

[0035] The start time of a close cluster is the starting point of the continuous time period corresponding to the close cluster, which is used to calculate the cluster duration. Specifically, the start time of a close cluster is the next instant after the earlier of two consecutive close clusters whose intensities are not greater than a preset threshold. If the close cluster begins at the beginning of the entire time series, then the start time is the start time of the entire time series.

[0036] The end time of a close cluster is the end time of the continuous time period corresponding to the close cluster, and is used to calculate the cluster duration. Specifically, the end time of a close cluster is the later time among two adjacent consecutive close clusters whose intensities are not greater than a preset threshold. If the close cluster terminates at the end of the entire time series, then the end time is the end time of the entire time series.

[0037] Cluster duration is the duration of a single near cluster, used to reflect the length of a single near eye activity; specifically, the cluster duration is the time difference obtained by subtracting the start time of the near cluster from the end time of the near cluster, and the result is rounded to one decimal place.

[0038] The cluster duration sample set is a dataset composed of the cluster durations of all terminated close clusters, which is used to calculate risk quantification indicators. Specifically, it involves traversing all terminated close clusters, extracting the cluster duration of each close cluster, and arranging these duration values ​​in chronological or size order to form the cluster duration sample set.

[0039] In one embodiment of the present invention, a risk quantification index is calculated based on a specific quantile of the cluster duration sample set, including: Determine the median and 90th percentile of the duration of all clusters in the cluster duration sample set; Calculate the ratio of the 90th percentile to the median; Performing the natural logarithm operation on the constant five yields the first logarithm result; The second logarithm is obtained by taking the natural logarithm of the ratio of the 90th percentile to the median. The risk quantification index is obtained by comparing the first logarithmic result with the second logarithmic result.

[0040] The median is the value in the middle of all cluster durations in the cluster duration sample set after sorting them by numerical value. It is not affected by extreme values ​​and can stably reflect the middle level of cluster duration. Specifically, the first step is to sort all cluster durations in the cluster duration sample set by numerical value from smallest to largest. The second step is to take the single value in the middle position after sorting if the number of samples is odd, and take the average of the two middle values ​​after sorting as the median if the number of samples is even.

[0041] The 90th percentile is the value that corresponds to the cumulative percentage of all cluster durations in the cluster duration sample set after sorting by numerical value. It is used to capture the upper tail feature of the cluster duration distribution. Specifically, the first step is to sort all cluster durations in the cluster duration sample set by numerical value from smallest to largest. The second step is to calculate the value at the (number of samples multiplied by 90%) position after sorting. If the result is an integer, the value at that position is taken. If it is a decimal, the value at that position is rounded up and taken as the 90th percentile.

[0042] The ratio of the 90th percentile to the median is an indicator that reflects the degree of deviation of the upper tail of the cluster duration distribution from the middle level. The larger the ratio, the heavier the upper tail and the higher the proportion of long-duration close-range clusters. Specifically, the ratio of the 90th percentile to the median is obtained by dividing the value of the 90th percentile by the value of the median, and the result is rounded to two decimal places.

[0043] The constant five is a fixed value of five, used as a benchmark for logarithmic operations to match the mapping relationship between quantile ratios and risk quantification; specifically, the value of the constant five is fixed at five and is directly used for natural logarithm operations.

[0044] The first logarithmic result is the value of constant five after natural logarithmic operation, which is used as the numerator in the risk quantification index calculation. Specifically, the constant five is calculated using the natural logarithmic operation method, and the result is the first logarithmic result, which is rounded to three decimal places.

[0045] The second logarithmic result is the ratio of the 90th percentile to the median after natural logarithmic calculation, which is used as the denominator in the risk quantification index calculation. Specifically, the ratio of the 90th percentile to the median is calculated using the natural logarithmic method, and the result is the second logarithmic result, which is rounded to three decimal places.

[0046] The risk quantification index is a dimensionless core indicator that characterizes the intensity of near-field eye use risk. The smaller the value, the more significant the heavy tail of the cluster duration distribution, and the higher the risk of a single long-term near-field eye use. Specifically, the risk quantification index is obtained by dividing the first logarithm result by the second logarithm result, and the result is rounded to two decimal places. This design accurately captures the extreme value proportion of near-field cluster duration through a specific combination of the median and the ninetieth percentile and logarithmic transformation, which is adapted to the physiological mechanism that long-term near-field use is riskier than short-term multiple uses.

[0047] Heavy-tailed distribution is a phenomenon in statistical distributions where the probability of extreme values ​​at the tail is significantly higher than that in a normal distribution. In the sample set of cluster durations, this manifests as follows: most near-field clusters have durations within the relatively short, normal range (corresponding to the main part of the distribution), but there are a small number of long-duration near-field clusters with durations far exceeding the average level (corresponding to the heavy-tailed part of the distribution). These long-duration clusters have a higher probability of occurrence than expected under normal distributions and do not decay rapidly with data accumulation. Their impact on the risk of near-field vision is far greater than that of ordinary short-duration clusters, making them a key distribution characteristic to capture when quantifying the risk of continuous near-field vision use.

[0048] In one embodiment of the present invention, recording the duration of the current close-knit cluster as the current cluster age includes: Set the time interval between adjacent video frames; Based on the judgment result of whether the continuous close-range intensity is greater than a preset threshold, it is determined whether the current moment is in a close-range cluster; The current cluster age is set to record the duration of the current nearest clusters; If the current time is within a close-range cluster, the current cluster age at the previous time is added to the time interval between adjacent video frames to obtain the current cluster age at the current time. If the current time is not within a close cluster, then set the current cluster age to zero.

[0049] The time interval between adjacent video frames refers to the time difference between two consecutive frames in a video sequence, used to accumulate the duration of the current near cluster. Specifically, the time interval between adjacent video frames is set according to the video frame rate. If the video frame rate is 30 frames per second, the value is approximately 0.033 seconds, and if it is 25 frames per second, the value is 0.04 seconds. The frame rate of common classroom monitoring videos is adapted to this range to ensure accurate accumulation of duration.

[0050] Whether the current time is in a near cluster is a binary judgment result representing the current eye use status, with a value of yes or no, used to trigger the accumulation or reset of the current cluster age; specifically, the continuous near intensity at the current time is compared with a preset threshold. If the continuous near intensity is greater than the preset threshold, it is judged that the current time is in a near cluster, otherwise it is judged that the current time is not in a near cluster.

[0051] The current cluster age is a quantitative value that records the duration of the current near cluster, reflecting the continuity of the current near-field eye use behavior. Specifically, the current cluster age is initialized to zero. If it is determined that the current moment is within a near cluster, the current cluster age at the current moment is equal to the current cluster age at the previous moment plus the time interval between adjacent video frames. If it is determined that the current moment is not within a near cluster, the current cluster age is reset to zero. This design updates in real time through frame interval recursion, adapting to the discontinuity of near clusters and accurately tracking the current continuous near-field state.

[0052] In one embodiment of the present invention, the state of reinforcement learning is constituted by a risk quantification index and the current cluster age, voice prompts and no prompts are used as actions, and changes in the risk quantification index, intervention costs, and potential adjustments based on the current cluster age constitute the reward. The reinforcement learning policy is updated using a policy gradient method, including: Perform a logarithmic transformation on the current cluster age; The risk quantification index and the current cluster age after logarithmic transformation are combined to form the state of reinforcement learning; Define a set of actions, including two types of actions: voice prompts and no prompts, and define action variables to take values ​​for the two types of actions; Set the settlement interval for changes in risk quantification indicators; The difference between the current risk quantification index and the risk quantification index at the previous settlement time is calculated as the risk quantification index change item; Set intervention cost weights, and calculate intervention cost items based on intervention cost weights and action variables; Define the potential energy coefficient and discount factor, and define the potential energy function based on the logarithmic transformation result of the current cluster age; The potential energy adjustment term is calculated based on the discount factor and the values ​​of the potential energy function at the next and current time steps; The total return is obtained by adding the risk quantification indicator change item, intervention cost item, and potential energy adjustment item. Define a policy probability function for selecting a voice prompt action, where the input to the policy probability function is the state of reinforcement learning; Calculate the discounted return based on the total return and the discount factor; Set the learning rate; The policy parameters are updated by combining the probability logarithmic discount reward for selecting the voice prompt action and the learning rate using the policy gradient method.

[0053] The current cluster age after logarithmic transformation is a dimensionless value obtained by logarithmically transforming the current cluster age. It is used to reduce the impact of extreme cluster ages on reinforcement learning states and improve state stability. Specifically, the current cluster age is increased by one, and then the result is calculated using the natural logarithm. The result is the current cluster age after logarithmic transformation. The addition operation is to avoid the logarithmic operation being meaningless when the current cluster age is zero.

[0054] The state in reinforcement learning is a two-dimensional state vector that represents the current eye risk and the progress of near vision, and is used to provide a basis for decision-making for reinforcement learning strategies. Specifically, the two-dimensional vector formed by sequentially combining the value of the risk quantification index with the value of the current cluster age after logarithmic transformation is the state of reinforcement learning.

[0055] The action set is the collection of all actions that reinforcement learning can execute. It contains only two mutually exclusive actions to meet the binary decision-making needs of eye intervention. Specifically, the action set always includes two actions: voice prompt and no prompt, which are directly used as optional actions for policy decision-making.

[0056] Action variables are quantified variables that map specific actions in the action set. They take discrete values ​​to simplify strategy calculations. Specifically, action variables take values ​​of zero or one, where zero corresponds to no reminder action and one corresponds to voice reminder action.

[0057] The settlement interval for changes in risk quantification indicators is the time period for calculating changes in risk quantification indicators, used to periodically assess risk change trends. Specifically, the settlement interval for changes in risk quantification indicators is set to 30 to 60 seconds. This range can capture risk changes in a timely manner without causing calculation redundancy due to excessively short intervals, and is suitable for the accumulation rate of classroom eye use risks.

[0058] The risk quantification indicator change item is a numerical value that represents the magnitude of change in eye use risk within a specific period. A positive value represents a decrease in risk, and a negative value represents an increase in risk. Specifically, the risk quantification indicator change item is obtained by subtracting the risk quantification indicator value at the previous settlement time from the current risk quantification indicator value.

[0059] Intervention cost weight is a coefficient that measures the cost of voice reminder intervention and is used to balance the intervention effect and user interference. Specifically, the intervention cost weight is set to 0.1 to 0.3. This range can both constrain excessive reminders and prevent necessary intervention from being suppressed due to excessive weight. It can be adjusted according to the reminder frequency allowed in the classroom.

[0060] Intervention cost item is a numerical value that quantifies the cost of voice reminder intervention, used to reflect the negative cost of intervention in the reward; specifically, the intervention cost item is obtained by multiplying the value of the intervention cost weight by the value of the action variable.

[0061] The potential energy coefficient is a constant that adjusts the strength of the potential energy function and is used to control the degree of influence of the potential energy adjustment term on the total reward. Specifically, the potential energy coefficient is set to 0.5 to 1.5, a range that can accommodate different students' sensitivity to the duration of close proximity, ensuring that the potential energy adjustment term effectively guides the strategy.

[0062] The discount factor is a constant that measures the importance of future returns, with a value between zero and one, used to balance immediate returns and long-term returns. Specifically, the discount factor is set to 0.8 to 0.95, a range that can take into account both the effects of short-term interventions and the accumulation of long-term risks, thus adapting to the time-series decision-making characteristics of reinforcement learning.

[0063] The potential energy function is a function that maps the current cluster age state and is used to transform the long-term benefits of interrupting long-term short-range clusters into an immediate return signal. Specifically, the output value of the potential energy function is obtained by multiplying the value of the potential energy coefficient by the value of the current cluster age after logarithmic transformation.

[0064] The potential energy adjustment term is a numerical value that characterizes the change in the potential energy function and is used to reflect the impact of the current action on future risk in the total return. Specifically, the potential energy adjustment term is obtained by multiplying the value of the discount factor by the output value of the potential energy function at the next time step and then subtracting the output value of the potential energy function at the current time step.

[0065] Total reward is the core feedback signal of reinforcement learning. The larger the value, the better the decision-making effect of the current action. Specifically, the total reward is the sum of the value of the risk quantification indicator change item, the value of the intervention cost item, and the value of the potential energy adjustment item. This design integrates risk improvement, intervention cost, and long-term benefits, and solves the problem of delayed reward for continuous eye use risk.

[0066] The policy probability function for selecting a voice prompt action is a mapping function that outputs the probability of selecting the voice prompt action. The input is the state of reinforcement learning, and the output is a probability value between 0 and 1. Specifically, the policy probability function adopts the form of a logistic function, and the calculation process is: divide one by (one plus the negative of the natural exponent (policy parameter zero plus policy parameter one multiplied by the risk quantification index value plus policy parameter two multiplied by the current cluster age value after logarithmic transformation)). The result is the probability of selecting the voice prompt action.

[0067] Discounted return is a numerical value that quantifies the long-term cumulative return of the current action and is used to guide the strategy to pursue long-term optimality. Specifically, the discounted return is calculated recursively. The discounted return at the current moment is equal to the total return at the current moment plus the value of the discount factor multiplied by the discounted return value at the next moment. The calculation is iterated until the nearest cluster ends.

[0068] The learning rate is a constant that controls the step size of the reinforcement learning policy parameters, used to balance the policy update speed and stability. Specifically, the learning rate is set to 0.001 to 0.01. This range can avoid policy oscillation caused by too large a step size or slow learning caused by too small a step size, and is suitable for the policy iteration needs of classroom scenarios.

[0069] The policy parameters are the parameters to be optimized in the policy probability function for selecting the voice prompt action. They are three-dimensional vectors used to fit the mapping relationship between the state and the action probability. Specifically, the policy parameters are initialized to random small values ​​near zero. When updating, the original policy parameter value is added to the learning rate value and multiplied by (the natural logarithm of the probability of selecting the voice prompt action multiplied by the discounted reward value) to obtain the updated policy parameters.

[0070] The policy gradient method is an algorithm for optimizing the parameters of a reinforcement learning policy. Its core is to adjust the parameters along the direction of increasing reward. Specifically, the execution steps of the policy gradient method are as follows: first, calculate the natural logarithm of the probability of choosing the voice prompt action, then multiply it by the discounted reward to obtain the gradient weight, and finally weight the gradient weight with the learning rate and add it to the original policy parameters to complete one parameter update.

[0071] In one embodiment of the present invention, the decision to provide a voice prompt or not to provide a prompt is output based on the updated reinforcement learning strategy, including: Based on the updated reinforcement learning strategy, calculate the policy probability of selecting the voice prompt action in the current reinforcement learning state; Set a decision threshold between zero and one; Compare the policy probability of voice prompt actions with the magnitude of the decision threshold; If the policy probability of the voice prompt action is greater than or equal to the decision threshold, then output the voice prompt. If the policy probability of the voice prompt action is less than the decision threshold, then output "no prompt".

[0072] The policy probability of choosing the voice prompt action is the likelihood of executing the voice prompt action in the current state, output by the updated reinforcement learning policy, and its value is between zero and one. Specifically, it is calculated using the logistic function, which is one divided by (one plus the negative of the natural exponent (the updated policy parameter zero plus the policy parameter one multiplied by the risk quantification index value plus the policy parameter two multiplied by the current cluster age value after logarithmic transformation)). The result is the policy probability of choosing the voice prompt action.

[0073] The decision threshold is a critical probability value used to determine whether to execute a voice reminder action. It ranges between zero and one and is used to convert the strategy probability into a binary decision. Specifically, the decision threshold is set to 0.5 by default, which is suitable for the balanced binary decision requirement of voice reminder / no reminder. If the frequency of reminders needs to be reduced in the classroom (such as in a self-study scenario), it can be adjusted to 0.6 to 0.7. If stronger intervention is needed (such as in a scenario with younger students), it can be adjusted to 0.3 to 0.4.

[0074] The core logic for determining whether to execute a voice reminder is to compare the policy probability of the voice reminder action with the decision threshold. The output is a binary result of greater than, equal to, or less than. Specifically, the policy probability value of selecting the voice reminder action is compared with the decision threshold value. If the policy probability value is higher than or equal to the decision threshold value, the result is greater than or equal to; if the policy probability value is lower than the decision threshold value, the result is less than.

[0075] Outputting voice prompts is the final intervention action in reinforcement learning. It refers to the educational robot issuing voice prompt signals to guide students to adjust their eye distance or posture. Specifically, when the strategy probability of the voice prompt action is greater than or equal to the decision threshold, the educational robot is triggered to output a voice prompt. The voice content can be set to a concise prompt such as "Please maintain an appropriate reading distance".

[0076] Outputting no prompts is a non-intervention decision in reinforcement learning, meaning that the educational robot does not issue voice prompts to avoid unnecessary interference. Specifically, when the policy probability of the voice prompt action is less than the decision threshold, the educational robot remains silent and does not perform any voice prompt action.

[0077] Example 2: Figure 2 As shown, the reinforcement learning-based educational robot behavior evaluation system, applied to any of the reinforcement learning-based educational robot behavior evaluation methods described herein, includes: The baseline module acquires video sequences, calculates the relative eye-to-book distance based on facial feature scales and the book area in the video sequences, and generates the relative accommodation demand intensity based on the statistical baseline of the relative eye-to-book distance. The cluster set construction module constructs continuous proximity intensity based on the relative adjustment demand intensity, defines continuous intervals with continuous proximity intensity greater than a preset threshold as proximity clusters, and obtains a cluster duration sample set. The cluster set construction module calculates a risk quantification index based on a specific quantile of the cluster duration sample set; it records the duration of the current nearby clusters as the current cluster age. The strategy update module uses risk quantification indicators and the current cluster age to form the state of reinforcement learning, voice prompts and no prompts as actions, and changes in risk quantification indicators, intervention costs, and potential energy adjustments based on the current cluster age as rewards. The reinforcement learning strategy is updated through the strategy gradient method. The intervention decision module outputs a decision on whether to provide a voice prompt or not, based on the updated reinforcement learning strategy.

[0078] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. A method for evaluating the behavior of educational robots based on reinforcement learning, characterized in that, include: Collect video sequences and calculate the relative eye-to-book distance based on facial feature scales and the book area in the video sequences; Based on a statistical baseline of relative eye-to-book distance, the relative intensity of adjustment demand is generated; A continuous near-range intensity is constructed based on the relative adjustment demand intensity. The continuous intervals where the continuous near-range intensity is greater than a preset threshold are defined as near-range clusters, and a sample set of cluster durations is obtained. Based on specific quantiles of the cluster duration sample set, a risk quantification index is calculated; the duration of the current nearby cluster is recorded as the current cluster age. The state of reinforcement learning is defined by risk quantification indicators and the current cluster age. Voice prompts and no prompts are used as actions. The reward is defined by changes in risk quantification indicators, intervention costs, and potential energy adjustments based on the current cluster age. The reinforcement learning strategy is updated through a policy gradient method. Based on the updated reinforcement learning strategy, output a decision on whether to provide a voice prompt or not.

2. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 1, characterized in that, Acquire video sequences, and calculate the relative eye-to-book distance based on facial feature scales and the book area within the video sequences, including: Extract the pixel coordinates of the center of the left pupil and the center of the right pupil from each frame of the video sequence; Calculate the straight-line distance between the center of the left pupil and the center of the right pupil as the pupillary distance in pixels; The book region is obtained by performing region segmentation on the book in the same frame, and the set of boundary pixels of the book region is determined. Calculate the pixel coordinates of the midpoint between the two eyes based on the pixel coordinates of the center of the left pupil and the center of the right pupil; Calculate the straight-line distance from the pixel coordinates of the midpoint of both eyes to each pixel in the set of pixels at the boundary of the book area, and select the smallest straight-line distance as the target pixel distance; The ratio of the target pixel distance to the interpupillary distance pixel is used as the relative eye-book distance.

3. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 2, characterized in that, Based on a statistical baseline of relative eye-to-book distance, the relative intensity of accommodation demand is generated, including: The original value of the adjustment demand intensity is obtained by taking the reciprocal of the relative eye-to-book distance; Determine the statistical baseline time length and define a time interval with the current time as the endpoint and a duration equal to the statistical baseline time length; Extract all raw quantities of adjustment demand intensity within the time interval, and calculate the median of all raw quantities of adjustment demand intensity as the statistical baseline; The relative adjustment demand intensity is obtained by calculating the ratio between the original quantity of adjustment demand intensity and the statistical baseline.

4. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 3, characterized in that, A continuous proximity intensity is constructed based on the relative adjustment demand intensity. Continuous intervals where the continuous proximity intensity is greater than a preset threshold are defined as proximity clusters. A sample set of cluster durations is obtained, including: The continuous short-range intensity is constructed by taking the difference between the relative adjustment demand intensity and the unit constant and taking the result that is not less than zero. Set a preset threshold; Determine whether the continuous near-field intensity is greater than a preset threshold, and mark all moments when the continuous near-field intensity is greater than the preset threshold; Determine the continuous time interval between two consecutive moments when the intensity of the proximity is no greater than a preset threshold, define this continuous time interval as a proximity cluster, and record the start and end times of each proximity cluster. The cluster duration of the corresponding close cluster is obtained by calculating the difference between the end time and the start time of each close cluster. Collect the cluster duration of all nearby clusters to form a cluster duration sample set.

5. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 4, characterized in that, Based on specific quantiles of the cluster duration sample set, risk quantification indicators are calculated, including: Determine the median and 90th percentile of the duration of all clusters in the cluster duration sample set; Calculate the ratio of the 90th percentile to the median; Performing the natural logarithm operation on the constant five yields the first logarithm result; The second logarithm is obtained by taking the natural logarithm of the ratio of the 90th percentile to the median. The risk quantification index is obtained by comparing the first logarithmic result with the second logarithmic result.

6. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 5, characterized in that, Record the duration of the current nearest cluster as the current cluster age, including: Set the time interval between adjacent video frames; Based on the judgment result of whether the continuous close-range intensity is greater than a preset threshold, it is determined whether the current moment is in a close-range cluster; The current cluster age is set to record the duration of the current nearest clusters; If the current time is within a close-range cluster, the current cluster age at the previous time is added to the time interval between adjacent video frames to obtain the current cluster age at the current time. If the current time is not within a close cluster, then set the current cluster age to zero.

7. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 6, characterized in that, The reinforcement learning state is defined by a risk quantification index and the current cluster age. Actions are voice prompts and no prompts. Rewards are determined by changes in the risk quantification index, intervention costs, and potential adjustments based on the current cluster age. The reinforcement learning policy is updated using a policy gradient method, including: Perform a logarithmic transformation on the current cluster age; The risk quantification index and the current cluster age after logarithmic transformation are combined to form the state of reinforcement learning; Define a set of actions, including two types of actions: voice prompts and no prompts, and define action variables to take values ​​for the two types of actions; Set the settlement interval for changes in risk quantification indicators; The difference between the current risk quantification index and the risk quantification index at the previous settlement time is calculated as the risk quantification index change item; Set intervention cost weights, and calculate intervention cost items based on intervention cost weights and action variables; Define the potential energy coefficient and discount factor, and define the potential energy function based on the logarithmic transformation result of the current cluster age; The potential energy adjustment term is calculated based on the discount factor and the values ​​of the potential energy function at the next and current time steps; The total return is obtained by adding the risk quantification indicator change item, intervention cost item, and potential energy adjustment item. Define a policy probability function for selecting a voice prompt action, where the input to the policy probability function is the state of reinforcement learning; Calculate the discounted return based on the total return and the discount factor; Set the learning rate; The policy parameters are updated by combining the probability logarithmic discount reward for selecting the voice prompt action and the learning rate using the policy gradient method.

8. The method for evaluating the behavior of educational robots based on reinforcement learning according to claim 7, characterized in that, Based on the updated reinforcement learning strategy, output a decision on whether to provide a voice prompt or not, including: Based on the updated reinforcement learning strategy, calculate the policy probability of selecting the voice prompt action in the current reinforcement learning state; Set a decision threshold between zero and one; Compare the policy probability of voice prompt actions with the magnitude of the decision threshold; If the policy probability of the voice prompt action is greater than or equal to the decision threshold, then output the voice prompt. If the policy probability of the voice prompt action is less than the decision threshold, then output "no prompt".

9. A reinforcement learning-based educational robot behavior evaluation system, applied in the reinforcement learning-based educational robot behavior evaluation method according to any one of claims 1-8, characterized in that, include: The baseline module acquires video sequences and calculates the relative eye-to-book distance based on facial feature scales and the book area in the video sequences. Based on a statistical baseline of relative eye-to-book distance, the relative intensity of adjustment demand is generated; The cluster set construction module constructs continuous proximity intensity based on the relative adjustment demand intensity, defines continuous intervals with continuous proximity intensity greater than a preset threshold as proximity clusters, and obtains a cluster duration sample set. The cluster set construction module calculates a risk quantification index based on a specific quantile of the cluster duration sample set; it records the duration of the current nearby clusters as the current cluster age. The strategy update module uses risk quantification indicators and the current cluster age to form the state of reinforcement learning, voice prompts and no prompts as actions, and changes in risk quantification indicators, intervention costs, and potential energy adjustments based on the current cluster age as rewards. The reinforcement learning strategy is updated through the strategy gradient method. The intervention decision module outputs a decision on whether to provide a voice prompt or not, based on the updated reinforcement learning strategy.

Images