Blink monitoring method and system in strong light environment, medium and product

Abstract

A blink monitoring method and system in a strong light environment, a medium and a product, the method comprising: a blink monitoring system determines a user's eyelid contour and a user's pupil visible portion from a user's eye image, predicts a user's pupil invisible portion based on user's complete pupil information, and determines a user's eye image; then calculating a pupil visible proportion, an effective pixel proportion and an eyelid height proportion; when the difference between the pupil visible proportion and the effective pixel proportion is within a preset error range, the blink monitoring system calculates the user eye closing degree corresponding to the user eye image according to the first weight of the eyelid height-width proportion and the second weight of the pupil visible proportion. In a preset time period, the blink monitoring system obtains a plurality of user eye images with the same time interval, and the user eye closing degree of each user eye image is determined in sequence, so that the blink condition of the user in the preset time period is evaluated. By implementing the method, the blinking index of the user can be accurately obtained, so that the fatigue state of the user is predicted.

Description

Technical Field

[0001] The present application relates to the field of blink monitoring, and in particular to a blink monitoring method, system, medium and product in a strong light environment. Background Art

[0002] With the rapid development of science and technology, people's demand for monitoring physiological characteristics continues to rise. Among them, blink monitoring analyzes the user's blinking behavior to reflect the user's fatigue status. In strong light environments, such as outdoor work scenes or driving, in order to ensure personnel safety and improve work efficiency, it is crucial to accurately monitor people's blinking conditions.

[0003] Currently, in the blink monitoring technology under strong light environment, the blink monitoring system obtains the eye image through the infrared camera, and after denoising the eye image, it segments the eyelid area and the pupil area, and calculates the contour changes of the two areas as the features of blink detection.

[0004] However, with the increasing complexity of application scenarios and the continuous improvement of monitoring accuracy requirements, related technologies have gradually exposed many defects. Since the area of ​​the pupil blocked by the eyelid is not taken into account, the eye closure degree is determined only by the pupil area visible in the eye image, which makes it difficult to accurately judge whether the eyes are closed. Ultimately, the calculated blinking index is not accurate enough, making it difficult to predict the user's fatigue state. Summary of the invention

[0005] The present application provides a blink monitoring method, system, medium and product in a strong light environment, which are used to accurately obtain the user's blink index and thus predict the user's fatigue state.

[0006] In a first aspect, the present application provides a blink monitoring method in a strong light environment, which is applied to a blink monitoring system, and the method comprises: periodically acquiring multiple user eye images taken by an infrared camera within a preset time period; determining the user's eyelid contour and the visible part of the user's pupil from the user's eye image; predicting the invisible part of the user's pupil through a preset pupil positioning algorithm based on the visible part of the user's pupil and pre-acquired complete pupil information of the user, the complete pupil information of the user including shape features and size features of the complete pupil of the user, the visible part of the user's pupil and the invisible part of the user's pupil The method comprises the following steps: calculating the pupil visible ratio and the effective pixel ratio, wherein the pupil visible ratio is determined by the area of ​​the visible part of the user's pupil and the area of ​​the invisible part of the user's pupil, and the effective pixel ratio is determined by the number of pixels in the visible part of the user's pupil and the number of pixels in the invisible part of the user's pupil; if the difference between the pupil visible ratio and the effective pixel ratio is not within a preset error range, marking the user's eye image as unavailable; if the difference between the pupil visible ratio and the effective pixel ratio is within the preset error range, determining the eye image of the user according to the first weight of the eyelid height-width ratio and the pupil. a second weight of the pupil visible ratio, calculating the degree of eye closure of the user corresponding to the user eye image, the value of the first weight and the value of the second weight added to 1, the value of the first weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, the value of the second weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, and the eyelid height-to-width ratio is determined by the user's eyelid contour; arranging the multiple user eye images in the order of acquisition time to obtain a user eye image sequence, in which the user eye images in the user eye image sequence correspond to the degree of eye closure of the user one-to-one; based on the user eye image sequence, determining the user's blinking index within the preset time length, the blinking index at least includes the user's eye closure ratio and the user's longest continuous eye closure time, the user's eye closure ratio is determined according to the total number of user eye images and the number of user eye closure images, the user eye closure image is used to represent the user eye image whose eye closure degree exceeds the preset degree threshold, and the user's longest continuous eye closure time is determined according to the cumulative time interval of consecutive user eye closure images in the user eye image sequence.

[0007] By adopting the above technical solution, within a preset time length, the blink monitoring system obtains multiple user eye images with the same time interval to evaluate the user's blinking within the preset time length. The blink monitoring system determines the user's eyelid contour and the visible part of the user's pupil in the user's eye image to predict the invisible part of the user's pupil, which solves the problem that the pupil may be partially blocked or difficult to fully identify in a strong light environment, thereby obtaining complete user pupil information, which is convenient for subsequent calculation of specific pupil visible ratio and effective pixel ratio. When the difference between the pupil visible ratio and the effective pixel ratio is not within the preset error range, it means that the pupil visible ratio and the effective pixel ratio are greatly different, indicating that the user's eye image quality is low, and continuing to use the user's eye image for subsequent pupil analysis may result in inaccurate results. When the difference between the pupil visible ratio and the effective pixel ratio is within the preset error range, it means that the user's eye image quality is high, there is no serious distortion or noise, and the user's eye image can be used for subsequent blinking state analysis. Due to the different areas of the visible part of the user's pupil, the methods used to calculate the degree of user eye closure are different. When the area of ​​the visible portion of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, the degree of eye closure of the user can be determined only based on the visible pupil ratio. This is because pupil information is usually more accurate than eyelid information in the above case, and changes in the pupil more directly reflect the actual opening and closing state of the eye. While simplifying the calculation, the processing speed is improved and a high accuracy is maintained. When the area of ​​the visible portion of the user's pupil is greater than or equal to the preset second area threshold, the degree of eye closure of the user can be determined only based on the eyelid height-to-width ratio. This is because when the pupil is overexposed, it may mean that the eye is in an extremely open state or there are abnormal lighting conditions. In the above case, the position and shape of the eyelid can more accurately reflect the actual opening and closing degree of the eye, and can avoid erroneous judgment caused by abnormal pupil expansion or contraction; when the area of ​​the visible part of the user's pupil does not meet the above two conditions, the user's eye closure degree corresponding to the user's eye image can be calculated according to the first weight of the eyelid height-to-width ratio and the second weight of the pupil visible ratio. This is because when the pupil is partially visible or the lighting conditions are complex, combining the eyelid information and the pupil information can provide a more comprehensive and accurate eye state assessment. Through weight adjustment, the blink monitoring system can flexibly balance the importance of the eyelid information and the pupil information to adapt to different environmental conditions. In the blink monitoring system, multiple user eye images are arranged in the order of acquisition time to obtain a user eye image sequence. Each user eye image corresponds to a user eye closure degree. The user eye image sequence reflects the changes in the user's eye opening and closing state over time, so that the user's blink index can be accurately calculated. Based on the accurate blink index, the user's fatigue state can be predicted.

[0008] In combination with some embodiments of the first aspect, in some embodiments, after the step of determining the user's eyelid contour and the visible part of the user's pupil from the user's eye image, the method further includes: if the area of ​​the visible part of the user's pupil is greater than the preset first area threshold and less than or equal to the preset second area threshold, then determining that the second weight of the visible pupil ratio is inversely correlated with the area of ​​the visible part of the user's pupil; if the area of ​​the visible part of the user's pupil is less than or equal to the preset first area threshold and greater than or equal to the preset third area threshold, then determining that the second weight of the visible pupil ratio is positively correlated with the area of ​​the visible part of the user's pupil; if the area of ​​the visible part of the user's pupil is less than the preset third area threshold, then marking the user's eye image as unavailable.

[0009] By adopting the above technical solution, when the area of ​​the visible part of the user's pupil is greater than the preset first area threshold and less than or equal to the preset second area threshold, the second weight of the visible pupil ratio is inversely correlated with the area of ​​the visible part of the user's pupil, indicating that the larger the area of ​​the visible part of the user's pupil, the smaller the corresponding second weight, and vice versa. This is because when the area of ​​the visible part of the user's pupil is larger, the possibility of eye closure is lower, so the influence of the visible pupil ratio is relatively reduced in the evaluation of the degree of eye closure of the user. When the area of ​​the visible part of the user's pupil is less than or equal to the preset first area threshold and greater than or equal to the preset third area threshold, the second weight of the visible pupil ratio is positively correlated with the area of ​​the visible part of the user's pupil, indicating that the larger the area of ​​the visible part of the user's pupil, the larger the corresponding second weight, and vice versa. This is because when the area of ​​the visible part of the user's pupil is smaller, the influence of the pupil visibility on judging the degree of eye closure of the user becomes more important. When the area of ​​the visible part of the user's pupil is smaller than the preset third area threshold, the user's eye image is marked as unavailable, indicating that the area of ​​the visible part of the user's pupil is extremely small and the pupil is almost invisible, so that the user's eye image cannot be effectively used to evaluate the degree of eye closure of the user. In the above manner, the blink monitoring system can dynamically adjust the second weight of the visible pupil ratio according to the area of ​​the visible part of the user's pupil, thereby improving the accuracy of blink monitoring.

[0010] In combination with some embodiments of the first aspect, in some embodiments, determining the blink index of the user within the preset time period based on the user's eye image sequence specifically includes: determining the user's blink fatigue evaluation standard under different light intensities based on the user's historical blink data, the user's historical blink data including the user's blink index data under the different light intensities within the preset time period, the user's blink fatigue evaluation standard including the correspondence between different blink index threshold intervals and corresponding fatigue degrees; when the light intensity difference between the first light intensity and the second light intensity exceeds the preset light intensity difference threshold, obtaining the light switching adaptation time, the light adaptation time being used to indicate the switching of the first light intensity to the second light intensity. The light switching adaptation time is the sum of the duration of the illumination intensity and the preset human eye adaptation time; if the illumination switching adaptation time does not exceed the preset time threshold, the user eye image obtained during the illumination switching time is marked as unavailable; if the illumination switching adaptation time exceeds the preset time threshold, the user's blink index within the illumination switching adaptation time is determined according to the user eye image sequence within the illumination switching adaptation time; the user's blink index within the preset time is predicted according to multiple user blink indices within the light switching adaptation time and multiple user blink indices within the illumination stabilization time, and the illumination stabilization time is used to indicate that the illumination intensity difference between the first illumination intensity and the second illumination intensity is lower than the preset illumination intensity difference threshold.

[0011] By adopting the above technical solution, the blink monitoring system determines the user's blinking habits based on the user's historical blinking data, thereby formulating a user blinking fatigue assessment standard for different light intensities, making the blink assessment closer to the user's actual situation. When the blink monitoring system detects a significant change in light intensity (i.e., the light intensity difference between the first light intensity and the second light intensity exceeds the preset light intensity difference threshold), the duration of the significant change in light intensity and the preset light adaptation time for the human eye (i.e., light switching adaptation time) are obtained. If the light switching adaptation time is short (that is, the light switching adaptation time does not exceed the preset time threshold), it means that the human eye may need to adapt to the new light intensity through physiological reactions such as pupil adjustment and changes in the retina's sensitivity to light. For example, strong light may cause the pupil to contract and reduce the amount of light entering the eye, while weak light may cause the pupil to dilate. Since the human eye has not yet adapted to the new light intensity, the user's blinking behavior may be affected. The user's eye images acquired within the light switching adaptation time are used for blink assessment, which may result in unstable or inaccurate blink monitoring. Therefore, the blink monitoring system will mark these user eye images as unavailable to avoid affecting the prediction of the user's fatigue state. It is necessary to wait until the human eye adapts to the new light intensity before starting to collect and analyze user eye images, thereby improving the reliability of blink monitoring. If the light switching adaptation time is long (i.e. the light switching adaptation time exceeds the preset time threshold), it means that the human eye can adapt smoothly to the change of light intensity and will not produce a strong physiological reaction of blinking, which has little impact on blink monitoring. The blink monitoring system can determine the user's blinking index within the light switching adaptation time by obtaining the user's eye image within the light switching adaptation time. After dividing the preset time by light intensity, the blink monitoring system deletes the user's eye image corresponding to the shorter light switching adaptation time (i.e. the light switching adaptation time does not exceed the preset time threshold), and combines the user's eye image corresponding to the longer light switching adaptation time (i.e. the light switching adaptation time exceeds the preset time threshold) and the user's eye image corresponding to the stable light time (i.e. the time when the light intensity does not change significantly), to predict the user's blinking index within the preset time, so that the blink monitoring result is more accurate.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the user's eye closure ratio is determined based on the total number of user eye images and the number of user eye closure images, the user eye closure images are used to represent user eye images in which the degree of eye closure of the user exceeds a preset degree threshold, and the user's longest continuous eye closure time is determined based on the cumulative time intervals of consecutive user eye closure images in the user eye image sequence; the user's eye closure index within the preset time length is predicted based on multiple user blinking indicators within the lighting switching adaptation time length and multiple user blinking indicators within the lighting stabilization time length, specifically including: determining the user's eye closure ratio and the user's longest continuous eye closure time corresponding to the lighting switching adaptation time length, as well as the user's eye closure ratio and the user's longest continuous eye closure time corresponding to the lighting stabilization time length; The sum of the first user's eye-closing ratio and the second user's eye-closing ratio is divided by the preset duration to obtain the user's eye-closing ratio within the preset duration, and the first user's eye-closing ratio is used to represent the product of the lighting switching adaptation duration and the corresponding user's eye-closing ratio, and the second user's eye-closing ratio is used to represent the product of the lighting stabilization duration and the corresponding user's eye-closing ratio; the sum of the first user's longest continuous eye-closing time and the second user's longest continuous eye-closing time is divided by the preset duration to obtain the user's longest continuous eye-closing time within the preset duration, and the first user's longest continuous eye-closing time is used to represent the product of the lighting switching adaptation duration and the corresponding user's longest continuous eye-closing time, and the second user's longest continuous eye-closing time is used to represent the product of the lighting stabilization duration and the corresponding user's longest continuous eye-closing time.

[0013] By adopting the above technical solution, the preset duration is divided into multiple light switching adaptation durations and multiple light stabilization durations. The blink monitoring system obtains user blink indicators corresponding to multiple light switching adaptation durations and user blink indicators corresponding to multiple light stabilization durations, that is, the user's blinking response under different lighting conditions, so that the blink evaluation is closer to the user's actual situation, thereby more accurately evaluating the user's fatigue level.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after the step of determining the user's blink index within the preset time length based on multiple blink indexes of the user within the light switching adaptation time lengths and multiple light stabilization time lengths, the method further includes: determining multiple blink index deviation values ​​and multiple blink index change trends based on the user's blink fatigue evaluation criteria under different light intensities and the user's blink index within the preset time length; inputting the multiple blink index deviation values ​​and the multiple blink index change trends into a preset fatigue type classification model to obtain the user fatigue type, and the fatigue type classification model is used to output the probability distribution of the fatigue type.

[0015] By adopting the above technical solution, the blink monitoring system compares the user's blinking indicators within a preset time period (multiple user blinking indicators corresponding to the light switching adaptation time period and multiple user blinking indicators corresponding to the light stabilization time period) with the user blinking fatigue evaluation criteria under different light intensities, determines the blinking indicator deviation value of the user within each light switching adaptation time period and light stabilization time period, and determines the blinking indicator change trend in chronological order to evaluate the user's fatigue type. The blinking indicator deviation value reflects the degree of difference between the user's blinking behavior and normal conditions, while the blinking indicator change trend shows the changes in the user's blinking behavior over time. The blink monitoring system inputs multiple blinking indicator deviation values ​​and the multiple blinking indicator change trends into a preset fatigue type classification model, and the fatigue type classification model can output the probability distribution of fatigue types, thereby improving the accuracy of fatigue type detection.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: sending user fatigue information to a fleet control center, the user fatigue information including a user ID and a corresponding user fatigue type and user fatigue degree; when a growth rate of the fleet's driving fatigue trend is greater than a preset threshold, receiving a stop driving instruction sent by the fleet control center, the fleet's driving fatigue trend is determined by the fleet control center based on multiple pieces of user fatigue information; when the growth rate of the fleet's driving fatigue trend is less than the preset threshold, receiving personalized fatigue management suggestions sent by the fleet control center, the personalized fatigue management suggestions are determined by the fleet control center based on user information, user fatigue type and user fatigue degree.

[0017] By adopting the above technical solution, the blink monitoring system sends user fatigue information to the fleet control center in real time, so that the fleet control center can continuously monitor the fleet's driving fatigue trend. Once the growth rate of the fleet's driving fatigue trend exceeds the preset threshold, it means that many users are fatigued. The fleet control center will immediately trigger the early warning mechanism and send a stop driving command to the blink monitoring system. When the growth rate of the fleet's driving fatigue trend is lower than the preset threshold, it means that many users are fatigued. The fleet control center will provide personalized fatigue management suggestions based on the specific situation of each user (including user information, user fatigue type and user fatigue level). This real-time monitoring and early warning function effectively prevents traffic accidents caused by fatigue driving and ensures road safety.

[0018] In combination with some embodiments of the first aspect, in some embodiments, if the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, calculating the degree of eye closure of the user corresponding to the user eye image according to a first weight of the eyelid height-to-width ratio and a second weight of the visible pupil ratio, specifically including: inputting the eyelid height-to-width ratio, the visible pupil ratio, the first weight, and the second weight into a preset eye closure degree calculation formula to obtain the degree of eye closure of the user corresponding to the user eye image; The calculation formula for the degree of eye closure is: C = αe -EAR +βlog(1+PVR) In the formula, C is the degree of eye closure of the user corresponding to the user's eye image, EAR is the ratio of the eyelid height to width, α is the first weight, PVR is the visible pupil ratio, and β is the second weight.

[0019] By adopting the above technical solution, when the eyelid height-to-width ratio is small, that is, when the eyes are close to a closed state, the exponential decay e is used. -EAR The effect of the eyelid aspect ratio on the degree of eye closure of the user can be magnified because when the eyelid aspect ratio decreases (the eyes are more closed), the -EAR The value of will increase, thereby enhancing the sensitivity to the eye closure state. The exponential function grows very fast when the eyelid height ratio is close to 0 (completely closed), which can help the blink monitoring system quickly respond to the eye closing action and is an effective way to capture rapid changes. When the pupil is fully visible, the pupil visibility ratio is high; when the pupil is partially or completely blocked by the eyelid, the pupil visibility ratio is low. Using the logarithmic function log(1+PVR) can help the blink monitoring system be more sensitive to changes when the pupil visibility ratio is low (that is, when the pupil is less visible). The logarithmic function grows faster when the pupil visibility ratio is close to 0 (the pupil is not visible), which can enhance the blink monitoring system's ability to capture the beginning of the eye closing process. Through this eye closure degree calculation formula, the accuracy of the user's eye closure degree calculation is improved.

[0020] In a second aspect, an embodiment of the present application provides a blink monitoring system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the blink monitoring system to perform the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a blink monitoring system, enables the blink monitoring system to perform the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a blink monitoring system, the blink monitoring system executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] It can be understood that the blink monitoring system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Since a method is adopted to predict the invisible part of the user's pupil based on the visible part of the user's pupil and the pre-acquired complete pupil information of the user, the complete part of the user's pupil can be accurately obtained in a strong light environment, which effectively solves the problem of inaccurate blink detection caused by relying only on the visible pupil area to determine the degree of eye closure in related technologies, thereby achieving a more accurate blink state analysis. This method not only overcomes the difficulty of partial pupil occlusion or difficulty in identification caused by strong light, but also improves the accuracy of pupil recognition by presetting the pupil positioning algorithm. Compared with traditional methods, the present invention can maintain a stable monitoring effect under various complex lighting conditions, greatly improves the reliability and applicability of blink detection, and lays a solid foundation for subsequent fatigue state assessment.

[0025] 2. Since a method is adopted to judge the quality of the user's eye image based on the difference between the visible pupil ratio and the effective pixel ratio, high-quality user eye images can be effectively screened out for subsequent analysis, effectively solving the problem that low-quality images may be used in related technologies to cause inaccurate analysis results, thereby achieving more reliable calculation of blink indicators. This method cleverly balances image quality requirements and data availability by setting a preset error range, which not only ensures the accuracy of the analysis, but also avoids insufficient data due to excessive screening. Compared with traditional methods, the present invention can better cope with image acquisition problems in various complex environments, significantly improves the anti-interference ability and data quality of blink monitoring, and provides more reliable data support for subsequent fatigue status assessment.

[0026] 3. Since the method of dynamically adjusting the eyelid height-to-width ratio and the pupil visible ratio weight according to the area of ​​the user's visible pupil is adopted, it is possible to flexibly select the most appropriate features for analysis under different eye states, effectively solving the analysis deviation problem that may be caused by fixed features in related technologies, and thus achieving a more accurate and comprehensive evaluation of the degree of eye closure. This method achieves refined processing of different degrees of eye opening and closing states by setting multiple area thresholds, which not only fully utilizes the accuracy of pupil information, but also takes into account the stability of eyelid information. Compared with traditional methods, the present invention can better adapt to various eye opening and closing states, significantly improves the accuracy and reliability of blink detection, and provides a more comprehensive and accurate basis for subsequent fatigue state evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of a user's eye image in an embodiment of the present application; Figure 2 It is a flow chart of a blink monitoring method in a strong light environment in an embodiment of the present application; Figure 3 is another flow chart of the blink monitoring method in a strong light environment in an embodiment of the present application; Figure 4 It is a schematic diagram of the structure of a physical device of the blink monitoring system in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0030] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0031] Master Zhang is a long-distance truck driver who needs to drive for more than 12 hours a day. During a transportation mission across the Gobi Desert, strong sunlight shone directly on the car window, making the light in the cockpit extremely dazzling, causing Master Zhang's eyes to feel extremely uncomfortable and he had to blink frequently to relieve it. However, due to long-term exposure to strong light, his eyes gradually became dry and even had temporary blurred vision, which not only affected his driving performance, but also greatly increased safety risks.

[0032] At the same time, logistics companies have also noticed that drivers driving in strong light environments generally report increased fatigue, and some even have slight vision impairment. Therefore, a blink monitoring system is needed to monitor the driver's eye condition in real time in strong light environments to predict the driver's fatigue level and provide effective intervention measures in a timely manner. However, traditional blink monitoring methods often do not work well in strong light environments. Since the area of ​​the pupil blocked by the eyelid is not taken into account, the eye closure degree is determined only by the pupil area visible in the eye image, making it difficult to accurately determine whether the eyes are closed. Ultimately, the calculated blink index is not accurate enough, making it difficult to predict the driver's fatigue state.

[0033] After the logistics company introduced the blink monitoring system of the present invention, the drivers' work experience has been significantly improved. This blink monitoring system determines the user's eyelid contour and the visible part of the user's pupil from the user's eye image, predicts the invisible part of the user's pupil based on the user's complete pupil information, and then calculates the visible pupil ratio, the effective pixel ratio, and the eyelid height ratio. When the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, the blink monitoring system calculates the degree of user eye closure corresponding to the user's eye image based on the first weight of the eyelid height-to-width ratio and the second weight of the visible pupil ratio. Within a preset duration, the blink monitoring system obtains multiple user eye images with the same time interval, and determines the degree of user eye closure for each user eye image in turn to evaluate the user's blinking within the preset duration.

[0034] It can be seen that due to the method of predicting the invisible part of the user's pupil based on the visible part of the user's pupil and the pre-acquired complete pupil information of the user, the complete part of the user's pupil can be accurately obtained in a strong light environment, which effectively solves the problem of inaccurate blink detection caused by relying solely on the visible pupil area to determine the degree of eye closure in the related art, thereby achieving a more accurate blink state analysis. This method not only overcomes the difficulty of partial pupil occlusion or difficulty in identification caused by strong light, but also improves the accuracy of pupil recognition by presetting the pupil positioning algorithm. Compared with traditional methods, the present invention can maintain a stable monitoring effect under various complex lighting conditions, greatly improves the reliability and applicability of blink detection, and lays a solid foundation for subsequent fatigue state assessment.

[0035] Since a method is adopted to judge the quality of the user's eye image based on the difference between the visible pupil ratio and the effective pixel ratio, it is possible to effectively screen out high-quality user eye images for subsequent analysis, effectively solving the problem that the related technology may use low-quality images to cause inaccurate analysis results, thereby achieving a more reliable calculation of blink indicators. This method cleverly balances image quality requirements and data availability by setting a preset error range, which not only ensures the accuracy of the analysis, but also avoids insufficient data due to excessive screening. Compared with traditional methods, the present invention can better cope with image acquisition problems in various complex environments, significantly improves the anti-interference ability and data quality of blink monitoring, and provides more reliable data support for subsequent fatigue status assessment.

[0036] Since a method of dynamically adjusting the eyelid height-to-width ratio and the pupil visible ratio weight according to the area of ​​the user's pupil visible part is adopted, it is possible to flexibly select the most appropriate features for analysis under different eye states, effectively solving the problem of analysis deviation that may be caused by fixed features in related technologies, thereby achieving a more accurate and comprehensive evaluation of the degree of eye closure. This method achieves refined processing of different degrees of eye opening and closing states by setting multiple area thresholds, which not only fully utilizes the accuracy of pupil information, but also takes into account the stability of eyelid information. Compared with traditional methods, the present invention can better adapt to various eye opening and closing states, significantly improves the accuracy and reliability of blink detection, and provides a more comprehensive and accurate basis for subsequent fatigue state evaluation.

[0037] As shown in Figure (1), there are three user eye images (a), (b), and (c), among which the large black circle constitutes the complete eyelid contour of the user, the small black circle constitutes the complete user pupil, the upper small black circle and the large black circle constitute the invisible part of the user's pupil, and the lower large black circle and the small black circle constitute the visible part of the user's pupil.

[0038] In Figure (a), the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, and the degree of user's eye closure can be determined only based on the eyelid height-to-width ratio; in Figure (b), the area of ​​the visible part of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, and the degree of user's eye closure can be determined only based on the pupil visible ratio; in Figure (c), the area of ​​the visible part of the user's pupil is less than the preset first area threshold, and the degree of user's eye closure corresponding to the user's eye image is calculated based on the first weight of the eyelid height-to-width ratio and the second weight of the pupil visible ratio.

[0039] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 2 , is a flow chart of a blink monitoring method in a strong light environment in an embodiment of the present application.

[0040] S201, periodically acquiring multiple user eye images taken by an infrared camera within a preset time period; The preset duration refers to a pre-set fixed time period, which is used to define the duration of blink monitoring, and can be several hours or longer, which is not limited here. Periodic acquisition refers to repeated image acquisition at fixed time intervals, which can be 0.1 seconds or 0.2 seconds, which is not limited here. An infrared camera is an image capturing device used to capture images of the user's eyes. User eye images refer to images containing the user's eyes and the surrounding area, which are used for subsequent blink analysis.

[0041] The blink monitoring system performs this step when it starts blink monitoring. First, the blink monitoring system sets a preset duration, such as 9 hours. Then, the blink monitoring system starts the infrared camera and sets a fixed acquisition interval, such as once every 0.1 seconds. During the entire preset duration (such as 9 hours), the blink monitoring system will continuously acquire user eye images according to the set acquisition interval (once every 0.1 seconds). For example, if the preset duration is 30 minutes and the acquisition interval is once every 1 second, the blink monitoring system will acquire 1,800 user eye images within these 30 minutes.

[0042] S202: determining the eyelid contour of the user and the visible part of the pupil of the user from the user eye image; The user's eyelid contour refers to the edge line of the user's eyelid, which is usually formed by connecting the contours of the upper eyelid and the lower eyelid. The user's pupil visible portion refers to the pupil area that can be directly observed in the user's eye image, which refers to the pupil portion that is not blocked by the eyelid.

[0043] The blink monitoring system performs this step immediately after acquiring each frame of the user's eye image. First, the blink monitoring system preprocesses the acquired user's eye image, such as denoising, contrast enhancement, etc., to improve image quality. Then, the blink monitoring system uses an edge detection algorithm (such as the Canny algorithm) to identify the user's eyelids. For the upper and lower eyelids, the blink monitoring system detects their edges separately and obtains the complete user's eyelid contour through curve fitting. Next, the blink monitoring system uses algorithms such as threshold segmentation or region growing to identify the user's pupil. Since the user's pupil may be partially obscured in some cases, the blink monitoring system will identify the visible part of the user's pupil and accurately calculate its area and position.

[0044] S203, predicting the invisible part of the user's pupil by a preset pupil positioning algorithm based on the visible part of the user's pupil and the pre-acquired complete pupil information of the user, wherein the complete pupil information of the user includes shape features and size features of the complete pupil of the user, and the visible part of the user's pupil and the invisible part of the user's pupil constitute a complete pupil of the user; Among them, the pre-acquired user complete pupil information refers to the information of the user's complete pupil obtained through calibration or a large amount of historical data collection before the start of blink monitoring. The shape feature refers to the geometric shape of the user's complete pupil, which is usually circular or elliptical. The size feature represents the size of the user's complete pupil, including the diameter or the length of the major and minor axes. The preset pupil positioning algorithm refers to an algorithm used to infer the position and shape of the complete user's pupil, which can be an algorithm based on a geometric model or a machine learning model, which is not limited here. The invisible part of the user's pupil refers to the pupil area that is blocked by the eyelids or cannot be directly observed in the user's eye image for other reasons.

[0045] The blink monitoring system performs this step after determining the visible part of the user's pupil. First, the blink monitoring system calls the pre-stored user's complete pupil information and compares the visible part of the user's pupil identified in the user's eye image with the user's complete pupil information. Then, the blink monitoring system uses a preset pupil positioning algorithm to infer the position and shape of the complete user's pupil after combining the shape characteristics, size characteristics of the user's complete pupil, and the curvature, area and position of the visible part of the user's pupil. In this process, the blink monitoring system considers the position and shape of the user's eyelids, because the position and shape of the user's eyelids affect the way the pupil is occluded. The blink monitoring system may generate multiple possible hypotheses of the complete user's pupil, and then select the one that best matches the known information and current image features as the final prediction result. In this way, the blink monitoring system can still accurately estimate the position and shape of the complete user's pupil when the pupil is partially occluded, providing more comprehensive information for subsequent blink analysis.

[0046] S204, calculating the pupil visibility ratio and the effective pixel ratio, where the pupil visibility ratio is determined by the area of ​​the visible part of the user's pupil and the area of ​​the invisible part of the user's pupil, and the effective pixel ratio is determined by the number of pixels in the visible part of the user's pupil and the number of pixels in the invisible part of the user's pupil; Among them, the visible pupil ratio refers to the ratio of the visible pupil area to the entire pupil area, which is used to indicate the degree of pupil exposure. The effective pixel ratio refers to the ratio of effective pixels in the entire pupil area (including the visible part of the user's pupil and the invisible part of the user's pupil), which is used to measure image quality. The area of ​​the visible part of the user's pupil refers to the area of ​​the pupil area directly observed in the user's eye image. The area of ​​the invisible part of the user's pupil refers to the area of ​​the pupil area that is blocked or cannot be directly observed. The number of pixels refers to the total number of pixels that make up a specific area, which is used to quantify the size of a specific area in the user's eye image.

[0047] The blink monitoring system performs this step after predicting the complete user pupil. First, the blink monitoring system calculates the pupil visibility ratio by dividing the area of ​​the visible part of the user's pupil by the predicted area of ​​the complete user's pupil (including the visible part of the user's pupil and the invisible part of the user's pupil). For example, if the area of ​​the visible part of the user's pupil is 60 square centimeters and the area of ​​the complete user's pupil is 100 square centimeters, then the pupil visibility ratio is 0.6. Next, the blink monitoring system counts the number of pixels in the visible part of the user's pupil, estimates the number of pixels in the invisible part of the user's pupil (based on the predicted complete user pupil), adds the two together to get the total number of pixels, and the blink monitoring system evaluates which of these pixels are valid (for example, pixels that are not affected by noise, blur or other interference) to calculate the ratio of the number of valid pixels to the total number of pixels to get the effective pixel ratio.

[0048] S205: if the difference between the visible pupil ratio and the effective pixel ratio is not within a preset error range, mark the user eye image as unavailable; The difference refers to the absolute value of the result of subtracting the effective pixel ratio from the pupil visible ratio, which is used to measure the consistency between the pupil visible ratio and the effective pixel ratio. The preset error range represents a predefined acceptable error interval, which is used to determine whether the difference is within a reasonable range.

[0049] The blink monitoring system performs this step immediately after calculating the visible pupil ratio and the effective pixel ratio. First, the blink monitoring system calculates the difference between the visible pupil ratio and the effective pixel ratio. For example, if the visible pupil ratio is 0.8 and the effective pixel ratio is 0.75, the difference is 0.05. Then, the blink monitoring system compares this difference with a preset error range. This preset error range may be determined based on a large amount of experimental data, such as ±0.1. If the difference falls within this preset error range, the blink monitoring system determines that the image quality is good and continues with subsequent processing. If the difference exceeds this preset error range, the blink monitoring system will mark the user's eye image as unusable because the user's eye image may have severe noise, blur or other quality problems. The user's eye images marked as unusable will be excluded from subsequent blink analysis to ensure the accuracy of the analysis results. The blink monitoring system can also record the number of images marked as unusable. If this number of images exceeds a certain threshold, it may trigger the blink monitoring system to recalibrate or remind the user to adjust the position of the infrared camera.

[0050] S206. If the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, the degree of eye closure of the user corresponding to the user eye image is calculated according to the first weight of the eyelid height-to-width ratio and the second weight of the visible pupil ratio, the value of the first weight added to the value of the second weight is 1, the value of the first weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, the value of the second weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, and the eyelid height-to-width ratio is determined by the eyelid contour of the user; Among them, the eyelid height-to-width ratio refers to the ratio of the eyelid opening height to the width, which is used to indicate the degree of eye opening. The first weight refers to the importance coefficient assigned to the eyelid height-to-width ratio. The second weight refers to the importance coefficient assigned to the pupil visible ratio. The user's eye closure degree indicates the degree to which the user's eyes are closed. Usually, when the user's eye closure degree is 80%, the eyes are considered to be closed. The preset first area threshold and the preset second area threshold refer to two pre-set area boundaries used to determine the weight distribution, which are usually determined based on a large amount of experimental data and are not limited here.

[0051] The blink monitoring system performs this step immediately after confirming that the user's eye image is available. The blink monitoring system first checks the area of ​​the visible portion of the user's pupil and compares it with a preset first area threshold and a preset second area threshold. For example, the preset first area threshold may be 30% of the area of ​​the complete user's pupil, and the preset second area threshold may be 70% of the area of ​​the complete user's pupil. According to the comparison result, the blink monitoring system dynamically adjusts the first weight and the second weight. For example, if the area of ​​the visible part of the user's pupil is less than the preset first area threshold, the first weight of the eyelid height-to-width ratio and the second weight of the pupil visible ratio are used to calculate the degree of eye closure corresponding to the user's eye image. The sum of the values ​​of the first weight and the second weight is 1, but the values ​​of the first weight and the second weight are not 0. If the area of ​​the visible part of the user's pupil is greater than the first preset area threshold and less than the preset second area threshold, the first weight of the eyelid height-to-width ratio is set to 0, and the second weight of the pupil visible ratio is set to 1, and the degree of eye closure corresponding to the user's eye image is determined only based on the pupil visible ratio. If the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, the first weight of the eyelid height-to-width ratio is set to 1, and the second weight of the pupil visible ratio is set to 0, and the degree of eye closure corresponding to the user's eye image is determined only based on the eyelid height-to-width ratio. The blink monitoring system weighted averages these two indicators according to their respective weights to obtain a value between 0 and 1, which indicates the degree of eye closure of the user. 0 may indicate fully open, and 1 indicates fully closed. This dynamic weight adjustment method enables the blink monitoring system to accurately assess the user's eye closure degree under different eye states.

[0052] Optionally, in general, according to the first weight of the eyelid height-to-width ratio and the second weight of the pupil visible ratio, the calculation of the user's eye closure degree corresponding to the user's eye image can also be achieved in the following manner: input the eyelid height-to-width ratio, the pupil visible ratio, the first weight and the second weight into a preset eye closure degree calculation formula to obtain the user's eye closure degree corresponding to the user's eye image; the eye closure degree calculation formula is: C = αe -EAR +βlog(1+PVR), C is the degree of eye closure corresponding to the user's eye image, EAR is the eyelid height-to-width ratio, α is the first weight, PVR is the pupil visible ratio, and β is the second weight.

[0053] When the eyelid height-to-width ratio is small, that is, when the eyes are close to being closed, an exponential decay is used. -EAR The effect of the eyelid aspect ratio on the degree of eye closure of the user can be magnified because when the eyelid aspect ratio decreases (the eyes are more closed), the -EAR The value of will increase, thereby enhancing the sensitivity to the eye closure state. The exponential function grows very fast when the eyelid height ratio is close to 0 (fully closed), which can help the blink monitoring system respond quickly to the eye closing movement and is an effective way to capture rapid changes. When the pupil is fully visible, the pupil visible ratio is high; when the pupil is partially or completely blocked by the eyelid, the pupil visible ratio is low. Using the logarithmic function log(1+PVR) can help the blink monitoring system be more sensitive to changes when the pupil visible ratio is low (that is, when the pupil is less visible). The logarithmic function grows faster when the pupil visible ratio is close to 0 (the pupil is not visible), which can enhance the blink monitoring system's ability to capture the beginning of the eye closing process.

[0054] The preset eye closure degree calculation formula refers to a mathematical expression for calculating the degree of eye closure of the user by comprehensively considering the eyelid height-width ratio and the pupil visible ratio. Exponential decay refers to a mathematical relationship in which the dependent variable decreases exponentially as the independent variable increases, and is used to amplify the impact of small changes. The logarithmic function refers to a function in the form of y=log(x), whose growth rate slows down as x increases, and is used to amplify the sensitivity of changes within a certain range.

[0055] The blink monitoring system performs this step after acquiring the user's eye image and calculating the eyelid height-to-width ratio and the pupil visible ratio. Specifically, the blink monitoring system first determines the values ​​of the first weight and the second weight, which may be set based on a large amount of experimental data and expert experience. For example, α may be set to 0.7 and β may be set to 0.3, indicating that the eyelid height-to-width ratio accounts for 70% of the weight in the calculation, while the pupil visible ratio accounts for 30% of the weight.

[0056] Then, the blink monitoring system substitutes the calculated eyelid height-to-width ratio (EAR), pupil visible ratio (PVR), and the first weight (α) and the second weight (β) into the preset eye closure degree calculation formula to obtain the user's eye closure degree corresponding to the user's eye image. The blink monitoring system performs this calculation for each frame of the user's eye image to obtain a continuous sequence of the user's eye closure degree, and further analyzes the user's blink indicators.

[0057] S207, arranging the plurality of user eye images in a time sequence of acquisition to obtain a user eye image sequence, wherein the user eye images in the user eye image sequence correspond one to one to the degree of eye closure of the user; The time sequence refers to the ordering according to the acquisition time of the user eye images. The user eye image sequence refers to a series of user eye images arranged in time sequence. One-to-one correspondence means that each user eye image has a corresponding user eye closure degree.

[0058] The blink monitoring system performs this step after completing the processing of all individual user eye images. First, the blink monitoring system checks the timestamps of all processed user eye images, which are usually recorded when the images are acquired. Then, the blink monitoring system sorts the user eye images from early to late according to these timestamps to form a sequence of user eye images. During this process, the blink monitoring system retains the degree of user eye closure corresponding to each user eye image to ensure that they maintain the correct correspondence with the user eye image. If any user eye images are marked as unavailable in the previous step, the blink monitoring system may choose to skip or delete these unavailable user eye images. After the sorting is completed, the blink monitoring system obtains a time series data structure containing user eye images and the degree of user eye closure.

[0059] S208. Based on the user eye image sequence, determine the user's blinking index within the preset time length, the blinking index at least including the user's eye closure ratio and the user's longest continuous eye closure time, the user's eye closure ratio is determined based on the total number of user eye images and the number of user eye closure images, the user eye closure image is used to represent the user's eye image with a degree of eye closure exceeding a preset degree threshold, and the user's longest continuous eye closure time is determined based on the cumulative time intervals of consecutive user eye closure images in the user eye image sequence.

[0060] Among them, the blink index refers to a series of numerical values ​​used to quantify the user's blinking behavior, which is used to evaluate the user's eye state and potential fatigue level, including at least the user's eye closure ratio and the user's longest duration. The user's eye closure ratio indicates the proportion of time that the user's eyes are closed within a preset duration, which is used to reflect the frequency of the user's eyes closing. The user's longest continuous eye closure time refers to the longest time that the user's eyes are closed at a single time within a preset duration, which is used to identify possible drowsiness or distraction. The total number of user eye images refers to the number of all valid user eye images collected within a preset duration. The number of user eye closure images indicates the number of user eye images in which the degree of eye closure of the user exceeds the preset degree threshold. The preset degree threshold refers to the critical value of the degree of closure used to determine whether the user's eyes are in a closed state. The time interval refers to the time difference between two adjacent frames of user eye images.

[0061] The blink monitoring system performs this step immediately after obtaining the user's eye image sequence. First, the blink monitoring system calls a preset degree threshold, which may be determined based on a large amount of experimental data. For example, it may be set to 0.8 (indicating that when the user's eyes are closed more than 70%, they are judged to be closed). Then, the blink monitoring system traverses the entire user eye image sequence and counts the total number of user eye images and the number of user closed eye images. In this process, the blink monitoring system records the timestamp of each user eye image that is judged to be closed.

[0062] Next, the blink monitoring system calculates the user's eye closing ratio. The user's eye closing ratio can be obtained by dividing the number of user eye closing images by the total number of user eye images, which is not limited here. For example, if 200 frames of user eye images out of 1000 frames of user eye images are determined to be closed eyes, then the user's eye closing ratio is 20%.

[0063] Then, the blink monitoring system starts to calculate the longest continuous eye closure time of the user by finding the longest continuous user eye closure image segment in the user image sequence. For each continuous user eye closure image segment, the blink monitoring system calculates its duration (the timestamp of the last frame minus the timestamp of the first frame). The blink monitoring system compares the duration of all continuous user eye closure image segments and finds the longest one as the longest continuous eye closure time of the user.

[0064] In addition, the blinking system may also calculate other related indicators, such as average blinking interval (by calculating the average time interval between two consecutive blinks), blinking frequency (number of blinks per minute), etc., which are not limited here. These indicators together constitute a comprehensive blinking behavior evaluation system.

[0065] Optionally, in general, determining the user's blinking index within the preset time period based on the user's eye image sequence can also be achieved in the following manner: determining the user's blinking fatigue evaluation standard under different light intensities based on the user's historical blinking data, the user's historical blinking data including the user's blinking index data under the different light intensities within the preset time period, the user's blinking fatigue evaluation standard including the correspondence between different blinking index threshold intervals and corresponding fatigue degrees; when the light intensity difference between the first light intensity and the second light intensity exceeds the preset light intensity difference threshold, obtaining the light switching adaptation time, the light adaptation time being used to indicate the switching of the first light intensity to the second light intensity the sum of the duration of the light switching adaptation time and the preset human eye adaptation time; if the light switching adaptation time does not exceed the preset time threshold, the user eye image acquired during the light switching adaptation time is marked as unavailable; if the light switching adaptation time exceeds the preset time threshold, the user's blink index within the light switching adaptation time is determined according to the user eye image sequence within the light switching adaptation time; the user's blink index within the preset time is predicted according to multiple blink indices of the users within the light switching adaptation time and multiple blink indices of the users within the light stabilization time, and the light stabilization time is used to indicate that the light intensity difference between the first light intensity and the second light intensity is lower than the preset light intensity difference threshold.

[0066] Among them, the user's historical blink data refers to the record of the user's blinking behavior under different light intensities in the past period of time. Light intensity refers to the brightness of the light in the environment, usually in lux. The user blink fatigue assessment standard under different light intensities refers to a set of blink reference indicators for assessing the user's fatigue level specially formulated for users according to different light intensities. The light switching adaptation time refers to the time when switching from one light intensity to another light intensity plus the time required for the human eye to adapt to the new light intensity. The preset light intensity difference threshold refers to the preset critical value of the light intensity change, which is used to determine whether the light change is significant. The preset human eye adaptation light time refers to the time required for the human eye to adapt from one light environment to another light environment, which is usually determined based on the light intensity difference of the light intensity switch. The preset time threshold refers to the preset minimum effective time, which is used to determine whether the user's eye image is available within the light switching adaptation time. The light stability time refers to a period of time when the light intensity does not change much.

[0067] The blink detection system performs these steps when the user is engaged in activities that require sustained visual attention, such as driving for long periods of time, especially in environments where lighting conditions may change significantly, such as traveling through tunnels or going from indoors to outdoors.

[0068] First, the blink monitoring system establishes a user blink fatigue evaluation standard under different light intensities based on the user's historical blink data. This user blink fatigue evaluation standard under different light intensities takes into account the user's blink behavior characteristics under different light intensities, including blink frequency, blink duration, and blink completeness, which are not limited here. Here is an example: When the ambient light changes significantly (i.e., the difference in light intensity between the first light intensity and the second light intensity exceeds the preset light intensity difference threshold), the blink monitoring system will calculate the light switching adaptation time. The blink monitoring system can continuously monitor the changes in light intensity through the light sensor. When it is detected that the light intensity (first light intensity) begins to change significantly, the blink monitoring system starts the timer and continuously monitors the trend of light intensity changes until the light intensity reaches a new stable level (i.e., the second light intensity). The timing is stopped. This process may be instantaneous (such as suddenly turning on the car lights) or gradual (such as driving into a tunnel). The blink monitoring system will accurately record the duration of this change process. At the same time, the blink monitoring system adds the preset human eye adaptation time to the time of switching from one light intensity to another to obtain the total light switching adaptation time. The following is a preset human eye adaptation time determined according to the light intensity difference of the light intensity switch: Light intensity difference (lux) Preset time for human eyes to adapt to light (seconds) 0-100 5 101-500 10 501-1000 20 1001-5000 30 5001-10000 45 10001-50000 60 50001+ 90 Table 1 The blink monitoring system will look up the corresponding preset human eye adaptation time in the above table based on the actual measured light intensity difference. For example, if the light intensity changes from 500 lux to 6000 lux, the light intensity difference is 5500 lux, and the blink monitoring system will select 45 seconds as the preset human eye adaptation time.

[0069] If the light switching adaptation time is short (compared with the preset time threshold corresponding to the preset human eye adaptation time), the blink monitoring system will mark the user's eye images during this period as "unavailable" because these user eye images may not accurately reflect the user's actual blinking status. If the light switching adaptation time is long enough (compared with the preset time threshold corresponding to the preset human eye adaptation time), the blink monitoring system will analyze the blinking behavior during this period. Here is an example: Assume that the user drives a truck under sunlight from 9:00 a.m. to 6:00 p.m. to perform transportation tasks. The changes in light intensity from 9:00 a.m. to 6:00 p.m. are as follows: Table 2 It can be seen that, assuming that the preset light intensity difference threshold is 2000 lux, the preset human eye adaptation time corresponding to Table 1 determines the preset time threshold to be 20s, and the light switching time obtained when the light intensity difference exceeds the preset light intensity difference threshold exceeds the preset time threshold, indicating that under the changes in light intensity in the above table, the human eye has time to adapt smoothly to the changes in light intensity and will not produce a strong physiological reaction of blinking, which has little impact on blink monitoring. The blink monitoring system can determine the user's blinking index within the light switching adaptation time by obtaining the user's eye image within the light switching adaptation time.

[0070] In the following cases, if the light switching adaptation time does not exceed the preset time threshold, it means that the human eye may need to adapt to the new light intensity through physiological reactions such as pupil adjustment and changes in the retina's sensitivity to light. For example, strong light may cause the pupil to contract and reduce the amount of light entering the eye, while weak light may cause the pupil to dilate. Since the human eye has not yet adapted to the new light intensity, the user's blinking behavior may be affected. The user's eye images obtained during the light switching adaptation time are used for blink assessment, which may result in unstable or inaccurate blink monitoring.

[0071] During a transport mission across the mountains, Master Li was driving a heavy truck on a sunny highway. As Master Li approached the entrance of a long mountain tunnel, the blink monitoring system detected that his blinking behavior began to change. At the moment when Master Li's truck entered the tunnel, he suddenly transitioned from a strong sunlight environment to a dark tunnel environment. The blink monitoring system immediately captured that Master Li's blinking frequency dropped sharply, while the duration of each blink was significantly prolonged. This change indicates that Master Li's eyes are undergoing a rapid adaptation process from bright to dark environments, including a series of physiological reactions such as pupil dilation and increased sensitivity of retinal photoreceptor cells. The blink monitoring system intelligently identified this special situation and determined that Master Li's blinking behavior may be significantly affected in this rapidly changing light environment. Since the user's eye images acquired during this period may cause unstable or inaccurate blink monitoring results, the blink monitoring system automatically marked the user's eye images collected during this period as "unavailable". This marking process lasted for about 15 seconds, which was the critical period for Master Li's eyes to adapt to the dark environment in the tunnel. The blink monitoring system marks the user's eye images within these 15 seconds as "unavailable" to avoid misjudgment due to sudden changes in the environment. When the blink system detects that Master Li's blinking behavior has gradually stabilized, indicating that his eyes have basically adapted to the light environment in the tunnel, it starts collecting and analyzing Master Li's user eye images again. At this time, Master Li's blinking frequency and the duration of each blink have returned to a relatively normal range, although it may still be slightly different from before entering the tunnel.

[0072] During a cross-provincial transportation mission, Master Wang was driving a large truck and crossing a mountainous road section. At that time, Master Wang was driving in a long tunnel with dim ambient light. The blink monitoring system had been steadily recording and analyzing Master Wang's blink data to assess his fatigue status. When Master Wang's truck approached the tunnel exit, the blink monitoring system detected that his blinking behavior began to change. At the moment when the truck drove out of the tunnel, Master Wang suddenly transitioned from the dark tunnel environment to the strong sunlight outside. The blink monitoring system immediately captured that Master Wang's blinking frequency increased sharply, while the duration of each blink was significantly shortened. This change indicates that Master Wang's eyes are undergoing a rapid adaptation process from dark to bright environment, including a series of physiological reactions such as pupil contraction and reduced sensitivity of retinal photoreceptor cells. The blink monitoring system intelligently identified this special situation and determined that Master Wang's blinking behavior may be significantly affected in this rapidly changing light environment. Since the user's eye images acquired during this period may cause unstable or inaccurate blink monitoring results, the blink monitoring system automatically marks the user's eye images collected during this period as "unavailable". This marking process lasted for about 10 seconds, which was the critical period for Master Wang's eyes to adapt to the strong sunlight outside. The blink monitoring system marked the user's eye images within these 10 seconds as "unavailable" to avoid misjudgment due to sudden changes in the environment. When the blink system detected that Master Wang's blinking behavior gradually stabilized, indicating that his eyes had basically adapted to the strong light environment outside, it started to collect and analyze Master Wang's user eye images again. At this time, Master Wang's blinking frequency and the duration of each blink returned to a relatively normal range, although it may still be slightly different from that in the tunnel.

[0073] Through this intelligent data processing method, the blink monitoring system ensures that the assessment of the user's fatigue status remains highly reliable under special circumstances where the light changes dramatically. It not only improves the adaptability of the blink system in various complex lighting environments, but also provides more reliable protection for the user's driving safety.

[0074] Finally, the blink monitoring system comprehensively considers the blink data during the light switching period and the light stable period to predict the user's blink index for the entire preset duration. This method can more accurately assess the user's fatigue state in a changing light environment, thereby providing more precise warnings and suggestions.

[0075] Optionally, under normal circumstances, the prediction of the user's blink index within the preset time length based on multiple users' blink indexes within the light switching adaptation time length and multiple users' blink indexes within the light stabilization time length can be achieved in the following way: determine the user's eye closing ratio and the user's longest continuous eye closing time corresponding to the light switching adaptation time length, as well as the user's eye closing ratio and the user's longest continuous eye closing time corresponding to the light stabilization time length; divide the sum of the first user's eye closing ratio and the second user's eye closing ratio by the preset time length, to obtain the user's eye closing ratio within the preset time length, and the first user's eye closing ratio is used to represent the light switching adaptation time length. The product of the lighting switching adaptation time and the corresponding user eye-closing ratio, and the second user eye-closing ratio is used to represent the product of the lighting stabilization time and the corresponding user eye-closing ratio; the sum of the first user's longest continuous eye-closing time and the second user's longest continuous eye-closing time is divided by the preset time to obtain the user's longest continuous eye-closing time within the preset time, the first user's longest continuous eye-closing time is used to represent the product of the lighting switching adaptation time and the corresponding user's longest continuous eye-closing time, and the second user's longest continuous eye-closing time is used to represent the product of the lighting stabilization time and the corresponding user's longest continuous eye-closing time.

[0076] Other specific implementation methods of predicting the user's blink indicators within the preset time period based on multiple user blink indicators within the light switching adaptation time periods and multiple user blink indicators within the light stabilization time periods are not limited here. By dividing the preset time period into multiple light switching adaptation time periods and multiple light stabilization time periods, the blink monitoring system obtains the user's blink indicators corresponding to the multiple light switching adaptation time periods and the user's blink indicators corresponding to the multiple light stabilization time periods, that is, the user's blinking response under different lighting conditions, so that the blink evaluation is closer to the user's actual situation, thereby more accurately evaluating the user's fatigue level.

[0077] The following is a supplement to the scenario of this embodiment.

[0078] Each vehicle is equipped with a blink monitoring system to monitor the user's blinking behavior. The blink monitoring system will obtain user fatigue information based on the user's blinking behavior and send it to the fleet control center. The fleet control center can manage the user fatigue information of all drivers. For example, in a certain high-altitude area, after receiving information from multiple drivers, the fleet control center found that many drivers had abnormal blinking and increased fatigue. The fleet control center immediately analyzed the characteristics of this area and found that it might be related to the strong ultraviolet radiation on the plateau. The fleet control center immediately sent early warning information to drivers who were about to enter the specific high-altitude area and provided targeted suggestions, such as wearing sunglasses with high protection factors and shortening driving time appropriately.

[0079] The fleet control center can also provide personalized recommendations for strong light protection and fatigue management for different drivers based on their age, vision, light sensitivity and other personal information. For example, for Mr. Chen, who has poor eyesight, the fleet control center generates a recommendation for him to "take a 10-minute break every hour in strong light areas", while for Mr. Liu, who has good eyesight, it is recommended that he can extend his driving time appropriately, but increase his blinking frequency and perform eye massage regularly.

[0080] These optimizations not only further improve driving safety in strong light environments, effectively prevent traffic accidents caused by fatigue driving, ensure road safety, but also greatly improve drivers' job satisfaction.

[0081] After combining the above scenarios, the following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the blink monitoring method in a strong light environment in an embodiment of the present application.

[0082] S301, periodically acquiring multiple user eye images taken by an infrared camera within a preset time period; For details, please refer to step S201, which will not be described in detail here.

[0083] S302, determining the eyelid contour of the user and the visible part of the pupil of the user from the user eye image; For details, please refer to step S202, which will not be described in detail here.

[0084] S303: if the area of ​​the visible portion of the user's pupil is greater than the preset first area threshold and less than or equal to the preset second area threshold, determine that the second weight of the pupil visibility ratio is inversely correlated with the area of ​​the visible portion of the user's pupil; Among them, the preset first area threshold and the preset second area threshold refer to two key values ​​used to divide the area of ​​the visible part of the user's pupil, which are used to determine the weight calculation strategy. Anti-correlation means that there is a negative correlation between two variables, that is, when one variable increases, the other variable decreases. In the embodiment of the present application, when the area of ​​the visible part of the user's pupil is greater than the preset first area threshold and less than or equal to the preset second area threshold, the larger the area of ​​the visible part of the user's pupil, the smaller the second weight of the visible pupil ratio, and conversely, the smaller the area of ​​the visible part of the user's pupil, the larger the second weight of the visible pupil ratio.

[0085] The blink monitoring system performs this step after acquiring the user's eye image and calculating the area of ​​the visible part of the user's pupil. First, the blink monitoring system compares the calculated area of ​​the visible part of the user's pupil with the preset first area threshold and the preset second area threshold. Assume that the preset first area threshold is 60 square pixels and the preset second area threshold is 90 square pixels. If the area of ​​the visible part of the user's pupil falls within this range, the blink monitoring system will establish an anti-correlated pupil visible ratio weight calculation model.

[0086] For example, the blink monitoring system may use the following formula to calculate the second weight of the pupil visibility ratio: second weight = 1-(the area of ​​the visible part of the user's pupil-the preset first area threshold) / (the preset second area threshold-the preset first area threshold). In this way, when the area of ​​the visible part of the user's pupil is the preset first area threshold, the second weight is 1; when the area of ​​the visible part of the user's pupil is the preset second area threshold, the second weight is 0. The purpose of this step is to reduce the influence of the pupil visibility ratio on the calculation of the user's eye closure degree when most of the pupil is visible, because in this case, the eyelid height-width ratio may more accurately reflect the opening and closing state of the eyes.

[0087] S304: if the area of ​​the visible portion of the user's pupil is less than or equal to the preset first area threshold and greater than or equal to the preset third area threshold, determine that the second weight of the pupil visibility ratio is positively correlated with the area of ​​the visible portion of the user's pupil; Among them, the preset third area threshold refers to another key value used to divide the range of the visible pupil area, and together with the preset first area threshold mentioned above, is used to determine different weight calculation strategies, and the preset third area threshold is less than the preset first area threshold and less than the preset second area threshold. Positive correlation means that there is a positive correlation between two variables, that is, when one variable increases, the other variable also increases. In the embodiment of the present application, when the area of ​​the visible part of the user's pupil is less than or equal to the preset first area threshold and greater than or equal to the preset third area threshold, the larger the area of ​​the visible part of the user's pupil, the greater the second weight of the visible pupil ratio, and conversely, the smaller the area of ​​the visible part of the user's pupil, the smaller the second weight of the visible pupil ratio.

[0088] The blink monitoring system performs this step after acquiring the user's eye image and calculating the area of ​​the visible part of the user's pupil. First, the blink monitoring system compares the calculated area of ​​the visible part of the user's pupil with the preset first area threshold and the preset third area threshold. Assume that the preset first area threshold is 60 square pixels (the same as in step S303), and the preset third area threshold is 30 square pixels. If the area of ​​the visible part of the user's pupil falls within this range, the blink monitoring system will establish a positively correlated pupil visible ratio weight calculation model.

[0089] For example, the blink monitoring system may use the following formula to calculate the second weight of the pupil visibility ratio: second weight = (preset first area threshold - area of ​​the visible part of the user's pupil) / (preset first area threshold - preset third area threshold). In this way, when the area of ​​the visible part of the user's pupil is the preset first area threshold, the second weight is 0; when the area of ​​the visible part of the user's pupil is the preset third area threshold, the weight is 1. The purpose of this step is to increase the influence of the pupil visibility ratio on the calculation of the eye closure degree when the pupil is partially blocked, because in this case, the visibility of the pupil may more accurately reflect the opening and closing state of the eye.

[0090] S305: if the area of ​​the visible portion of the user's pupil is smaller than the preset third area threshold, marking the user's eye image as unusable; Among them, unavailable means that the quality of the user's eye image is not sufficient for effective blink analysis and needs to be excluded from subsequent processing. After the blink monitoring system completes the judgment of steps S303 and S304, if both conditions are not met, this step is executed. The blink monitoring system compares the calculated area of ​​the visible part of the user's pupil with the preset third area threshold. If the area of ​​the visible part of the user's pupil is smaller than the preset third area threshold (for example, 30 square pixels), the blink monitoring system will mark the current user's eye image as unavailable.

[0091] The purpose of this step is to filter out poor quality images. When the visible area of ​​the user's pupil is too small, it may be caused by almost completely closed eyes, severe occlusion, or image quality problems. In these cases, the user's eye image may not provide enough information to accurately determine the degree of eye closure.

[0092] The blink monitoring system may mark these unusable user eye images in an internal database or log, recording their timestamps and possible reasons (such as "the visible area of ​​the user's pupil is too small"). These user eye images marked as unusable will be excluded from subsequent analysis to ensure the accuracy of the analysis results. At the same time, the blink monitoring system may trigger a warning or error message to remind the operator or user that the quality of the current user eye image is poor and the position of the infrared camera may need to be adjusted.

[0093] S306, predicting the invisible part of the user's pupil by a preset pupil positioning algorithm based on the visible part of the user's pupil and the pre-acquired complete pupil information of the user, wherein the complete pupil information of the user includes shape features and size features of the complete pupil of the user, and the visible part of the user's pupil and the invisible part of the user's pupil constitute a complete pupil of the user; For details, please refer to step S203, which will not be described in detail here.

[0094] S307, calculating the pupil visibility ratio and the effective pixel ratio, where the pupil visibility ratio is determined by the area of ​​the visible part of the user's pupil and the area of ​​the invisible part of the user's pupil, and the effective pixel ratio is determined by the number of pixels in the visible part of the user's pupil and the number of pixels in the invisible part of the user's pupil; For details, please refer to step S204, which will not be described in detail here.

[0095] S308: If the difference between the visible pupil ratio and the effective pixel ratio is not within a preset error range, mark the user eye image as unavailable; For details, please refer to step S205, which will not be described in detail here.

[0096] S309, if the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, then the degree of eye closure of the user corresponding to the user eye image is calculated according to the first weight of the eyelid height-to-width ratio and the second weight of the pupil visibility ratio, the value of the first weight added to the value of the second weight is 1, the value of the first weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, the value of the second weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, and the eyelid height-to-width ratio is determined by the user's eyelid contour; For details, please refer to step S206, which will not be described in detail here.

[0097] S310, arranging the plurality of user eye images in a time sequence of acquisition to obtain a user eye image sequence, wherein the user eye images in the user eye image sequence correspond one to one to the degree of eye closure of the user; For details, please refer to step S207, which will not be described in detail here.

[0098] S311. Based on the user eye image sequence, determine the user's blinking index within the preset time length, and the blinking index includes at least the user's eye closing ratio and the user's longest continuous eye closing time. The user's eye closing ratio is determined based on the total number of user eye images and the number of user closed eye images. The user closed eye image is used to represent the user's eye image with a degree of eye closure exceeding a preset degree threshold. The user's longest continuous eye closing time is determined based on the cumulative time intervals of consecutive user closed eye images in the user eye image sequence.

[0099] For details, please refer to step S208, which will not be described in detail here.

[0100] S312, determining a plurality of blink indicator deviation values ​​and a plurality of blink indicator change trends according to the user blink fatigue evaluation standard under the different light intensities and the user's blink indicators within the preset time period; Among them, the user blink fatigue assessment standard under different light intensities refers to a set of blink reference indicators specially formulated for users to assess the user's fatigue level according to different light intensities. The blink indicator deviation value refers to the degree of difference between the user's actual blink indicator value and the standard blink indicator value. The blink indicator change trend refers to the direction and magnitude of the change of these blink indicators over time.

[0101] The blink monitoring system performs this step after completing the blinking behavior monitoring of the user within the preset time. First, the blink monitoring system selects the user's blink fatigue assessment standard based on the light intensity of the current environment. For example, in a bright environment, the normal blinking frequency may be 15-20 times per minute, while in a dim environment, the blinking frequency may be reduced to 10-15 times per minute.

[0102] Next, the blink monitoring system will compare the user's blink data collected within a preset period of time (such as the past hour) with the user's blink fatigue assessment criteria under different light intensities. The blink monitoring system may calculate multiple blink indicator deviation values, such as: Blink frequency deviation: the difference between the actual blink frequency and the standard blink frequency; Blink duration deviation value: the difference between the actual blink duration and the standard blink duration; Blink completeness deviation: the difference between the actual blink completeness and the standard blink completeness; (Other blink indicator deviation values)… At the same time, the blink monitoring system also analyzes the changing trends of these blink indicators, such as: Whether the blinking frequency shows an increasing or decreasing trend; Whether the duration of blinks shows an increasing or decreasing trend; Whether the completeness of the blink shows an increasing or decreasing trend; (Other blink indicator trends)… These blink indicator deviation values ​​and blink indicator change trends will be quantified into specific values, which may be expressed as percentages or specific units (such as times / minute, milliseconds, etc.). The blink monitoring system will temporarily store these calculation results in memory for use in subsequent steps.

[0103] S313, inputting the multiple blink indicator deviation values ​​and the multiple blink indicator change trends into a preset fatigue type classification model to obtain the user fatigue type, and the fatigue type classification model is used to output a probability distribution of the fatigue type; Among them, the fatigue type classification model refers to a machine learning model used to determine the user's fatigue type based on the input blink indicator data (multiple blink indicator deviation values ​​and multiple blink indicator change trends). The probability distribution of fatigue types refers to the quantitative evaluation result of the fatigue type classification model on the possibility of different fatigue types. The user fatigue type refers to the classification result of the user's current fatigue state obtained based on the blink behavior analysis.

[0104] The blink monitoring system performs this step immediately after completing step S312, that is, calculating multiple blink indicator deviation values ​​and multiple blink indicator change trends. Specifically, the blink monitoring system first prepares input data, which includes the multiple blink indicator deviation values ​​and multiple blink indicator change trends calculated in S312. For example, the input data may include: Blink frequency deviation value: +2 times / minute; Blink duration deviation value: -50 milliseconds; Blink completeness deviation value: -10%; Blink frequency change trend: increasing, with a slope of 0.1 times / minute / hour; Blink duration change trend: decreasing, with a slope of -5 milliseconds / hour; Blink completeness change trend: decreasing, with a slope of -1% / hour; The blink monitoring system will input these input data into a pre-trained fatigue type classification model. This fatigue type classification model may be a multi-classification machine learning model, such as a random forest, support vector machine, or deep neural network. The fatigue type classification model will process these input data and output a probability distribution representing the probability that the user's current state belongs to various fatigue types.

[0105] For example, fatigue types may include: mild visual fatigue, moderate visual fatigue, severe visual fatigue, mild mental fatigue, moderate mental fatigue, severe mental fatigue, etc. The output of the fatigue type classification model may be a probability distribution like this: Mild visual fatigue: 20%; Moderate visual fatigue: 50%; Severe visual fatigue: 10%; Mild mental fatigue: 15%; Moderate mental fatigue: 5%; Severe mental fatigue: 0%; The blink monitoring system will determine the most likely user fatigue type based on this probability distribution. In this example, the probability of moderate visual fatigue is the highest (50%), so the blink monitoring system may determine the user fatigue type as moderate visual fatigue.

[0106] The following are the steps to build a fatigue type classification model based on deep learning: First, the blink monitoring system collects the historical blink indicator deviation values ​​and historical blink indicator change trends of the user within the past preset time period (for example, within 6 months). In addition, the blink monitoring system records the historical blink indicator deviation values ​​and historical blink indicator change trends within the past preset time period (for example, within 6 months) corresponding to the historical user fatigue type probability distribution. The historical user fatigue type probability distribution can be obtained through expert evaluation and is not limited here. The blink monitoring system stores the collected historical blink indicator deviation values ​​and historical blink indicator change trends and the corresponding historical user fatigue type probability distribution in the data set D, and the format of each data is (historical blink indicator deviation value, historical blink indicator change trend, historical user fatigue type probability distribution). Among them, the historical blink indicator deviation value and the historical blink indicator change trend are the input features of the model training, and the historical user fatigue type probability distribution is the output feature of the model training.

[0107] Then, the blink monitoring system builds a LSTM-based recurrent neural network, which includes an input layer, two LSTM hidden layers, a fully connected layer, and an output layer. The input layer inputs the historical blink indicator deviation value and the historical blink indicator change trend. The number of hidden layer nodes is set to 64, and the number of fully connected layer nodes is set to 32. The output layer outputs the probability distribution of historical user fatigue types.

[0108] Next, the blink monitoring system uses the Adam optimizer, the learning rate is set to 0.001, and the training batch size is 32. It can also be set according to the actual situation and is not limited here. 80% of the historical data is divided into a training set and 20% is divided into a validation set. 100 epochs are trained and the model with the highest accuracy of the validation set is saved. It can also be set according to the actual situation and is not limited here. Epoch is the process in which the training data set passes through the neural network once. In machine learning and deep learning, epoch is a unit used to measure the number of times the entire training set is repeatedly learned. Specifically, when the neural network completes a forward calculation and back propagation process, that is, all data has been processed by the network once, this completes an epoch. The blink monitoring system uses binary cross entropy as the loss function and uses Early Stopping to prevent overfitting. When the value of the loss function exceeds the preset function threshold, the model training is determined to be completed and the fatigue type classification model is obtained. Early Stopping is a technology in deep learning and machine learning to prevent model overfitting. It determines when to stop training by monitoring the performance of the model on the validation set.

[0109] Finally, the blink monitoring system inputs the input features in the validation set into the fatigue type classification model, and then obtains the predicted output of the fatigue type classification model. The predicted output of the fatigue type classification model is compared with the actual output features in the validation set, and some performance indicators such as accuracy, precision, recall, F1 score, mean square error (MSE), etc. are used to evaluate the performance of the fatigue type classification model. According to the performance of the fatigue type classification model on the validation set, the parameters of the fatigue type classification model are adjusted, including adjusting the learning rate, changing the model complexity (such as increasing or decreasing the number of layers or nodes of the neural network), modifying the regularization strength, etc. This process may require multiple iterations, each adjustment based on the previous learning results, so as to optimize the fatigue type classification model.

[0110] S314, sending user fatigue information to the fleet control center, the user fatigue information including the user ID and the corresponding user fatigue type and user fatigue degree; Among them, user fatigue information refers to a set of data that describes the user's current fatigue state, including the user ID and the corresponding user fatigue type and user fatigue degree. The user ID represents a code or identifier used to uniquely identify each user. The user fatigue type refers to the classification result of the user's current fatigue state obtained through blinking behavior analysis, such as visual fatigue, mental fatigue, etc. The user fatigue degree is used to indicate the severity of fatigue, which may include mild, moderate, severe and other levels. The fleet control center refers to the central management terminal responsible for monitoring and managing the operation status of the entire fleet.

[0111] First, the blink monitoring system integrates user fatigue information, including obtaining user ID, user fatigue type, and user fatigue level. For example, the user ID may be "Driver_A001", the user fatigue type may be "visual fatigue", and the user fatigue level may be "severe". The blink monitoring system packages this user fatigue information into a data packet, which may be in JSON format, as shown below: {"user_id":"Driver_A001","fatigue_type":"visual","fatigue_level":"moderate"}. Then, the blink monitoring system establishes a communication connection with the fleet control center, which may be through 4G, 5G or the in-vehicle Wi-Fi network. The blink monitoring system sends this data packet to the fleet control center using a predefined communication protocol. After sending, the blink monitoring system may wait for a confirmation message to ensure that the data packet has been successfully received by the fleet control center. If no confirmation is received, the blink monitoring system may try to resend the data packet.

[0112] S315, when the growth rate of the driving fatigue trend of the fleet is greater than a preset threshold, receiving a stop driving instruction sent by the fleet control center, the driving fatigue trend of the fleet being determined by the fleet control center based on fatigue information of multiple users; The fleet driving fatigue trend refers to the overall change in the fatigue status of the entire fleet driver group. The growth rate indicates how fast the fleet driving fatigue trend changes over time. The preset threshold refers to a pre-set critical value used to determine whether the growth of the fleet driving fatigue trend has reached a level that requires action. The stop driving instruction refers to a command requiring the driver to stop driving and take rest measures.

[0113] After sending user fatigue information, the blink monitoring system keeps the communication channel with the fleet control center open and is ready to receive messages from the fleet control center at any time. When the fleet control center analyzes the overall fatigue trend (i.e., the fleet driving fatigue trend) based on the user fatigue information collected from multiple drivers, and finds that the growth rate exceeds the preset threshold (for example, if the preset threshold is 50%, and the current growth rate reaches 55%), it means that there are a large number of fatigued people in the fleet, and the fleet control center will send a stop driving instruction to all relevant vehicles. The fleet driving fatigue trend can be determined by calculating the total number of users and the total number of fatigued users. The total number of users refers to the total number of user IDs in the user fatigue information, and the total number of fatigued users refers to the total number of users whose user fatigue level exceeds the preset fatigue level threshold. The fleet driving fatigue trend can also be achieved in other ways, which are not limited here.

[0114] When the blink monitoring system receives a stop driving command, it will immediately trigger an alarm inside the vehicle, which may include an audible alarm and a visual prompt on the dashboard. At the same time, the blink monitoring system may display detailed parking and rest instructions on the in-vehicle display, such as the location of the nearest rest area, the recommended rest time, etc. The blink monitoring system will also continue to monitor the driver's response. If it is found that the driver does not respond to the stop driving command in time, it may escalate the alarm level or directly contact the emergency service department.

[0115] S316. When the growth rate of the driving fatigue trend of the fleet is less than the preset threshold, receiving personalized fatigue management suggestions sent by the fleet control center, where the personalized fatigue management suggestions are determined by the fleet control center based on user information, user fatigue type, and user fatigue degree.

[0116] Among them, personalized fatigue management suggestions refer to methods and measures to relieve fatigue customized according to the specific situation of each driver. User information refers to comprehensive data including user personal characteristics, driving habits, etc., which is not limited here.

[0117] When the fleet control center analyzes the overall fatigue trend (i.e., the fleet driving fatigue trend) based on the user fatigue information collected from multiple drivers and finds that the growth rate is lower than the preset threshold (for example, if the preset threshold is 50%, and the current growth rate reaches 40%), it means that there are not many fatigued people in the fleet, and the fleet control center will generate personalized fatigue management suggestions based on the specific situation of each driver. After receiving these personalized fatigue management suggestions, the blink monitoring system will present them to the driver, including displaying text suggestions on the display screen, such as "It is recommended that you take a 15-minute break at the next service area", or broadcasting personalized fatigue management suggestions by voice. If the personalized fatigue management suggestions contain specific operational instructions, such as eye relaxation exercises, the blink monitoring system may start an interactive guidance program to guide the driver to complete these exercises. At the same time, the blink monitoring system will continue to monitor the driver's blinking behavior and evaluate the effectiveness of these personalized fatigue management suggestions. If it is found that the personalized fatigue management suggestions do not significantly improve the driver's fatigue status, the blink monitoring system may feedback this situation to the fleet control center and request updated suggestions or more stringent intervention measures.

[0118] The blink monitoring system in the embodiment of the present invention is described below from the perspective of hardware processing. Figure 4 , is a schematic diagram of a physical device structure of a blink monitoring system in an embodiment of the present application.

[0119] It should be noted that Figure 4 The structure of the blink monitoring system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0120] like Figure 4 As shown, the blink monitoring system includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method described in the above embodiment. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402 and the RAM 403 are connected to each other through the bus 404. The input / output (I / O) interface 405 is also connected to the bus 404.

[0121] The following components are connected to the I / O interface 405: an input section 406 including an audio input device, a button switch, etc.; an output section 407 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0122] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, various functions defined in the present invention are performed.

[0123] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0125] Specifically, the blink monitoring system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the blink monitoring method in a strong light environment provided by the above embodiment is implemented.

[0126] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the blink monitoring system described in the above embodiment; or may exist independently without being assembled into the blink monitoring system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the blink monitoring system, the blink monitoring system implements the blink monitoring method in a strong light environment provided in the above embodiment.

[0127] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0128] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0129] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A blink monitoring method in a strong light environment, characterized in that: Applied to a blink monitoring system, the method comprises: Periodically obtaining multiple user eye images taken by an infrared camera within a preset time period; Determine, from the user's eye image, a user's eyelid contour and a visible portion of the user's pupil; Based on the visible part of the user's pupil and the pre-acquired complete pupil information of the user, predicting the invisible part of the user's pupil by a preset pupil positioning algorithm, wherein the complete pupil information of the user includes shape features and size features of the complete pupil of the user, and the visible part of the user's pupil and the invisible part of the user's pupil constitute a complete user pupil; Calculate the pupil visible ratio and the effective pixel ratio, wherein the pupil visible ratio is determined by the area of ​​the visible part of the user's pupil and the area of ​​the invisible part of the user's pupil, and the effective pixel ratio is determined by the number of pixels in the visible part of the user's pupil and the number of pixels in the invisible part of the user's pupil; If the difference between the visible pupil ratio and the effective pixel ratio is not within a preset error range, marking the user eye image as unavailable; If the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, the degree of user eye closure corresponding to the user eye image is calculated according to the first weight of the eyelid height-to-width ratio and the second weight of the pupil visible ratio, the value of the first weight and the value of the second weight are added to be 1, the value of the first weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset first area threshold and less than the preset second area threshold, the value of the second weight is 0 when the area of ​​the visible part of the user's pupil is greater than or equal to the preset second area threshold, and the eyelid height-to-width ratio is determined by the user's eyelid contour; Arranging the plurality of user eye images according to a time sequence of acquisition to obtain a user eye image sequence, wherein the user eye images in the user eye image sequence correspond one to one to the degree of eye closure of the user; Based on the user eye image sequence, determine the user's blinking index within the preset time length, the blinking index at least includes the user's eye closure ratio and the user's longest continuous eye closure time, the user's eye closure ratio is determined according to the total number of user eye images and the number of user closed eye images, the user closed eye image is used to represent the user's eye image with a degree of eye closure exceeding a preset degree threshold, and the user's longest continuous eye closure time is determined based on the cumulative time intervals of consecutive user closed eye images in the user eye image sequence.

2. The method according to claim 1, characterized in that: After the step of determining the eyelid contour of the user and the visible part of the pupil of the user from the eye image of the user, the method further comprises: If the area of ​​the visible portion of the user's pupil is greater than the preset first area threshold and less than or equal to the preset second area threshold, determining that the second weight of the visible pupil ratio is inversely correlated with the area of ​​the visible portion of the user's pupil; If the area of ​​the visible part of the user's pupil is less than or equal to the preset first area threshold and greater than or equal to the preset third area threshold, the second weight of the pupil visibility ratio is determined to be positively correlated with the area of ​​the visible part of the user's pupil; if the area of ​​the visible part of the user's pupil is less than the preset third area threshold, the user's eye image is marked as unavailable.

3. The method according to any one of claims 1 to 2, characterized in that: The determining, based on the user eye image sequence, a blink index of the user within the preset time period specifically includes: Determine a user blink fatigue assessment standard under different light intensities based on the user's historical blink data, wherein the user's historical blink data includes blink index data of the user under the different light intensities within the preset time period, and the user blink fatigue assessment standard includes a correspondence between different blink index threshold intervals and corresponding fatigue levels; When the illumination intensity difference between the first illumination intensity and the second illumination intensity exceeds a preset illumination intensity difference threshold, obtaining the illumination switching adaptation time, where the illumination adaptation time is used to represent the sum of the time for the first illumination intensity to switch to the second illumination intensity and the preset human eye adaptation illumination time; If the illumination switching adaptation time does not exceed the preset time threshold, the user eye image acquired within the illumination switching time is marked as unusable; If the illumination switching adaptation time exceeds the preset time threshold, determining the user's blink index within the illumination switching adaptation time according to the user's eye image sequence within the illumination switching adaptation time; The blink index of the user within the preset time period is predicted based on the blink index of the user within multiple lighting switching adaptation time periods and the blink index of the user within multiple lighting stabilization time periods, and the lighting stabilization time period is used to indicate that the light intensity difference between the first lighting intensity and the second lighting intensity is lower than the preset light intensity difference threshold.

4. The method according to claim 3, characterized in that The user eye-closing ratio is determined according to the total number of user eye images and the number of user eye-closing images, the user eye-closing image is used to represent the user eye image whose eye closure degree exceeds a preset degree threshold, and the user's longest continuous eye-closing time is determined according to the cumulative time intervals of consecutive user eye-closing images in the user eye image sequence; The predicting the user's blink index within the preset time period according to the multiple blink indexes of the user within the illumination switching adaptation time period and the multiple blink indexes of the user within the illumination stabilization time period specifically includes: Determine the user eye closing ratio and the user's longest continuous eye closing time corresponding to the illumination switching adaptation time, and the user eye closing ratio and the user's longest continuous eye closing time corresponding to the illumination stabilization time; The sum of the first user's eye-closing ratio and the second user's eye-closing ratio is divided by the preset duration to obtain the user's eye-closing ratio within the preset duration, the first user's eye-closing ratio is used to represent the product of the lighting switching adaptation duration and the corresponding user's eye-closing ratio, and the second user's eye-closing ratio is used to represent the product of the lighting stabilization duration and the corresponding user's eye-closing ratio; the sum of the first user's longest continuous eye-closing time and the second user's longest continuous eye-closing time is divided by the preset duration to obtain the user's longest continuous eye-closing time within the preset duration, the first user's longest continuous eye-closing time is used to represent the product of the lighting switching adaptation duration and the corresponding user's longest continuous eye-closing time, and the second user's longest continuous eye-closing time is used to represent the product of the lighting stabilization duration and the corresponding user's longest continuous eye-closing time.

5. The method according to claim 3, characterized in that: After the step of determining the user's blink index within the preset duration according to the plurality of illumination switching adaptation durations and the user's blink index within the plurality of illumination stabilization durations, the method further comprises: Determining a plurality of blink indicator deviation values ​​and a plurality of blink indicator change trends according to the user blink fatigue evaluation criteria under the different light intensities and the user's blink indicators within the preset time period; The multiple blink index deviation values ​​and the multiple blink index change trends are input into a preset fatigue type classification model to obtain the user fatigue type, and the fatigue type classification model is used to output the probability distribution of the fatigue type.

6. The method according to claim 5, characterized in that The method further comprises: Sending user fatigue information to the fleet control center, the user fatigue information including the user ID and the corresponding user fatigue type and user fatigue degree; When the growth rate of the driving fatigue trend of the fleet is greater than a preset threshold, receiving a stop driving instruction sent by the fleet control center, wherein the driving fatigue trend of the fleet is determined by the fleet control center according to the fatigue information of the plurality of users; When the growth rate of the driving fatigue trend of the fleet is less than the preset threshold, a personalized fatigue management suggestion sent by the fleet control center is received, where the personalized fatigue management suggestion is determined by the fleet control center according to user information, user fatigue type and user fatigue degree.

7. The method according to claim 1, characterized in that If the difference between the visible pupil ratio and the effective pixel ratio is within the preset error range, calculating the degree of eye closure of the user corresponding to the user eye image according to the first weight of the eyelid height-to-width ratio and the second weight of the visible pupil ratio, specifically including: Inputting the eyelid height-to-width ratio, the pupil visible ratio, the first weight, and the second weight into a preset eye closure degree calculation formula to obtain the user eye closure degree corresponding to the user eye image; The eye closure degree calculation formula is: C = αe -EAR +βlog(1+PVR) In the formula, C is the degree of eye closure of the user corresponding to the user eye image, EAR is the eyelid height-to-width ratio, α is the first weight, PVR is the pupil visible ratio, and β is the second weight.

8. A blink monitoring system, characterized in that: The blink monitoring system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the blink monitoring system to execute the method described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a blink monitoring system, the blink monitoring system is caused to perform the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a blink monitoring system, the blink monitoring system is caused to perform the method according to any one of claims 1 to 7.

Images