Fatigue early warning method and device based on visual projection and active appearance model, terminal and medium

Abstract

The invention provides a fatigue early warning method and system based on visual projection and an active appearance model, a terminal and a medium. The method comprises the following steps: carrying out posture straightening preprocessing on a to-be-processed image; performing preliminary positioning on the preprocessed image to be processed based on an integral projection algorithm to extract and obtain an eyebrow region; based on an active appearance model algorithm, performing accurate positioning on the eyebrow region obtained through preliminary positioning so as to extract an eye region; and according to the extracted eye region, determining an eye region area change trend under a plurality of preset acquisition frame rates, so as to perform fatigue early warning judgment according to the eye region area change trend. The monitoring effect can be achieved on the premise that driving of the driver is not affected, and the fatigue driving monitoring effect can be accurately conducted on the driver in real time.

Description

Technical Field

[0001] The present application relates to the field of intelligent judgment of fatigue driving and image processing technology, and in particular to a fatigue warning method, device, terminal and medium based on visual projection and active appearance model. Background Art

[0002] Existing fatigue driving detection methods are mainly divided into contact detection methods and non-contact detection methods, such as heart rate monitoring, lane departure monitoring, head position monitoring, steering wheel monitoring, etc.

[0003] However, both contact detection methods and non-contact detection methods have their own disadvantages: (1) Since the driving environment in which the driver is in is highly restrictive, the existing contact detection methods cannot play a good role and cannot achieve the purpose of preventing fatigue driving to a large extent; (2) The existing non-contact detection methods are difficult to meet the high-coefficient driving safety performance. In order to achieve the supervision effect without affecting the driver's driving, a real-time and accurate driver fatigue driving monitoring system is required.

[0004] With the rise and popularization of new energy vehicles, the requirements for fatigue driving detection are higher, and the existing contact detection methods and non-contact detection methods are difficult to meet the requirements. For example: (1) Today's new energy vehicle fatigue warning detection methods are very restrictive. Contact detection methods cannot play a good role and cannot achieve the purpose of preventing fatigue driving to a large extent. They may also affect the driver's normal driving and have a low safety index. (2) Today's new energy vehicle fatigue warning detection methods have low sensitivity, accuracy, and predictiveness. Existing non-contact detection methods are difficult to meet the high-coefficient driving safety performance. (3) The detection system for new energy vehicles is expensive, takes a long time to detect, and has poor real-time performance.

[0005] Therefore, providing a technical solution that can achieve the supervision effect without affecting the driver's driving and can also monitor the driver's fatigue driving in real time and accurately is a technical challenge commonly faced in this field. Summary of the invention

[0006] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a fatigue warning method, device, terminal and medium based on visual projection and active appearance model, so as to solve the problem of how to provide a technical solution that can achieve the supervision effect without affecting the driver's driving and can monitor the driver's fatigue driving in real time and accurately.

[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a fatigue warning method based on visual projection and active appearance model, including: performing posture correction preprocessing on the image to be processed; performing preliminary positioning of the preprocessed image to be processed based on the integral projection algorithm to extract the eyebrow and eye area; performing precise positioning of the eyebrow and eye area obtained by the preliminary positioning based on the active appearance model algorithm to extract the eye area; determining the eye area area change trend under multiple preset acquisition frame rates based on the extracted eye area, so as to make a fatigue warning judgment based on the eye area area change trend.

[0008] In some embodiments of the first aspect of the present application, the process of performing posture correction preprocessing on the image to be processed includes: extracting the human eye contour from the image to be processed based on the grayscale integral projection algorithm; locating the two end corner points of the human eye contour to calculate the slope data of the two end corner points, and calculating the corresponding inclination angle data based on them; and performing posture correction on the image to be processed according to the inclination angle data so that the human eye in the image is horizontal.

[0009] In some embodiments of the first aspect of the present application, the pre-processed image to be processed is initially positioned based on the integral projection algorithm to extract the eyebrow and eye area, which includes: performing horizontal grayscale integral projection processing on the image to be processed and taking the two points with the smallest extreme points in the horizontal grayscale integral projection diagram as the human eye area and eyebrow area, thereby locating the vertical coordinate of the human eye area; performing vertical grayscale integral projection processing on the image to be processed and taking the points with the minimum values ​​on both sides of the symmetry axis of the symmetrical waveform in the vertical grayscale integral projection diagram as the center points of the left and right eyes, thereby locating the horizontal coordinate of the human eye area.

[0010] In some embodiments of the first aspect of the present application, the active appearance model-based algorithm accurately locates the eyebrow and eye region obtained by preliminary positioning to extract the eye region, and the process includes: separately constructing a shape model and a texture model of the eyebrow and eye region to combine to form an active appearance model; adjusting the shape and texture of the active appearance model corresponding to the eyebrow and eye region accordingly by adjusting a preset model parameter vector, so that the eyebrow and eye region is adjusted to the human eye region.

[0011] In some embodiments of the first aspect of the present application, the process of constructing the shape model includes: normalizing the scale and rotation differences of the shape of the sample image based on the Prucker analysis method to obtain an aligned sample shape vector set; and constructing a shape model on the sample shape vector set based on the principal component analysis method.

[0012] In some embodiments of the first aspect of the present application, the shape and texture of the corresponding active appearance model are adjusted by the following formula: B=Q*C; wherein Q represents the feature vector obtained by principal component analysis, C represents the parameter vector of the appearance model; and B represents the adjusted feature vector.

[0013] In some embodiments of the first aspect of the present application, the eye area change trend under multiple preset acquisition frame rates is determined based on the extracted eye area, which includes: constructing a human eye change curve graph based on the acquired multiple preset acquisition frame rates and the eye area area values ​​under each acquisition frame rate to illustrate the eye area area change trend.

[0014] In some embodiments of the first aspect of the present application, the method of determining fatigue warning based on the trend of changes in the eye area includes: selecting the maximum eye area value from the human eye change curve graph, and comparing the other eye area values ​​in the human eye change curve graph with the maximum eye area value, comparing the obtained ratio with the critical value of the human eye closure state, and determining whether to issue a fatigue warning based on the comparison result.

[0015] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a fatigue warning device based on visual projection and active appearance model, including: a preprocessing module, used to perform posture correction preprocessing on the image to be processed; an integral projection processing module, used to preliminarily locate the preprocessed image to be processed based on the integral projection algorithm to extract the eyebrow and eye area; an active appearance model processing module, used to accurately locate the eyebrow and eye area obtained by the preliminary positioning based on the active appearance model algorithm to extract the eye area; a fatigue warning judgment module, used to determine the eye area area change trend under multiple preset acquisition frame rates based on the extracted eye area, so as to make a fatigue warning judgment based on the eye area area change trend.

[0016] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the fatigue warning method based on visual projection and active appearance model is implemented.

[0017] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides an electronic terminal, including: a processor and a memory; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the fatigue warning method based on visual projection and active appearance model.

[0018] As described above, the fatigue warning method, device, terminal and medium based on visual projection and active appearance model of the present application have the following beneficial effects:

[0019] (1) Compared with the contact detection method, the present invention is more convenient, safer and provides a more comfortable driving experience.

[0020] (2) The present application is a fatigue warning method for new energy vehicles based on visual projection and active appearance model (AAM) with higher sensitivity, higher accuracy and higher predictive power, which perfectly meets the high coefficient safety performance requirements of driving.

[0021] (3) This application can well match the characteristics of new energy vehicle system detection system, such as low cost, shorter detection time, and stronger real-time performance.

[0022] In summary, the present application can not only achieve the supervision effect without affecting the driver's driving, but also monitor the driver's fatigue driving in real time and accurately, achieving a predictive effect with high accuracy, high sensitivity and high safety factor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Shown is a flow chart of a fatigue warning method based on visual projection and active appearance model in one embodiment of the present application.

[0024] Figure 2 Shown is a schematic diagram of the process of performing posture preprocessing on an image to be processed in one embodiment of the present application.

[0025] Figure 3 Shown is a schematic diagram of a process for performing preliminary positioning of a pre-processed image to be processed based on an integral projection algorithm in one embodiment of the present application.

[0026] Figure 4A Shown is a schematic diagram of a horizontal grayscale integral projection diagram in an embodiment of the present application.

[0027] Figure 4B Shown is a schematic diagram of a vertical grayscale integral projection diagram in an embodiment of the present application.

[0028] Figure 5 Shown is a schematic diagram of a process for performing precise positioning based on an active appearance model algorithm in one embodiment of the present application.

[0029] Figure 6 Shown is a schematic diagram of a human eye change curve in one embodiment of the present application.

[0030] Figure 7 Shown is a structural schematic diagram of a fatigue warning device based on visual projection and active appearance model in one embodiment of the present application.

[0031] Figure 8 Shown is a schematic diagram of the structure of an electronic terminal in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0033] It should be noted that in the following description, with reference to the accompanying drawings, several embodiments of the present application are described in the accompanying drawings. It should be understood that other embodiments may also be used, and mechanical composition, structure, electrical and operational changes may be made without departing from the spirit and scope of the present application. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used here are only for describing specific embodiments and are not intended to limit the present application. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0034] In this application, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", "fix", "hold" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0035] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0036] In order to solve the problems in the above-mentioned background technology, the present invention provides a fatigue warning method, system, terminal and medium based on visual projection and active appearance model, which can achieve the effect of supervision without affecting the driver's driving, and can also monitor the driver's fatigue driving in real time and accurately. In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention is further described in detail through the following embodiments and in combination with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the invention.

[0037] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0038] <1> AAM (Active Appearance Model): A face recognition algorithm based on a statistical shape model, which can model a face and achieve face recognition and tracking through model training and optimization. The basic idea of ​​the AAM algorithm is to divide a face into two parts: shape and texture, and achieve face recognition and tracking through shape and texture modeling; the shape model consists of a set of key points, and the texture model consists of a set of feature vectors.

[0039] <2> Integral projection: Each frame of the image in the image sequence is integrally projected in the horizontal and vertical directions to form two integral projection vectors.

[0040] The embodiment of the present invention provides a fatigue warning method based on visual projection and active appearance model, a system for the fatigue warning method based on visual projection and active appearance model, and a storage medium storing an executable program for implementing the fatigue warning method based on visual projection and active appearance model. As for the implementation of the fatigue warning method based on visual projection and active appearance model, the embodiment of the present invention will explain an exemplary implementation scenario of fatigue warning based on visual projection and active appearance model.

[0041] like Figure 1 As shown, a flowchart of a fatigue warning method based on visual projection and active appearance model in an embodiment of the present invention is shown. The fatigue warning method based on visual projection and active appearance model in this embodiment mainly includes the following steps:

[0042] Step S1: Perform posture correction preprocessing on the image to be processed.

[0043] It should be noted that, during driving, due to the special environment in the cab, the driver's head will inevitably move and change the driving posture, which may cause the face image captured by the camera to move or shake, and the integral projection cannot accurately detect the human eyes in the tilted image. In view of this, the embodiment of the present invention performs a pre-processing operation of posture correction before grayscale integral projection.

[0044] Since the tilted human eye has certain geometric characteristics, the tilted human eye image can be straightened using geometric knowledge. The specific process of posture preprocessing of the image to be processed is as follows: Figure 2 As shown, the following steps are included:

[0045] Step S1A: extracting the human eye contour from the image to be processed based on the grayscale integral projection algorithm.

[0046] Specifically, before determining the human eye contour, it is necessary to determine the face area first, and the target detection algorithm can be used to determine the face area from the image to be processed. Take the AdaBoost-based fast target detection algorithm as an example: AdaBoost (Adaptive Boosting) is an iterative algorithm that continuously trains weak classifiers on the training set and combines these weak classifiers to form a strong classifier. The AdaBoost-based fast target detection algorithm can train face targets and collect positive and negative samples. The positive sample is a segmented face target, and the negative sample can be any image that does not contain a positive sample in the image. The positive and negative samples are imported into the classifier for training.

[0047] After determining the face area, the grayscale integral projection algorithm is used to extract the human eye contour in the face area. Specifically, according to the distribution of facial organs, the human eye is in the upper half of the face, so the upper half of the face area can be intercepted first for processing. The grayscale value of the eye area in the face image is usually smaller than the grayscale value of the surrounding area. Using this feature, the integral projection method is often used to locate the eye and extract the human eye contour accordingly.

[0048] Step S1B: locating the eye corner points at both ends of the human eye contour to calculate the slope data of the eye corner points at both ends, and then calculating the corresponding inclination angle data accordingly.

[0049] Specifically, after locating the two end corners of the human eye contour, namely the left end corner and the right end corner, the coordinates of the two end corners are (X1, Y1) and (X2, Y2), respectively. Then the slope k and the tilt angle β are expressed as:

[0050] K=(Y2-Y1) / (X2-X1); Formula (1)

[0051] β = arctan(k); Formula (2)

[0052] Step S1C: adjusting the posture of the image to be processed according to the tilt angle data so that the human eyes in the image are horizontal.

[0053] Specifically, after obtaining the tilt angle data, the image to be processed can be straightened according to the tilt angle data. The line connecting the two end corners of the human eye in the straightened image is a horizontal straight line, while the line connecting the two end corners of the human eye in the unstraightened image is a tilted line.

[0054] Step S2: Preliminary positioning of the preprocessed image to be processed is performed based on the integral projection algorithm to extract the eyebrow and eye region.

[0055] The integral projection algorithm usually performs a preliminary positioning of the human eye based on the grayscale distribution characteristics of the face, and the algorithm can roughly determine the eyebrow area. The grayscale of the facial organs of the face is usually lower than that of other parts, so these organs of the face will produce a minimum value, and then the decision criteria are created using the correlation between the grayscale curves at each minimum value position. The integral projection algorithm includes horizontal grayscale integral projection and vertical grayscale integral projection. In the sample object, the grayscale value at the point (x, y) is defined as G(x, y), the horizontal grayscale integral projection within the [y1, y2] region is H(x), and the vertical grayscale integral projection within the [x1, x2] region is V(y). The expressions of H(x) and V(y) are as follows:

[0056]

[0057]

[0058] After facial image positioning and preprocessing, the integral projection algorithm is used. A certain peak appears in the facial area in the positioning projection result image, and the approximate position of this peak is at the critical left-right direction corresponding to the edge of the face in the left-right direction, which causes the grayscale value to differ between the facial area and the adjacent area.

[0059] Specifically, the pre-processed image to be processed is initially positioned based on the integral projection algorithm to extract the eyebrow area. The process is as follows: Figure 3 Shown include:

[0060] Step S2A: performing horizontal grayscale integral projection processing on the image to be processed and using the two points with the smallest extreme value points in the horizontal grayscale integral projection diagram as the human eye area and eyebrow area, thereby locating the vertical coordinate of the human eye area.

[0061] For ease of understanding, combined Figure 4ATake the horizontal grayscale integral projection diagram as an example: the human eye area and eyebrow area are the two areas closest to black. After the horizontal grayscale integral projection, these two areas correspond to the two points with the smallest extreme points in the horizontal grayscale integral projection diagram. Among them, since the height of the human eye area is lower than the eyebrow area, the point with the smaller extreme point among the two points with the smallest extreme points corresponds to the eye area, and the other point corresponds to the eyebrow area. Therefore, the vertical coordinate of the human eye area can be obtained by the horizontal grayscale integral projection.

[0062] At the left and right edges of the face area, the overall facial brightness value is significantly reduced, and the overall projection result obtained by the integral projection graph shows an obvious peak. Therefore, the final result of the left and right boundaries of the face area can be obtained by obtaining the left and right boundaries of the projection graph.

[0063] Step S2B: Perform vertical grayscale integral projection processing on the image to be processed and use the points with minimum values ​​on both sides of the symmetry axis of the symmetrical waveform in the vertical grayscale integral projection diagram as the center points of the left and right eyes, thereby locating the horizontal coordinate of the human eye area.

[0064] For ease of understanding, combined Figure 4B The vertical grayscale integral projection diagram is used as an example to illustrate: when using vertical integral projection to locate facial images, a symmetrical waveform will appear. With the center point as the boundary, two minimum values ​​will be seen on both sides of the center point of the symmetry axis. These two minimum value points correspond to the center points of the left and right eyes in the facial image, respectively, and are used to locate the horizontal coordinate of the human eye area.

[0065] Step S3: Based on the active appearance model algorithm, the eyebrow and eye region obtained by preliminary positioning is accurately positioned to extract the eye region.

[0066] Generally speaking, the ordinate of the human eye area is determined by horizontal integral projection, and the abscissa of the human eye area is determined by vertical integral projection. The position of the abscissa and ordinate can determine the range of the human eye. However, although the integral projection method can be used to preliminarily locate the human eye area, the eyebrow area of ​​the human body will cause interference, resulting in low accuracy of the selected human eye area. Therefore, the active appearance model algorithm (AAM algorithm) is used for further precise positioning.

[0067] In the embodiment of the present invention, the eyebrow and eye region obtained by preliminary positioning is accurately positioned based on the active appearance model algorithm to extract the eye region, and the process is as follows: Figure 5 The steps shown include the following:

[0068] Step S5A: constructing the eyebrow and eye region shape model and texture model respectively to form an active appearance model.

[0069] The active appearance model algorithm (AAM algorithm) is mainly divided into two stages, the model building stage and the model matching stage. The model building stage includes building a shape model and a texture model for the training samples, and then combining the two models to form an active appearance model. Among them, the model matching stage refers to the process of finding the best matching target in the current frame image of the established active appearance model in the video sequence.

[0070] In the embodiment of the present invention, the shape model and the texture model of the eyebrow and eye region are respectively constructed to form an active appearance model in combination, wherein the shape model and the texture model are independently constructed.

[0071] In some examples, sample selection and calibration are performed before building the model, which includes sample collection, sample point selection and sample data processing. Sample collection refers to the need to collect samples of the target in the process of building an active appearance model. Generally speaking, the more samples there are, the better the prediction effect of the active appearance model, but the longer the model building time is. Sample point selection means that after collecting the samples, only a series of images of target information are obtained, and to build an active appearance model, the target needs to be extracted from the sample by manual calibration to remove the background. Sample data processing refers to saving the position of the sample points in the form of vectors through a program after manually calibrating the points in the sample.

[0072] The process of constructing the shape model includes: normalizing the scale and rotation differences of the shape of the sample image based on the Prucker analysis method to obtain an aligned sample shape vector set; and constructing the shape model on the sample shape vector set based on the principal component analysis method.

[0073] Specifically, in the process of building the shape model, the shape is represented by a vector S. If there are K key points represented in a face, the shape vector is as follows:

[0074] S=(X0,Y0,X1,Y1,X2,Y2...,X K-1 , Y K-1 ); Formula (5)

[0075] Therefore, the shape vectors of all samples will constitute a corresponding vector set. The difference in the face of the sample is manifested in scale and rotation, so the shape of the sample is normalized by Procrustes Analysis to normalize the differences in scale and rotation between the samples, thereby obtaining an aligned sample shape vector set. It should be understood that Procrustes Analysis is an analysis method for shape distribution, and its principle is to find a standard shape through continuous iteration, and use the least squares method to find the affine change of each sample shape to this standard shape.

[0076] Furthermore, subsequent modeling operations can be performed on the sample shape vector set. The projection of the sample to the shape subspace mainly adopts the principal component analysis method, and its formula is as follows:

[0077] S=S+P S b S ; Formula (6)

[0078] Among them, P S represents a matrix composed of a set of orthogonal basis vectors, which represent the shape changes in the sample set; the parameter vector b S Represents the coordinates of the sample in the shape subspace.

[0079] After the shape model is constructed, the corresponding texture model is established. From each sample, the experiment can obtain a texture vector g, and the pixels in the vector sample are arranged according to the grayscale value. The principal component analysis method can be used to project the sample texture, and the area is the corresponding texture subspace. The formula is as follows:

[0080] g=g+P g b g ; Formula (7)

[0081] Among them, g represents the average texture, which refers to the subject's explanation of the consistency of the texture in the sample; P g represents a matrix composed of a set of orthogonal basis vectors, which represent the texture changes in the sample set; the parameter vector b g Represents the coordinates of the sample in the texture subspace.

[0082] Step S5B: adjusting the shape and texture of the active appearance model corresponding to the eyebrow and brow region by adjusting the preset model parameter vector, so that the eyebrow and brow region is adjusted to the human eye region.

[0083] In the embodiment of the present invention, the shape and texture of the model can be adjusted by the following formula:

[0084] B = Q * C;

[0085] Among them, Q represents the feature vector obtained by the principal component analysis method, C represents the parameter vector of the appearance model; B represents the adjusted feature vector; by controlling the parameter vector, the shape and texture of the model can be adjusted, so as to complete the modeling of the active appearance model algorithm, and then apply the face model to make the model closer to the target face image by changing the parameter vector C.

[0086] After roughly locating the face image through the integral projection algorithm, the area that needs to be detected in the face is greatly reduced. Combined with the active appearance model algorithm, the amount of calculation can be reduced further, and the human eye area can be accurately extracted from the eyebrow area, and the needs of secondary accurate and efficient eye positioning can be met. In this way, not only the system detection time is greatly reduced and the detection efficiency is improved, but also the real-time requirements in fatigue driving research are met.

[0087] After completing the above steps, the use of visual projection combined with AAM can be used to locate key points of the face, breaking through the limitations of the sample model when training the database, and making the system run faster and more accurately. To facilitate the description of the advantages of the embodiments of the present invention, the key point positioning algorithm listed in Table 1 below is compared with the grayscale integral projection combined with AAM algorithm in terms of accuracy and detection speed:

[0088] Table 1: Performance comparison of different eye location algorithms

[0089]

[0090] Step S4: determining the eye area area change trend under a plurality of preset acquisition frame rates according to the extracted eye area, so as to make fatigue warning judgment according to the eye area area change trend.

[0091] It should be noted that after completing the above steps, the present invention uses the human eye change curve method to quickly determine fatigue. Under normal circumstances, a person will have an average of 10 to 18 blink reflexes every 60 seconds, and there is often an interval of 2 to 5 seconds between two blinks. The time required for a blink is 0.1 to 0.3 seconds. When a person's eyes are focused or the mind is highly focused, the blink reflex will decrease by an average of 4 to 5 times per minute. After a person shows fatigue, the blinking rate will increase and the time required will slow down. Therefore, the experiment can summarize and count the human eye data, and the blinking frequency data can be used to determine the driver's fatigue state. The human eye change curve, for example Figure 6 shown.

[0092] In some examples, the eye area area change trend under multiple preset acquisition frame rates is determined based on the extracted eye area, which includes: constructing a human eye change curve graph based on the acquired multiple preset acquisition frame rates and the eye area area values ​​under each acquisition frame rate to illustrate the eye area area change trend.

[0093] In some examples, the method of determining fatigue warning based on the eye area area change trend includes: selecting the maximum eye area value from the human eye change curve graph, and comparing the other eye area values ​​in the human eye change curve graph with the maximum eye area value, comparing the obtained ratio with the critical value of the human eye closure state, and determining whether to issue a fatigue warning based on the comparison result.

[0094] Specifically, first assume that the maximum area of ​​the eye region is M, and the ratios of the other eye region area values ​​in the human eye change curve diagram to the maximum value M are calculated respectively, so the ratio n corresponds to the corresponding size parameter of the human eye, and its value should be between 1 and 1; n is compared with the critical value of the human eye closure state (for example, 0.28 can be selected in the embodiment of the present invention); if it is not greater than 0.28, it means that the human eye is not closed and there is no need to issue a fatigue warning; if it is greater than 0.28, it means that the human eye is closed and a fatigue warning needs to be issued.

[0095] In the above, the implementation process and principle of a fatigue warning method based on visual projection and active appearance model provided by the present invention are explained in detail. In the following, another fatigue warning device, medium and terminal based on visual projection and active appearance model provided by the present invention are further described in combination with specific embodiments.

[0096] like Figure 7 , which shows a structural schematic diagram of a fatigue warning device based on visual projection and active appearance model in an embodiment of the present invention.

[0097] In the embodiment of the present invention, the fatigue warning device 700 specifically includes a pre-processing module 701 , an integral projection processing module 702 , an active appearance model processing module 703 and a fatigue warning determination module 704 .

[0098] The preprocessing module 701 is used to perform posture correction preprocessing on the image to be processed. The integral projection processing module 702 is used to perform preliminary positioning of the preprocessed image to be processed based on the integral projection algorithm to extract the eyebrow and eye area. The active appearance model processing module 703 is used to accurately position the eyebrow and eye area obtained by preliminary positioning based on the active appearance model algorithm to extract the eye area. The fatigue warning judgment module 704 is used to determine the eye area area change trend under multiple preset acquisition frame rates based on the extracted eye area, so as to perform fatigue warning judgment based on the eye area area change trend.

[0099] In some examples, the process of the preprocessing module 701 performing posture correction preprocessing on the image to be processed includes: extracting the human eye contour from the image to be processed based on the grayscale integral projection algorithm; locating the two end corner points of the human eye contour to calculate the slope data of the two end corner points, and calculating the corresponding inclination angle data based on them; and performing posture correction on the image to be processed according to the inclination angle data so that the human eye in the image is horizontal.

[0100] In some examples, the integral projection processing module 702 performs preliminary positioning on the preprocessed image to be processed to extract the eyebrow and eye area, which includes: performing horizontal grayscale integral projection processing on the image to be processed and taking the two points with the smallest extreme points in the horizontal grayscale integral projection diagram as the human eye area and eyebrow area, thereby locating the vertical coordinate of the human eye area; performing vertical grayscale integral projection processing on the image to be processed and taking the points with the minimum values ​​on both sides of the symmetry axis of the symmetrical waveform in the vertical grayscale integral projection diagram as the center points of the left and right eyes, thereby locating the horizontal coordinate of the human eye area.

[0101] In some examples, the active appearance model processing module 703 accurately locates the eyebrow and eye area obtained by the preliminary positioning based on the active appearance model algorithm to extract the eye area, and the process includes: separately constructing the shape model and texture model of the eyebrow and eye area to combine to form an active appearance model; by adjusting the preset model parameter vector, the shape and texture of the active appearance model corresponding to the eyebrow and eye area are adjusted accordingly, so that the eyebrow and eye area is adjusted to the human eye area.

[0102] Furthermore, the process of constructing the shape model includes: normalizing the scale and rotation differences of the shapes of the sample images based on the Prucker analysis method to obtain an aligned sample shape vector set; and constructing the shape model on the sample shape vector set based on the principal component analysis method.

[0103] Furthermore, the active appearance model processing module 703 adjusts the shape and texture of the corresponding active appearance model by the following formula: B=Q*C; wherein Q represents the feature vector obtained by principal component analysis, C represents the parameter vector of the appearance model; and B represents the adjusted feature vector.

[0104] In some examples, the fatigue warning determination module 704 determines the eye area area change trend under multiple preset acquisition frame rates based on the extracted eye area, including: constructing a human eye change curve graph based on the acquired multiple preset acquisition frame rates and the eye area area values ​​under each acquisition frame rate to illustrate the eye area area change trend.

[0105] Furthermore, the fatigue warning determination module 704 performs fatigue warning determination according to the eye area area change trend, including: selecting the maximum eye area value according to the human eye change curve diagram, and comparing the other eye area values ​​in the human eye change curve diagram with the maximum eye area value, comparing the obtained ratio with the critical value of the human eye closure state, and determining whether to issue a fatigue warning according to the comparison result.

[0106] It should be noted that: the fatigue warning device based on visual projection and active appearance model provided in the above embodiment only uses the division of the above program modules as an example when performing fatigue warning based on visual projection and active appearance model. In actual application, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the fatigue warning device based on visual projection and active appearance model provided in the above embodiment and the fatigue warning method embodiment based on visual projection and active appearance model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0107] The fatigue warning method based on visual projection and active appearance model provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 8 , is an optional hardware structure diagram of an electronic terminal 800 provided in an embodiment of the present invention. The terminal 800 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The electronic terminal 800 includes: at least one processor 801, a memory 802, at least one network interface 804 and a user interface 806. The various components in the device are coupled together through a bus system 805. It can be understood that the bus system 805 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 805 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 8 In the specification, various buses are labeled as bus systems.

[0108] The user interface 806 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0109] It is understood that the memory 802 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0110] The memory 802 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 800. Examples of these data include: any executable program used to operate on the electronic terminal 800, such as an operating system 8021 and an application 8022; the operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 8022 can include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The fatigue warning method based on visual projection and active appearance model provided in the embodiment of the present invention can be included in the application 8022.

[0111] The method disclosed in the above embodiment of the present invention can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 801 or the instruction in the form of software. The above processor 801 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 801 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0112] In an exemplary embodiment, the electronic terminal 800 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0113] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0114] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0115] In summary, the present application provides a fatigue warning method, device, terminal and medium based on visual projection and active appearance model. Compared with the contact detection method, the present application is more convenient, safer and more comfortable to drive; it is a fatigue warning method for new energy vehicles based on visual projection and active appearance model (AAM) with higher sensitivity, higher accuracy and higher predictability, which perfectly meets the high coefficient safety performance requirements of driving; it can well match the characteristics of new energy vehicle system detection system, such as low cost, shorter detection time and stronger real-time performance; in short, the present application can achieve the supervision effect without affecting the driver's driving, and can also monitor the driver's fatigue driving in real time and accurately, achieving the predictive effect of high accuracy, high sensitivity and high safety factor. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0116] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A fatigue warning method based on visual projection and active appearance model, characterized in that: include: Perform posture correction preprocessing on the image to be processed; Based on the integral projection algorithm, the pre-processed image to be processed is preliminarily positioned to extract the eyebrow and eye area; Based on the active appearance model algorithm, the eyebrow and eye region obtained by preliminary positioning is accurately positioned to extract the eye region; The eye area change trend under a plurality of preset acquisition frame rates is determined according to the extracted eye area, so as to make a fatigue warning judgment according to the eye area change trend.

2. The fatigue warning method based on visual projection and active appearance model according to claim 1 is characterized in that: The process of performing posture correction preprocessing on the image to be processed comprises: Extracting a human eye contour from the image to be processed based on a grayscale integral projection algorithm; Locating the eye corner points at both ends of the human eye contour to calculate the slope data of the eye corner points at both ends, and calculating the corresponding inclination angle data accordingly; The image to be processed is corrected according to the tilt angle data so that the eyes of the human body in the image are horizontal.

3. The fatigue warning method based on visual projection and active appearance model according to claim 1 is characterized in that: The method of performing preliminary positioning of the pre-processed image to be processed based on the integral projection algorithm to extract the eyebrow and eye region includes: Performing horizontal grayscale integral projection processing on the image to be processed and using the two points with the smallest extreme value in the horizontal grayscale integral projection graph as the human eye region and the eyebrow region, thereby locating the vertical coordinate of the human eye region; The image to be processed is subjected to vertical grayscale integral projection processing, and the points with minimum values ​​on both sides of the symmetry axis of the symmetric waveform in the vertical grayscale integral projection diagram are used as the center points of the left and right eyes, thereby locating the horizontal coordinate of the human eye area.

4. The fatigue warning method based on visual projection and active appearance model according to claim 1 is characterized in that: The process of accurately locating the eyebrow and eye region obtained by preliminary positioning based on the active appearance model algorithm to extract the eye region includes: Separately constructing the eyebrow and eye region shape model and texture model to combine and form an active appearance model; The shape and texture of the active appearance model corresponding to the eyebrow and brow region are adjusted accordingly by adjusting the preset model parameter vector, so that the eyebrow and brow region is adjusted to the human eye region.

5. The fatigue warning method based on visual projection and active appearance model according to claim 4 is characterized in that: The process of constructing the shape model includes: normalizing the scale and rotation differences of the shape of the sample image based on the Prucker analysis method to obtain an aligned sample shape vector set; and constructing the shape model on the sample shape vector set based on the principal component analysis method.

6. The fatigue warning method based on visual projection and active appearance model according to claim 4 is characterized in that: The shape and texture of the corresponding active surface model are adjusted by the following formula: B = Q * C; Among them, Q represents the eigenvector obtained by principal component analysis, C represents the parameter vector of the apparent model; and B represents the adjusted eigenvector.

7. The fatigue warning method based on visual projection and active appearance model according to claim 1 is characterized in that: The eye area change trend under multiple preset acquisition frame rates is determined according to the extracted eye area, which includes: constructing a human eye change curve diagram according to the obtained multiple preset acquisition frame rates and the eye area area values ​​under each acquisition frame rate to illustrate the eye area area change trend.

8. The fatigue warning method based on visual projection and active appearance model according to claim 7 is characterized in that: The method of determining fatigue warning based on the eye area area change trend includes: selecting the maximum eye area value from the human eye change curve diagram, and comparing the other eye area values ​​in the human eye change curve diagram with the maximum eye area value, comparing the obtained ratio with the critical value of the human eye closure state, and determining whether to issue a fatigue warning based on the comparison result.

9. A fatigue warning device based on visual projection and active appearance model, characterized in that: include: A preprocessing module, used for performing posture correction preprocessing on the image to be processed; An integral projection processing module is used to perform preliminary positioning of the pre-processed image to be processed based on the integral projection algorithm to extract the eyebrow and eye area; An active appearance model processing module, used for accurately locating the eyebrow and eye region obtained by preliminary positioning based on an active appearance model algorithm to extract the eye region; The fatigue warning determination module is used to determine the eye area area change trend under multiple preset acquisition frame rates according to the extracted eye area, so as to perform fatigue warning determination according to the eye area area change trend.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the fatigue warning method based on visual projection and active appearance model described in any one of claims 1 to 8 is implemented.

11. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the terminal executes the fatigue warning method based on visual projection and active appearance model as described in any one of claims 1 to 8.

Images