Automatic driving human eye detection method and system based on visual identification, terminal and medium

Abstract

The invention provides an automatic driving human eye detection method and system based on visual identification, a terminal and a medium, and the method comprises the steps: firstly carrying out the human eye inclination correction of a collected image, and then carrying out the preliminary positioning of the corrected image through an integral projection algorithm. And then, secondary positioning is performed according to the preliminary positioning information by means of the shape model, so that a more accurate human eye detection result can be obtained. According to the method, data are acquired only by depending on a visual sensor, and simplified key data are input into a deep learning network for detection through data processing and screening. According to the method, a complex mapping relation does not need to be constructed, and a large amount of marked data is not needed for network training, so that the operand is remarkably reduced. Moreover, the system detection time is greatly shortened, the detection efficiency is improved, and the harsh requirement for real-time performance in intelligent safe driving research is met.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, system, terminal, and medium for detecting human eyes in autonomous driving based on visual recognition. Background Technology

[0002] Autonomous driving technology, as a key application of artificial intelligence, has made significant progress. However, many problems remain in the development of autonomous vehicles, the most important of which is safety. When people ride in autonomous vehicles, the driver must constantly monitor road conditions and the vehicle to ensure the safety of both the driver and pedestrians. If the driver is distracted, not looking ahead, drowsy, or has a medical condition, traffic accidents may occur. Therefore, to improve the safety and reliability of autonomous vehicles, real-time monitoring of the driver's eye movements has been implemented.

[0003] When implementing human eye detection, autonomous driving systems typically utilize a combination of sensors, including cameras, LiDAR, and millimeter-wave radar. However, parameter transfer between sensors can introduce cascading errors, increasing data processing complexity and systemic errors. Another strategy relies on deep neural network models to process massive amounts of data, but this is not only time-consuming but also places higher demands on computing resources. Furthermore, deep neural network models require the construction of complex mapping relationships to accurately identify and analyze human eye features, which undoubtedly increases the difficulty and cost of model construction and training. Crucially, the training and optimization of this model depend heavily on a large amount of high-quality labeled data, which is extremely difficult to obtain in practical applications, and the labeling process is also extremely time-consuming and labor-intensive. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method, system, terminal and medium for human eye detection in autonomous driving based on visual recognition, which solves the technical problems of low detection accuracy and low detection efficiency caused by large errors in parameter transmission from multiple sensors, high difficulty in model construction and difficulty in obtaining high-quality labeled data in the human eye detection technology used in existing autonomous driving systems.

[0005] To achieve the above and other related objectives, the present invention provides a visual recognition-based method for human eye detection in autonomous driving. The method includes: real-time acquisition of a driver's facial image in the cab of a target vehicle; performing human eye tilt correction on the acquired driver's facial image to obtain a corrected image; performing preliminary human eye localization on each corrected image based on an integral projection algorithm, and extracting preliminary human eye localization information from each corrected image; and obtaining a human eye detection result from the corrected image with the extracted preliminary human eye localization information based on a shape model.

[0006] In one embodiment of the present invention, the data transmission command corresponds to a transmission type, which includes: data address, data bit width, and data transmission length.

[0007] In one embodiment of the present invention, the step of performing human eye tilt correction on the acquired driver facial image to obtain a corrected image includes: performing human eye contour detection on the currently acquired driver facial image and extracting the position information of multiple key points; calculating the eye corner curvature using the position information of each key point and calculating the illumination tilt angle based on the eye corner curvature; and performing geometric correction on the corresponding driver facial image based on the calculated illumination tilt angle to obtain a corrected image.

[0008] In one embodiment of the present invention, the preliminary human eye localization of each corrected image based on the integral projection algorithm and the extraction of preliminary human eye localization information of each corrected image include: performing horizontal integral projection and vertical integral projection on each corrected image, and obtaining the coordinate information of the two eyes initially determined in the corrected image.

[0009] In one embodiment of the present invention, the step of performing horizontal and vertical integral projection on each corrected image and obtaining the coordinate information of two human eyes in the corrected image includes: performing horizontal and vertical integral projection on the current corrected image to obtain corresponding horizontal and vertical integral projection result images; extracting a minimum point of the relevant waveform in the horizontal integral projection image to obtain the corresponding preliminary ordinates of the two human eyes; and extracting two minimum points on both sides of the central axis of symmetry of the relevant symmetrical waveform in the vertical integral projection image of the current corrected image to obtain the corresponding preliminary abscissas of the two human eyes.

[0010] In one embodiment of the present invention, obtaining human eye detection results from the corrected images from which preliminary human eye positioning information is extracted based on a shape model includes: selecting multiple images as target images from the corrected images from which preliminary human eye positioning information is extracted within a set time period; and performing human eye detection based on each input target image according to the shape model to obtain the corresponding human eye detection results.

[0011] In one embodiment of the present invention, the human eye detection based on the input target images includes: constructing a shape vector set for each input target image; wherein the shape vector set includes: Euclidean distance vectors involving key points of each human eye, vectors under scale measurement, and vectors under curl measurement; normalizing the differences in scale and curl of the constructed shape vector sets of each target image by a three-layer sample analysis method to obtain an aligned shape vector set; and outputting the human eye detection result based on the model parameter vectors and the aligned shape vector sets of each target image.

[0012] In one embodiment of the present invention, the model parameter vector is adjusted during shape model training. The adjustment method includes: constructing shape vector sets for driver face image samples marked with key points of each human eye, and obtaining the aligned shape vector set of each driver face image sample by normalizing the differences in scale and curl through a three-layer sample analysis method, and then adjusting the model parameter vector based on the difference between the aligned shape vector set of each driver face image sample and the corresponding target human eye vector; wherein, the target human eye vector includes: the aligned shape vector set involving the key points of each human eye marked in the corresponding driver face image sample.

[0013] To achieve the above and other related objectives, this invention provides a vision-based autonomous driving human eye detection system. The system includes: an image acquisition module for real-time acquisition of driver facial images within the cab of a target vehicle; a correction module connected to the image acquisition module for performing human eye tilt correction on the acquired driver facial images to obtain corrected images; an integral projection module connected to the correction module for preliminary human eye localization of each corrected image based on an integral projection algorithm, and extracting preliminary human eye localization information from each corrected image; and a detection module connected to the integral projection module for obtaining human eye detection results from the corrected images with extracted preliminary human eye localization information based on a shape model.

[0014] To achieve the above and other related objectives, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store a computer program; the one or more processors are connected to the memories and are used to run the computer program to execute the vision recognition-based autonomous driving human eye detection method.

[0015] To achieve the above and other related objectives, the present invention provides a computer-readable storage medium storing a computer program that is executed by one or more processors to perform the method.

[0016] As described above, this invention provides a vision-based method, system, terminal, and medium for human eye detection in autonomous driving, offering the following advantages: First, the invention corrects for human eye tilt in the acquired images, then uses an integral projection algorithm to perform preliminary localization on the corrected images. Next, a shape model is used to perform secondary localization based on the preliminary localization information, resulting in more accurate human eye detection. This invention relies solely on visual sensors to collect data, and through data processing and filtering, the simplified key data is input into a deep learning network for detection. This method eliminates the need to construct complex mapping relationships or require a large amount of labeled data for network training, significantly reducing computational load. Furthermore, it not only drastically shortens system detection time and improves detection efficiency but also meets the stringent real-time requirements of intelligent safe driving research. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of a vision recognition-based autonomous driving human eye detection method according to an embodiment of the present invention.

[0018] Figure 2 The diagram shown is a schematic representation of relevant waveforms in a horizontal integral projection diagram according to an embodiment of the present invention.

[0019] Figure 3 The diagram shown is a schematic representation of the relevant symmetrical waveforms in a vertical integral projection diagram according to an embodiment of the present invention.

[0020] Figure 4 The diagram shown is a structural schematic of an autonomous driving human eye detection system based on visual recognition according to an embodiment of the present invention.

[0021] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0024] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.

[0025] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.

[0026] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0027] This invention provides a vision-based method for human eye detection in autonomous driving. First, the acquired image undergoes eye tilt correction, followed by preliminary localization using an integral projection algorithm. Then, a shape model is used to perform secondary localization based on the preliminary localization information, resulting in more accurate eye detection. This invention relies solely on visual sensors to collect data, and through data processing and filtering, the simplified key data is input into a deep learning network to achieve human eye detection. This method eliminates the need for complex mapping relationships in the model and extensive network training with labeled data, significantly reducing computational load. Furthermore, it not only drastically shortens system detection time and improves detection efficiency but also meets the stringent real-time requirements of intelligent safe driving research.

[0028] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0029] like Figure 1 This is a flowchart illustrating a vision recognition-based human eye detection method for autonomous driving, as shown in an embodiment of the present invention.

[0030] The method includes:

[0031] Step S1: Real-time acquisition of facial images of the driver inside the target vehicle's cab.

[0032] In one embodiment, a visual sensor device installed inside or outside the vehicle's driver's cab (e.g., close to the inside of the window or through the window using an external camera) is used to continuously and instantly capture images of the driver's face in the target vehicle being driven; the visual sensor device can be a camera, a smart visual sensor, or other sensor devices.

[0033] Step S2: Perform eye tilt correction on the acquired driver's facial image to obtain a corrected image.

[0034] In one embodiment, due to the unique environment inside the driver's cabin, the driver's head inevitably moves, altering their posture. This can cause the captured facial image to shift or shake, resulting in the integral projection failing to accurately detect the tilted facial portion of the image. Therefore, geometric correction is first performed on the driver's face. Since tilted eyes possess certain geometric characteristics, geometric knowledge can be used to geometrically correct the tilted facial portion.

[0035] Step S2 includes:

[0036] The system performs eye contour detection on the currently acquired driver facial image and extracts the positional information of multiple key points; for example, an edge detection algorithm is used for eye contour detection. It should be noted that key points are eye contour points, and the combination of these key points reflects the shape of the eye. Multiple key points can be randomly obtained as needed, for example, five points can be used: X1, X2, X3, X4, and X5.

[0037] The corner radius of the eye is calculated using the positional information of each key point, and the illumination tilt angle is calculated based on the corner radius. Specifically, the corner radius ε is the ratio of the human eye contour shape formed by the positional information of each key point relative to a standard angle (such as 360°), multiplied by the interocular distance (IOD, i.e., the distance between the two eyes) and normalized. The illumination tilt angle β is calculated based on the corner radius. Since the change in the corner radius ε reflects the degree of tilt of the face, the illumination tilt angle β can be obtained by calculating the arctangent value of the corner radius ε. This angle represents the degree of tilt of the face relative to the image plane. For example, using five key points X1, X2, X3, X4, and X5, the contour shape formed is (X1, X2, X3, X4, X5), and the corner radius ε and illumination tilt angle β can be expressed as:

[0038] ￡=(X1, X2, X3, X4, X5) / 360° *IOD; (1)

[0039] β=arc tan (￡); (2)

[0040] According to the calculated illumination tilt angle, the corresponding driver's facial image is geometrically corrected to obtain a corrected image. Specifically, based on the calculated illumination tilt angle β, a geometric transformation matrix (such as affine transformation, perspective transformation, etc.) is determined and applied to the original driver's facial image to obtain the corrected image. After geometric correction, the positions of the two eyes in the image are almost on the same straight line.

[0041] Step S3: Based on the integral projection algorithm, perform preliminary human eye localization on each corrected image and extract the preliminary human eye localization information of each corrected image.

[0042] In one embodiment, an integral projection method is selected to roughly determine the face region, typically based on the dark distribution characteristics of the face to initially locate the eyes. Facial organs are darker than other parts of the face, so the locations of these organs will generate a minimum value. Then, the correlation between the gray-scale curves at each minimum value location is used to create a decision criterion.

[0043] Step S3 includes: performing horizontal integral projection and vertical integral projection on each corrected image, and obtaining the coordinate information of the corresponding two human eyes.

[0044] Specifically, horizontal integral projection is an image processing technique that generates a projection curve by calculating the sum of the gray values ​​of each row of an image. This curve reflects the gray-level distribution of the image in the horizontal direction. When there are obvious edges or structural changes in the image, peaks or troughs will appear on the projection curve.

[0045] In the corrected image, the gray value at point (x, y) is G(x, y), and the projection of the horizontal product over the region [y1, y2] is H(x), where the expression for H(x) is:

[0046]

[0047] This expression gives the average gray value of all pixels in the interval [y1, y2] at each x-coordinate in the image.

[0048] For the vertical product projection within the region [x1, x2], let V(y) be the expression:

[0049]

[0050] This expression gives the average gray value of all pixels in the interval [x1, x2] at each y-coordinate in the image.

[0051] In one embodiment, the step of performing horizontal and vertical integral projections on each corrected image to obtain the coordinate information of the corresponding two human eyes includes:

[0052] The current corrected image is subjected to horizontal and vertical integral projections to obtain corresponding horizontal and vertical integral projection result images. These two result images respectively demonstrate the accumulation of grayscale values ​​in the horizontal and vertical directions of the image, presented as projection curves. Because the overall brightness value of the face decreases significantly at the left and right edges of the facial region, distinct peaks appear on the horizontal integral projection curve. By identifying the positions of these peaks, the left and right boundaries of the facial region in the image can be determined, thereby defining the extent of the entire facial region.

[0053] Extract a minimum point from the relevant waveform in the horizontal integral projection image to obtain the corresponding preliminary ordinates of the two eyes. Specifically, within the facial region of the horizontal integral projection result image, find the minimum points in the waveform. Since the eye region has the lowest grayscale value (close to black), these minimum points will appear on the projection curve after horizontal projection. Because tilt correction has been performed previously, ensuring the two eyes are on the same horizontal line, by extracting the ordinates of these minimum points (i.e., their vertical positions), the preliminary ordinates of the two eyes can be obtained, such as... Figure 2 .

[0054] Extract the two minimum points on either side of the central axis of symmetry of the relevant symmetrical waveform in the vertical integral projection image of the current corrected image to obtain the corresponding ordinates of the two eyes. Specifically, within the facial region of the vertical integral projection result image, due to the symmetry of the face, the projection curve usually presents a symmetrical waveform. Find two minimum points on either side of the central axis of symmetry of the symmetrical waveform. These two points correspond to the horizontal positions of the center points of the left and right eyes, respectively. By extracting these two minimum points, the preliminary abscissas of the two eyes can be obtained, such as... Figure 3 .

[0055] Step S4: Based on the shape model, obtain the human eye detection result from the corrected image from which preliminary human eye localization information has been extracted.

[0056] In one embodiment, the vertical coordinate of the eye is typically determined using horizontal integral projection, and the horizontal coordinate is determined using vertical integral projection. Knowing the horizontal and vertical coordinates allows for a rough determination of the eye's range. The eye region can be initially located using integral projection, thus roughly outlining the eyebrow and eye area. However, the presence of the eyebrow region can interfere with this process. For example, if the driver's eyes are closed, the minimum value in the facial waveforms of both the horizontal and vertical integral projections corresponds to the eyebrow position, resulting in inaccurate results. To achieve more precise eye localization, further processing using a shape model is necessary.

[0057] Step S4 includes:

[0058] Multiple images are selected as target images from the corrected images from which preliminary human eye localization information has been extracted within a set time period. Specifically, since images are acquired continuously in real time, multiple images are captured consecutively within a short period of time, and multiple corrected images from which preliminary human eye localization information is extracted. Therefore, it is necessary to select a portion of these corrected images as target images for subsequent processing. For example, if preliminary human eye localization information is extracted from 10 images in five minutes, multiple images can be selected from these 10 images based on requirements, reducing the amount of computation. To simplify processing, a portion of the images can be randomly selected as target images. Alternatively, selection can be based on factors such as image quality (e.g., high clarity, low noise) and the difference between the preliminary human eye localization information and the standard localization information, which helps improve the accuracy of human eye detection.

[0059] Based on the shape model, human eye detection is performed according to the preliminary human eye localization information of each input target image to obtain the corresponding human eye detection results.

[0060] In one embodiment, the step of performing human eye detection based on the preliminary human eye localization information of each input target image includes:

[0061] A shape vector set is constructed for each input target image; wherein, the shape vector set includes: Euclidean distance vectors, scale-measured vectors, and curl-measured vectors for each human eye keypoint; it should be noted that each human eye keypoint is pre-defined, and the positions of the human eye keypoints may be different for each image. If a face has k human eye keypoints, the specific shape vector set is as follows:

[0062] The Euclidean distance vector S of the target image is represented as:

[0063] S=(X0, Y0, X1, Y1, X2, Y2, ..., X k-1 Y k-1 (5)

[0064] Among them, X i Y i The coordinates of the (i+1)th key point are represented by the vector S, which can be used to represent the Euclidean distance.

[0065] The vector V representing the scale of the target image is obtained from the coordinate vector S of the target image as follows:

[0066] V=(L0,I0,L1,I1,L2,I2,…,L k-1 I k-1 (6)

[0067] Where L represents the cosine distance between the same human eye keypoints in the current target image and the previous target image under scale measurement, and I represents the angle between the same human eye keypoints in the current target image and the previous target image. Scale measurement fully considers the degree of eye opening and closing, and can determine whether the driver is blinking or closing their eyes, and thus determine whether they are driving under fatigue.

[0068] Given the coordinate vector S of the target image, the vector P, representing the curl of the target image, can be expressed as:

[0069] P = (U0, D0, U1, D1, U2, D2, …, U k-1 D k-1 (7)

[0070] Where U represents the cosine distance between the current target image and the two preceding target images at the same keypoint, under the curl measure, and D represents the curl between the current target image and the two preceding target images at the same keypoint. The curl measure fully considers the degree of eye rotation and can determine whether the driver is looking straight ahead and not driving with their eyes crossed.

[0071] It should be noted that all target images are arranged in order. In the shape vector set of the first target image, only the coordinate vector S has a value, while the vector V under scale measurement and the vector P under curl measurement are 0. In the shape vector set of the second target image, only the coordinate vector S and the vector V under scale measurement have values, while the vector P under curl measurement is 0. From the third image onwards, all three types of vectors have values. Therefore, at least 3 target images are selected after initial positioning.

[0072] Differences in facial features within images manifest as scale and rotation, necessitating shape normalization using a three-layer sampling method. By minimizing the Euclidean distance that overcomes all keypoints, the scale and rotation differences between samples are normalized, resulting in an aligned set of sample shape vectors. Three-layer sampling is a technique for processing and analyzing image data, particularly facial image data. It aligns samples by normalizing scale and rotation differences to facilitate subsequent modeling and analysis.

[0073] The shape vector sets of each target image are normalized for scale and curl differences using a three-layer sample analysis method to obtain an aligned shape vector set; the aligned shape vector set S' is represented as:

[0074] S ′ =V+P; (8)

[0075] The human eye detection results are output based on the model parameter vector and the alignment shape vector set of each target image. The human eye detection results should include the qualified or unqualified results of human eye detection within a set time period, and may also include the precise location information of the identified human eyes.

[0076] Specifically, by multiplying the aligned shape vector of the input target image with the model parameter vector and comparing it with the target human eye vector, the eye detection of each image can be determined by judging whether the difference is within a set range. Within a set time period, if most of the eye detection results of all calibrated images are qualified, a qualified eye result is output, indicating that the driver is driving safely; if there are many unqualified results, an unqualified eye result is output, which may indicate unsafe driving behavior, such as closing the eyes or fatigue, and an alarm operation will be triggered.

[0077] In one embodiment, the model parameter vector is obtained during shape model training, and the method of obtaining it includes:

[0078] Acquire driver facial image samples labeled with key points of each person's eyes; these images will be used in the subsequent shape model training process.

[0079] Construct shape vector sets for each driver's facial image sample. These vector sets include: Euclidean distance vectors involving key points of each person's eyes, vectors measured by scale, and vectors measured by curl.

[0080] The shape vector sets of each driver's facial image sample were normalized by scale and curl differences using a three-layer sample analysis method to obtain the aligned shape vector sets of each sample.

[0081] The model parameter vector is adjusted based on the difference between the aligned shape vector set of each driver's facial image sample and the corresponding target eye vector. Here, the target eye vector is composed of the aligned shape vector set of each eye key point marked in the corresponding driver's facial image sample. The result of multiplying the aligned shape vector set with the model parameter vector is made close to the target eye vector.

[0082] As can be seen from the above, the shape model does not require a large amount of labeled data or complex mapping relationships, and the data used for calculation is filtered in real time. Therefore, compared with the existing technology, it reduces the need for a large amount of labeled data, simplifies complex mapping relationships, reduces the amount of computation, and greatly improves detection efficiency.

[0083] Similar to the principles of the above embodiments, the present invention provides an autonomous driving human eye detection system based on visual recognition.

[0084] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0085] like Figure 4 This invention presents a schematic diagram of the structure of an autonomous driving human eye detection system based on vision recognition, as described in an embodiment of the present invention.

[0086] The system includes:

[0087] Image acquisition module 1 is used to acquire real-time facial images of the driver inside the target vehicle's cab;

[0088] Correction module 2, connected to the image acquisition module 1, is used to correct the human eye tilt of the acquired driver's facial image to obtain a corrected image;

[0089] The integral projection module 3, connected to the correction module 2, is used to perform preliminary human eye localization on each correction image based on the integral projection algorithm, and to extract preliminary human eye localization information from each correction image.

[0090] The detection module 4 is connected to the integral projection module 3 and is used to obtain the human eye detection result based on the shape model from the corrected image from which preliminary human eye positioning information has been extracted.

[0091] Since the implementation principle of the vision recognition-based autonomous driving human eye detection system has been described in the foregoing embodiments, it will not be repeated here.

[0092] The vision-based human eye detection method for autonomous driving provided in this invention can be implemented on the terminal side or the server side. Regarding the hardware structure of the electronic terminal, please refer to [link to relevant documentation]. Figure 5 This is a schematic diagram of an optional hardware structure of an electronic terminal 1000 provided in an embodiment of the present invention. The terminal 1000 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010, and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It is understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as bus systems.

[0093] The user interface 1009 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0094] It is understood that memory 1002 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0095] In this embodiment of the invention, the memory 1002 is used to store various types of data to support the operation of the terminal 1000. Examples of this data include: any executable program for operation on the terminal 1000, such as the operating system 10021 and application program 10022; the operating system 10021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 10022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The vision recognition-based autonomous driving human eye detection method provided in this embodiment of the invention can be included in the application program 10022.

[0096] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The processor 1001 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0097] In an exemplary embodiment, the terminal 1000 may be used to execute the aforementioned method by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0098] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0099] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0100] In summary, the vision-based autonomous driving eye detection method, system, terminal, and medium of this invention first corrects the eye tilt of the acquired image, and then uses an integral projection algorithm to perform preliminary localization on the corrected image. Next, a shape model is used to perform secondary localization based on the preliminary localization information to obtain more accurate eye detection results. This invention relies solely on visual sensors to collect data, and through data processing and filtering, the simplified key data is input into a deep learning network for detection. This method eliminates the need to construct complex mapping relationships or require a large amount of labeled data for network training, thus significantly reducing computational load. Furthermore, it not only greatly shortens the system detection time and improves detection efficiency, but also meets the stringent real-time requirements of intelligent safe driving research. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0101] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for detecting human eyes in autonomous driving based on visual recognition, characterized in that, The method includes: Real-time acquisition of facial images of the driver inside the target vehicle's cab; The collected driver facial images were subjected to eye tilt correction to obtain a corrected image; Based on the integral projection algorithm, preliminary human eye localization is performed on each corrected image, and preliminary human eye localization information of each corrected image is extracted; Based on the shape model, the human eye detection result is obtained from the corrected image from which preliminary human eye localization information is extracted.

2. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 1, characterized in that, The process of correcting the eye tilt of the acquired driver's facial images to obtain the corrected image includes: Perform eye contour detection on the currently acquired driver's facial image and extract the position information of multiple key points; The curvature of the corner of the eye is calculated using the location information of each key point, and the angle of illumination tilt is calculated based on the curvature of the corner of the eye. Based on the calculated illumination tilt angle, geometric correction is performed on the corresponding driver's facial image to obtain a corrected image.

3. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 1, characterized in that, The method based on integral projection algorithm performs preliminary human eye localization on each corrected image, and extracts preliminary human eye localization information for each corrected image, including: Perform horizontal and vertical integral projections on each corrected image to obtain the coordinate information of the two eyes initially identified in the corrected image.

4. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 3, characterized in that, The step of performing horizontal and vertical integral projections on each corrected image to obtain the coordinate information of the two human eyes in the corrected image includes: Perform horizontal and vertical integral projections on the current corrected image to obtain the corresponding horizontal and vertical integral projection result images. Extract a minimum point of the relevant waveform in the horizontal integral projection image to obtain the corresponding preliminary ordinate of the two human eyes; Extract the two minimum points on both sides of the central axis of symmetry of the relevant symmetrical waveform in the vertical integral projection of the current calibrated image, and obtain the corresponding preliminary x-coordinates of the two human eyes.

5. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 1, characterized in that, The method of obtaining human eye detection results from the corrected image based on the shape model, from which preliminary human eye localization information is extracted, includes: Multiple images are selected as target images from the corrected images from which preliminary human eye positioning information has been extracted within a set time period; Based on the shape model, human eye detection is performed on each input target image to obtain the corresponding human eye detection results.

6. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 5, characterized in that, The step of performing human eye detection based on each input target image includes: A shape vector set is constructed for each input target image; wherein, the shape vector set includes: Euclidean distance vectors involving key points of each human eye, vectors under scale measurement, and vectors under curl measurement; The shape vector sets of each target image are normalized by scale and curl differences using a three-layer sample analysis method to obtain aligned shape vector sets. The human eye detection results are output based on the model parameter vector and the alignment shape vector set of each target image.

7. The method for detecting human eyes in autonomous driving based on visual recognition as described in claim 6, characterized in that, The model parameter vector is adjusted during shape model training. The adjustment method includes: constructing shape vector sets for driver face image samples marked with key points of each human eye, and obtaining the aligned shape vector set of each driver face image sample by normalizing the differences in scale and curl through a three-layer sample analysis method. Then, the model parameter vector is adjusted based on the differences between the aligned shape vector sets of each driver face image sample and the corresponding target human eye vector. The target human eye vector includes: the aligned shape vector set involving the key points of each human eye marked in the corresponding driver face image sample.

8. A vision-based autonomous driving human eye detection system, characterized in that, The system includes: The image acquisition module is used to acquire real-time facial images of the driver inside the target vehicle's cab; The correction module, connected to the image acquisition module, is used to correct the human eye tilt in the acquired driver's facial image to obtain a corrected image; An integral projection module, connected to the correction module, is used to perform preliminary human eye localization on each correction image based on the integral projection algorithm, and to extract preliminary human eye localization information from each correction image. The detection module, connected to the integral projection module, is used to obtain human eye detection results from the corrected image from which preliminary human eye positioning information has been extracted, based on the shape model.

9. An electronic terminal, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by one or more processors, performs the method as described in any one of claims 1 to 7.

Images