Blink detection method and system

Abstract

To provide a blink detection method and system.SOLUTION: A method includes a step S41 for photographing a face by using a DVS camera to thereby acquire a DVS pixel stream, and a step S42 for integrating DVS pixels of the DVS pixel stream to form a plurality of DVS frames. Each DVS frame includes a plurality of first color pixels and a plurality of second color pixels, each first color pixel is associated with one or more DVS pixels showing a brightening event, and each second pixel is associated with one or more DVS pixels showing a darkening event. Also, the method includes a step S43 for determining whether or not blink movement exists in at least one DVS frame of the plurality of DVS frames. The step for determination includes a step for determining whether or not a pattern with a first color area and a second color area distributed vertically exists within an eye part region of at least one DVS frame.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a blink detection method and system, more particularly to a blink detection method and system based on a DVS camera or an event camera.The present invention further relates to a fatigue driving detection method and system, and a vehicle equipped with a blink detection system or a fatigue driving detection system. [Background technology]

[0002] Fatigue driving refers to a driver driving a vehicle in a fatigued state due to long-term continuous driving or other reasons. Fatigue driving is likely to cause serious traffic accidents, so it is necessary to detect the driver's fatigue state. A person's blinking pattern, such as blink duration and / or frequency, can indicate a person's state, such as the degree of drowsiness. In the fatigue driving detection method, the driver's blink detection is a very important process.

[0003] Currently, there are blink detection technologies using conventional cameras. However, the eye area is small on the entire face, and blinks are generally very fast, lasting only a fraction of a second. With most conventional cameras, it is difficult to analyze not only the frequency of blinks, but also the details of the blink pattern. For example, blink images captured by conventional cameras at 30 frames per second may be unclear. In addition, an entire blink usually lasts only a few frames, making it difficult to identify the exact duration of the blink. By adopting high-speed cameras with frame rates of over 100 frames per second, blink detection can be performed more accurately. However, high-speed cameras significantly increase costs and data processing. In addition, conventional cameras have difficulty taking clear blink photos under low-light conditions.

[0004] DVS cameras or event cameras are novel, bio-inspired asynchronous vision acquisition devices. Compared to standard conventional frame-based cameras, a single pixel of a DVS camera asynchronously reports an "event" stream of light intensity changes. Compared to conventional cameras, DVS cameras have several advantages: i) high time resolution (millisecond scale), ii) high dynamic range (140 dB, while the dynamic range of conventional cameras is 60 dB), and iii) low power consumption. However, currently there is no simple and effective method to perform blink detection based on DVS cameras. Summary of the Invention [Means for solving the problem]

[0005] An object of the present invention is to provide a blink detection method based on a DVS camera, which can simply and accurately detect blink frequency, blink duration, etc.

[0006] According to one aspect of the present invention, there is provided a blink detection method based on a DVS camera, comprising the steps of: photographing a face using a DVS camera, thereby obtaining a DVS pixel stream; integrating DVS pixels of the DVS pixel stream to form a plurality of DVS frames, each DVS frame of the plurality of DVS frames including a plurality of first color pixels and a plurality of second color pixels, each of the first color pixels being associated with one or more DVS pixels indicating a lightening event, and each of the second color pixels being associated with one or more DVS pixels indicating a darkening event; and determining whether or not a blinking action is present in at least one DVS frame of the plurality of DVS frames, wherein the step of determining whether or not a blinking action is present includes the step of determining whether or not a pattern in which first color regions and second color regions are distributed vertically exists within an eye region of the at least one DVS frame.

[0007] According to another aspect of the present invention, there is provided a blink detection system comprising a DVS camera used for imaging to obtain a DVS pixel stream, and a processor connected to the DVS camera and configured to perform the above-mentioned blink detection method.

[0008] According to another aspect of the present invention, there is provided a second pixel vehicle comprising a vehicle body and the above-described blink detection system mounted on the vehicle body. The present invention provides, for example, the following: (Item 1) capturing an image of a face using a DVS camera, thereby obtaining a DVS pixel stream; aggregating DVS pixels of the DVS pixel stream to form a plurality of DVS frames, each DVS frame of the plurality of DVS frames including a plurality of first color pixels and a plurality of second color pixels, each of the first color pixels associated with one or more DVS pixels indicative of a lightening event and each of the second color pixels associated with one or more DVS pixels indicative of a darkening event; determining whether a blinking motion is present in at least one DVS frame of the plurality of DVS frames; A blink detection method based on a DVS camera, wherein the step of determining whether or not a blinking action exists includes a step of determining whether or not a pattern in which a first color area and a second color area are distributed vertically exists within the eye area of ​​the at least one DVS frame. (Item 2) The blink detection method described in the above item, wherein each DVS frame among the plurality of DVS frames further includes a plurality of third color pixels, the third color pixels being not associated with DVS pixels indicating a lightening event or DVS pixels indicating a darkening event. (Item 3) The blink detection method according to any one of the preceding claims, wherein the step of determining whether or not a blinking action is present further includes a step of determining that an eye closing action is in progress when a first color area is above the second color area in the pattern. (Item 4) The method for detecting blinking according to any one of the preceding claims, wherein the step of determining whether or not a blinking action is present further includes a step of determining that an eye-opening action is in progress when a second color area in the pattern is above the first color area. (Item 5) The step of determining whether or not a blinking motion is present includes: calculating an average height difference for each DVS frame of the at least one DVS frame, the average height difference being an average height difference between first color pixels and second color pixels in the eye region of the at least one DVS frame; The blink detection method according to any one of the preceding items, further comprising a step of determining that a pattern in which a first color area and a second color area are distributed vertically exists in the eye area of ​​the at least one DVS frame when the absolute value of the difference in the average height is greater than a threshold value. (Item 6) The step of calculating the average height difference comprises: calculating a first average height of first color pixels in the eye region; calculating a second average height of second color pixels in the eye region; and calculating the difference in average heights by subtracting the second average height from the first average height. (Item 7) After the step of determining whether a blinking motion is present, the method further includes a step of calculating a blink duration, the step of calculating the blink duration comprising: identifying a first set of adjacent DVS frames from the plurality of DVS frames, the average height difference of each DVS frame of the first set of adjacent DVS frames being positive and having an absolute value greater than a first threshold; identifying a second set of adjacent DVS frames from the DVS frames subsequent to the first set of adjacent DVS frames of the plurality of DVS frames, wherein a difference in average height of each of the second set of adjacent DVS frames is negative and its absolute value is greater than a second threshold; identifying a first reference DVS frame from the first set of adjacent DVS frames; identifying a second reference DVS frame from the second set of adjacent DVS frames; The blink detection method according to any one of the preceding items, further comprising: calculating a blink duration based on a time difference between the capture time of the first reference DVS frame and the capture time of the second reference DVS frame. (Item 8) The blink detection method according to any one of the above items, wherein the first reference DVS frame is a temporally middle DVS frame among the first set of adjacent DVS frames, and the second reference DVS frame is a temporally middle DVS frame among the second set of adjacent DVS frames. (Item 9) The blink detection method according to any one of the above items, wherein the first reference DVS frame is the DVS frame having the largest absolute value of the average height difference among the first set of adjacent DVS frames, and the second reference DVS frame is the DVS frame having the largest absolute value of the average height difference among the second set of adjacent DVS frames. (Item 10) After the step of determining whether a blinking motion is present, the method further includes a step of identifying one or more complete blinks, the step of identifying one or more complete blinks comprising: identifying a first set of adjacent DVS frames from the plurality of DVS frames, the average height difference of each DVS frame of the first set of adjacent DVS frames being positive and greater than a first threshold; identifying a second set of adjacent DVS frames from the DVS frames subsequent to the first set of adjacent DVS frames of the plurality of DVS frames, wherein a difference in average height of each of the second set of adjacent DVS frames is negative and greater than a second threshold; A blink detection method according to any one of the preceding claims, comprising a step of identifying one or more complete blinks based on the identified first set of adjacent DVS frames and the identified second set of adjacent DVS frames. (Item 11) The blink detection method according to any one of the preceding items, further comprising a step of calculating a blink frequency using the shooting times of at least some of the DVS frames associated with the one or more complete blinks. (Item 12) The blink detection method according to any one of the preceding claims, further comprising a step of identifying the eye region within each DVS frame of the at least one DVS frame prior to the step of determining whether or not a blink action is present. (Item 13) A DVS camera used for capturing images to obtain a DVS pixel stream; A blink detection system comprising: a processor connected to the DVS camera and configured to execute the blink detection method described in any one of the preceding items. (Item 14) A vehicle body, A vehicle comprising: the blink detection system described above attached to the vehicle body. (Summary) A blink detection method and system are disclosed, which includes the steps of: using a DVS camera to capture a face image, thereby obtaining a DVS pixel stream; aggregating DVS pixels from the DVS pixel stream to form a plurality of DVS frames, each of the DVS frames including a plurality of first color pixels and a plurality of second color pixels, each of the first color pixels being associated with one or more DVS pixels indicative of a lightening event, and each of the second color pixels being associated with one or more DVS pixels indicative of a darkening event; and determining whether a blinking action is present in at least one DVS frame of the plurality of DVS frames, wherein the step of determining whether a blinking action is present includes the step of determining whether a pattern in which the first color region and the second color region are distributed vertically is present in the eye region of the at least one DVS frame. [Brief description of the drawings]

[0009] [Figure 1]1 is a schematic diagram of a blink detection system in accordance with one or more embodiments of the present invention. [Figure 2A] FIG. 2 is a diagram of multiple DVS frames during a blink process acquired by a blink detection system in accordance with one or more embodiments of the present invention. [Figure 2B] FIG. 2 is a diagram of multiple DVS frames during a blink process acquired by a blink detection system in accordance with one or more embodiments of the present invention. [Figure 2C] FIG. 2 is a diagram of multiple DVS frames during a blink process acquired by a blink detection system in accordance with one or more embodiments of the present invention. [Figure 2D] FIG. 2 is a diagram of multiple DVS frames during a blink process acquired by a blink detection system in accordance with one or more embodiments of the present invention. [Figure 2E] FIG. 2 is a diagram of multiple DVS frames during a blink process acquired by a blink detection system in accordance with one or more embodiments of the present invention. [Figure 3A] FIG. 2C is a partially enlarged view of FIG. 2B. [Figure 3B] FIG. 2E is a partially enlarged view of FIG. 2D. [Figure 4] 1 is a method for blink detection according to one or more embodiments of the present invention. [Diagram 5] Specific steps of determining blinking behavior (step S33) according to some embodiments of the present invention. [Figure 6] 1 is a method for blink detection according to one or more embodiments of the present invention. [Figure 7] 1 is a method for blink detection according to one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following detailed description of the embodiments of the present invention is given in the drawings, in which the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are illustrative and are only for explaining the present invention, and should not be understood as limiting the present invention.

[0011] As used herein, the terms "DVS camera," "event camera," "dynamic vision sensor," and "DVS" are used interchangeably and all refer to an event-based camera. The terms "DVS pixel," "pixel event," and "event" are used interchangeably and can refer to a change in light intensity captured by a DVS camera. The term "frame" or "DVS frame" of a DVS camera refers to a DVS pixel captured by the DVS in a timespan or an image or frame with a certain number of pixels integrated into it. The term "capture time" of a DVS frame refers to the time when the DVS pixel is captured or captured, for example, the midpoint of the timespan of the DVS frame, or the midpoint of the timestamps of all the DVS pixels in the DVS frame.

[0012] Unless otherwise defined, technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the present invention. In the description of the present invention, the orientation or positional relationship indicated by terms such as "center", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like is based on the orientation or positional relationship shown in the drawings and is merely for the purpose of making the present invention easier to explain and simplifying the description, and does not indicate or imply that the subject device or element has a specific orientation and should be configured and operated in a specific orientation, and therefore should not be understood as limiting the present invention. Furthermore, the terms "first" and "second" are used for explanatory purposes only and should not be understood as indicating or implying relative importance.

[0013] DVS camera captures only the change in light intensity and then creates asynchronous DVS pixels (pixel events). DVS pixels or pixel events can be represented using pixel or event data, typically in the form of [x, y, t, p], where x and y represent the x and y coordinates of the event, t represents a timestamp indicating the event, and p represents the polarity of the event, indicating whether the change in light intensity is brightening or darkening. In one or more embodiments, if the light intensity brightens, p is +1, and if the light intensity becomes weak, p is -1. Compared with traditional cameras, DVS has advantages such as low latency, no motion blur, high dynamic range, and low power consumption.

[0014] Unlike conventional cameras that create complete images or frames that contain continuous pixels, DVS cameras only create asynchronous DVS pixels that are discrete and non-continuous in space and time, so DVS pixels (pixel events) captured by DVS cameras need to be integrated in a time span to generate a DVS frame. There are various ways that a DVS frame can be created, such as integrating DVS pixels in a fixed or variable time span, or integrating a certain number of DVS pixels to form a DVS frame. The method of integrating DVS pixels into a DVS frame is well known, and will not be described here.

[0015] The present invention provides a blink detection method based on a DVS camera. The blink detection method performs blink detection by determining whether a unique pattern characteristic exists in a DVS frame. In some embodiments, the blink detection method determines whether a blink action is in progress by determining whether a pattern of first and second color regions distributed vertically exists in an eye region of a DVS frame. In some embodiments, the blink detection method determines whether a blink action is in progress by calculating an average height difference between first and second color pixels in an eye region, and if the absolute value of the average height difference is greater than a threshold, determining that a pattern of first and second color regions distributed vertically exists in an eye region of the DVS frame. In some embodiments, the blink detection method further detects blinking, eye opening or eye closing. In some embodiments, the blink detection method can further calculate a blink frequency and / or a blink duration.

[0016] In some embodiments of the present invention, eye blinking, eye opening and eye closing actions can be accurately determined by simply calculating the average height of the first color pixels and the average height of the second color pixels in the eye area of ​​the DVS frame and the difference between the heights of the first and second color pixels. The method of the present invention requires a very low amount of calculation compared to conventional image processing methods. In addition, the amount of calculation is further reduced because only the positions where the light intensity changes occur in the DVS frame have the first color pixels or the second color pixels. Therefore, the method of the present invention is particularly suitable for image processing at high frame rates (e.g., 200 frames per second), thereby ensuring the accuracy of blink detection.

[0017] 1 is a schematic diagram of a blink detection system 100 in accordance with one or more embodiments of the present invention. As shown, the blink detection system 100 in accordance with the present invention includes a DVS camera 102 that is used to capture a face 106 of a person 104. The blink detection system 100 further includes a processor (not shown) that is coupled to the DVS camera 102 and that is used to receive and process the DVS pixel stream from the DVS camera 102.

[0018] 2A-2E are diagrams of a number of DVS frames during a blinking process captured by the blink detection system 100 according to the present invention, where FIG. 2A shows a DVS frame before blinking, FIG. 2B shows a DVS frame when the eyes are just closing, FIG. 2C shows a DVS frame when the eyes are substantially completely closed, and FIG. 2D and FIG. 2E show DVS frames when the eyes are just opening. In the DVS frames of 2A-2E, white pixel points indicate a brightening of light intensity, black pixel points indicate a darkening of light intensity, and gray pixel points indicate no change in light intensity. The pixel colors of the DVS frames of the present invention may not be limited to black, white, and gray. For example, in the DVS frames according to another embodiment of the present invention, the first color pixel points indicate a brightening of light intensity, the second color pixel points indicate a darkening of light intensity, and the third color pixel points indicate no change in light intensity. The DVS frames according to another embodiment of the present invention may not be actual images as shown in FIG. 2A-2E, but may simply be a concept of a virtual frame including DVS pixels of a time span. In this case, the DVS frame may only include first and second color pixels, but not third color pixels.

[0019] Those skilled in the art should understand that Figures 2A-2E are merely a few example DVS frames captured by blink detection system 100, and that the number of DVS frames actually captured by blink detection system 100 may be much greater than those shown in Figures 2A-2E. Each DVS frame shown in Figures 2A-2E is a DVS frame formed by integrating DVS pixels captured by DVS camera 102 within a 5 ms time span. Those skilled in the art should understand that a DVS frame may be formed by integrating DVS pixels over a time span other than 5 ms, or may be formed by integrating a fixed number of DVS pixels.

[0020] As a result of research, the inventors have found that the DVS frame taken during the process of closing and opening the eyes has very distinctive features. As shown in Figure 2B, Figure 2D and Figure 2E, during the process of opening and closing the eyes, the eye area of ​​the DVS frame has a very clear pattern of white and black areas distributed up and down. Specifically, as shown in Figure 2B, during the process of closing the eyes, the white area in the pattern of the eye area of ​​the DVS frame is at the top and the black area is at the bottom, whereas, as shown in Figure 2D and Figure 2E, during the process of opening the eyes, the white area in the pattern of the eye area of ​​the DVS frame is at the bottom and the black area is at the top. In Figure 2A before the start of blinking and Figure C when the eyes are completely closed, the eye area of ​​the DVS frame does not contain such a pattern.

[0021] FIG. 3A is a partial enlarged view of FIG. 2B, and FIG. 3B is a partial enlarged view of FIG. 2D. As shown in FIG. 3A, in the eye region 302a of the DVS frame taken during the process of closing the eyes, the eye is substantially divided into two parts, upper and lower, the upper half 312a of the eye is white, and the lower half 314a of the eye is black. In other words, the upper half 312a of the eye exhibits a substantially horizontal white strip, and the lower half 314a of the eye exhibits a substantially horizontal black strip. As shown in FIG. 3B, in the eye region 302b of the DVS frame taken during the process of closing the eyes, the eye is substantially divided into two parts, upper and lower, the upper half 312b of the eye is black, and the lower half 304b of the eye is white. In other words, the upper half 312b of the eye exhibits a substantially horizontal black strip, and the lower half 314b of the eye exhibits a substantially horizontal white strip.

[0022] Since the eye pattern of the DVS frame captured during the process of opening and closing the eyes has a very distinctive feature (white and black areas distributed vertically), the inventors use the distinctive feature in the DVS frame to identify the blinking or eye opening and closing actions. In one or more embodiments of the present invention, when a pattern of black and white areas distributed vertically exists in the eye area of ​​the DVS frame, it can be determined that a blinking action is in progress. In another one or more embodiments of the present invention, when white and black areas are distributed vertically in the eye area of ​​the DVS frame, and the white area is above the black area, it is determined that an eye closing action is in progress. In one or more embodiments of the present invention, when white and black areas are distributed vertically in the eye area of ​​the DVS frame, and the black area is above the white area, it is determined that an eye closing action is in progress.

[0023] In some embodiments of the present invention, it is possible to determine whether or not a pattern in which black and white regions are distributed vertically exists in the DVS frame by using some known image processing techniques, and when a pattern in which white and black regions are distributed vertically exists in the eye region of the DVS frame, it is possible to determine that a blinking action is in progress. In another embodiment of the present invention, it is possible to determine whether or not a pattern in which white and black regions are distributed vertically and the white region is above the black region exists in the eye region of the DVS frame by using some known image processing techniques, and when such a pattern exists, it is possible to determine that an eye closing action is in progress. In another embodiment of the present invention, it is possible to determine whether or not a pattern in which white and black regions are distributed vertically and the black region is above the white region exists in the eye region of the DVS frame by using some known image processing techniques, and when such a pattern exists, it is possible to determine that an eye opening action is in progress. In some embodiments of the present invention, it is possible to identify the eye region in the DVS frame before and / or during the above determination. Identifying the eye region in the DVS can be realized by using some known image processing techniques.

[0024] The inventors of the present invention further provide a simple and effective method for determining blinking, opening or closing of eyes based on the pattern characteristics in the DVS frame during blinking, opening or closing of eyes. As shown in Fig. 3A, the upper half 312a of the eye of the DVS frame shows a substantially horizontal white strip, and the lower half 314a of the eye shows a substantially horizontal black strip, so that the average height of all white pixels of the eye of the DVS frame is high and the average height of all black pixels is low. As shown in Fig. 3B, the upper half 312b of the eye of the DVS frame shows a substantially horizontal black strip, and the lower half 314b of the eye shows a substantially horizontal white strip, so that the average height of all white pixels of the eye of the DVS frame is low and the average height of all black pixels is high. In DVS frames other than opening or closing the eyes (for example, the DVS frames shown in FIG. 2A and FIG. 2C), the distribution of white pixels and black pixels in the eye portion of the DVS frame is relatively random, and the average height of all white pixels and the average height of all black pixels in the eye portion of the DVS frame are basically the same. That is, the inventors have found that it is possible to determine whether a blinking action, an eye opening action, or an eye closing action is in progress based on the relationship between the average height of white pixels and the average height of black pixels in the eye portion of the DVS frame. When the absolute value of the difference between the average height of white pixels and the average height of black pixels in the eye portion of the DVS frame is greater than a threshold value, it can be determined that a pattern in which white areas and black areas are distributed up and down in the eye portion of the DVS frame exists, that is, a blinking action is in progress. When the average height of white pixels in the eye portion of the DVS frame is greater than the average height of black pixels, and the absolute value of the height difference is greater than a threshold value, it can be determined that a pattern in which white areas and black areas are distributed up and down in the eye portion of the DVS frame, and the white areas are above the black areas exists, that is, a closing action is in progress.When the average height of the white pixels in the eye area of ​​the DVS frame is smaller than the average height of the black pixels, and the absolute value of the height difference is greater than a threshold value, it can be determined that a pattern exists in which white and black areas are distributed vertically in the eye area of ​​the DVS frame and the white areas are below the black areas, that is, the eye-opening action is in progress.

[0025] In the embodiment of the present invention, the blinking, opening and closing of the eyes can be accurately determined by simply calculating the average height of all white pixels and the average height of all black pixels in the eye of the DVS frame and the difference between the heights of the two. The method of the present invention requires a very small amount of calculation compared with the conventional image processing method. In addition, due to the characteristics of the DVS camera, only the positions where the light intensity changes in the DVS frame have white pixels or black pixels, and the positions where the light intensity does not change are gray pixels, which further reduces the amount of calculation.

[0026] 2A-2E, a DVS frame is formed by integrating DVS pixels captured by the DVS camera 102 within a time span of 5 ms. Thus, in this embodiment, 200 DVS frames can be generated per second. As described above, the method of the present invention requires very little computation when determining blinking, eye opening, and eye closing, and therefore can guarantee high frame rate (e.g., 200 frames per second) image processing.

[0027] In the embodiment of the present invention, since the frame rate is high (for example, 200 frames per second), more DVS frames can be obtained in each blink process, and more DVS frames can be obtained in each eye-opening process or each eye-closing process. Therefore, the blink detection method of the present invention can clearly determine the complete process of first closing the eyes and then opening the eyes in the blink process, and can calculate the blink frequency more accurately. In addition, the blink duration can be calculated relatively accurately by analyzing the process of first closing the eyes and then opening the eyes in the entire blink process. The two indicators of blink frequency and blink duration are very useful for determining a person's mental state, such as whether or not he is in a fatigued driving state.

[0028] FIG. 4 is a method for blink detection according to one or more embodiments of the present invention. In step S41, a DVS camera is used to capture a person's face and obtain a DVS pixel stream. The part of the face captured by the DVS camera includes at least the eye. Unlike a conventional camera, the DVS camera only captures changes in light intensity and then creates asynchronous DVS pixels (pixel events). These DVS pixels may be discrete and non-continuous in space and time. The DVS pixels that are continuously generated during the capture process of the DVS camera and may be discrete and non-continuous in space and time are referred to herein as a DVS pixel stream. In step S42, the DVS pixels of the DVS pixel stream are integrated or combined to form multiple DVS frames. In some embodiments, the DVS pixels (pixel events) captured by the DVS camera within a fixed time span may be integrated or combined to generate one DVS frame. In other embodiments, the DVS pixels within a variable time span may be integrated or combined to generate one DVS frame. In another embodiment, a certain number of DVS pixels may be integrated or combined to generate one DVS frame. In step S43, it is determined whether a blinking action exists based on the formed DVS frame. In some embodiments, if there is a pattern in which white and black areas are distributed vertically in the eye area of ​​the DVS frame, it is determined that there is a blinking action. In one or more embodiments of the present invention, the step of determining the blinking action includes the steps of calculating a first average height of white pixels in the eye area, calculating a second average height of black pixels in the eye area, subtracting the second average height of black pixels from the first average height of white pixels in the eye area to obtain an average height difference, and determining that there is a pattern in which white and black areas are distributed vertically in the eye area of ​​the DVS frame, that is, there is a blinking action, when the absolute value of the average height difference is greater than a threshold value. The threshold value can be determined by experiment.In one or more embodiments, the threshold may be a value related to the eye height, such as 1 / 8, 1 / 10, 1 / 15, 1 / 20, etc. of the eye height.

[0029] FIG. 5 is a specific step of determining a blinking action (step S33) according to some embodiments of the present invention. In the embodiment of FIG. 5, the step of determining a blinking action includes step S51 of determining an eye closing action and step S52 of determining an eye opening action. In the embodiment of FIG. 5, in a plurality of DVS frames generated from the DVS pixel stream of the DVS camera, the eye closing action is determined first, and then the eye opening action is determined in the subsequent DVS frames. This allows detection of the entire blinking process, and the blinking process to be detected more accurately. In step S51, if there is a pattern in which white areas and black areas are distributed vertically in the eye area of ​​the DVS frame, and the white area is above the black area, it is determined that the eye closing action is in progress. In step S52, if there is a pattern in which white areas and black areas are distributed vertically in the eye area of ​​the DVS frame, and the white area is below the black area, it is determined that the eye opening action is in progress. In one or more other embodiments of the present invention, in step S51, when the absolute value of the average height difference between the average height of the white pixels and the average height of the black pixels in the eye region of the DVS frame is greater than a threshold value, and the average height of the white pixels is greater than the average height of the black pixels, it is determined that a pattern in which the white regions and the black regions are distributed vertically in the eye region of the DVS frame and the white regions are above the black regions exists, that is, an eye closing action is in progress. In step S52, when the absolute value of the average height difference between the average height of the white pixels and the average height of the black pixels in the eye region of the DVS frame is greater than a threshold value, and the average height of the white pixels is smaller than the average height of the black pixels, it is determined that a pattern in which the white regions and the black regions are distributed vertically in the eye region of the DVS frame and the white regions are below the black regions exists, that is, an eye opening action is in progress. In some embodiments of the present invention, the threshold value for determining the eye closing action and the threshold value for determining the eye opening action may be the same or different.

[0030] In one or more embodiments of the present invention, step S51 identifies a plurality of adjacent DVS frames with closed eyes (a first set of adjacent DVS frames) from a plurality of DVS frames generated from the DVS pixel stream of a DVS camera, and the difference between the average height of white pixels and black pixels of each DVS frame with closed eyes is greater than a threshold, and the average height of white pixels is greater than the average height of black pixels. Step S52 identifies a plurality of adjacent DVS frames with open eyes (a second set of adjacent DVS frames) from the subsequent DVS frames of the plurality of DVS frames with closed eyes, and the difference between the average height of white pixels and black pixels of each DVS frame with open eyes is greater than a threshold, and the average height of white pixels is less than the average height of black pixels. Thus, steps S51 and S52 identify one complete blink including a plurality of DVS frames with closed eyes and a plurality of DVS frames with open eyes.

[0031] Fig. 6 is a blink detection method according to one or more embodiments of the present invention. Steps S61, S62, and S63 of the blink detection method shown in Fig. 6 are basically the same as or similar to steps S41, S42, and S43 shown in Fig. 4, and will not be described here. In step S64, step S63 is repeatedly executed (or steps S61 to S63 are repeatedly executed) to identify multiple blinks, and the blink frequency is calculated by combining the shooting times of each DVS frame.

[0032] FIG. 7 is a blink detection method according to one or more embodiments of the present invention. Steps S71, S72 and S73 of the blink detection method shown in FIG. 7 are basically the same as or similar to steps S41, S42 and S43 shown in FIG. 4, and are not described here. In step S74, a reference DVS frame for closing eyes (first reference DVS frame) is identified from the plurality of DVS frames for closing eyes (first set of adjacent DVS frames) identified in step S51, and a reference DVS frame for opening eyes (second reference DVS frame) is identified from the plurality of DVS frames for opening eyes (second set of adjacent DVS frames) identified in step S52, and then a blink duration is identified based on the difference in shooting time between the reference DVS frame for closing eyes and the reference DVS frame for opening eyes. The actual blink duration is generally longer than the difference in shooting time between the reference DVS frame for closing eyes and the reference DVS frame for opening eyes. In some embodiments of the present invention, the relationship between the blink duration and the difference in shooting time can be identified by experiment. In some embodiments according to the present invention, blink duration may be a linear function of the difference in acquisition time between the reference DVS frame with eyes closed and the reference DVS frame with eyes open.

[0033] In some embodiments of the present invention, the reference DVS frame with eyes closed or the reference DVS frame with eyes open is the DVS frame with the most obvious pattern shown in Figures 3A and 3B during the process of closing and opening the eyes. In these embodiments, the reference DVS frame with eyes closed is the DVS frame with the largest absolute value of the average height difference between white pixels and black pixels in multiple DVS frames with eyes closed, and the reference DVS frame with eyes open is the DVS frame with the largest absolute value of the average height difference between white pixels and black pixels in multiple DVS frames with eyes open.

[0034] In another embodiment of the present invention, the reference DVS frame for closing eyes is the DVS frame with the middle shooting time among the multiple DVS frames for closing eyes, and the reference DVS frame for opening eyes is the DVS frame with the middle shooting time among the multiple DVS frames for opening eyes.In another embodiment of the present invention, the reference DVS frame for closing eyes can be another DVS frame among the multiple DVS frames for closing eyes, and the reference DVS frame for opening eyes can be another DVS frame among the multiple DVS frames for opening eyes.

[0035] In one or more embodiments of the present invention, the blink detection method further comprises a step of identifying an eye region in the DVS frame prior to the step of determining whether a blink action is present.

[0036] The blink detection shown in Figure 6 is used to calculate the blink frequency, and the blink detection shown in Figure 7 is used to calculate the blink duration. Those skilled in the art should understand that the methods of Figure 6 and Figure 7 may be combined together. That is, in one or more embodiments of the present invention, the blink detection not only includes calculating the blink frequency, but also includes calculating the blink duration.

[0037] Aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of a fully-integrated hardware embodiment, a fully-integrated software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code stored thereon.

[0038] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage media include portable computer magnetic disks, hard disks, random access memories (RAMs), read-only memories (ROMs), programmable erasable programmable read-only memories (EPROMs or flash memories), compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the context of this specification, the computer readable storage medium may be any tangible medium that can contain or store a program for use with or in combination with a command execution system, apparatus, or device.

[0039] Aspects of the present invention are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be noted that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program commands. These computer program commands are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to generate an apparatus, and the commands executed by the processor of the computer or other programmable data processing apparatus enable implementation of the specified functions / operations in one or more blocks of the flowcharts and / or block diagrams. Such a processor may be, but is not limited to, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable processor or gate array.

[0040] The present invention can be practiced in the following manner.

[0041] (Item 1) A blink detection method based on a DVS camera, comprising: a step of photographing a face using a DVS camera, thereby obtaining a DVS pixel stream; a step of integrating DVS pixels of the DVS pixel stream to form a plurality of DVS frames, each DVS frame of the plurality of DVS frames including a plurality of first color pixels and a plurality of second color pixels, each of the first color pixels being associated with one or more DVS pixels indicating a lightening event, and each of the second color pixels being associated with one or more DVS pixels indicating a darkening event; and a step of determining whether or not a blinking action is present in at least one DVS frame of the plurality of DVS frames, wherein the step of determining whether or not a blinking action is present includes a step of determining whether or not a pattern in which first color regions and second color regions are distributed vertically exists within the eye region of the at least one DVS frame.

[0042] (Item 2) The blink detection method described in Item 1, wherein each DVS frame among the plurality of DVS frames further includes a plurality of third color pixels, the third color pixels being not associated with DVS pixels indicating a lightening event or DVS pixels indicating a darkening event.

[0043] (Item 3) The blink detection method according to any one of items 1 to 2, wherein the step of determining whether or not a blinking action is present further includes a step of determining that an eye closing action is in progress when a first color area is above the second color area in the pattern.

[0044] (Item 4) The blink detection method according to any one of items 1 to 3, wherein the step of determining whether or not a blinking action is present further includes a step of determining that an eye-opening action is in progress when a second color area in the pattern is above the first color area.

[0045] (Item 5) The step of determining whether or not a blinking motion is present includes: calculating an average height difference for each DVS frame of the at least one DVS frame, the average height difference being an average height difference between first color pixels and second color pixels in the eye region of a corresponding DVS frame; The blink detection method according to any one of items 1 to 4, further comprising a step of determining, when the absolute value of the difference in the average heights is greater than a threshold value, that a pattern in which the first color area and the second color area are distributed vertically exists in the eye area of ​​the corresponding DVS frame.

[0046] (Item 6) The step of calculating the average height difference is calculating a first average height of first color pixels in the eye region; calculating a second average height of second color pixels in the eye region; 6. The blink detection method according to any one of items 1 to 5, further comprising: determining a difference between a first average height and a second average height to calculate the difference in average heights.

[0047] (Item 7) After the step of determining whether or not a blinking motion is present, the method further includes a step of calculating a blink duration, and the step of calculating the blink duration includes: identifying a first set of adjacent DVS frames from the plurality of DVS frames, the average height difference of each DVS frame of the first set of adjacent DVS frames being positive and having an absolute value greater than a first threshold; identifying a second set of adjacent DVS frames from the DVS frames subsequent to the first set of adjacent DVS frames of the plurality of DVS frames, wherein a difference in average height of each of the second set of adjacent DVS frames is negative and its absolute value is greater than a second threshold; identifying a first reference DVS frame from the first set of adjacent DVS frames; identifying a second reference DVS frame from the second set of adjacent DVS frames; 7. The blink detection method according to any one of items 1 to 6, further comprising: calculating a blink duration based on a time difference between an imaging time of the first reference DVS frame and an imaging time of the second reference DVS frame.

[0048] (Item 8) A blink detection method according to any one of items 1 to 7, wherein the first reference DVS frame is a temporally middle DVS frame among the first set of adjacent DVS frames, and the second reference DVS frame is a temporally middle DVS frame among the second set of adjacent DVS frames.

[0049] (Item 9) A blink detection method according to any one of items 1 to 8, wherein the first reference DVS frame is a DVS frame among the first set of adjacent DVS frames having the largest absolute value of the average height difference, and the second reference DVS frame is a DVS frame among the second set of adjacent DVS frames having the largest absolute value of the average height difference.

[0050] (Item 10) After the step of determining whether or not a blinking motion is present, the method further includes a step of identifying a plurality of complete blinks, and the step of identifying each complete blink of the plurality of complete blinks includes: identifying a first set of adjacent DVS frames from the plurality of DVS frames, the average height difference of each DVS frame of the first set of adjacent DVS frames being positive and greater than a first threshold; identifying a second set of adjacent DVS frames from the DVS frames subsequent to the first set of adjacent DVS frames of the plurality of DVS frames, wherein a difference in average height of each of the second set of adjacent DVS frames is negative and greater than a second threshold; A blink detection method according to any one of claims 1 to 9, further comprising a step of identifying one complete blink based on the identified first set of adjacent DVS frames and the identified second set of adjacent DVS frames.

[0051] (Item 11) The blink detection method according to any one of items 1 to 10, further comprising a step of calculating a blink frequency using the shooting times of at least some of the DVS frames associated with the multiple complete blinks.

[0052] (Item 12) The blink detection method according to any one of items 1 to 11, further comprising a step of identifying the eye area within each DVS frame of the at least one DVS frame prior to the step of determining whether or not a blink action is present.

[0053] (Item 13) A blink detection system comprising: a DVS camera used for capturing images to obtain a DVS pixel stream; and a processor connected to the DVS camera and configured to execute the blink detection method described in any one of items 1 to 12.

[0054] (Item 14) A vehicle comprising a vehicle body and the blink detection system described in item 13 attached to the vehicle body.

[0055] The above description is merely an illustrative embodiment for explaining the principles of the present invention, and does not limit the scope of protection of the present invention. Those skilled in the art can make various modifications and improvements without departing from the spirit and essence of the present invention, and these modifications and improvements are also within the scope of protection of the present invention.

Claims

1. The steps include: capturing a face using a DVS camera and obtaining a DVS pixel stream; The steps include: integrating the DVS pixels of the DVS pixel stream to form a plurality of DVS frames, each of the plurality of DVS frames including a plurality of first color pixels and a plurality of second color pixels, each of the first color pixels being associated with one or more DVS pixels indicating a brightening event, and each of the second pixels being associated with one or more DVS pixels indicating a darkening event; The step includes determining whether or not a blinking motion is present in at least one of the plurality of DVS frames, A blink detection method based on a DVS camera, comprising the step of determining whether or not a blinking motion is present, wherein the step of determining whether or not a pattern exists in which a first color region and a second color region are distributed vertically within the eye region of at least one DVS frame.

2. The blink detection method according to claim 1, wherein each of the plurality of DVS frames further includes a plurality of third-color pixels, and the third-color pixels are not associated with a DVS pixel indicating a brightening event or a DVS pixel indicating a darkening event.

3. The blink detection method according to claim 1, further comprising the step of determining whether or not a blinking motion is present, which is the step of determining that an eye-closing motion is in progress when the first color region is above the second color region in the pattern.

4. The blink detection method according to claim 1, further comprising the step of determining whether or not a blinking motion is present, which is the step of determining that an eye-opening motion is in progress when the second color region is above the first color region in the pattern.

5. The step of determining whether or not a blinking motion is present is: The steps include: calculating the difference in average height of each DVS frame among the at least one DVS frame, wherein the difference in average height is the difference in average height between the first color pixel and the second color pixel in the eye region of the at least one DVS frame; A blink detection method according to any one of claims 1 to 4, further comprising the step of determining that a pattern exists in which a first color region and a second color region are distributed vertically in the eye region of at least one DVS frame when the absolute value of the difference in average height is greater than a threshold.

6. The step of calculating the difference in average height is, The steps include: calculating the first average height of the first color pixels in the eye region; The steps include calculating the second average height of the second color pixels in the eye region, A blink detection method according to claim 5, comprising the step of calculating the difference in average heights by subtracting a second average height from a first average height.

7. The process includes a step of determining whether or not a blinking motion is present, followed by a step of calculating the blinking duration, the step of calculating the blinking duration is: A first pair of adjacent DVS frames is identified from the plurality of DVS frames, and the difference in the average height of each DVS frame in the first pair of adjacent DVS frames is positive and its absolute value is greater than a first threshold, which is a step. A second set of adjacent DVS frames is identified from the DVS frames following the first set of adjacent DVS frames of the plurality of DVS frames, and the difference in the average height of each DVS frame in the second set of adjacent DVS frames is negative and its absolute value is greater than the second threshold, and this is a step. The steps include identifying a first reference DVS frame from the first set of adjacent DVS frames, The steps include identifying a second reference DVS frame from the second set of adjacent DVS frames, A blink detection method according to claim 5, comprising the step of calculating blink duration based on the time difference between the capture time of a first reference DVS frame and the capture time of a second reference DVS frame.

8. The blink detection method according to claim 7, wherein the first reference DVS frame is the DVS frame that is in the middle of time among the first set of adjacent DVS frames, and the second reference DVS frame is the DVS frame that is in the middle of time among the second set of adjacent DVS frames.

9. The blink detection method according to claim 7, wherein the first reference DVS frame is the DVS frame among the first pair of adjacent DVS frames in which the absolute value of the difference in average height is the largest, and the second reference DVS frame is the DVS frame among the second pair of adjacent DVS frames in which the absolute value of the difference in average height is the largest.

10. The step of determining whether or not a blinking motion is present further includes a step of identifying one or more complete blinks, the step of identifying one or more complete blinks is: A first pair of adjacent DVS frames is identified from the plurality of DVS frames, and the difference in the average height of each DVS frame in the first pair of adjacent DVS frames is positive and greater than a first threshold. A second set of adjacent DVS frames is identified from the DVS frames following the first set of adjacent DVS frames of the plurality of DVS frames, and the difference in the average height of each DVS frame in the second set of adjacent DVS frames is negative and greater than a second threshold, and A blink detection method according to claim 5, comprising the step of identifying one or more complete blinks based on a first set of adjacent DVS frames identified and a second set of adjacent DVS frames identified.

11. The blink detection method according to claim 10, further comprising the step of calculating the blink frequency using the capture time of at least some of the DVS frames associated with the one or more complete blinks.

12. A blink detection method according to any one of claims 1 to 4, further comprising the step of identifying the eye region within each DVS frame of at least one DVS frame, prior to the step of determining whether or not a blinking motion is present.

13. A DVS camera used to capture a DVS pixel stream, A blink detection system comprising: a processor connected to the DVS camera and configured to perform the blink detection method according to any one of claims 1 to 4.

14. The vehicle body and A vehicle comprising the blink detection system according to claim 13, which is attached to the vehicle body.

Images