Image processing method, computer program, and image processing device

Abstract

To provide an image processing method and apparatus capable of quickly restoring eye coordinates when a detector has failed tracking of eyes or an eye region, in various applications for tracking a face principally based on pupils or on eyes and a nose.SOLUTION: A method with an image processing apparatus includes: acquiring an image frame; tracking a face region of a user based on first prior information obtained from at least one previous frame of the image frame; in response to a determination that the tracking of the face region has failed, setting a scan region in the image frame based on second prior information obtained from the at least one previous frame; and detecting the face region in the image frame based on the scan region.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The following embodiments relate to a video processing method, a computer program, and a video processing device. [Background technology]

[0002] The camera-based eye tracking technology may be used in many fields, such as a gaze-tracking-based glasses-free 3D super multi-view display and / or a HUD (Head Up Display). The performance of the camera-based eye tracking technology depends on the image quality of the camera and / or the eye tracking method. The stability of the camera-based eye tracking technology is reduced due to backlight, strong sunlight, dark low-light environment, and environment with sudden changes in roughness such as passing through a tunnel in the driving environment and / or the movement of the driver. Considering the actual usage environment of an Augmented Reality 3D HUD (Head Up Display) during driving, there is a need for a method of quickly redetecting the eye position and acquiring the eye coordinates when tracking of the user's eyes fails due to the influence of the user's movement or roughness. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2000-070225A Summary of the Invention [Problem to be solved by the invention]

[0004] According to one embodiment, in various applications of face tracking around the pupil center or around the eyes and nose, the object is to quickly recover eye coordinates when the detector fails to track the eye or eye region.

[0005] According to one embodiment, when face tracking fails, the speed of detecting eye coordinates is improved by limiting the scan area for detecting eyes or eye areas from a video frame based on prior information (e.g., a previously detected face area) instead of scanning the entire video frame.

[0006] According to one embodiment, if detection of a face area fails even in a reset scan area, the size of the window in which the scan area is set is gradually expanded to reduce the latency when re-acquiring eye coordinates. [Means for solving the problem]

[0007] According to one embodiment, a video processing method includes a step of acquiring a video frame, a step of tracking a user's facial area based on first prior information acquired from at least one previous frame of the video frame, a step of setting a scan area in the video frame based on second prior information acquired from the at least one previous frame after tracking the facial area based on the first prior information, and a step of detecting the facial area from the video frame based on the scan area.

[0008] The second prior information may include information regarding the at least one previous scan area based on detection of the face area being performed from the at least one previous frame, and the step of setting the scan area may include a step of re-setting the scan area to an area expanded from the at least one previous scan area.

[0009] The step of resetting the scan area may include a step of resetting the scan area by sequentially expanding a size of a window for setting the previous scan area based on whether tracking of the face area from the at least one previous scan area has failed.

[0010] The step of resetting the scan area by successively expanding the size of the window may include a step of successively expanding the size of a window for setting the previous scan area in response to the number of times tracking of the face area from the previous scan area has failed, and a step of resetting the scan area according to the size of the successively expanded window.

[0011] The step of sequentially expanding the size of the window in accordance with the number of times tracking of the face area has failed may include at least one of the steps of: expanding the size of the window to a first window size obtained by expanding the previous scan area upward, downward, left, and right based on a determination that tracking of the face area has failed once; expanding the size of the window to a second window size obtained by expanding the scan area left and right based on the first window based on a determination that tracking of the face area has failed twice; and expanding the size of the window to a third window size obtained by expanding the scan area upward and downward based on the second window based on a determination that tracking of the face area has failed three times.

[0012] The image processing method may further include setting an initial scan window corresponding to the scan area based on coordinates of a center of a user's pupil accumulated in the at least one previous frame.

[0013] The image processing method may further include a step of selecting an initial scan window corresponding to the scan area from among a plurality of candidate windows based on facial features of a user included in the image frame, and a step of setting the initial scan area using the initial scan window.

[0014] The step of selecting the initial scan window may include selecting the initial scan window from the plurality of candidate windows based on a position of a camera that captured the video frame and statistical position coordinates of the user.

[0015] The step of tracking the user's facial region may include the steps of aligning a plurality of predetermined feature points to a plurality of feature locations included in the facial region, and tracking the user's face based on the aligned plurality of predetermined feature points.

[0016] The aligning step may include mapping the predetermined feature points based on image information within the face region.

[0017] The aligning step may include aligning the predetermined feature points to a plurality of features contained within the face region and adjacent regions of the face region.

[0018] The first prior information may include at least one of coordinates of a pupil center of the user accumulated in the at least one previous frame, position coordinates of feature points corresponding to the face of the user in the at least one previous frame, and position coordinates of feature points corresponding to the eyes and nose of the user in the at least one previous frame.

[0019] The step of tracking the user's facial area may include the steps of generating a tracking map corresponding to the facial area based on the first prior information, and tracking the user's facial area according to the tracking map.

[0020] The step of generating the tracking map may include generating the tracking map based on a movable range of the face region based on the first prior information in the video frame.

[0021] The image processing method may further include outputting information regarding the detected face region of the user.

[0022] The step of outputting information regarding the user's face may include a step of outputting information regarding at least one of the positions of the pupils and nose included in the scan area, a viewpoint based on the position of the pupils, and a facial expression of the user expressed in the scan area.

[0023] The image frames may include at least one of color and infrared image frames.

[0024] According to one embodiment, a computer program stored on a computer-readable recording medium storing instructions that, when executed by at least one processor, cause the at least one processor to execute a video processing method, the video processing method includes a step of acquiring a video frame, a step of tracking a user's facial area based on first prior information acquired from at least one previous frame of the video frame, a step of setting a scan area in the video frame based on second prior information acquired from the at least one previous frame after tracking the facial area based on the first prior information, and a step of detecting the facial area from the video frame based on the scan area.

[0025] In one embodiment, the video processing device includes a sensor that acquires a video frame, a processor that tracks a user's facial area based on first prior information acquired from at least one previous frame of the video frame, and after tracking the facial area based on the first prior information, sets a scan area in the video frame based on second prior information acquired from the at least one previous frame, and detects the facial area from the video frame based on the scan area, and a display that outputs information regarding the detected facial area of ​​the user.

[0026] The second prior information includes information regarding the at least one previous scan area based on detection of the face area being performed from the at least one previous frame, and the processor can reconfigure the scan area to an area extending from the at least one previous scan area.

[0027] The processor can reconfigure the scan area by sequentially expanding the size of a window that sets the previous scan area based on whether tracking of the face area from the at least one previous scan area has failed.

[0028] The processor can sequentially expand the size of a window in which the previous scan area is set in response to the number of times tracking of the face area from the previous scan area has failed, and can re-set the scan area according to the size of the sequentially expanded window.

[0029] The processor may, based on a determination that tracking of the face area has failed once, expand the size of the window to a first window size that extends the previous scan area up, down, left, and right; based on a determination that tracking of the face area has failed twice, expand the size of the window to a second window size that extends the scan area left and right based on the first window; and based on a determination that tracking of the face area has failed three times, expand the size of the window to a third window size that extends the scan area up and down based on the second window.

[0030] The processor may set an initial scan window corresponding to the scan area based on coordinates of a user's pupil center accumulated in the at least one previous frame.

[0031] The processor can select an initial scan window corresponding to the scan area from among a plurality of candidate windows based on facial features of the user included in the video frame, and set the initial scan area using the initial scan window.

[0032] The processor may select the initial scan window from the plurality of candidate windows based on a position of a camera that captured the video frame and statistical position coordinates of the user.

[0033] The processor may align a plurality of predetermined feature points to a plurality of feature locations included in the face region, and track the user's face based on the aligned plurality of predetermined feature points.

[0034] The processor may map the predetermined feature points based on image information within the face region.

[0035] The processor may align the predetermined feature points to features contained within the face region and adjacent regions of the face region.

[0036] The first prior information may include at least one of coordinates of a pupil center of the user accumulated in the at least one previous frame, position coordinates of feature points corresponding to the face of the user in the at least one previous frame, and position coordinates of feature points corresponding to the eyes and nose of the user in the at least one previous frame.

[0037] The processor may generate a tracking map corresponding to the face region based on the first prior information, and track the face region of the user according to the tracking map.

[0038] The processor may generate the tracking map based on a movable range of the face region based on the first prior information in the video frame.

[0039] The display may output information regarding at least one of the positions of the pupils and nose included in the scan area, a viewpoint based on the pupil positions, and a facial expression of the user expressed in the scan area.

[0040] The image frames may include at least one of color and infrared image frames. Effect of the Invention

[0041] According to one aspect, in various applications of face tracking around the pupil center or around the eyes and nose, eye coordinates can be quickly recovered if the detector fails to track the eye or eye region.

[0042] According to one aspect, when face tracking fails, the speed of detecting eye coordinates can be improved by limiting the scan area for detecting eyes or eye regions from a video frame based on prior information (e.g., previously detected face regions) instead of scanning the entire video frame.

[0043] According to one embodiment, if detection of a face area fails even in a reset scan area, the size of the window in which the scan area is set can be gradually expanded to reduce latency when re-acquiring eye coordinates. [Brief description of the drawings]

[0044] [Figure 1] 4A to 4C are diagrams illustrating a process of tracking and detecting a face area according to an embodiment; [Diagram 2] 1 is a flowchart illustrating an image processing method according to an embodiment. [Diagram 3] 13 is a flowchart illustrating a video processing method according to another embodiment. [Figure 4] 13 is a flowchart illustrating a video processing method according to another embodiment. [Diagram 5]11A and 11B are diagrams illustrating a method for setting a position of an initial scan window according to an embodiment. [Figure 6] 11A and 11B are diagrams for explaining a method for setting a scan area according to an embodiment. [Figure 7] 1 is a diagram illustrating a method for tracking a user's face region according to an embodiment. [Figure 8] 1 is a block diagram of a video processing device according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] Specific structural or functional descriptions disclosed herein are merely exemplary for purposes of describing embodiments, and the embodiments may be embodied in various different forms and are not limited to the embodiments set forth herein.

[0046] Although terms such as first or second may be used to describe multiple components, such terms should be construed only for the purpose of distinguishing one component from the other components. For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0047] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, the terms "comprise" or "have" and the like indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0048] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and should not be interpreted as ideal or overly formal unless expressly defined in this specification.

[0049] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and duplicated description thereof will be omitted.

[0050] The embodiments described below may be utilized to track a user's eyes using an infrared camera or an RGB camera and output eye coordinates when using a glasses-free 3D monitor, a glasses-free 3D tablet / smartphone, a 3D head-up display (HUD) for a vehicle, etc. Also, the embodiments may be implemented, for example, in the form of a software algorithm in a chip in a monitor, or in the form of an app in a tablet / smartphone, or may be implemented as a hardware eye tracking device. The embodiments may be applied, for example, to autonomous vehicles, intelligent vehicles, smartphones, mobile devices, etc. Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals presented in each drawing indicate the same members.

[0051] 1 is a diagram illustrating a process of tracking and detecting a face region according to an embodiment of the present invention, which illustrates a process in which an image processing device according to an embodiment of the present invention scans an image frame 110 to detect a face region and tracks eyes and a nose included in the face region.

[0052] For example, in various situations of tracking a face around the pupil center or around the eyes and nose, assume that the eye detector fails to track the eyes or eye area in the video frame 110. If the user's eye tracking fails in the video frame 110, the video processing device can detect the user's eye and nose area, as in the video 120, using the scan window 115. As will be described in detail below, the scan window 115 is a window that sets a scan area in the video frame. For convenience of explanation, it is expressed as "tracking failed". From the viewpoint of performing tracking without excess or deficiency, it is preferable to expand the scan area and perform tracking after it is determined that the tracking has actually failed. However, it will be clear to those skilled in the art that it is not essential that the tracking has actually failed. For example, even if the initial tracking has not failed, the scan area may be expanded and tracking may be performed thereafter. In addition, the expansion of the scan area and tracking may be repeated multiple times regardless of the success or failure of tracking.

[0053] The image processing device may align a plurality of predetermined feature points at positions corresponding to the face, corresponding to the eye region and nose region included in the scan window 115 in the image 120. The predetermined feature points may be feature points corresponding to key points indicating facial features, such as the eyes and nose. The feature points may be displayed in the form of dots (●) as displayed in the images 120 and 130, as stars (*), or in various other forms. In the exemplary embodiment shown in FIG. 1, eleven feature points may be aligned at corresponding positions on the face.

[0054] The image processing device can track the user's face based on the aligned feature points in the image 120 and extract the user's pupils and / or the user's face area as in the image 130.

[0055] If extraction of the user's eye and / or user's face regions from the image 130 is unsuccessful, the image processor scans 140 the image frames 110 again.

[0056] When face tracking is lost and scanning is performed again to detect a face region, the image processing device according to an embodiment may improve the detection speed of eye coordinates by limiting the scan area for detecting eyes or eye areas from the image frame based on prior information (e.g., a face area or scan area previously detected, etc.) instead of scanning the entire image frame. In an embodiment, when the driver is captured using a fixed camera and is within a limited range of movement, such as when the driver in the vehicle is photographed, the scan area in the image frame is limited, thereby improving the detection speed of eyes or faces. In addition, according to an embodiment, when the face region fails to be detected from the limited scan area, the size of a window for resetting the scan area is sequentially expanded, thereby reducing the latency in detecting eye coordinates.

[0057] 2 is a flowchart showing an image processing method according to an embodiment. Referring to FIG. 2, an image processing device according to an embodiment acquires an image frame (S210). The image frame includes, for example, a color image frame and / or an infrared image frame. The image frame may be, for example, an image of a driver captured by a camera, an image sensor, etc. installed in a vehicle.

[0058] The image processing device tracks the user's face region based on first prior information acquired from at least one frame prior to the image frame acquired in step S210 (S220). If the image frame is acquired at a t viewpoint, the at least one prior frame is acquired from a previous viewpoint (e.g., t-1 viewpoint) or multiple previous viewpoints (e.g., t-1 viewpoint, t-2 viewpoint, t-3 viewpoint) based on the viewpoint at which the image frame is acquired. The first prior information may include, for example, coordinates of at least one pupil center of the user accumulated in at least one previous frame, position coordinates of feature points corresponding to the user's face in at least one previous frame, and position coordinates of feature points corresponding to the user's eyes and nose in at least one previous frame.

[0059] In step S220, the image processing device can align the predetermined feature points to the feature parts of the face area and track the user's face based on the aligned feature points. The "feature parts" refer to some parts or areas included in the face area of ​​the image frame, and may include, for example, the eyes, nose, mouth, eyebrows, glasses, etc. Here, the image processing device aligns the feature points to the feature parts included in the face area and / or adjacent areas of the face area. The image processing device can move (or map) the predetermined feature points based on image information in the face area.

[0060] In step S220, the image processing device determines whether tracking of the user's face area has been successful or failed based on the first prior information. A method for tracking the face area by the image processing device will be described in detail with reference to FIG.

[0061] In response to the determination that the facial region has not been tracked in step S220, the image processing device sets a scan area in the image frame based on second prior information acquired from at least one previous frame (S230). The second prior information may include, for example, information on at least one previous scan area based on the fact that the facial region has been detected in the at least one previous frame. The at least one previous scan area may be, for example, an initial scan area set in an initial scan window.

[0062] For ease of explanation, a window for setting a scan area in an image frame is referred to as a "scan window," and a scan window that is first used to set a scan area in a corresponding image frame is referred to as an "initial scan window." The scan area set by the initial scan window may be referred to as an "initial scan area." A method for setting the position and size of the initial scan window will be described in detail with reference to FIG. 5 below.

[0063] In one embodiment, "setting a scan area" may be understood to include not only setting a scan area, but also setting an initial scan area, adjusting or re-setting a scan area, etc.

[0064] In step S230, the image processing device resets the scan area, for example, by expanding the previous scan area. The image processing device can reset the scan area by sequentially expanding the size of the window in which the scan area is set based on whether tracking of the face area from the scan area has failed. The method in which the image processing device sets or resets the scan area will be described in detail with reference to FIG. 6 below.

[0065] The video processing device detects a face area from the video frame based on the scan area set in step S230 (S240). According to an embodiment, the video processing device outputs information about the face area detected in step S240. Here, the information about the user's face area may include, for example, the positions of the pupils and nose included in the scan area, a viewpoint based on the pupil positions, and a facial expression of the user expressed in the scan area. The video processing device can output information about the face area explicitly or implicitly. The expression "explicitly outputting information about the face area" means, for example, performing an operation such as displaying the positions of the pupils included in the face area and / or the facial expression expressed in the face area on a screen, and / or outputting the positions of the pupils included in the face area and / or the facial expression displayed in the face area by audio to a screen. Alternatively, the expression "implicitly outputting information about the face area" means, for example, performing an operation such as adjusting an image displayed on a HUD (Head Up Display) based on the positions of the pupils included in the face area and a viewpoint based on the pupil positions, or providing a service corresponding to the facial expression expressed in the face area.

[0066] 3 is a flowchart showing an image processing method according to another embodiment. Referring to FIG. 3, an image processing device according to an embodiment acquires an n-th image frame from a camera (S310). The image frame may be, for example, an RGB color image frame or an infrared (IR) image frame.

[0067] The image processing device determines whether the eyes and nose are detected from the previous (n-1)th image frame (S320). Here, the image processing device determines whether the eyes and nose are detected within the initial scan area of ​​the previous (n-1)th image frame, for example. In response to the determination that the eyes and nose are not detected in step S320, the image processing device sets or adjusts the scan area based on prior information to detect the eyes or the eyes and nose (S370), and aligns predetermined feature points (e.g., 11 feature points shown in FIG. 1) with the eyes or the eyes and nose detected in step S370 (S330).

[0068] Meanwhile, in response to the determination that the eyes and nose are detected in step S320, the image processing device aligns predetermined feature points, for example, to the detected eyes or the detected nose (S330). For example, the image processing device aligns predetermined feature points to a plurality of feature portions of the detected eyes or the detected eyes and nose included in the scan area or an area adjacent to the scan area in step S330. The predetermined feature points may be, for example, three feature points for each of the eyes, one feature point in the center between the eyes, one feature point at the tip of the nose, and three feature points for the mouth (or three feature points for the nose).

[0069] In step S330, the image processing device aligns the plurality of feature points to a plurality of feature parts included in the face region and the region adjacent to the face region. The image processing device moves the plurality of predetermined feature points based on image information in the face region to align them to the plurality of feature parts. The image processing device may recognize the positions of the plurality of feature parts corresponding to the user's eyes and nose from the face region of the image frame based on various methods such as a Supervised Descent Method (SDM) method that aligns feature points to the shape of an image using a descent vector learned from an initial shape configuration, an Active Shape Model (ASM) method, an Active Appearance Model (AAM) method, or a Constrained Local Models (CLM) method that aligns feature points based on a principal component analysis (PCA) of shape and shape. The image processing device aligns the plurality of predetermined feature points by moving (or mapping) them to the positions of the recognized plurality of feature parts. For example, if the image frame is an initial image frame, the plurality of feature points before alignment correspond to the average positions of the feature parts of the plurality of users. Also, if the image frame is not an initial image frame, the plurality of feature points before alignment corresponds to the plurality of feature points aligned based on the previous image frame.

[0070] The image processing device verifies the alignment result corresponding to a combination of a plurality of features according to the aligned feature points (S340). The image processing device verifies the alignment result of a face area corresponding to a combination of a plurality of features (e.g., eyes and nose) in the scan area based on image information in the scan area.

[0071] The image processing device verifies whether or not a plurality of feature parts in the scan area are a class corresponding to a combination of eyes and a nose, based on image information in the scan area. The image processing device can verify whether or not the scan area is a face class, for example, by using a verifier based on a scale invariant feature transform (SIFT) feature. Here, the "SIFT feature" can be acquired through the following two steps. The image processing device extracts candidate feature points with local maximum or minimum brightness in a scale space according to an image pyramid from the image data of the scan area, filters out feature points with low brightness ratios, and selects feature points to be used for image matching. The image processing device acquires a directional component based on the gradient of a surrounding area centered on the selected feature point, resets a region of interest for the acquired directional component, detects the size of the feature point, and generates a descriptor. Here, the descriptor corresponds to the SIFT feature. The "verifier" is a classifier that aligns feature points corresponding to the key points of the eyes and nose of each face stored in the training video DB with the face area of ​​the training video frame and learns SIFT features extracted from each aligned feature point. The verifier verifies whether the face area in which the feature points are aligned corresponds to an actual face class using video information within the face area of ​​the video frame. The verifier may be, for example, a support vector machine classifier (SVM). The verifier can be called a "face checker" in that it verifies the presence or absence of alignment with respect to the face area.

[0072] The image processing device determines whether or not the eyes are detected based on the verification result of step S340 (S350). In response to the determination that the eyes are not tracked in step S350, the image processing device sets a scan area based on the prior information and detects the eyes or the eyes and nose (S370).

[0073] In response to determining in step S350 that the eyes have been tracked, the video processor outputs eye coordinates or pupil coordinates (S360).

[0074] 4 is a flowchart showing an image processing method according to another embodiment. Referring to FIG. 4, an image processing device according to an embodiment determines whether a face region including eyes and nose is detected from an image frame (e.g., nth frame) (S410). If the eyes and nose are detected in step S410, the image processing device aligns a plurality of predetermined feature points with a plurality of feature parts of the face region (e.g., eyes, nose, middle between the eyes, pupils, etc.) (S440). The image processing device tracks the user's face region based on the plurality of feature points aligned in step S440 (S450).

[0075] On the other hand, if the face area is not detected in step S410, the image processing device sets a scan area in the image frame based on the prior information acquired in at least one previous frame (e.g., n-1 frame) of the image frame (S420). Here, the scan area may be set by a window for setting the scan area.

[0076] The image processing device determines whether a face area is detected from the scan area set in step S420, in other words, whether the face area is successfully detected from a video frame, such as the n-th frame, by using the set scan area (S430). In response to determining that the face area is successfully detected in step S430, the image processing device aligns a plurality of predetermined feature points to a plurality of feature portions of the face area (S440). The image processing device tracks the user's face area based on the plurality of feature points aligned in step S440 (S450).

[0077] Alternatively, in response to the determination that the face area has failed to be detected in step S430, the image processing device may reset the scan area by sequentially expanding the size of the window in which the scan area is set. In response to the determination that the face area has failed to be detected in step S430, the image processing device expands the size of the scan window (S460). In step S460, the image processing device may reset the scan area by sequentially expanding the size of the window in which the scan area is set based on whether or not the face area has failed to be tracked from the scan area. The image processing device resets the scan area in response to the size of the window sequentially expanded in step S460 (S420). The image processing device may reset the scan area by repeating the sequential expansion of the size of the scan window in response to the number of times the face area has failed to be tracked in the scan area. For example, the image processing device may reset the scan area and sequentially expand the size of the scan window each time the face area has failed to be tracked in the scan area.

[0078] In one embodiment, the number of iterations for expanding the size of the scan window may be preset by a user to a certain number (e.g., 3 or 4 times), and may be repeated until the face area is successfully tracked from the scan area, in other words, until the size of the scan window is expanded to the entire video frame. A method for the video processing device to set or reset the scan area will be specifically described with reference to the following FIG. 6.

[0079] 5 is a diagram illustrating a method for setting a position of an initial scan window according to an embodiment of the present invention, in which an initial scan window 517 is set in an image frame 510 according to an embodiment of the present invention.

[0080] The image processing device according to an embodiment selects an initial scan window 517 corresponding to a scan area from among a plurality of candidate windows 515 based on features of the user's face included in the face area of ​​the image frame 510, such as eyes, nose, mouth, eyebrows, and glasses. The image processing device selects the initial scan window 517 from among the plurality of candidate windows 515 based on, for example, the position of the camera that captured the image frame 510 and the statistical position coordinates of the user. Here, in an embodiment of face detection in a driving environment, the statistical position coordinates of the user are, for example, position coordinates obtained by averaging position coordinates to which a user sitting in the driver's seat of a vehicle can move on average. The image processing device can set the initial scan area according to the position of the initial scan window 517.

[0081] According to an embodiment, the image processing device sets an initial scan window 517 for a corresponding image frame based on the coordinates of the user's pupil center accumulated in at least one previous frame. For example, the image processing device can set the initial scan window 517 to an area with left and right margins based on the coordinates of the user's pupil center.

[0082] 6 is a diagram illustrating a method for setting a scan area according to an embodiment of the present invention, which illustrates an image 610 in which an initial scan window 611, a first window 615, an image 630 in which a second window 635, and an image 650 in which a third window 655 are set, according to an embodiment of the present invention.

[0083] An image processing device according to one embodiment can set a scan area by gradually expanding the size of a window for setting the scan area based on whether tracking of a face area in the scan area has failed.

[0084] For example, if tracking of a face region using the scan window fails once, the image processing device may expand the size of the scan window to the size of a first window 615 as shown in the image 610. The size of the first window 615 is, for example, a size obtained by expanding the initial scan region corresponding to the initial scan window 611 shown in Fig. 6 to the top, bottom, left, and right. The size of the first window 615 may be, for example, a size obtained by expanding the initial scan region to the top, bottom, left, and right by 5% each.

[0085] For example, if the face region tracking fails twice, the image processing device expands the size of the scan window to the size of the second window 635 as shown in the image 630 of FIG. 6. The size of the second window 635 is, for example, a size obtained by expanding the scan region to the left and right based on the first window 615 shown in FIG. 6. The size of the second window 635 is, for example, a size obtained by expanding the left and right sides of the first window 615 by a distance from the center between the eyes to each eye in the initial scan window 611 plus a certain margin (for example, 2 mm). For example, assume that the distance from the center between the eyes to the right eye (or the left eye) is 3.5 cm. This distance is obtained by averaging the distances from the middle part between the eyes to the right eye (or the left eye) of multiple users. In this case, the size of the second window 635 is a size obtained by expanding the left and right sides of the first window 615 by 3.7 cm.

[0086] For example, if tracking of the face region fails three times, the image processing device may expand the size of the scan window by the size of a third window 655 as shown in the image 650 of FIG. 6. The size of the third window 655 is, for example, a size obtained by expanding the scan region up and down based on the second window 635 shown in FIG. 6. The size of the third window 655 is, for example, a size obtained by expanding the second window 635 up and down by a distance of "the distance from the center between the eyes to each eye in the initial scan window 611 + a certain margin (for example, 2 mm)". For example, it is assumed that the distance from the center between the eyes to the right eye (or the left eye) is 3.5 cm. In this case, the size of the third window 655 is a size obtained by expanding the second window 635 up and down by 3.7 cm.

[0087] 6, the size of the first window 615 corresponds to the size of the initial scan window 611 expanded upward, downward, left and right, the size of the second window 635 corresponds to the size of the first window 615 expanded left and right, and the size of the third window 655 corresponds to the size of the second window 635 expanded up and down, but these are merely examples provided for illustrative purposes and are not limited to the expansion directions described herein, it is understood that each window may be expanded in any direction, upward, downward, left, right, or combination thereof, or in any other direction.

[0088] 7 is a diagram illustrating a method for tracking a user's face region according to an embodiment. Referring to FIG. 7, a video frame 710 and a tracking map 720 generated based on a previous frame of the video frame 710 according to an embodiment are shown.

[0089] An image processing device according to an embodiment generates a tracking map 720 corresponding to a face region of an image frame 710 based on prior information (e.g., first prior information) acquired from at least one previous frame of the image frame 710. The first prior information may include, for example, coordinates of at least one pupil center of a user accumulated in at least one previous frame, position coordinates of feature points corresponding to the user's face in at least one previous frame, and position coordinates of feature points corresponding to the user's eyes and nose in at least one previous frame.

[0090] The video processing device may determine a movement range of the movable face area in the video frame 710 based on, for example, the first prior information. Here, the movement range of the movable face area may be understood to include not only the movement range of the face area when the driver moves the upper body or head left and right while sitting in the driver's seat as shown in Fig. 7, but also the movement range of the face area when the driver rotates the upper body or head back and forth.

[0091] The video processing device generates a tracking map 720 based on the range of movement of the face region. The tracking map 720 may include coordinates corresponding to the maximum range of movement of the face region with respect to the top, bottom, left, and right directions to which the face region can move. The video processing device can track the user's face region using the tracking map 720.

[0092] Fig. 8 is a block diagram of an image processing device according to an embodiment. Referring to Fig. 8, an image processing device 800 according to an embodiment includes a sensor 810, a processor 830, a memory 850, a communication interface 870, and a display 890. The sensor 810, the processor 830, the memory 850, the communication interface 870, and the display 890 can communicate with each other via a communication bus 805.

[0093] The sensor 810 acquires video frames. The sensor 810 may be, for example, an image sensor, a vision sensor, or an infrared camera that captures input video with infrared illumination. The video frames may include, for example, a video of a user's face or a video of a user traveling in a vehicle.

[0094] The processor 830 tracks a face region of the user based on first prior information obtained from at least one previous frame of the video frame. In response to determining that tracking of the face region based on the first prior information is unsuccessful, the processor 830 sets a scan region in the video frame based on second prior information obtained from at least one previous frame. The processor 830 detects the face region from the video frame based on the scan region. The number of processors 830 may be one or more.

[0095] The processor 830 can perform at least one of the methods or an algorithm corresponding to at least one of the methods described above with reference to Figures 1 to 7. The processor 830 executes a program and controls the video processing device 800. The program code executed by the processor 830 may be stored in the memory 850. The processor 830 is configured with, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit).

[0096] The memory 850 stores the video frame acquired by the sensor 810, the first prior information and the second prior information acquired by the processor 830 from at least one previous frame of the video frame, the face region of the user acquired by the processor 830, and / or information related to the face region, etc. The memory 850 may also store information related to the face region detected by the processor 830. The memory 850 may be a volatile memory or a non-volatile memory.

[0097] The communication interface 870 receives video frames from outside the image processing device 800. The communication interface 870 outputs information related to the face area and / or the user's face area detected by the processor 830. The communication interface 870 receives video frames captured outside the image processing device 800 or information of various sensors received from outside the image processing device 800.

[0098] The display 890 displays the processing results of the processor 830, such as information related to the user's face area. For example, when the image processing device 800 is installed in a vehicle, the display 890 may be configured as a head-up display (HUD) provided in the vehicle.

[0099] The above embodiments may be implemented using hardware components, software components, or a combination of hardware and software components. For example, the devices and components described herein may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or other devices that execute and respond to instructions. The processing device executes an operating system (OS) and one or more software applications that run on the operating system. The processing device also accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the processing device may be described as being one in which only one processing device is used, but one skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0100] Software may include computer programs, codes, instructions, or any combination of one or more thereof, capable of configuring or instructing a processing device to operate as desired, either independently or in combination. The software and / or data may be embodied in any type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, either permanently or temporarily, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed across computer systems coupled to a network, and may be stored and executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

[0101] The method according to the present invention may be embodied in the form of program instructions to be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and the program instructions may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code, such as generated by a compiler, but also high-level language code executed by a computer using an interpreter, and the like. The hardware devices may be configured to operate as one or more software modules to perform the operations shown in the present invention, and vice versa.

[0102] Although the embodiments have been described above with reference to limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, or may be replaced or substituted with other components or equivalents to achieve appropriate results. [Explanation of symbols]

[0103] 800: Image processing equipment 805:Communication bus 810: Sensor 830: Processor 850:Memory 870:Communication interface 890: Display

Claims

1. acquiring a video frame; tracking a face region of a user based on first prior information obtained from at least one previous frame of the video frame; after tracking the face region based on the first prior information, setting a scan region in the video frame based on second prior information obtained from the at least one previous frame; detecting the face area from the video frame based on the scan area; The step of setting the scan area includes: selecting an initial scan window corresponding to the scan area from among a plurality of candidate windows based on facial features of the user included in the video frame, the initial scan window being selected from among the plurality of candidate windows based on a position of a camera that captured the video frame and a statistical position coordinate of the user; setting an initial scan area by the initial scan window; A video processing method comprising:

2. The second prior information includes information about at least one previous scan area obtained based on the detection of the face area being performed from the at least one previous frame; The image processing method of claim 1 , wherein the step of setting the scan area comprises the step of resetting the scan area to an area obtained by expanding the at least one previous scan area.

3. The image processing method of claim 2, wherein the step of resetting the scan area includes a step of resetting the scan area by sequentially expanding a size of a window for setting the previous scan area based on whether or not tracking of the face area has failed from the at least one previous scan area.

4. The step of resetting the scan area by successively expanding the size of the window includes: gradually expanding a size of a window for setting the previous scan area according to the number of times that tracking of the face area from the previous scan area has failed; resetting the scan area according to the size of the sequentially expanded window; The image processing method according to claim 3 , comprising:

5. The step of sequentially expanding the size of the window in response to the number of times tracking of the face region has failed includes: expanding the size of the window to a first window size obtained by expanding the previous scan area upward, downward, left, and right based on a determination that the face area has been tracked unsuccessfully once; expanding the size of the window to a size of a second window obtained by expanding the scanning area to the left and right based on the first window based on a determination that tracking of the face area has failed twice; expanding the size of the window to a size of a third window obtained by expanding the scanning area vertically based on the second window based on the determination that tracking of the face area has failed three times; The image processing method according to claim 4 , comprising at least one of the following steps:

6. The image processing method according to any one of claims 1 to 5, further comprising a step of setting an initial scan window corresponding to the scan area based on coordinates of a user's pupil center accumulated in the at least one previous frame.

7. The step of tracking a face region of the user includes: aligning a plurality of predetermined feature points with a plurality of feature portions included in the face region; tracking the user's face based on the aligned predetermined feature points; The image processing method according to any one of claims 1 to 6, comprising:

8. The image processing method according to claim 7 , wherein the aligning step includes the step of mapping the predetermined plurality of feature points based on image information within the face region.

9. The image processing method of claim 7 , wherein the aligning step includes a step of aligning the predetermined plurality of feature points to a plurality of feature portions included in the face region and adjacent regions of the face region.

10. 6. The image processing method of claim 1, wherein the first prior information includes at least one of coordinates of a pupil center of the user accumulated in the at least one previous frame, position coordinates of feature points corresponding to the user's face in the at least one previous frame, and position coordinates of feature points corresponding to the user's eyes and nose in the at least one previous frame.

11. A method for acquiring a video frame, comprising: tracking a face region of a user based on first prior information obtained from at least one previous frame of the video frame; after tracking the face region based on the first prior information, setting a scan region in the video frame based on second prior information obtained from the at least one previous frame; detecting the face area from the video frame based on the scan area; and tracking the user's face region includes: generating a tracking map corresponding to the face region based on the first prior information, the tracking map being generated based on a movable range of the face region in the video frame determined based on the first prior information; tracking a facial region of the user according to the tracking map; A video processing method comprising:

12. The video processing method of claim 1 , further comprising the step of outputting information regarding the detected face region of the user.

13. The image processing method of claim 12, wherein the step of outputting information about the user's face includes a step of outputting information about at least one of the positions of pupils and nose included in the scan area, a viewpoint based on the positions of the pupils, and a facial expression of the user expressed in the scan area.

14. The image processing method of claim 1 , wherein the image frames include at least one of color image frames and infrared image frames.

15. 1. A computer program stored on a computer-readable recording medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform a video processing method, comprising: The image processing method includes: acquiring a video frame; tracking a face region of a user based on first prior information obtained from at least one previous frame of the video frame; after tracking the face region based on the first prior information, setting a scan region in the video frame based on second prior information obtained from the at least one previous frame; detecting the face area from the video frame based on the scan area; The step of setting the scan area includes: selecting an initial scan window corresponding to the scan area from among a plurality of candidate windows based on facial features of the user included in the video frame, the initial scan window being selected from among the plurality of candidate windows based on a position of a camera that captured the video frame and a statistical position coordinate of the user; setting an initial scan area by the initial scan window; A computer program comprising:

16. a sensor for acquiring video frames; a processor that performs the steps of: tracking a face area of ​​a user based on first prior information obtained from at least one previous frame of the video frame; after tracking the face area based on the first prior information, setting a scan area in the video frame based on second prior information obtained from the at least one previous frame; and detecting the face area from the video frame based on the scan area; a display that outputs information regarding the detected facial region of the user; The step of setting the scan area includes: selecting an initial scan window corresponding to the scan area from among a plurality of candidate windows based on facial features of the user included in the video frame, the initial scan window being selected from among the plurality of candidate windows based on a position of a camera that captured the video frame and a statistical position coordinate of the user; setting an initial scan area by the initial scan window; 2. A video processing device comprising:

17. The second prior information is information about at least one previous scan area obtained based on which the face area detection is performed from the at least one previous frame; The image processing device of claim 16 , wherein the processor resets the scan area to an area that expands the at least one previous scan area.

18. The image processing device according to claim 17 , wherein the processor resets the scan area by sequentially expanding a size of a window that sets the previous scan area based on whether tracking of the face area from the at least one previous scan area has failed.

19. The image processing device of claim 18, wherein the processor sequentially expands a size of a window in which the previous scan area is set in response to a number of times that tracking of the face area from the previous scan area has failed, and resets the scan area according to the size of the sequentially expanded window.

20. The processor, Based on a determination that the face region has failed to be tracked once, the size of the window is expanded to a first window size obtained by expanding the previous scan region upward, downward, left, and right; Based on a determination that the tracking of the face area has failed twice, a size of the window is expanded based on the first window by a size of a second window obtained by expanding the scan area to the left and right; The image processing device according to claim 19 , further comprising: a step of: expanding a size of the window based on the second window to a size of a third window obtained by expanding the scan area vertically based on a determination that tracking of the face area has failed three times.

21. The image processing device according to any one of claims 16 to 20, wherein the processor sets an initial scan window corresponding to the scan area based on coordinates of a user's pupil center accumulated in the at least one previous frame.

22. The image processing device according to any one of claims 16 to 21, wherein the processor aligns a plurality of predetermined feature points to a plurality of characteristic parts included in the face region, and tracks the face of the user based on the aligned plurality of predetermined feature points.

23. The image processing device according to claim 22 , wherein the processor causes the predetermined feature points to be mapped based on image information within the face region.

24. The image processing apparatus of claim 22 , wherein the processor aligns the predetermined feature points to a plurality of features contained within the face region and adjacent regions of the face region.

25. 21. The image processing device of claim 16, wherein the first prior information includes at least one of coordinates of a pupil center of the user accumulated in the at least one previous frame, position coordinates of feature points corresponding to the user's face in the at least one previous frame, and position coordinates of feature points corresponding to the user's eyes and nose in the at least one previous frame.

26. A sensor for acquiring video frames; a processor that performs the steps of: tracking a face area of ​​a user based on first prior information obtained from at least one previous frame of the video frame; after tracking the face area based on the first prior information, setting a scan area in the video frame based on second prior information obtained from the at least one previous frame; and detecting the face area from the video frame based on the scan area; a display that outputs information regarding the detected facial region of the user; and tracking the user's face region includes: generating a tracking map corresponding to the face region based on the first prior information, the tracking map being generated based on a movable range of the face region in the video frame determined based on the first prior information; tracking a facial region of the user according to the tracking map; 2. A video processing device comprising:

27. The image processing device according to claim 16 , wherein the display outputs information regarding at least one of the positions of an eye and a nose included in the scan area, a viewpoint based on the position of the eye, and a facial expression of the user expressed in the scan area.

28. The image processing device of claim 16 , wherein the image frames include at least one of color image frames and infrared image frames.

Images