Target acquisition and tracking system and method

Abstract

The present invention relates to a target acquisition and tracking system and method, and more specifically, to a target acquisition and tracking system and method capable of automatically identifying and tracking a specific target within a sensor image based on the gaze of a combat system operator. A target acquisition and tracking system is provided, comprising: an image display unit that displays information received through various sensors or detection equipment according to an embodiment of the present invention; an eye tracking unit that tracks the direction of an operator's gaze in real time and converts it into coordinate data; an image processing unit that identifies a target by mapping the target sensing image and the coordinate information of the eye tracking unit; and a tracking module unit that continues tracking of the identified target.

Description

Technology Field

[0001] The present invention relates to a target acquisition and tracking system and method, and more specifically, to a target acquisition and tracking system and method capable of automatically identifying and tracking a specific target within a sensor image based on the line of sight of a combat system operator. Background Technology

[0003] Conventional Combat Management Systems (CMS) have primarily utilized methods of selecting and designating targets within an image using physical input devices such as video feeds, joysticks, or touchpads. In this input method, the operator must identify a target on the screen and then physically manipulate the cursor to move it to the target's location. This process requires a lengthy procedure involving the eyes, brain, hands, input devices, and cursor movement.

[0004] In terms of speed, there is a time delay between the time it takes to recognize a target with the eyes and the time it takes to manipulate the input device with the hands. In other words, while the eyes can process information by moving in increments of hundreds of milliseconds per second, hand movements are relatively slow and require precise control. For high-speed moving targets, a problem arises where the target has already moved to a different location while the cursor is being moved to an accurate position.

[0005] Furthermore, inexperienced operators face issues where operational efficiency is reduced due to a lack of proficiency in manipulating physical input devices. Significant training time is required to skillfully handle physical input devices such as joysticks or touchpads. Particularly during high-speed maneuvers, even slight hand tremors or excessive manipulation can lead to fatal errors. Inexperienced operators may require multiple attempts or miss targets because they lack the hand-eye coordination to accurately designate them with the cursor, even if they visually recognize the target.

[0006] In multi-target situations, priority determination, target switching, and continuous designation must be performed simultaneously; however, inexperienced operators experience a rapid increase in cognitive load while handling these complex tasks. Problems arise where other targets are missed while designating one, and operators struggle to grasp the overall tactical situation due to their concentration on manipulating input devices.

[0007] In other words, conventional target tracking systems face problems such as the time required to physically move a cursor to select high-speed maneuvering targets, or reduced responsiveness to such targets due to errors. Furthermore, the sequential execution of target selection, designation, and tracking commands leads to limitations in operational throughput, resulting in reduced operational efficiency, particularly in multi-target situations. Such delayed target designation can delay the engagement timing via sensors and weapon systems, presenting a critical disadvantage that reduces tactical effectiveness. Prior art literature

[0009] KR 10-2349854 The problem to be solved

[0010] The present invention provides a target acquisition and tracking system and method that can solve the aforementioned conventional problems by using eye tracking technology that uses the operator's gaze as a target input signal to calculate gaze coordinates on the screen in real time and match them with object information within the image to automatically designate the object the gaze is directed toward as a target candidate, thereby eliminating mechanical operation steps mediated by the hand and enabling automatic target designation and tracking without a separate cursor movement process by using the eye, which has the fastest reaction speed, as a direct input tool. Furthermore, by minimizing the procedure between observation and operation, it enables a delay-free response even in situations involving high-speed maneuvering targets or multiple targets simultaneously, and improves operation efficiency and mission execution speed by allowing even unskilled users to make accurate inputs solely through intuitive gaze, as it does not require separate operation training or proficiency. means of solving the problem

[0012] A target acquisition and tracking system is provided, comprising: an image display unit that displays information received through various sensors or detection equipment according to an embodiment of the present invention; an eye tracking unit that tracks the direction of an operator's gaze in real time and converts it into coordinate data; an image processing unit that identifies a target by mapping the target sensing image and the coordinate information of the eye tracking unit; and a tracking module unit that continues tracking of the identified target.

[0013] The above-mentioned eye tracking unit may include: an eye sensing unit that senses the operator's gaze; an eye coordinate calculation unit that calculates image coordinates corresponding to the sensed gaze; a calibration unit that finally corrects the calculated image coordinates; and an eye output unit that outputs the corrected gaze image coordinates.

[0014] The above gaze sensing unit may include a gaze illumination unit that illuminates the operator's face area; a gaze image acquisition unit that acquires an image of the operator's face area; and an image conversion unit that converts the image.

[0015] The above gaze coordinate calculation unit may include: an image detection unit that detects the position of a face and eyes within image information; a gaze vector calculation unit that calculates a gaze vector from the detected eye information; and a coordinate calculation unit that calculates gaze image coordinates by mapping the gaze vector to image coordinates.

[0016] The image processing unit may include: an ROI setting unit that generates an ROI within a sensor image based on line-of-sight image coordinates; a target identification unit that identifies a target object within the ROI area; a target information calculation unit that calculates identified target information; and a target output unit that outputs target information.

[0017] The above ROI setting unit receives gaze coordinates based on the sensor image through the gaze output unit and sets an ROI area centered on the received gaze coordinates. The ROI area can be formed as a circle with a radius of 0.1 to 5% of the total number of pixels when the total number of pixels is 100, or as a circle with a radius of 5 to 10% of the length of the long side of the screen when the long side length is 100.

[0018] The target identification unit may include: an identification area generation unit that generates an identification area based on an ROI area; a candidate detection unit that detects object candidates within the identification area; a target verification unit that classifies targets from the detected candidate objects; and a target display unit that displays the verified targets within an image.

[0019] The tracking module may include a tracking input unit that receives target information from the image processing unit; a tracking state management unit that tracks target movement and manages the state; a multi-target management unit that manages and adjusts priorities for multiple targets; and a tracking output unit that outputs target tracking results.

[0020] The tracking input unit determines whether it is a new target based on the target's location distance, size difference, and external shape difference, and the tracking status management unit can manage the target's acquisition status, stable tracking status, temporary loss status, and tracking failure or termination status.

[0021] The above tracking module may further include a tracking prediction unit that predicts the state of the target based on target information.

[0022] The above tracking prediction unit calculates the expected position of a target based on a Kalman filter prediction and compares the predicted target position value, i.e., the prediction information, with the tracking value of the normal tracking state of the tracking state management unit, i.e., the tracking information; if the difference exceeds the error, it can be determined as an outlier or the error can be corrected through re-searching.

[0023] It may further include a user interface section that performs tracking maintenance and tracking release on a target based on operator input.

[0024] The above user interface unit recognizes the operator's voice through a voice recognition module and performs an action according to the recognized voice, is equipped with a gesture recognition module to perform a predetermined action according to the operator's gesture, and can notify the operator of the corresponding action via voice or on the screen before the action is performed through a feedback module.

[0025] It may further include a gaze intent analysis unit that identifies the operator's observation intent from the operator's gaze pattern.

[0026] The above-mentioned gaze intention analysis unit is equipped with an analysis module that analyzes gaze patterns to analyze gaze duration, gaze repeatability, and gaze movement path as time-series data, thereby enabling raising the priority of a target, i.e., an object, emphasizing the ROI, immediately designating a target, or performing rapid tracking of a target.

[0027] It may further include a gaze guidance marker unit to guide gaze toward an abnormal object based on sensor information.

[0028] The above-mentioned gaze guidance marker unit generates a visual marker on the image display unit to guide the operator's gaze to the relevant area when there are abnormal signs or targets requiring priority verification among the objects detected and classified by the image processing unit, and can display the marker based on target importance, risk level, and approach speed.

[0029] In addition, the present invention provides a target acquisition and tracking method comprising: a process of tracking the direction of an operator's gaze according to the present invention and converting it into image coordinate data; a process of identifying and selecting a target by matching a target sensing image and image coordinate data; a process of performing tracking on the identified target; and a process of maintaining tracking and releasing tracking on the target according to the operator's input. Effects of the invention

[0031] According to an embodiment of the present invention, by using the operator's gaze as a direct input signal, mechanical operation steps mediated by the hand or joystick are eliminated, and the reaction speed of the eye is utilized directly for target designation, thereby enabling a significantly faster response compared to existing methods.

[0032] In addition, by calculating and correcting gaze coordinates in real time and matching them with object information within the image, objects are automatically designated as target candidates simply by the user looking at them, enabling immediate target acquisition without cursor movement or menu selection.

[0033] In addition, by integrating observation and manipulation actions into a single process and minimizing procedural steps, it is possible to enable delay-free target selection and continuous tracking even in situations where high-speed maneuvering targets or multiple targets appear simultaneously.

[0034] In addition, the gaze-based interface does not require separate equipment operation training or proficiency, allowing even beginners to perform accurate command inputs simply through intuitive observation, thereby improving overall operational efficiency and mission execution speed.

[0035] In addition, the structure integrating eye tracking, gaze intent analysis, automatic target designation, and high-speed tracking can be combined with various advanced services in the future, such as intelligent situational awareness, fatigue detection, UI optimization, and automated mission support. Brief explanation of the drawing

[0037] FIG. 1 is a drawing for illustrating a target acquisition and tracking system according to an embodiment of the present invention. FIG. 2 is a conceptual block diagram for explaining the eye tracking unit. FIG. 3 is a conceptual block diagram for explaining a gaze sensing unit. FIG. 4 is a conceptual block diagram for explaining the line of sight coordinate calculation unit. FIG. 5 is a conceptual block diagram for explaining an image processing unit. FIG. 6 is a conceptual block diagram for explaining a target identification unit. FIG. 7 is a conceptual block diagram illustrating the tracking module section. FIG. 8 is a flowchart illustrating a target acquisition and tracking method according to an embodiment of the present invention. FIG. 9 is a drawing for illustrating a target acquisition and tracking system according to one variant of the present invention. FIG. 10 is a drawing for illustrating a target acquisition and tracking system according to another variant of the present invention. Specific details for implementing the invention

[0038] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments of the present invention are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. To explain the invention in detail, the drawings may be exaggerated, and like reference numerals in the drawings refer to like elements.

[0040] FIG. 1 is a diagram illustrating a target acquisition and tracking system according to an embodiment of the present invention. In addition, FIG. 2 is a conceptual block diagram illustrating a gaze tracking unit, FIG. 3 is a conceptual block diagram illustrating a gaze sensing unit, FIG. 4 is a conceptual block diagram illustrating a gaze coordinate calculation unit, FIG. 5 is a conceptual block diagram illustrating an image processing unit, FIG. 6 is a conceptual block diagram illustrating a target identification unit, and FIG. 7 is a conceptual block diagram illustrating a tracking module unit.

[0041] Referring to FIGS. 1 to 7, a target acquisition and tracking system according to an embodiment of the present invention includes an image display unit (100) that displays information received through various sensors or detection equipment, an eye tracking unit (200) that tracks the direction of an operator's gaze in real time and converts it into coordinate data, an image processing unit (300) that identifies a target by mapping the target sensing image and the coordinate information of the eye tracking unit (200), a tracking module unit (400) that continues tracking of the identified target, and a user interface unit (500) that performs tracking maintenance and tracking release for the target according to the operator's input.

[0042] The target acquisition and tracking system according to the present embodiment analyzes the operator's gaze information in real time through the linkage between the eye-tracking device and the sensor image processing module, and automatically designates and tracks a target based on this, thereby enabling faster and more intuitive target tracking and management than existing physical operation methods, and maximizing operational efficiency in high-speed maneuvering target and multi-target situations.

[0043] The image display unit (100) visualizes information collected through various detection devices and sensors and provides it to the operator. The image display unit (100) uses a high-resolution display to effectively display information received from electro-optical tracking equipment (EOTS), infrared search and tracking equipment (IRST), camera images, radar, etc., as an image without delay. It is possible to use various types of monitors as the image display unit (100), and it is also possible to use a tactical status board. The image display unit (100) is equipped with a graphic overlay function so that target information, aiming lines, and system status information can be superimposed and displayed on the sensor images.

[0044] Through this, video collected from multiple sensors or tracking equipment can be streamed in real time, and effects such as marking points for target tracking or changing the color of the target box upon successful tracking can be applied. The video display unit (100) can intuitively display the target tracking status, ROI area, risk level, and priority, which are the outputs of the eye tracking unit (200), video processing unit (300), and tracking module unit (400).

[0045] The eye tracking unit (200) is effective in capturing the operator's eye movements and converting the gaze point into precise screen coordinates.

[0046] The eye tracking unit (200) includes an eye sensing unit (210) that senses the operator's gaze, an eye coordinate calculation unit (220) that calculates image coordinates corresponding to the sensed gaze, a calibration unit (230) that finally corrects the calculated image coordinates, and an eye output unit (240) that outputs the corrected gaze image coordinates.

[0047] It is effective for the gaze sensing unit (210) to actually photograph or measure the operator's eyes and face to sense image data and state information necessary for gaze estimation. It is effective for the gaze sensing unit (210) to precisely capture the operator's eye movements and distinguish between rapid gaze movement and fixation to acquire high-speed, high-resolution eye image data. Accuracy can be improved through binocular tracking.

[0048] The gaze sensing unit (210) of this example is effective in including a near-infrared LED or pattern light and a high-speed camera or a high-resolution camera. Through this, it is effective to illuminate the eye surface with IR light and acquire an image of it with the camera.

[0049] The gaze sensing unit (210) includes a gaze lighting unit (211) that illuminates the operator's face area, a gaze image acquisition unit (212) that acquires an image of the operator's face area, and an image conversion unit (213) that converts the image.

[0050] The gaze illumination unit (211) is effective in creating a corneal reflection point by shining near-infrared light onto the pupil, cornea, and iris. The gaze illumination unit (211) is effective in including an IR LED or a near-infrared light device. This is effective in forming a reflection point on the eye surface and minimizing the influence of external light.

[0051] The gaze image acquisition unit (212) is effective in capturing the operator's eyes and face area in continuous frames using a high-speed camera. The gaze image acquisition unit (212) is effective in using a camera capable of low-latency shooting of 120 to 250 Hz or higher using an IR camera and / or an RGB camera.

[0052] The image conversion unit (213) stores image information captured by the gaze image acquisition unit (212) and assigns a time stamp to each frame to synchronize with the system clock. This allows the time reference to be aligned with the gaze coordinate calculation unit (220) and the image processing unit (300). By applying an IR wavelength bandpass filter, noise can be removed to respond to environmental changes. The image conversion unit (213) provides image information to the gaze coordinate calculation unit (220).

[0053] The gaze coordinate calculation unit (220) receives image information of the eyes and face transmitted from the gaze sensing unit (210), extracts features such as the pupil and the reflection point of the cornea, and is capable of calculating a gaze vector and gaze image coordinates on the screen based on this.

[0054] The gaze coordinate calculation unit (220) includes an image detection unit (221) that detects the position of a face and eyes within image information, a gaze vector calculation unit (222) that calculates a gaze vector from the detected eye information, and a coordinate calculation unit (223) that maps the gaze vector to image coordinates to calculate the gaze image coordinates.

[0055] The image detection unit (221) is effective in finding face and eye regions within the image information and detecting the pupil center and corneal reflection point. At this time, it is desirable to extract information from both eyes simultaneously.

[0056] The image detection unit (221) is effective in finding face patterns through a contrast method or an edge direction distribution method. Of course, it is also possible to detect face regions within image information using deep learning-based face detection technology. At this time, it is also possible to detect the face and the locations of landmarks such as eyes, nose, and mouth simultaneously. Afterward, it is desirable for the image detection unit to extract an eye region within the detected face region. To do this, it is also possible to estimate the upper part of the face detection region as the eye region.

[0057] After calculating the eye region, the image detection unit (221) detects the center of the pupil. Since the pupil appears as a low-luminance region, it is possible to perform pupil location detection through binarization, convert the eye region to grayscale, filter pupil candidates based on area and shape, and then estimate the center point as the center of the pupil.

[0058] The image detection unit (221) can detect corneal reflection points by finding a strong brightness area in the eye region, generating a small cluster code through connection component analysis of bright pixels, and filtering candidate points based on area, major axis and minor axis ratios and average brightness for the generated candidates to calculate the remaining cluster center points as corneal reflection point coordinates.

[0059] The image detection unit (221) is capable of detecting eye opening, eye closing, gaze deviation, blinking, and head movements within the image information. It is desirable to apply this to detect abnormalities or subsequent analysis.

[0060] The gaze vector calculation unit (222) can estimate the gaze vector using the vector between the detected pupil center and the corneal reflection point. That is, the cornea is estimated as a spherical mirror, and the gaze direction can be determined using the relative position between the reflection point generated from the IR light source and the pupil center. Since the gaze vector has a linear relationship with the rotation of the eye (gaze direction), it may be possible to estimate the gaze direction by learning a mapping function with screen coordinates through calibration.

[0061] The gaze vector calculation unit (222) may estimate the gaze vector by applying a 3D eye model-based algorithm. That is, the gaze vector is estimated by assuming the eye as a 3D spherical model and geometrically considering the positions of the camera and IR light source. The 3D position of the reflection point is estimated by back-projecting the pixels observed in the camera image into the camera ray to intersect with the corneal model, and the 2D coordinates of the pupil center are back-projected into the camera coordinate system, i.e., normalized ray, and then the actual position of the pupil can be calculated by estimating the intersection point with the spherical model of the eye. Afterwards, it is preferable to define the vector going out from the center of the eye or the center of the pupil as the gaze direction.

[0062] The gaze vector calculation unit (222) can calculate the gaze vector using camera reference coordinates.

[0063] The coordinate calculation unit (223) effectively converts the calculated view vectors into 2D coordinates by matching them to image coordinates.

[0064] It is preferable for the coordinate calculation unit (223) to convert the gaze vector calculated based on the camera into an actual screen or image coordinate system. That is, it is possible to match the camera coordinate system centered on the camera sensor, the pixel-unit image coordinate system, and the screen coordinate system of the image display unit with the sensor image coordinate system by considering them.

[0065] The coordinate calculation unit (223) checks the camera position and the gaze vector, and defines the screen, that is, the screen, as a single 3D plane to calculate image coordinates through the intersection of the gaze rays. That is, the gaze rays are defined, the screen plane is defined, and then the intersection of them is calculated. Afterwards, the intersection is converted into 2D coordinates based on two axes on the screen plane, and then it is possible to calculate the coordinates by scaling them into pixel coordinates according to the resolution. Of course, the error in the coordinates calculated through the geometric model can also be minimized through data-based correction using a learned mapping function.

[0066] The coordinate calculation unit (223) may be capable of performing coordinate transformations corresponding to sensor image coordinates. In order to use the gaze coordinates as the ROI of the sensor image, it may be necessary to convert the coordinates of the image display unit (100) into sensor image coordinates. To do this, it is effective to inversely map the pixel area corresponding to the sensor area among the areas displayed on the display, i.e., the image display unit. Of course, in the case of multi-sensor fusion, it is desirable to manage projection and alignment matrices for each sensor to map the gaze coordinates to the corresponding sensor image.

[0067] It is desirable for the calibration unit (230) to perform correction on the image coordinates calculated by the gaze coordinate calculation unit (220). By doing so, it is effective to suppress variations depending on the operator and environment, thereby enabling the optimal gaze tracking function to operate.

[0068] The calibration unit (230) can perform multiple reference setting operations when the system is initially started or when an operator is positioned in front of the image display unit (100). That is, it is desirable to display multiple initial reference points on the image display unit (100) and, when the operator gazes at each reference point, store the calculated gaze coordinate value at that moment and the actual reference point location as pairs. It is effective for the calibration unit (230) to estimate parameters for coordinate transformation using the pairs collected through a matching model. Through this, it is effective to calculate the average error and the maximum error by applying the learned parameters to the recalculated coordinates and the actual coordinates.

[0069] Through such a calibration unit (230), changes caused by changes in head position, changes in lighting, etc., can be reduced, and a natural gaze pattern can be corrected. In other words, it is possible to offset differences in operator characteristics and secure consistent accuracy.

[0070] The gaze coordinate output unit (220) provides the final gaze image coordinates generated through correction by the calibration line unit (230) to the outside, namely the image processing unit (300) and the tracking module unit (400).

[0071] It is desirable for the gaze output unit (240) to package the gaze image coordinates and related metadata according to a predefined message format. It is desirable to provide the packaged gaze data to the outside through wired or wireless communication methods.

[0072] The image processing unit (300) maps the coordinates received from the gaze tracking unit (200) to the sensor image to identify targets within the area being viewed by the operator. It is also possible to set a region of interest (ROI) based on the gaze image coordinates, search for objects within the image centered on this, and designate the objects as target candidates.

[0073] The image processing unit (300) includes an ROI setting unit (310) that generates an ROI within a sensor image based on line-of-sight image coordinates, a target identification unit (320) that identifies a target object within the ROI area, a target information calculation unit (330) that calculates identified target information, and a target output unit (340) that outputs target information.

[0074] The ROI setting unit (310) is effective in setting the region of interest within the sensor image using the gaze image coordinates.

[0075] The ROI setting unit (310) can receive gaze coordinates based on the sensor image through the gaze output unit (240). It is effective to set an ROI area centered on the provided gaze coordinates. At this time, it is desirable to convert the gaze coordinates into sensor image coordinates and manage them. In this example, the ROI area is formed in a circular shape, but it is not limited to this and can be formed in a polygonal or elliptical shape. The following description focuses on forming it in a circular shape.

[0076] By setting the region of interest through the ROI setting unit (310), the amount of computation can be significantly reduced, thereby improving performance for real-time target detection and reducing errors in target search, and enabling rapid response by prioritizing the processing of the region of interest even for high-speed maneuvers and multiple targets.

[0077] It is effective to form the ROI area as a circle with a radius of 0.1 to 5% of the total number of pixels, assuming the total number of pixels is 100. This allows for the identification of the area the gaze is directed toward, and in particular, enables the accurate setting of the perception point through the reflection of resolution.

[0078] Not limited to this, the ROI area can be set as a circular shape with a radius of 5 to 10% of the screen's long side length when the screen's long side length is set to 100. By setting the ROI area in proportion to the screen length in this way, if it is smaller than the above range, the size of the area of ​​interest becomes too small, making it difficult to set an accurate target, and if it is larger than the above range, there may be a disadvantage of delaying subsequent processing time due to the designation of too many areas.

[0079] The target identification unit (320) includes an identification area generation unit (321) that generates an identification area based on an ROI area, a candidate detection unit (322) that detects object candidates within the identification area, a target confirmation unit (323) that classifies targets from the detected candidate objects, and a target display unit (324) that displays the confirmed targets within the image.

[0080] The identification area generation unit (321) effectively processes and normalizes only the image frames corresponding to the ROI area in the sensor image information to maximize the computational efficiency of the subsequent process.

[0081] The identification area generation unit (321) effectively uses the ROI area as the identification area and generates a subframe by newly processing the image frame corresponding to this identification area. At this time, it is effective to take a margin area of ​​-5% to +5% based on the ROI area of ​​the identification area. Through this, target distortion that may occur during frame processing or defects caused by calculation or processing errors can be reduced.

[0082] The identification area generation unit (321) can adjust filtering, brightness, or histograms through image preprocessing techniques, and can edit the image of the area through resolution changes or recycling algorithms. In addition, the exact location can be specified through coordinate mapping of the RIO area.

[0083] The candidate detection unit (322) is effective in extracting a group of target candidates within an identification area. The candidate detection unit is effective in extracting a group of target candidates through a background difference method, a motion detection method, or a bold detection method. Of course, it is not limited to these methods, and it is also possible to perform object detection based on deep learning.

[0084] Since the candidate detection unit (322) analyzes only the identification area set based on the ROI area rather than the entire area, it is possible to select candidates quickly with less computation.

[0085] The candidate detection unit (322) can assign weights to the center point area of ​​the ROI region, i.e., the line of sight image coordinates, when selecting a candidate target.

[0086] The target identification unit (323) is effective in finally identifying and selecting the actual target among multiple target candidates detected through the candidate detection unit (322). The target identification unit (323) may identify the target through a CNN or deep learning model, calculate the distance from the rear center based on gaze data through gaze-based intent modeling, generate the target by utilizing gaze time or gaze pattern, and also generate the target based on rules including speed, approach direction, and distance among object candidates. The target identification unit (323) of this example may use any one of these methods or combine the methods.

[0087] It is desirable for the target identification unit (323) to classify targets, reflect their priority, and then evaluate the threat level to identify the targets. That is, it is effective to classify whether the candidate object is an aircraft, a ship, or a warhead, assign weights based on how far or close the candidate object is from the line-of-sight center, and evaluate the risk level based on its speed, size, and pattern to select the target. That is, it is desirable for the target identification unit (323) to select one object with the highest score among the candidate objects as the target. In addition, in the case of multiple targets, it is effective to mark an object larger than the reference value as the target based on the reference value.

[0088] Since the target verification unit (323) calculates the target using only the already filtered candidate list, the calculation speed can be fast, and it is possible to determine what the target intended by the operator is.

[0089] The target display unit (324) is effective in visually displaying the target confirmed by the target verification unit (323) on the actual image. This allows the operator to visually confirm the target at the point they are looking at.

[0090] The target display unit (324) can display a box, reticle, or icon around the target so that the operator can visually recognize it in the actual image. In the case of multiple targets, it is effective for the target display unit to display the mark more boldly as the score increases.

[0091] The target information calculation unit (330) is effective in calculating quantitative target information that can be used in the tracking and combat management system for the target calculated by the target identification unit (320).

[0092] It is desirable for the target information calculation unit (330) to calculate the position and size of the target within the image, as well as the velocity, direction, and acceleration of the target. It is also possible to analyze the size, thermal characteristics, and signal patterns of the target by estimating the target attributes and state. The target information calculation unit (330) may also assign a target ID to each target. Through the target information calculation unit (330), it is possible to manage targets consistently through structured information rather than as a simple point in the image.

[0093] The target output unit (340) can package the target-related information calculated by the target verification unit (323) and the target information calculation unit (330) into a single package and provide it to an external module via wired or wireless communication. That is, the target output unit (340) can package the target ID, location, speed, direction, type, threat level, and reliability information, and provide this to the tracking module unit (400), the combat management system, and the image display unit (100), etc. In the case of multiple targets, it is also possible to provide additional priority values.

[0094] The tracking module (400) locks on to continuously detect and identify targets identified by the image processing unit, assigns a separate tracking ID to each target in the case of multiple targets, manages the status, and displays the tracking results on the image display unit or provides them in real time to the CMS and weapon control unit.

[0095] The tracking module (400) includes a tracking input unit (410) that receives target information from an image processing unit (300), a tracking state management unit (420) that tracks target movement and manages the state, a multi-target management unit (430) that manages multiple targets and adjusts priorities, and a tracking output unit (440) that outputs target tracking results.

[0096] In addition, the tracking module (400) may have a tracking prediction unit (450) that predicts the state of a target based on target information, and may improve the positional accuracy by linking with the tracking state management unit (420).

[0097] The tracking input unit (410) receives target information input from the image processing unit (300) and assigns a new tracking ID to the target to perform a tracking session or match it with an existing tracking object. It is effective to receive target information provided through the target identification unit (320) of the image processing unit (300) and process and store it so that it can be utilized by the tracking status management unit (420).

[0098] It is desirable for the tracking input unit (410) to receive target location, target class, and detection reliability information on the screen or sensor image coordinates. It is possible to improve reliability and accuracy by matching time information regarding the movement of the target through time stamp information.

[0099] The tracking input unit (410) effectively assigns a unique tracking ID when the target information is a new target compared with the target currently being tracked. At this time, the comparison of the new target can be classified by reflecting the distance, the external characteristics of the target, and the results of the eye-tracking. In addition, it is also possible to determine whether it is a new target based on the positional distance, size difference, and external difference of the target using a separate association algorithm technology.

[0100] It is effective for the tracking status management unit (420) to track target movement and continuously manage and track the target status. It is desirable to track the status of each target, such as its position and speed, over time, and to manage whether the current status of the target is newly acquired, stable tracking, or lost.

[0101] The tracking state management unit (420) can define a state vector for a target. The state vector may include position coordinates on the image coordinates and velocity, that is, position change per frame. Additionally, if necessary, the state vector can be defined by including information on the target's size, acceleration, and direction angle.

[0102] The tracking status management unit (420) can calculate the final tracking value by utilizing the observation values ​​provided by the image processing unit (300) and the image display unit (100). Of course, it is also possible to calculate the final tracking value by reflecting the value predicted through the tracking prediction unit (450). Using this tracking value, it is possible to express the position of the target moving smoothly.

[0103] The tracking status management unit (420) is equipped with multiple status information and can manage the current tracking status. At this time, it is possible to include an acquisition status, a stable tracking status, a temporary loss status, and a tracking failure or termination status.

[0104] When a new target is input through the tracking input unit (410), it enters an acquisition state, and subsequently, if it is observed continuously and stably for more than a certain number of frames, it can be managed as a stable tracking state, indicating that the current tracking is stable. Additionally, if there is no observation value in a specific frame or if the quality of the frame deteriorates, it can be displayed as a temporary loss state, indicating a temporary error. Subsequently, if a new observation is successfully made at the predicted position or a position close to it (error range) within the frame that is input again, it can be returned to a stable tracking state. Furthermore, if the target is not tracked, i.e., observed, in a number of consecutive frames, it is possible to manage the tracking state as an observation failure or termination state.

[0105] In this example, if an abnormality occurs in the frame temporarily, it is not determined to be a tracking failure, and tracking is continued using the predicted value of the tracking prediction unit (450). When the frame is normalized, if the target is tracked near the predicted value, it is managed in a normal state, and if the frame is normalized but the target is not tracked, it is possible to make it a tracking failure or terminated state. Tracking can be maintained based on prediction without terminating tracking even in situations where it is temporarily obscured or there is sensor noise.

[0106] It is desirable for the multi-target management unit (430) to perform tracking on multiple targets in parallel and to allocate more resources to important targets by assigning priorities based on line of sight, threat level, and mission criteria.

[0107] The multi-target management unit (430) can determine new targets requiring new tracking by evaluating the correlation between the list of target candidates received from the tracking input unit (410) for each frame and the list of existing tracking objects. Of course, as mentioned above, the new target information determined by the tracking input unit can be received directly.

[0108] It is desirable for the multi-target management unit (430) to determine the priority of multiple targets and allocate tracking resources according to the priority. At this time, it is effective to determine the priority first based on the gaze time and recent gaze count and the distance between the gaze and the center of the target by the gaze tracking unit, and to determine it secondarily based on the relative speed, direction, and type of the target. That is, if multiple targets are selected through the first determination, it is effective to determine the final priority through the second determination. Through this, it is possible to track the movement of targets with higher priority more detailedly through higher frame update analysis. In addition, it is possible to make the display range of targets with higher priority different from other targets through the tracking output unit.

[0109] The tracking output unit (440) provides the tracking results of the target to the image display unit, CMS, or weapon control unit, and it is desirable to display them in a visualized manner, such as a bounding box, on the image display unit. It is effective for the tracking output unit (440) to continuously output the tracking of the target's movement after the target display based on the output of the preceding image processing unit (300).

[0110] The tracking output unit (440) can output the ID, speed, direction, and tracking trajectory of the target. It is also possible to output the position coordinates and azimuth within the image coordinates, and to output the tracking status, priority, and threat level. The tracking output unit (440) can display a bounding box or reticle around the target within the image display unit (100), or display a marker at the center point, and it is also possible to display the characteristics of the target being tracked on the screen by displaying different colors or different line thicknesses.

[0111] The tracking prediction unit (450) predicts the future location and speed of the target based on input information and, in conjunction with the tracking status management unit (420), maintains lock-on to the target even if observation is lost, and can achieve re-search and target search stabilization.

[0112] The tracking prediction unit (450) effectively calculates the predicted position of the next frame or several frames after the current state, as with the tracking state management unit (420), through Kalman filter-based prediction. To this end, it is possible to use a state transition matrix including a constant velocity model.

[0113] The tracking prediction unit (450) can compare the predicted target position value, i.e., the prediction information, with the tracking value of the normal tracking state of the target state management unit (420), i.e., the tracking information, and if the difference exceeds the error, it can determine it as an outlier or correct the error through re-searching.

[0114] The user interface unit (500) is effective in generating control signals for system control by inputting the operator's movements or voice. That is, it collects command inputs to assist and control gaze-based target tracking operations, and based on this, can perform target tracking or operations that match the operator's intention.

[0115] It is effective for the user interface unit (500) to be equipped with a voice recognition module. Through this, it can recognize the operator's voice and perform actions according to the recognized voice. It is also possible to be equipped with a gesture recognition module to perform actions determined by the operator's gestures. Through a feedback module, the corresponding action can be notified via voice or on the screen before the action is performed. When the operator gives a command via voice or gesture through such modules, the user interface first verifies the intent. If the user says "track" in voice, the eye tracking unit and the image processing unit first examine whether there is a targeted object, i.e., a target. Subsequently, if there is a target, it is recognized as a command to track that target, and accordingly, the tracking module can be activated to proceed with tracking that target. However, if there is no target, it is possible to perform a request to designate a target, i.e., a target to be tracked.

[0116] In other words, it is desirable to first determine the user's input intent based on eye-tracking results and perform actions according to that determination. Additionally, it is possible to transmit a command after verification and provide feedback to the operator via voice or screen regarding whether the command transmission was successfully executed.

[0117] The following describes a target acquisition and tracking method using a target acquisition and tracking system. Descriptions that overlap with the embodiments described above are omitted. Descriptions that are redundant with the embodiments described above are omitted, and the technology described below may be applied to the embodiments described above.

[0119] FIG. 8 is a flowchart illustrating a target acquisition and tracking method according to an embodiment of the present invention.

[0120] As illustrated in FIG. 8, the target acquisition and tracking method according to the present example includes a process of tracking the operator's line of sight and converting it into image coordinate data (S110), a process of identifying and selecting a target by matching the target sensing image and the image coordinate data (S120), and a process of performing tracking on the identified target (S130). Additionally, it may further include a process of maintaining tracking and releasing tracking on the target according to the operator's input (S140).

[0121] The process of converting image coordinate data by tracking the operator's gaze direction (S110) is effectively performed by sensing the operator's gaze using a camera and a light source, calculating image coordinates based on the sensed gaze, and then correcting and outputting them. To achieve this, it is effective to locate the face and eye regions within the image information and to detect the pupil center and the corneal reflection point. It is desirable to estimate the gaze vector using the vector between the detected pupil center and the corneal reflection point.

[0122] It is desirable to generate image coordinate data by converting the gaze vector into an actual screen or associative coordinate system.

[0123] The process of identifying a target by matching target sensing images with image coordinate data (S120) sets a region of interest within the sensor area based on image coordinate data derived from the line of sight. Subsequently, objects are detected only within the region of interest and set as target candidates. If there is only one object, the detected object becomes the target; if there are multiple objects, it is effective to classify the targets and select a target by evaluating their priority and threat level.

[0124] The process of tracking an identified target (S130) performs continuous detection and verification of the target based on the identified target information. First, it is determined whether the identified target is a new target or a target already being tracked. If it is a target already being tracked, the movement of the target is continuously tracked; if it is a new target, a target ID is assigned, and tracking operations are performed based on the target's location and speed. When tracking multiple targets, it is effective to allocate tracking resources according to the priority of the targets and to provide visibility to targets with higher priority on the screen.

[0125] The present invention is not limited to the description above and various modifications are possible. The following description describes a target acquisition and tracking system according to other variations of the present invention. Descriptions in the following description that overlap with the embodiments described above are omitted. Furthermore, the technology described below may be applied to the embodiments described above.

[0127] FIG. 9 is a drawing for illustrating a target acquisition and tracking system according to one variant of the present invention.

[0128] Referring to FIG. 9, a target acquisition and tracking system according to a modified example of the present invention includes an image display unit (600) that displays information received through various sensors or detection equipment, an eye tracking unit (700) that tracks the direction of the operator's gaze in real time and converts it into coordinate data, an eye gaze intention analysis unit (800) that confirms the operator's observation intention from the operator's gaze pattern, an image processing unit (900) that identifies targets and assigns priorities based on the operator's observation intention, coordinate data, and target sensing images, and a tracking module unit (1000) that continues tracking of the identified targets.

[0129] In this modified example, it is possible not only to extract image coordinate data by tracking the operator's gaze and extracting a direction vector based on the direction of movement, but also to identify the operator's gaze movement pattern from the gaze tracking results to determine the observation intent and provide this to the image processing unit to assign priority to the target.

[0130] The following describes the eye gaze intention analysis unit (800) of the example.

[0131] The gaze intent analysis unit (800) can determine whether the operator is simply looking at an area of ​​the screen or has an intention to select, observe, direct, and track a specific object, i.e., a target, and determine and assign the priority of the actions to be performed by the image processing unit (900) and the tracking module unit (1000).

[0132] It is effective for the gaze intent analysis unit (800) to be equipped with an analysis module that analyzes gaze patterns. This makes it possible to analyze gaze duration, gaze repeatability, gaze movement path, etc., as time-series data. As a result, it becomes possible to analyze gaze patterns. Furthermore, based on the results of the gaze pattern analysis, it is desirable to recognize the intention regarding whether the pattern is intended to select a target, to search for a target, or to make meaningless gaze movements.

[0133] It is effective to convey the operator's perceived intent by classifying it according to importance. For example, it is effective to elevate the priority of a short, repeated gaze to search for candidate targets and to emphasize the ROI. Conversely, it is effective to designate the center of the gaze as a target to indicate an intention to select an object for a long period. In the case of a sudden stop following a gaze movement, it is possible to enable rapid target discovery, allowing tracking to be performed via the tracking module simultaneously with target selection. Through this, it is possible to elevate the priority of a target (object), emphasize the ROI, immediately designate a target, or perform rapid tracking of the target, based on the intent analysis results of the gaze intent analysis unit.

[0135] FIG. 10 is a drawing for illustrating a target acquisition and tracking system according to another variation of the present invention.

[0136] Referring to FIG. 10, a target acquisition and tracking system according to an embodiment of the present invention includes an image display unit (1100) that displays information received through various sensors or detection equipment, an eye tracking unit (1200) that tracks the operator's gaze direction in real time and converts it into coordinate data, an image processing unit (1300) that identifies a target by mapping the target sensing image and the coordinate information of the eye tracking unit (1200), a tracking module unit (1400) that continues tracking of the identified target, and an eye guidance marker unit (1500) for guiding gaze toward an abnormal object based on sensor information.

[0137] Through this, if an abnormal object is present in the information collected from detection devices and sensors but cannot be identified by the operator's line of sight, it is possible to form a marker in the corresponding area on the image display to guide the operator's gaze toward it, thereby enabling rapid target identification and tracking.

[0138] The gaze guidance marker unit (1500) is effective in guiding the operator's attention to a target by generating a visual marker on the image display unit so that the operator's gaze reaches the area when there are abnormal signs among the objects detected and classified by the image processing unit (1300) or when there is a target that needs to be checked first.

[0139] It is preferable for the gaze guidance marker unit (1500) to be equipped with a separate abnormal object judgment module. Through this, it is possible to select a target for verification based on sensor information and image processing results. As mentioned above, it is preferable to judge objects as abnormal objects when the risk level increases depending on their speed, direction, and type. In addition, it is effective to improve visibility by displaying the marker shape, blinking, color, size, movement, etc., in the area corresponding to the target's location on the screen through a marker generation module. Furthermore, it is preferable for the marker to be removed when the operator's gaze reaches that area.

[0140] This enables the priority identification of high-risk targets based on target importance, risk, and approach speed, prevents the risk of operators missing targets, and optimizes eye fatigue by removing markers in conjunction with eye response.

[0141] In the foregoing, preferred embodiments of the present invention have been described and illustrated using specific terms, but such terms are intended solely to clarify the present invention, and it is obvious that various modifications and changes may be made to the embodiments and described terms of the present invention without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. Explanation of the symbols

[0143] 100, 600, 1100: Video display unit 200, 700, 1200: Eye tracking unit 210: Gaze Sensing Unit 220: Gaze Coordinate Calculation Unit 230: Calibration unit 240: Sight output unit 300, 900, 1300: Image processing unit 310: ROI setting unit 320: Target Identification Unit 330: Target Information Calculation Unit 340: Target output unit 400, 1000, 1400: Tracking module unit 410: Trace Input Unit 420: Trace Status Management Unit 430: Multi-target Management Unit 440: Tracking Output Unit 450: Tracking and Prediction Unit 500: User Interface Unit 800: Gaze Intention Analysis Unit 1500: Gaze Guidance Marker Unit

Claims

Claim 1 A target acquisition and tracking system comprising: an image display unit that displays information received through various sensors or detection equipment; an eye tracking unit that tracks the operator's gaze direction in real time and converts it into coordinate data; an image processing unit that identifies a target by mapping the target sensing image and the coordinate information of the eye tracking unit; a tracking module unit that continues tracking of the identified target; and an eye guidance marker unit for guiding gaze toward an abnormal object based on sensor information. Claim 2 A target acquisition and tracking system according to claim 1, wherein the gaze tracking unit comprises: a gaze sensing unit that senses the gaze of an operator; a gaze coordinate calculation unit that calculates image coordinates corresponding to the sensed gaze; a calibration unit that finally corrects the calculated image coordinates; and a gaze output unit that outputs the corrected gaze image coordinates. Claim 3 A target acquisition and tracking system according to claim 2, wherein the gaze sensing unit comprises: a gaze illumination unit that illuminates the face area of ​​an operator; a gaze image acquisition unit that acquires an image of the face area of ​​an operator; and an image conversion unit that converts the image. Claim 4 A target acquisition and tracking system according to claim 2, wherein the gaze coordinate calculation unit comprises: an image detection unit that detects face and eye positions within image information; a gaze vector calculation unit that calculates a gaze vector from the detected eye information; and a coordinate calculation unit that maps the gaze vector to image coordinates to calculate gaze image coordinates. Claim 5 A target acquisition and tracking system according to claim 1, wherein the image processing unit comprises: an ROI setting unit that generates an ROI within a sensor image based on line-of-sight image coordinates; a target identification unit that identifies a target object within an ROI area; a target information calculation unit that calculates identified target information; and a target output unit that outputs target information. Claim 6 A target acquisition and tracking system according to claim 5, wherein the ROI setting unit receives gaze coordinates based on a sensor image through a gaze output unit and sets an ROI area centered on the received gaze coordinates, wherein the ROI area is formed as a circle with a radius of 0.1 to 5% of the total number of pixels when the total number of pixels is 100, or is set as a circular shape with a radius of 5 to 10% of the length of the long side of the screen when the long side length is 100. Claim 7 A target acquisition and tracking system according to claim 5, wherein the target identification unit comprises: an identification area generation unit that generates an identification area based on an ROI area; a candidate detection unit that detects object candidates within the identification area; a target confirmation unit that classifies targets from the detected candidate objects; and a target display unit that displays the confirmed targets in an image. Claim 8 A target acquisition and tracking system according to claim 1, wherein the tracking module comprises: a tracking input unit that receives target information from the image processing unit; a tracking state management unit that tracks target movement and manages the state thereof; a multi-target management unit that manages and adjusts priorities for multiple targets; and a tracking output unit that outputs target tracking results. Claim 9 A target acquisition and tracking system according to claim 8, wherein the tracking input unit determines whether it is a new target based on the position distance, size difference, and external shape difference of the target, and the tracking status management unit manages the acquisition status, stable tracking status, temporary loss status, and tracking failure or termination status of the target. Claim 10 A target acquisition and tracking system according to claim 8, wherein the tracking module further comprises a tracking prediction unit that predicts the state of a target based on target information. Claim 11 A target acquisition and tracking system according to claim 10, wherein the tracking prediction unit calculates the expected position of a target based on a Kalman filter prediction, and compares the predicted target position value, i.e., the prediction information, with the tracking value of the normal tracking state of the tracking state management unit, i.e., the tracking information, and if the difference exceeds an error, determines it as an outlier or corrects the error through re-searching. Claim 12 A target acquisition and tracking system according to claim 1, further comprising a user interface unit that performs tracking maintenance and tracking release on a target according to operator input. Claim 13 A target acquisition and tracking system according to claim 12, wherein the user interface unit recognizes the operator's voice through a voice recognition module and performs an action according to the recognized voice, is equipped with a gesture recognition module and performs a predetermined action according to the operator's gesture, and notifies the operator of the corresponding action via voice or on the screen before the action is performed through a feedback module. Claim 14 A target acquisition and tracking system according to claim 1, further comprising a gaze intention analysis unit that identifies the operator's observation intention from the operator's gaze pattern. Claim 15 A target acquisition and tracking system according to claim 14, wherein the gaze intention analysis unit comprises an analysis module that analyzes gaze patterns to analyze gaze duration, gaze repeatability, and gaze movement path as time-series data, thereby increasing the priority of a target, i.e., an object, emphasizing an ROI, immediately designating a target, or performing rapid tracking of a target. Claim 16 delete Claim 17 A target acquisition and tracking system according to claim 1, wherein the gaze guidance marker unit generates a visual marker on an image display unit to guide the operator's gaze to reach the area when there is an abnormal sign or a target requiring priority verification among the objects detected and classified by the image processing unit, and displays the marker based on target importance, risk level, and approach speed. Claim 18 delete

Images