Target tracking device and method

Abstract

The present invention relates to a target tracking device and method, comprising: a sonar capable of receiving a sound wave signal from a target and generating image data including the movement path of the target; a first artificial neural network that receives the image data from the sonar and restores a lost section in the movement path of the target; and a second artificial neural network that estimates the expected movement path of the target using the restored image data, thereby improving the quality of the image data of the sonar and enabling continuous tracking of the target.

Description

Technology Field

[0001] The present invention relates to a target tracking device and method, and more specifically, to a target tracking device and method capable of restoring a lost section from image data of a sonar and predicting the expected movement path of a target using an artificial neural network. Background Technology

[0002] SONAR (SOund NAvigation and Ranging), also known as sound wave detector, refers to equipment used to identify the existence, location, and characteristics of target objects underwater. There are two types: passive sonar and active sonar, each with unique characteristics and applications.

[0003] Active sonar detects objects by having a transmitter emit a random signal and a receiver receive the signal that is reflected back from the target object. The emitted signal has preset frequency characteristics, making detection easy, and distance can be measured by determining the return time.

[0004] In contrast, passive sonar is a detector consisting solely of a receiver that detects vibrations generated by objects themselves, such as a ship's engines and propellers. While passive sonar is simple to design and inexpensive to build, it receives signals from animals and other vessels without filtering, requiring a vast amount of data to distinguish only the desired signals.

[0005] Meanwhile, passive sonar passively receives acoustic signals from targets, arranges them along the azimuth and time axes, and provides Bearing-Time-Intensity (BTI) display images that represent the signal strength in grayscale. However, in the actual marine environment where passive sonar is used, problems frequently occur where the signal is heavily mixed with noise or the signal in certain sections is completely lost due to physical obstacles, marine noise, and interference between multiple targets. As a result, it becomes difficult to track the continuous movement path of a target, such as its trajectory, and there is a problem where the accuracy of target identification and motion prediction in subsequent stages is significantly reduced.

[0006] In addition, there is a problem in that separation and tracking errors frequently occur when there are many targets or when trajectories intersect or overlap, causing failure in the separation and tracking of targets. Prior art literature

[0007] KR10-2017-0082080 A The problem to be solved

[0008] The present invention provides a target tracking device and method capable of continuously tracking the movement path of a target.

[0009] The present invention provides a target tracking device and method capable of separating and tracking the movement paths of multiple targets and predicting their expected movement paths. means of solving the problem

[0010] A target tracking device according to an embodiment of the present invention may include: a sonar detector capable of receiving a sound wave signal from a target and generating image data including a movement path of the target; a first artificial neural network that receives the image data from the sonar detector and restores a lost section in the movement path of the target; and a second artificial neural network capable of estimating the expected movement path of the target using the restored image data.

[0011] The first artificial neural network is an artificial neural network trained to restore the lost section of image data, and the second artificial neural network may be an artificial neural network trained to estimate the expected movement path of the target over time.

[0012] The first artificial neural network above may have an encoder-decoder structure capable of extracting multi-resolution features to restore a signal region corresponding to the movement path of a target from the image data and combining the extracted multi-resolution features.

[0013] The above encoder-decoder structure may include a U-Net structure.

[0014] The above-mentioned first artificial neural network can control the learning rate using the Early Stopping and ReduceLROnPlateau techniques.

[0015] The second artificial neural network above can separate targets from the restored image data into instance units and perform an instance separation function to identify signal regions for each target.

[0016] The above second artificial neural network may include a Mask R-CNN and a Transformer-based network.

[0017] The second artificial neural network may include: a mask generator capable of generating a mask that is distinguished from the background by extracting a pixel region corresponding to a target from the restored image data; a coordinate extractor capable of extracting coordinate information of the target from the mask and extracting valid coordinate information by removing abnormal coordinate information from the coordinate information; and a movement path estimator capable of generating predicted movement path information by estimating the predicted movement path of the target for the valid coordinate information.

[0018] If there are multiple targets in the above restored image data,

[0019] The above mask generator can separate multiple targets into instances to generate a mask for each target, extract valid coordinate information from the mask for each target, and the above movement path estimator can generate expected movement path information for each target.

[0020] The above movement path estimator can perform image processing to output the movement path, mask, and predicted movement path for each target to a display device in the same color.

[0021] The apparatus may further include a dataset generation unit capable of generating an image dataset for training the first artificial neural network; and a preprocessing unit capable of normalizing the image dataset and classifying the normalized image dataset into batch units that can be input to the first artificial neural network according to predetermined classification criteria.

[0022] The above dataset generation unit can generate simulated image data by simulating image data of a target from the sonar, generate synthetic image data by synthesizing at least one of noise and a lost section with the simulated image data, and generate the simulated image data and the synthetic image data as the image dataset for training the first artificial neural network.

[0023] A method for tracking a target according to an embodiment of the present invention may include: receiving image data generated using an acoustic signal received from a target; generating corrected image data by restoring a lost section in the image data using a first artificial neural network; and generating an expected movement path of the target from the corrected image data using a second artificial neural network.

[0024] The process of generating the above-mentioned corrected image data may include a process of removing noise included in the image data and restoring lost sections using the first artificial neural network including an encoder-decoder structure.

[0025] The process of generating the predicted movement path of the target may include the process of estimating the predicted movement path of the target as a straight trajectory or a curved trajectory from the corrected image data using the second artificial neural network including a Mask R-CNN and a Transformer-based network.

[0026] The process of generating the expected movement path of the target may include: a process of generating a mask that is distinguished from the background by extracting a pixel area corresponding to the target from the corrected image data; a process of extracting coordinate information of the target from the mask; a process of extracting valid coordinate information by removing abnormal coordinate information from the coordinate information; and a process of estimating the expected movement path of the target based on the valid coordinate information according to a least squares criterion or a robust estimation technique.

[0027] When multiple targets exist in the correction image data, the process of generating the expected movement path of the targets may include: a process of separating the multiple targets into instances using the correction image data; a process of generating instance-specific masks from the correction image data; a process of extracting instance-specific coordinate information from the masks; a process of extracting instance-specific valid coordinate information by removing abnormal coordinate information that deviates from a predetermined removal criterion from the coordinate information; and a process of estimating the instance-specific expected movement path using the instance-specific valid coordinate information.

[0028] The method may include: a process of preparing an image dataset for training the first artificial neural network; a process of training the first artificial neural network to generate corrected image data by restoring a lost section from the image dataset; and a process of training the second artificial neural network to generate an expected movement path of a target from the corrected image data.

[0029] The process of preparing the above image dataset may include: receiving existing image data from the above sound detector; generating simulated image data that mimics the existing image data; generating synthetic image data by synthesizing at least one of virtual noise and a virtual loss section with the simulated image data; and storing the simulated image data and the synthetic image data as the above image dataset.

[0030] The process of training the first artificial neural network and the process of training the second artificial neural network can be performed using a learning control technique that improves learning stability and convergence.

[0031] The above learning control technique may include at least one of a technique for adjusting the learning rate and a technique for controlling the learning end time.

[0032] The above learning control technique may include at least one of an Adam optimizer, an early termination technique, and a learning rate reduction technique.

[0033] The method further includes a process of outputting the correction image data to a display device, and the outputting process may include a process of superimposing the boundary of the mask and the predicted movement path onto the correction image data.

[0034] In the above outputting process, when multiple targets exist in the image data, the movement path for each target, the boundary of the mask, and the predicted movement path can be output to the display device in the same color.

[0035] The above image data includes a Bearing-Time-Intensity (BTI) image in which the signal strength is expressed as a brightness value by arranging the sonar signals along the azimuth and time axes, and

[0036] The above outputting process may further include at least one of the following: a process of storing the result image data obtained by superimposing the boundary of the mask and the expected movement path onto the correction image data in a lossless image file format, and a process of transmitting the result image data to an external terminal in real time using a network. Effects of the invention

[0037] According to an embodiment of the present invention, a deep learning-based artificial neural network can be used to correct missing sections in image data received from a sonar to generate corrected image data with improved quality. Additionally, the expected movement path of a target can be estimated using the image data with the missing sections corrected. Therefore, a target can be accurately tracked using high-quality image data, and a target can be stably tracked by predicting its expected movement path.

[0038] Even in multi-target environments where targets frequently overlap and cross, target identification accuracy and tracking reliability can be improved by reliably separating each target and tracking them individually.

[0039] By implementing heterogeneous artificial neural networks in a modular pipeline structure, compatibility between target tracking devices is ensured, and flexible operation and maintenance are enabled without changing the overall structure, even during system expansion or function replacement. Brief explanation of the drawing

[0040] FIG. 1 is a block diagram schematically showing a target tracking device according to an embodiment of the present invention. Figure 2 is a structural diagram of the signal processing unit shown in Figure 1. FIG. 3 is a flowchart showing a method for tracking a target according to an embodiment of the present invention. FIG. 4 is a flowchart specifically showing the process of training an artificial neural network using a target tracking method according to an embodiment of the present invention. FIGS. 5 to 7 are simulation screens exemplarily showing the state of tracking the movement path of a target using a target tracking method according to an embodiment of the present invention. Specific details for implementing the invention

[0041] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms, and the embodiments of the present invention are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. To explain the invention in detail, the drawings may be exaggerated, and like reference numerals in the drawings refer to like elements.

[0043] FIG. 1 is a block diagram schematically showing a target tracking device according to an embodiment of the present invention, and FIG. 2 is a structural diagram of a signal processing unit shown in FIG. 1.

[0044] Referring to FIG. 1, a target tracking device according to an embodiment of the present invention may include a sonar detector (100) capable of receiving a sound wave signal from a target and generating image data including the movement path of the target, a first artificial neural network (231) capable of receiving the image data from the sonar detector (100) and restoring a lost section in the movement path of the target, and a second artificial neural network (232) capable of estimating the expected movement path of the target using the restored image data.

[0045] The sonar (100) refers to a sonar that detects, identifies, and tracks a target by receiving and analyzing sound signals naturally generated by the target, without emitting sound signals itself. The target is an object that generates or reflects sound signals in an underwater environment and becomes the subject of detection, identification, and tracking by the sonar (100), and may include submarines, surface ships (vessels), unmanned underwater vehicles (UUV, AUV), torpedoes, underwater structures, etc. The sonar (100) may be operated for the purpose of distinguishing between sound signals generated by the target and interference sources or noise, such as marine organisms, seabed topography, and environmental noise caused by waves and rainfall.

[0046] Such a sonar (100) is installed in a submarine, surface ship, unmanned underwater vehicle, underwater structure, etc., that performs a mission, and can receive and analyze sound wave signals generated by a target to generate image data including the target's movement path. The image data is information to be output to a display device (300) connected to the sonar (100), and refers to screen information, such as a Bearing-Time-Intensity (BTI) image, for which bearing information obtained by performing beamforming on the received sound wave signal is arranged along a time axis and the intensity of each bearing component is displayed as color or gray scale information (see FIGS. 5 to 7).

[0047] The first artificial neural network (231) can receive image data from the sonar (100) and restore the lost section in the target's movement path.

[0048] The first artificial neural network (231) is a convolutional neural network including an encoder-decoder structure and skip connections, and can perform pixel-unit segmentation of an input image. This first artificial neural network (231) may include a U-Net structure having an encoder-decoder structure. Additionally, the first artificial neural network (231) may be trained using an Adam optimizer. This first artificial neural network (231) can segment areas corresponding to targets by applying U-Net to image data received from a sonar (100). U-Net is a convolutional neural network (CNN) structure designed to perform pixel-unit segmentation in images, and can stably extract target orientation strips even in low signal-to-noise ratio environments. The first artificial neural network (231) may be trained using a learning control technique to improve learning stability and convergence. At this time, the learning control technique may include a technique for adjusting the learning rate and a technique for controlling the learning end time. For example, the first artificial neural network (231) can be trained by controlling the learning rate of the Adam optimizer using an early stopping technique and a learning rate reduction technique.

[0049] The second artificial neural network (232) can predict the expected movement path of the target using the restored image data. The second artificial neural network (232) may include a Mask R-CNN and a Transformer-based network. In this case, the second artificial neural network (232) can predict the expected movement path of the target by applying a RANSAC (Random Sample Consensus) based algorithm to the output of the second artificial neural network (232).

[0050] This second artificial neural network (232) may include a mask generator (not shown) capable of generating a mask that is distinguished from the background by extracting a pixel region corresponding to a target from the restored image data, a coordinate extractor (not shown) capable of extracting coordinate information of the target from the mask and removing abnormal coordinate information from the coordinate information to extract valid coordinate information, and a movement path estimator (not shown) capable of generating a predicted movement path information by estimating the predicted movement path of the target for the valid coordinate information.

[0051] If there are multiple targets in the restored image data, the mask generator can separate the multiple targets by instance and generate a mask for each target. Additionally, the coordinate extractor can extract valid coordinate information from the target-specific masks, and the movement path estimator can generate predicted movement path information for each target.

[0052] At this time, the movement path estimator can perform image processing to output the movement path, mask, and predicted movement path for each target to the display device in the same color.

[0053] A target tracking device according to an embodiment of the present invention may further include a dataset generation unit (210) capable of generating an image dataset for training the first artificial neural network, and a preprocessing unit (220) capable of normalizing the image dataset and classifying the normalized image dataset into batch units that can be input to the first artificial neural network (231) according to a predetermined classification criterion.

[0054] The first artificial neural network (231) and the second artificial neural network (232) can be configured as a pipeline structure that processes data continuously or sequentially from input to output, as shown in FIG. 2. The first artificial neural network (231) and the second artificial neural network (232) can be provided as a single signal processing unit (230).

[0055] The dataset generation unit (210) can generate simulated image data by simulating image data of a target from a sonar (100), generate synthetic image data by synthesizing at least one of noise and a lost section to the simulated image data, and generate the simulated image data and the synthetic image data as an image dataset for training the first artificial neural network. At this time, existing image data may also be used as an image dataset for training the first artificial neural network.

[0056] The dataset generation unit (210) can receive existing image data from the sonar (100) and generate simulated image data that mimics the existing image data. Additionally, the dataset generation unit (210) can generate synthetic image data by synthesizing at least one of virtual noise and a virtual loss section to the simulated image data. The dataset generation unit (210) can augment image data for training the first artificial neural network (231) by generating simulated image data and synthetic image data from the existing image data. This will be explained further later.

[0057] The dataset generation unit (210) may prepare simulated image data and synthetic image data, or existing image data, simulated image data and synthetic image data as a dataset to train the first artificial neural network (231), such as training data. This dataset may be random image data in which virtual or random noise and lost sections are synthesized.

[0058] The preprocessing unit (220) can normalize image data, such as a dataset, generated by the dataset generation unit (210), and classify the normalized dataset into batch units that can be input into the first artificial neural network (231). For example, the preprocessing unit (220) can normalize the grayscale of the image data to [0, 1]. Then, the normalized image data can be classified into batch units according to a predetermined classification criterion. At this time, the classification criterion may be various, such as the type of target, the signal reception environment such as the reception time of the target signal (sound wave signal) and the reception season, and the number of image data predetermined.

[0059] The storage unit (240) can store existing image data received from the sound detector (100), image data generated by the dataset generation unit (210), and image data preprocessed by the preprocessing unit (220). At this time, the storage unit (240) can store the image data generated by the dataset generation unit (210) and the image data preprocessed by the preprocessing unit (220) separately.

[0060] Additionally, the storage unit (240) can store various data generated during the process of training the first artificial neural network (231) and the second artificial neural network (232). Additionally, the storage unit (240) can store various data obtained using the trained first artificial neural network (231) and the trained second artificial neural network (232).

[0061] The communication unit (250) can transmit to an external terminal and receive from an external terminal. Here, the external terminal may be a personal terminal or a server.

[0062] The display device (300) can display or output the sound wave signal received from the target by the sonar (100) as image data including the target's movement path. The display device (300) can arrange the target's movement path and bearing information along the time axis and display the intensity of each bearing component in color or brightness.

[0063] Additionally, the display device (300) can display various image data generated or output from the first artificial neural network (231) and the second artificial neural network (232). At this time, the display device (300) may display at least two image data superimposed. For example, the display device (300) may superimpose image data received from the sonar (100) and image data generated or output from the first artificial neural network (231) and the second artificial neural network (232) (corrected image data, result image data, etc.) and output them.

[0064] Additionally, the display device (300) may extract the displayed image data into a lossless image file format and store it in the storage unit (240). For example, the display device (300) may extract the displayed image data into a still image and store it in the storage unit (240) in a PNG (Portable Network Graphics) file format.

[0065] The communication unit (250) can transmit result data to an external terminal in real time using methods such as RTSP (Real-Time Streaming Protocol) streaming.

[0066] The remaining components, excluding the aforementioned sonar (100) and display device (300), such as the signal processor (200), can be implemented as a single module. Accordingly, the signal processor (200) can be easily integrated into other tracking devices for tracking the movement path or trajectory of a target.

[0068] Hereinafter, a method for tracking a target according to an embodiment of the present invention will be described.

[0069] FIG. 3 is a flowchart showing a target tracking method according to an embodiment of the present invention, and FIG. 4 is a flowchart specifically showing the process of training an artificial neural network using a target tracking method according to an embodiment of the present invention.

[0070] Referring to FIG. 3, a target tracking method according to an embodiment of the present invention may include a process of receiving image data generated using an acoustic signal received from a target (S110), a process of generating corrected image data by restoring a lost section in the image data using a first artificial neural network (231) (S120), and a process of generating an expected movement path of a target from the corrected image data using a second artificial neural network (232) (S130).

[0071] In the following description, image data refers to sonar image data received from a sound detector (100), and corrected image data refers to sonar image data in which a lost section has been restored from the image data. Additionally, the target's movement path refers to the trajectory of the target moving as shown in the image data, and the target's predicted movement path refers to the trajectory in which the target is expected to move based on the corrected image data. The result image data refers to sonar image data in which the corrected image data and the predicted movement path are superimposed on the image data.

[0072] First, a first artificial neural network (231) can be trained to generate corrected image data by restoring the lost section in the image data, and a second artificial neural network (232) can be trained to generate the expected movement path of a target using the corrected image data.

[0073] Referring to FIG. 4, the process of training the first artificial neural network (231) and the second artificial neural network (232) will be explained.

[0074] An image dataset to be used as training data can be generated or prepared to train the first artificial neural network (231) and the second artificial neural network (232).

[0075] The dataset generation unit (210) can collect existing image data received from the sonar (100) to prepare an image dataset (S11). Here, the existing image data is a Bearing-Time-Intensity (BTI) image in which the signal strength is expressed as a brightness value by arranging the sonar signal along the azimuth and time axes. The dataset generation unit (210) can generate simulated image data by simulating the prepared existing image data. At this time, the simulated image data can be generated to have a loss section similar in form to the loss section included in the existing image data.

[0076] When simulated image data is generated, the dataset generation unit (210) can generate synthetic image data by additionally synthesizing noise and lost sections to the simulated image data. At this time, the dataset generation unit (210) can randomly generate noise and lost sections and synthesize arbitrary or virtual noise and lost sections to the simulated image data. The dataset generation unit (210) can generate synthetic image data by synthesizing at least one of the noise and lost sections to the simulated image data according to the number of image data to be used for training the artificial neural network.

[0077] For example, if there are 10 existing image data and the number of image data to train the first artificial neural network (231) is 5,000, the dataset generation unit (210) can generate 10 simulated image data that simulate the existing image data. Then, the dataset generation unit (210) can augment (S12) the image data to train the first artificial neural network (231) by generating synthetic image data by randomly generating virtual or arbitrary noise and virtual or arbitrary loss sections and synthesizing them with the simulated image data. The number of image data may be at least 5,000, and if necessary, about 6,000 to 10,000 image data may be secured to train the first artificial neural network (231).

[0078] When such image data, for example, an image dataset, is generated or prepared, the prepared image dataset, for example, an existing image dataset, can be stored in the storage unit (240).

[0079] A preprocessing unit (220) can be used to preprocess an image dataset, such as an existing image dataset (S13). The preprocessing process of the image dataset can normalize the grayscale of the image data, such as the image dataset, and classify the image dataset into batch units for input into the first artificial neural network (231). For example, the process of normalizing the image dataset can set the grayscale of the image data to [0, 1]. In addition, the process of classifying the image dataset into batch units can classify the normalized image data into batch units according to predetermined classification criteria, such as the type of target, the signal reception environment such as the reception time of the target signal (sound wave signal), the reception season, and the number of predetermined image data.

[0080] In this way, the classified image dataset can be stored in the storage unit (240), and can be stored in the storage unit (240) separately from the existing image dataset.

[0081] Afterwards, the first artificial neural network (231) can be trained (S14) using the preprocessed image dataset.

[0082] The process of training the first artificial neural network (231) can be performed by inputting an image dataset in batches to the input side of the first artificial neural network (231) to train it to generate corrected image data by removing virtual noise synthesized from the image data and restoring the lost sections. At this time, the first artificial neural network (231) is an artificial neural network having an encoder-decoder structure configured to restore images, and may include a U-Net structure. This first artificial neural network (231) can receive image data containing noise and lost sections, extract and fuse multi-scale features, and generate corrected image data while maintaining the structural characteristics of the original signal.

[0083] In the process of training the first artificial neural network (231), the Adam optimizer can be applied to improve learning stability and the convergence speed of the restoration performance. Here, the Adam optimizer is a stochastic gradient descent-based optimization algorithm that simultaneously estimates the first moment (average of the gradient) and the second moment (average of the squares of the gradient) to adaptively adjust the learning rate, thereby suppressing oscillations and divergence that may occur in the early stages of training and ensuring stable restoration performance even in a limited training data environment.

[0084] In addition, during the process of training the first artificial neural network (231), by applying an Early Stopping technique that controls the training end time according to the change in validation loss and a Reduce LR On Plateau technique that adaptively reduces the learning rate, overfitting can be prevented and stable convergence of the recovery performance can be induced.

[0085] When correction image data is generated in the first artificial neural network (231), the second artificial neural network (232) can be trained (S15) to generate an expected movement path of the target from the correction image data.

[0086] The process of training the second artificial neural network (232) can estimate the expected movement path of the target based on the correction image data generated from the first artificial neural network (231). In the process of training the second artificial neural network (232), the expected movement path of the target can be predicted using a network structure based on Mask R-CNN and Transformer, or an artificial neural network having a structure functionally equivalent thereto. At this time, a RANSAC (Random Sample Consensus) based algorithm is applied to the output of the second artificial neural network (232) to estimate the expected movement path of the target as a straight trajectory or a curved trajectory.

[0087] Specifically, the process of training the second artificial neural network (232) involves extracting a pixel area corresponding to the target from the corrected image data to generate a mask that is distinct from the background, extracting coordinate information of the target from the mask, removing abnormal coordinate information from the coordinate information to extract valid coordinate information, and then estimating the expected movement path of the target based on the valid coordinate information according to the least squares criterion or robust estimation technique.

[0088] Mask generation is the process of distinguishing the background and target signals in the signal regions corresponding to each target within the corrected image data. These masks can be utilized as foundational data for estimating and tracking the target's expected movement path in subsequent processes.

[0089] By extracting coordinate information of the target from the generated mask and removing outlier coordinate information unrelated to the target's movement path, such as the trajectory, valid coordinate information for estimating the target's expected movement path can be extracted. At this time, when removing outlier coordinate information, an iterative sampling model estimation technique based on RANSAC (Random Sample Consensus) is applied to detect and remove outlier coordinate information candidates based on the rate of change of bearing, time interval, and distribution of signal strength, thereby maintaining only the main signal (inlier) forming the trajectory as valid coordinate information.

[0090] Furthermore, using valid coordinate information, a linear or curved model can be selected based on the target's motion characteristics to approximate it in the form of a continuous curve or line, thereby estimating and generating the target's expected movement path (bearing trajectory). In this process, the model selection can be automatically determined based on the residual distribution or fitting error. By representing the target's continuous movement characteristics based on the signal distribution accumulated along the time axis rather than the instantaneous signal position of a single frame, a stable expected movement path or trajectory can be generated even in the presence of noise or transient signal loss. Consequently, the process of estimating the target's expected movement path can enhance the tracking stability of the target. Additionally, when there are multiple targets, it enables target-specific matching and tracking for extended periods.

[0091] Meanwhile, multiple targets may exist in the corrected image data. In this case, during the process of generating the expected movement path of the targets, the corrected image data can be used to separate the multiple targets into instances. Then, instance-specific masks are generated from the corrected image data, instance-specific coordinate information is extracted from the masks, and abnormal coordinate information that deviates from a predetermined removal criterion is removed from the coordinate information to extract valid coordinate information for each instance. Finally, the expected movement path for each instance can be estimated using the valid coordinate information for each instance.

[0092] In this way, the process of training the first artificial neural network (231) and the second artificial neural network (232) can be carried out stepwise or sequentially, including the restoration process and the estimation of the expected movement path.

[0093] The process of training the first artificial neural network (231) and the second artificial neural network (232) may include an optimization process to improve learning performance.

[0094] The optimization process involves iteratively adjusting the internal settings and processing conditions of each process. Specifically, it refers to adjusting overall system operation parameters—such as the input data processing method, processing intensity of the restoration and separation stages, target matching criteria, trajectory update conditions, and buffer operation method—to ensure a balanced performance of detection stability, tracking continuity, and reduction of false positives and omissions. This is not limited to specific algorithms or numerical values, but rather refers to a system-level adjustment process designed to simultaneously satisfy reliability and real-time performance in actual operating environments.

[0095] The optimization process can be performed by evaluating performance based on RMSE, SSIM, and Precision / Recall metrics using various image data used in the learning process.

[0096] When the first artificial neural network (231) and the second artificial neural network (232) are trained, the trained first artificial neural network (231) and the second artificial neural network (232) can be used to track a target.

[0097] FIGS. 5 to 7 are simulation screens that exemplarily show the state of tracking the movement path of a target using a target tracking method according to an embodiment of the present invention.

[0098] Image data of a target including a lost section as illustrated in FIG. 5 can be received (S110) from a sonar (100). The image data received from the sonar (100) can be input into a first artificial neural network (231), that is, a learned first artificial neural network (231).

[0099] The first artificial neural network (231) can restore the lost section from the input image data to generate corrected image data (S120) and output it to a display device (300) so that the movement path of the target appears continuously as shown in FIG. 6.

[0100] Correction image data generated in the first artificial neural network (231) is input into the second artificial neural network (232), and the second artificial neural network (232) can generate an expected movement path of the target (S120) from the correction image data.

[0101] Meanwhile, if there are multiple targets in the correction image data, the second artificial neural network (232) can separate the multiple targets by instance, generate an expected movement path for each instance, and process the image so that it is output to the display device (300) in different colors for each instance as shown in FIG. 7.

[0102] In this way, when outputting the processing results of the first artificial neural network (231) and the second artificial neural network (232) to the display device (300), the result image data can be output by superimposing the mask boundary and the predicted movement path onto the corrected image data. Additionally, if necessary, image data received from the sonar (100) can be additionally superimposed and output.

[0103] In addition, various data output to the display device (300), such as image data, corrected image data, mask boundaries, predicted movement paths, result image data, etc., can be stored in a lossless image file format and transmitted in real time to an external terminal using a network. This is intended to simultaneously secure real-time operational interoperability and the effect of preserving data for post-analysis.

[0105] In the foregoing, preferred embodiments of the present invention have been described and illustrated using specific terms, but such terms are intended solely to clarify the present invention, and it is obvious that various modifications and changes may be made to the embodiments and described terms of the present invention without departing from the technical spirit and scope of the following claims. Such modified embodiments should not be understood separately from the spirit and scope of the present invention, but should be considered to fall within the scope of the claims of the present invention. Explanation of the symbols

[0106] 100: Sonar 200: Signal processor 210: Dataset Generation Section 220: Preprocessing Section 230: Signal processing unit 231: First artificial neural network 232: Second Artificial Neural Network 240: Storage Unit 250: Communication unit 300: Display device

Claims

Claim 1 A target tracking device comprising: a sonar capable of receiving a sound wave signal from a target and generating image data including the movement path of the target; a first artificial neural network that receives the image data from the sonar and restores a lost section in the movement path of the target; and a second artificial neural network capable of estimating the expected movement path of the target using the restored image data, wherein the image data is a Bearing-Time-Intensity (BTI) image in which the signal of the sonar is arranged along an azimuth and time axis and the signal intensity is expressed as a brightness value, the first artificial neural network is a U-Net structure having an encoder-decoder structure capable of extracting multi-resolution features to restore a signal region corresponding to the movement path of the target from the image data and combining the extracted multi-resolution features, and the second artificial neural network includes a Mask R-CNN and Transformer-based network. Claim 2 A target tracking device according to claim 1, wherein the first artificial neural network is an artificial neural network trained to restore a lost section of image data, and the second artificial neural network is an artificial neural network trained to estimate the expected movement path of the target over time. Claim 3 delete Claim 4 delete Claim 5 In claim 2, the first artificial neural network is a target tracking device capable of controlling the learning rate using Early Stopping and ReduceLROnPlateau techniques. Claim 6 A target tracking device according to claim 2, wherein the second artificial neural network is capable of separating targets from the restored image data into instances and performing an instance separation function to identify signal regions for each target. Claim 7 delete Claim 8 A target tracking device according to claim 2, wherein the second artificial neural network comprises: a mask generator capable of generating a mask that is distinguished from the background by extracting a pixel region corresponding to a target from the restored image data; a coordinate extractor capable of extracting coordinate information of the target from the mask and extracting valid coordinate information by removing abnormal coordinate information from the coordinate information; and a movement path estimator capable of generating predicted movement path information by estimating the predicted movement path of the target for the valid coordinate information. Claim 9 A target tracking device according to claim 8, wherein, when there are multiple targets in the restored image data, the mask generator can separate the multiple targets by instance to generate a mask for each target and extract valid coordinate information from the mask for each target, and the movement path estimator can generate expected movement path information for each target. Claim 10 In claim 8, the movement path estimator is a target tracking device capable of image processing to output a target-specific movement path, a mask, and an expected movement path in the same color to a display device. Claim 11 A target tracking device according to claim 1, further comprising: a dataset generating unit capable of generating an image dataset for training the first artificial neural network; and a preprocessing unit capable of normalizing the image dataset and classifying the normalized image dataset into batch units that can be input to the first artificial neural network according to a predetermined classification criterion. Claim 12 A target tracking device according to claim 11, wherein the dataset generating unit can generate simulated image data by simulating image data of a target from the sonar, generate synthetic image data by synthesizing at least one of noise and a lost section with the simulated image data, and generate the simulated image data and the synthetic image data as the image dataset for training the first artificial neural network. Claim 13 A method for tracking a target using a target tracking device described in claim 1, comprising: receiving image data generated using an acoustic signal received from a target; generating corrected image data by restoring a lost section in the image data using a first artificial neural network; and generating an expected movement path of a target from the corrected image data using a second artificial neural network. Claim 14 A method for tracking a target according to claim 13, wherein the process of generating the corrected image data includes the process of removing noise included in the image data and restoring the lost section using the first artificial neural network. Claim 15 A method for tracking a target according to claim 13, wherein the process of generating the expected movement path of the target comprises the process of estimating the expected movement path of the target as a straight trajectory or a curved trajectory from the corrected image data using the second artificial neural network. Claim 16 A method for tracking a target according to claim 13, wherein the process of generating an expected movement path of the target comprises: a process of generating a mask that is distinguished from the background by extracting a pixel area corresponding to the target from the corrected image data; a process of extracting coordinate information of the target from the mask; a process of extracting valid coordinate information by removing abnormal coordinate information from the coordinate information; and a process of estimating the expected movement path of the target based on the valid coordinate information according to a least squares criterion or a robust estimation technique. Claim 17 A method for tracking a target according to claim 16, wherein when a plurality of targets exist in the corrected image data, the process of generating an expected movement path of the targets comprises: a process of separating the plurality of targets into instances using the corrected image data; a process of generating instance-specific masks from the corrected image data; a process of extracting instance-specific coordinate information from the masks; a process of extracting instance-specific valid coordinate information by removing abnormal coordinate information that deviates from a predetermined removal criterion from the coordinate information; and a process of estimating an instance-specific expected movement path using the instance-specific valid coordinate information. Claim 18 A method for tracking a target according to any one of claims 13 to 17, comprising: a process of preparing an image dataset for training a first artificial neural network; a process of training the first artificial neural network to generate corrected image data by restoring a lost section from the image dataset; and a process of training the second artificial neural network to generate an expected movement path of the target from the corrected image data. Claim 19 A method for tracking a target according to claim 18, wherein the process of preparing the image dataset comprises: receiving existing image data from a sonar; generating simulated image data that simulates the existing image data; generating synthesized image data by synthesizing at least one of virtual noise and a virtual loss section to the simulated image data; and storing the simulated image data and the synthesized image data as the image dataset. Claim 20 A method for tracking a target according to claim 18, wherein the process of training the first artificial neural network and the process of training the second artificial neural network are trained using a learning control technique that improves learning stability and convergence. Claim 21 A method for tracking a target according to claim 20, wherein the learning control technique comprises at least one of a technique for adjusting the learning rate and a technique for controlling the learning end time. Claim 22 A method for tracking a target according to claim 21, wherein the learning control technique comprises at least one of an Adam optimizer, an early termination technique, and a learning rate reduction technique. Claim 23 A method for tracking a target according to claim 17, further comprising a process of outputting the correction image data to a display device, wherein the outputting process comprises a process of superimposing the boundary of the mask and the predicted movement path onto the correction image data. Claim 24 In claim 23, the outputting process is a target tracking method in which, when a plurality of targets exist in the image data, the target-specific movement path, the boundary of the mask, and the predicted movement path are output to the display device in the same color. Claim 25 A method for tracking a target according to claim 23 or 24, wherein the outputting process further comprises at least one of the following: a process of storing result image data obtained by superimposing the boundary of the mask and the expected movement path onto the corrected image data in a lossless image file format, and a process of transmitting the result image data to an external terminal in real time using a network.

Images