Deep learning-based distant subject location identification system

Abstract

The present invention relates to a deep learning-based distant subject location identification system which can quickly and accurately track the exact location of a subject (object) in real time by using a deep learning location tracking algorithm on the basis of GPS location information collected from a tag device worn on the subject and infrared light source information of the tag device, collected through an infrared camera.

Description

Deep learning-based long-distance object location identification system

[0001] The present invention relates to a deep learning-based remote subject location identification system, and more specifically, to a deep learning-based remote subject location identification system that can quickly and accurately track the exact location of a subject in real time by using a deep learning location tracking algorithm based on GPS location information collected from a tag device worn on a subject (object) and infrared light source information of the tag device collected through an infrared camera.

[0002]

[0003] Recently, various techniques for searching and tracking objects based on artificial intelligence-based image processing / analysis techniques have been proposed, and the speed and sophistication of image processing have also become significantly faster.

[0004] However, moving object detection techniques in image processing systems still have stability issues. Furthermore, the detection process presents challenges in terms of accuracy and processing speed. Therefore, there is a need for technologies that can smoothly detect and track objects in images without requiring excessive computation.

[0005]

[0006] In order to solve the above problems, the present invention provides a deep learning-based remote subject location identification system that can quickly and accurately track the exact location of a subject in real time by using a deep learning location tracking algorithm based on GPS location information collected from a tag device worn on a subject (object) and infrared light source information of the tag device collected through an infrared camera.

[0007]

[0008] A deep learning-based long-distance object location identification system according to one embodiment of the present invention may include a body part (110), a tag device (120) worn on an object and generating GPS location information, and a location tracking unit (130) that performs object identification and location analysis on the GPS location information and infrared photographed image of the tag device (120) and then tracks the location movement status of the object.

[0009] In one embodiment, the body part (110) may include a housing (111), a stand (112) having one side mounted on the floor and to which the housing (111) is connected, a rotation motor (113) that rotates the housing (111) on the stand (112), an infrared camera (114) for photographing an infrared light source emitted from the tag device (120), and a transmitting / receiving antenna (115) for wirelessly communicating with the tag device (120).

[0010] In one embodiment, the body part (110) may further include a user terminal holder (116) on which a user terminal is placed and a tilting means (117) that allows the user terminal holder (116) to be angle-adjusted.

[0011] In one embodiment, based on a tracking signal of the object's position movement status through the position tracking unit (130), the rotational direction of the rotation motor (113) can be controlled in real time so that the front of the housing (111) always faces the object.

[0012] In one embodiment, the tag device (120) may include an infrared emitting unit (121) that emits an infrared light source, a GPS location information generating unit (122) that generates GPS location information, a wireless communication module (123) that wirelessly communicates with the transmitting / receiving antenna (115), and a battery (124) that supplies power to the infrared emitting unit (121), the GPS location information generating unit (123), and the wireless communication module (123).

[0013] In one embodiment, the infrared emitting unit (121) can emit an infrared light source of 850 nm.

[0014] In one embodiment, the battery (124) may support one or more of a wired charging method and a wireless charging method.

[0015] In one embodiment, the tag device (120) may have a wearable bracelet shape for a user.

[0016] In one embodiment, the position tracking unit (130) can perform object identification and position analysis based on the results of learning the object tracking model.

[0017]

[0018] According to one aspect of the present invention, there is an advantage in that the precise location of a subject can be quickly and accurately tracked in real time using a deep learning location tracking algorithm based on GPS location information collected from a tag device worn on a subject (object) regardless of the subject's remote or close location and infrared light source information of the tag device collected through an infrared camera.

[0019]

[0020] FIG. 1 is a diagram showing the configuration of a deep learning-based long-distance subject position identification system (100) according to one embodiment of the present invention.

[0021] Figure 2 is a drawing showing the body part (110) in more detail.

[0022] Figure 3 is a drawing showing a state in which the housing (111) is rotated by a rotary motor (113).

[0023] Figure 4 is a drawing showing a state in which the angle of the user terminal is adjusted through a tilting means (117).

[0024] Figure 5 is a drawing showing the tag device (120) in more detail.

[0025] Figures 6(a) and 6(b) are drawings specifically showing the process of modifying parameters in the process of learning an object tracking model in the position tracking unit (130).

[0026] Figure 7 is a drawing specifically showing the process of learning and verifying by distinguishing learning data and verification data during the process of learning an object tracking model in the location tracking unit (130).

[0027] Figure 8 is a diagram showing the process of tracking an object by predicting the object's movement speed and direction using an object tracking model.

[0028]

[0029] Hereinafter, preferred examples are presented to aid in understanding the present invention. However, the following examples are provided solely to facilitate a better understanding of the present invention, and the scope of the present invention is not limited by the examples.

[0030]

[0031] FIG. 1 is a drawing showing the configuration of a deep learning-based remote subject position identification system (100) according to one embodiment of the present invention, FIG. 2 is a drawing showing the body part (110) in more detail, FIG. 3 is a drawing showing a state in which a housing (111) is rotated by a rotation motor (113), FIG. 4 is a drawing showing a state in which a user terminal is adjusted in angle by a tilting means (117), and FIG. 5 is a drawing showing a tag device (120) in more detail.

[0032] Looking at FIGS. 1 to 5, the deep learning-based long-distance subject position identification system (100) according to the present invention can be largely configured to include a body part (110), a tag device (120), and a position tracking part (130).

[0033] First, the body part (110) is installed at a position a certain distance away from the object (subject), detects an infrared light source from a tag device (120) worn on the object, and performs Bluetooth-based wireless communication with the tag device (120) to receive GPS location information. In particular, the body part (110) can be position-corrected so that the user terminal always faces the object along the object's path based on an infrared shooting image, with a user terminal (such as a smartphone) mounted on one side thereof.

[0034] More specifically, the body part (110) is configured to include a housing (111), a stand (112), a rotation motor (113), an infrared camera (114), a transmitting / receiving antenna (115), a user terminal stand (116), and a tilting means (117).

[0035] The housing (111) is connected to the upper part of the lower stand (112), and at this time, the housing (111) itself is provided to be able to rotate on the stand (112) by a rotation motor (113) provided inside the housing (111).

[0036] The housing (111) is provided with an infrared camera (114) for photographing an object using infrared light and obtaining an infrared photographed image, as well as photographing an infrared light source emitted from a tag device (120).

[0037] The infrared camera (114) detects an infrared light source emitted from a tag device (120) worn on an object, and for this purpose, an infrared image captured by the infrared camera (114) is provided to a location tracking unit (130).

[0038] In addition, the housing (111) is provided with a transmitting / receiving antenna (115) for communicating with the tag device (120) through Bluetooth-based wireless communication and receiving GPS location information about the tag device (120). The GPS location information of the tag device (120) received through the transmitting / receiving antenna (115) is also provided to the location tracking unit (130).

[0039] The rotational motor (113) provided in the housing (111) can have its rotational direction controlled in real time so that the front of the housing (111) always faces the object based on the object's position movement status tracking signal from the position tracking unit (130) described later.

[0040] Here, the front of the housing (111) means the direction in which the user terminal mounted on the user terminal holder (116) of the housing (111) always faces the object. A user terminal holder (116) for mounting the user terminal is provided on the upper side of the housing (111), and the user terminal mounted on the user terminal holder (116) records the object's actions. At this time, since the housing (111) itself rotates by the rotation motor (113), when video recording starts through the camera of the user terminal, the object can be constantly tracked and recorded in the camera's field of view.

[0041] In addition, the user terminal holder (116) may be connected to the housing (111) via a tilting means (117). The tilting means (117) is provided to allow the user terminal holder (116) to be freely adjusted in angle. Although not shown in the drawing, in one embodiment, a separate rotation motor (not shown) is provided within the tilting means (117), so that the tilting angle may be automatically changed so that the camera of the user terminal can accurately capture the object based on the object's position movement status tracking signal of the position tracking unit (130).

[0042] In one embodiment, a battery (not shown) may be provided within the housing (111), and power may be supplied to a rotation motor (113), an infrared camera (114), and a transmitting / receiving antenna (115) mounted within the housing (111) through the battery. In addition, in one embodiment, the housing (111) may be connected to an external power source through a separate power cable, thereby being constantly supplied with power.

[0043]

[0044] The tag device (120) is formed in a form that can be worn on a moving object regardless of its form, such as a bracelet, anklet, necklace, or wristwatch, and has the function of emitting an infrared light source of 850 nm so that the infrared light source can be recognized by the infrared camera (114) of the body (110), and transmits GPS location information to the transmitting / receiving antenna (115).

[0045] More specifically, the tag device (120) is configured to include an infrared emitting unit (121), a GPS location information generating unit (122), a wireless communication module (123), and a battery (124).

[0046] The infrared emitting unit (121) emits an infrared light source of 850 nm so that it can be recognized by the infrared camera (114), and the GPS location information generating unit (122) generates GPS location information and transmits it to the transmitting / receiving antenna (114) via the wireless communication module (123).

[0047] That is, the infrared light source emitted through the infrared emitting unit (121), the infrared image captured through the infrared camera (114), and the GPS location information can be provided to the location tracking unit (130) described below, and the location tracking unit (130) can use this to accurately identify and track the location of an object.

[0048] In one embodiment, the battery (124) may support one or more of a conventional charging method or a wireless charging method, and in one embodiment, may be configured to be replaced with a new, fully charged battery. For example, if a player wearing the tag device (120) during a match experiences a low battery level, the player cannot recharge it immediately. In this case, the battery (124) of the tag device (10) itself can be immediately replaced with a new, fully charged battery, allowing the player to continue playing.

[0049]

[0050] Next, the location tracking unit (130) performs object identification and location analysis on the GPS location information and infrared imaging of the tag device (120), tracks the location movement status (movement path) of the object in real time, and then controls the rotation direction of the rotation motor (113) of the body unit (110) in real time based on this. This will be discussed as follows.

[0051] FIG. 6(a) and FIG. 6(b) are drawings specifically showing the process of modifying parameters in the process of learning an object tracking model in the position tracking unit (130), and FIG. 7 is a drawing specifically showing the process of learning and verifying by distinguishing learning data and verification data in the process of learning an object tracking model in the position tracking unit (130).

[0052] Looking at FIG. 6(a), FIG. 6(b) and FIG. 7, the position tracking unit (130) in the present invention uses an object tracking algorithm based on deep learning.

[0053] More specifically, the object tracking algorithm used in the position tracking unit (130) learns an object tracking model by utilizing a dataset labeled with information on the position (x, y) and size (w, h) of the object wearing the image and the tag device (120). When an infrared photographed image of an object is received through the infrared camera (114) discussed above, the position tracking unit (130) first identifies the position of the object appearing in the corresponding infrared photographed image based on the infrared light source emitted from the tag device (120), and then tracks it in real time using a deep learning-based object tracking model.

[0054] The object tracking model generates input data linked to a learning data reference frame and a number of past frames relative to the reference frame. At this time, the object tracking model can acquire past infrared images captured to include frames in which the tag device (120) appears as learning data. The object tracking model can generate input data by integrating input information derived from at least one past frame and each heat map image corresponding to the past frame.

[0055] Here, the term 'heat map image' refers to an image that visualizes certain information that can be expressed in color in the form of heat distribution from the original frame, and may mean an image that is processed (transformed) from the original frame to reflect location information (e.g., location of a specific body part, location of a center point, etc.) for each object detected (identified) in each frame.

[0056] Meanwhile, in one embodiment, input data associated with a reference frame and at least one past frame for the reference frame are partially extracted from the training image, and may correspond to a plurality of frames corresponding to a section of the image in which an arbitrary object appears. For example, assuming that a specific object appears in the training image and then disappears (exits) from frame N, frame N and the frames immediately preceding frame N may be selected as the reference frame and past frames, respectively, or frame N-1 and the frames immediately preceding frame N-1 may be selected as the reference frame and past frames, respectively.

[0057] In one embodiment, the object tracking model according to the present invention can obtain input information derived from a reference frame by sequentially passing the reference frame through a convolution layer, a batch normalization layer, and an activation layer. For example, the activation layer may be a layer that applies a ReLU function as an activation function.

[0058] In one embodiment, the object tracking model according to the present invention can integrate input information derived from each frame (image) by bit-by-bit into input data for training the object tracking model. For example, multiple bits corresponding to each of the multiple input pieces of information can be concatenated into a single sequence, or multiple bits can be summed at each corresponding bit position (order), but this is not limited thereto.

[0059] Meanwhile, the object tracking model of the present invention considers that, unlike detecting an object from an infrared photographed image, it is impossible to derive tracking information corresponding to an appearing object (e.g., a time-series movement (trajectory) of a bounding box partitioned to surround an appearing object, etc.) by considering only information of a single frame (e.g., a current frame) at a specific point in time in order to track an object. Accordingly, by integrating input information derived from a reference frame and at least one past frame for the reference frame and utilizing it as learning data, the object tracking model can easily learn changes between frames over time, and based on this result, can be implemented to continuously track an appearing object from a learning image.

[0060] That is, the object tracking model of the present invention can train a deep learning-based object tracking model that outputs information for tracking an object appearing in an infrared photographed image based on input data generated by integrating input information derived from multiple frames / images.

[0061] Meanwhile, in one embodiment, the object tracking model of the present invention may be learned based on a time series analysis algorithm such as an LSTM algorithm, an Attention algorithm, a Transformer algorithm, a BERT algorithm, etc. so that time series characteristics associated with objects appearing in infrared images can be utilized for object tracking.

[0062] The Long-Short Term Memory (LSTM) algorithm is a type of recurrent neural network (RNN) architecture used in deep learning. Unlike feedforward neural networks, it features feedback connections. Therefore, the LSTM algorithm can learn and process entire data sequences, not just single data points.

[0063] These LSTM algorithms are suitable for classifying, processing, and predicting predictions based on time series data, and LSTM has the advantage of solving the vanishing gradient problem that can occur during training using traditional RNNs.

[0064] The Attention algorithm is a technique that improves the performance of a deep learning model by having it focus on a specific vector. It can be understood as an algorithm implemented so that the model focuses (attentions) only on important parts during the learning process. In other words, the Attention algorithm allows the decoder to focus on different parts of the input sequence at each step when generating the output, and instead of encoding it into a single fixed context vector, it can learn how to generate a context vector at each step of the output. This can be understood as the model deciding which part to focus on in subsequent learning based on the input sequence and the results generated so far, and deciding which part to focus on can be performed through Softmax.

[0065] The Transformer algorithm may include an encoder-decoder structure that receives an input sequence from an encoder and outputs an output sequence from a decoder, similar to the conventional seq2seq architecture. However, unlike the conventional seq2seq architecture, multiple encoders and decoders may be provided. In addition, while the aforementioned Attention algorithm determines which of the hidden states of the encoder to focus on and utilizes it in decoder calculation, the Transformer algorithm does not connect the encoder and decoder, but uses the Attention algorithm only internally, and has the characteristic of being composed only of an Attention neural network, not an RNN such as LSTM or GRU.

[0066] The BERT (Bidirectional Encoder Representations from Transformers) algorithm is an algorithm that pre-trains a model with a large amount of unlabeled data and then performs transfer learning with labeled data that has a specific task. BERT's architecture partially uses the Transformer algorithm described above, but it makes transfer learning easier by slightly differentiating the architecture of the pre-training and fine-tuning stages. It is characterized by using only the Encoder part of the Transformer algorithm structure.

[0067] Meanwhile, the deep learning-based object tracking model disclosed in the present invention may be learned based on various types of artificial intelligence algorithms that are known in the past or may be developed in the future, such as a convolutional neural network.

[0068]

[0069] Figure 8 is a diagram showing the process of tracking an object by predicting the object's movement speed and direction using an object tracking model.

[0070] Looking at Figure 8, the infrared camera (114) of the body (110) takes infrared pictures of an object wearing a tag device (120) and provides the infrared pictures to the location tracking unit (130).

[0071] The location tracking unit (130) detects an infrared light source (light source) emitted from a tag device (120) from an infrared photographed image transmitted in real time, and simultaneously analyzes the location of the light source based on the GPS location information of the tag device (120). At this time, during the analysis process, the object tracking model outputs a predicted value according to the operation of the model's parameters and input data, and repeats the process of modifying and learning the parameters so that the error between the label data and the predicted value is reduced. Accordingly, the accuracy is improved as the intersection area between the predicted value and the label data increases.

[0072] Meanwhile, the object tracking model estimates the moving speed and direction of the object when the position of the light source is analyzed, and based on this, the object tracking model corrects the position of the light source so that the rotational direction and rotational speed of the rotational motor (113) of the body part (110) are controlled. Based on the correction result, the final position of the light source is calculated, and based on this, the rotational direction and rotational speed of the rotational motor (113) are controlled, thereby allowing the object to be tracked in real time.

[0073]

[0074] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0075]

[0076] [Explanation of symbols]

[0077] 100: Deep Learning-Based Long-Distance Object Location Identification System

[0078] 110: Body

[0079] 111: Housing

[0080] 112: Stand

[0081] 113: Rotary motor

[0082] 114: Infrared camera

[0083] 115: Transmitting and receiving antennas

[0084] 116: User terminal stand

[0085] 117: Tilting means

[0086] 120: Tag device

[0087] 121: Infrared emitter

[0088] 122: GPS location information generation unit

[0089] 123: Wireless communication module

[0090] 124: Battery

[0091] 130: Location Tracking Unit

Claims

1. Body (110); A tag device (120) worn on an object and generating GPS location information; and A deep learning-based remote subject location identification system, characterized by including a location tracking unit (130) that performs object identification and location analysis on the GPS location information and infrared photographed image of the tag device (120) and then tracks the location movement status of the object.

2. In paragraph 1, The above body part (110) is Housing (111); A stand (112) that is placed on the floor on one side and to which the housing (111) is connected; A rotation motor (113) that allows the above housing (111) to rotate on the stand (112); An infrared camera (114) for photographing an infrared light source emitted from the tag device (120); and A deep learning-based long-distance subject location identification system, characterized by including a transmitting / receiving antenna (115) for wireless communication with the tag device (120).

3. In paragraph 2, The above body part (110) is A user terminal holder (116) on which a user terminal is placed; and A deep learning-based remote subject position identification system, characterized in that it further includes a tilting means (117) for adjusting the angle of the user terminal stand (116).

4. In paragraph 2, A deep learning-based long-distance object position identification system, characterized in that the rotation direction of the rotation motor (113) is controlled in real time so that the front of the housing (111) always faces the object based on a position movement status tracking signal of the object through the position tracking unit (130).

5. In paragraph 2, The above tag device (120) is, An infrared emitting part (121) that emits infrared light; GPS location information generation unit (122) that generates GPS location information; A wireless communication module (123) that wirelessly communicates with the above transmitting and receiving antenna (115); and A deep learning-based long-distance subject location identification system, characterized by including a battery (124) that supplies power to the infrared emitting unit (121), GPS location information generating unit (123), and wireless communication module (123).

6. In paragraph 5, The above infrared emitting part (121) is A deep learning-based long-distance object location identification system characterized by emitting an 850 nm infrared light source.

7. In paragraph 5, The above battery (124) is, A deep learning-based long-distance subject location identification system characterized by supporting at least one charging method among a wired charging method and a wireless charging method.

8. In paragraph 2, The above tag device (120) is, A deep learning-based remote subject location identification system characterized by having a wearable bracelet shape for a user.

9. In paragraph 1, The above location tracking unit (130) A deep learning-based long-distance object location identification system characterized by performing object identification and location analysis based on the results of learning an object tracking model.

10. A body part including a housing, a stand on which one side is placed on the floor and to which the housing is connected, a rotation motor for rotating the housing on the stand, an infrared camera for photographing an infrared light source, a transmitting and receiving antenna, a user terminal stand on which a user terminal is placed, and a tilting means for adjusting the angle of the user terminal stand (116); A tag device worn on an object and generating GPS location information; and It includes a location tracking unit that performs object identification and location analysis from the GPS location information and infrared photographed image of the tag device, and then tracks the location movement status of the object. A deep learning-based long-distance object position identification system, characterized in that a rotation motor is provided within the tilting means, and the rotation motor rotates based on a position movement status tracking signal of an object by the position tracking unit, thereby automatically changing the tilting angle of the tilting means.

Images