Frame by frame feature matching based wandering detection method using multi modal video

Abstract

The present invention proposes a Frame by Frame Feature Matching-based loitering detection method using multimodal image information, which realizes a specific technical configuration that utilizes multimodal image information obtained through an IR camera and a low-cost LiDAR sensor to extract a person's pose-based skeleton, finds multiple center points in the skeleton, and performs loitering detection through feature matching.

Description

Technology Field

[0001] The present invention relates to a technology for detecting wandering, which refers to the behavior of wandering around aimlessly without a specific purpose or direction. Background Technology

[0002] One technology for detecting wandering, which refers to the behavior of roaming aimlessly without a specific purpose or direction, is detection technology utilizing computer vision algorithms.

[0003] Loitering detection systems using such computer vision algorithms have the disadvantage that detecting and localizing objects of interest is very difficult, and that they operate properly only if algorithms are designed specifically for each environment.

[0004] Accordingly, loitering detection technology utilizing deep learning models that aim to detect loitering through algorithms using deep learning techniques has emerged.

[0005] However, since algorithms using general deep learning techniques search for objects of interest only in specific backgrounds, existing deep learning models used in wandering detection systems have the disadvantage of being vulnerable to occlusion or changes in the surrounding environment (night, rain, etc.).

[0006] In particular, in environments such as hospital rooms in medical facilities and homes, the illuminance may be low for various reasons, such as lights being turned off. In such cases, it is impossible to take pictures with a standard camera, which significantly lowers the performance of deep learning. Therefore, it may be virtually impossible to detect loitering using existing deep learning models in loitering detection systems.

[0007] For this reason, multimodal techniques for detecting objects using various sensors have been introduced; however, there is a disadvantage in that smooth detection is difficult if occlusion occurs due to the surrounding environment or other objects when generating a bounding box for an object of interest or detecting it via a center point. Furthermore, setting a center point using a bounding box of a specific area presents the difficulty of determining the extent of that range.

[0008] Accordingly, the present invention proposes a Frame by Frame Feature Matching-based loitering detection technology utilizing multimodal image information. This technology involves extracting joint points using a deep learning algorithm that recognizes and represents human body parts, such as PoseNet, and then extracting multiple center points of the lines connecting these points. By finding corresponding points for the multiple center points per frame through Feature Matching and connecting the lines, movement patterns based on multiple center points can be identified. By utilizing the Feature Matching method for multiple center points, a specific threshold can be set, and loitering can be detected when the threshold is exceeded. The problem to be solved

[0009] The present invention was created in consideration of the above-mentioned circumstances, and the problem to be solved by the present invention is to identify wandering patterns and determine whether wandering is occurring by obtaining multiple center points from a bar connecting joint points extracted from a person's pose and connecting corresponding points for each frame of the image. means of solving the problem

[0010] A method for detecting loitering based on Frame by Frame Feature Matching using multimodal image information according to one aspect of the present invention for achieving the above objective comprises: an input step of inputting multimodal image information of a loitering detection target into a loitering detection model that has been previously trained using multimodal image information composed of an RGB image of an object, an infrared image, and a depth information visualization image; a first loitering detection result generation step in which, in the loitering detection model, an object of interest is detected regarding the input multimodal image information and a first loitering detection result is generated by determining whether the detected object is loitering; a second loitering detection result generation step in which, in the loitering detection model, outliers of an image frame re-implemented or predicted regarding the input multimodal image information are measured and a second loitering detection result is generated; and a final determination step in which, in the loitering detection model, a final determination is made regarding whether the object is loitering based on the first loitering detection result and the second loitering detection result through a previously trained decision-making algorithm.

[0011] Specifically, the loitering detection model may include a Feature Pyramid Network (FPN) designed to generate feature information used in the first and second loitering detection result generation steps by connecting RGB images, infrared images, and depth information visualization images acquired from an RGB sensor, an IR sensor, and a LiDAR sensor in parallel, and combining feature information extracted from each layer for each acquired RGB image, infrared image, and depth information visualization image into a single set of feature information including information of each image data type.

[0012] Specifically, in the above-mentioned loitering detection model, the FPN receives feature information extracted from each layer for each of the acquired RGB image, infrared image, and depth information visualization image as multimodal image information, and for each layer, combines the multimodal image information into a single set of feature information including information of each image data type, and can generate feature information according to a Top-down Pathway based on the combination result.

[0013] Specifically, the first loitering detection result generation step may include a first step of predicting a bounding box for the head and / or upper body of an object of interest, a second step of extracting a center point for the head and / or upper body using the predicted bounding box, and a third step of determining that a frame identified by tracking the movement path of the extracted center point for the head and / or upper body is a loitering frame.

[0014] Specifically, the third step determines whether there is a pre-set wandering boundary when a center point is extracted in the second step, and if there is a wandering boundary and the extracted center point is included, determines that the object is wandering within the specific frame from which the center point was extracted; if there is no wandering boundary, determines whether the connection shape formed by connecting the extracted center point and the previously extracted center points satisfies the conditions for use as a wandering boundary, and if satisfied, the connection shape can be set as a wandering boundary.

[0015] Specifically, the above decision-making algorithm may be an algorithm generated through deep learning training that causes a final judgment based on a first loitering detection result and a second loitering detection result generated for multimodal image information corresponding to loitering to converge on the loitering of an object.

[0016] Specifically, the final judgment step can, through the decision-making algorithm, finally determine as object loitering frames in which the outlier value predicted by the second loitering detection result is higher than the threshold value set through the learning process, among the frames determined to be the object loitering by the first loitering detection result.

[0017] A computer program stored on a medium, combined with hardware according to one aspect of the present invention for achieving the above objective, may be stored on the medium to execute: a first loitering detection result generation step, which, when multimodal image information of a loitering detection target is input, detects an object of interest with respect to the input multimodal image information and generates a first loitering detection result that determines whether the detected object is loitering; a second loitering detection result generation step, which measures an outlier of an image frame re-implemented or predicted with respect to the input multimodal image information and generates a second loitering detection result; and a final determination step, which finally determines whether the object is loitering based on the first loitering detection result and the second loitering detection result through a previously learned decision-making algorithm.

[0018] A Frame by Frame Feature Matching-based loitering detection system using multimodal image information according to one aspect of the present invention for achieving the above objective comprises: a first loitering detection result generating unit that, when multimodal image information composed of an RGB image, an infrared image, and a depth information visualization image that detects a loitering detection target is input, detects an object of interest with respect to the input multimodal image information and generates a first loitering detection result that determines whether the detected object is loitering; a second loitering detection result generating unit that generates a second loitering detection result by measuring an outlier of an image frame re-implemented or predicted with respect to the input multimodal image information; and a decision control unit that finally determines whether the object is loitering based on the first loitering detection result and the second loitering detection result through a previously learned decision algorithm.

[0019] Specifically, it may further include an FPN operating unit that operates a Feature Pyramid Network (FPN) designed to generate feature information utilized in the first and second loitering detection result generation units by connecting RGB images, infrared images, and depth information visualization images acquired from an RGB sensor, an IR sensor, and a LiDAR sensor in parallel, and combining feature information extracted from each layer for each acquired RGB image, infrared image, and depth information visualization image into a single set of feature information including information of each image data type.

[0020] Specifically, the FPN receives feature information extracted from each layer for each of the acquired RGB image, infrared image, and depth information visualization image as multimodal image information, combines the multimodal image information for each layer into a single set of feature information including information of each image data type, and generates feature information according to a Top-down Pathway based on the combination result.

[0021] Specifically, the first loitering detection result generation unit predicts a bounding box for the head and / or upper body of an object of interest, extracts a center point for the head and / or upper body using the predicted bounding box, and can determine that the frame identified by tracking the movement path of the extracted center point for the head and / or upper body is the object loitering.

[0022] Specifically, the first loitering detection result generation unit determines whether there is a pre-set loitering boundary when the center point is extracted, and if there is a loitering boundary, if the extracted center point is included, determines that the object is loitering in the specific frame from which the center point was extracted, and if there is no loitering boundary, determines whether the connection shape connecting the extracted center point and the previously extracted center points satisfies the conditions for being usable as a loitering boundary, and if satisfied, can set the connection shape as a loitering boundary.

[0023] Specifically, the above decision-making algorithm may be an algorithm generated through deep learning training that causes a final judgment based on a first loitering detection result and a second loitering detection result generated for multimodal image information corresponding to loitering to converge on the loitering of an object.

[0024] Specifically, the decision control unit can, through the decision algorithm, finally determine as object loitering a frame in which the outlier predicted by the second loitering detection result is higher than the threshold value set through the learning process, among the frames determined as object loitering by the first loitering detection result. Effects of the invention

[0025] Accordingly, according to the present invention, in loitering detection using deep learning techniques, by implementing a specific technology that combines multimodal image information through an IR camera and a low-cost LiDAR sensor and performs loitering detection through a decision-making algorithm, it is possible to realize a new Frame by Frame Feature Matching-based loitering detection technology using multimodal image information that can improve the reliability and accuracy of loitering detection.

[0026] Accordingly, the present invention provides not only the effect of providing loitering detection results with more robust performance and reliability by robustly detecting loitering in unspecified environments, such as occlusion, changes in the surrounding environment (night, rain, etc.), and changes in lighting, compared to existing loitering detection technologies, without excessive cost increase, but also the effect of leading to the diversification of service models based on loitering detection technology by accelerating and universalizing the dissemination of loitering detection technology. Brief explanation of the drawing

[0027] FIG. 1 is a block diagram showing the configuration of a Frame by Frame Feature Matching-based loitering detection system using multimodal image information according to one embodiment of the present invention. FIG. 2 is an example diagram showing the concept of deriving a final loitering detection result (final judgment) in the loitering detection model proposed in the present invention. FIG. 3 is an example diagram illustrating the flow of an FPN designed for more robust detection results proposed in the present invention. FIGS. 4 and FIGS. 5 are exemplary diagrams showing a process of determining whether wandering is occurring using the center point of an object of interest / the center point of a skeleton in the present invention. FIG. 6 is an example diagram showing the operation flow of a Frame by Frame Feature Matching-based loitering detection method using multimodal image information according to an embodiment of the present invention. Figure 7 is an example diagram illustrating the Frame by Frame Feature Matching-based loitering detection concept proposed in the present invention. Specific details for implementing the invention

[0028] Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

[0029] The present invention relates to a technology for detecting wandering, which refers to the behavior of wandering around aimlessly without a specific purpose or direction.

[0030] One technology for detecting wandering, which refers to the behavior of roaming aimlessly without a specific purpose or direction, is detection technology utilizing computer vision algorithms.

[0031] Loitering detection systems using such computer vision algorithms have the disadvantage that detecting and localizing objects of interest is very difficult, and that they operate properly only if algorithms are designed specifically for each environment.

[0032] Accordingly, loitering detection technology utilizing deep learning models that aim to detect loitering through algorithms using deep learning techniques has emerged.

[0033] However, since algorithms using general deep learning techniques search for objects of interest only in specific backgrounds, existing deep learning models used in wandering detection systems have the disadvantage of being vulnerable to occlusion or changes in the surrounding environment (night, rain, etc.).

[0034] In particular, in environments such as hospital rooms in medical facilities and homes, the illuminance may be low for various reasons, such as lights being turned off. In such cases, it is impossible to take pictures with a standard camera, which significantly lowers the performance of deep learning. Therefore, it may be virtually impossible to detect loitering using existing deep learning models in loitering detection systems.

[0035] For this reason, multimodal techniques for detecting objects using various sensors have been introduced; however, there is a disadvantage in that smooth detection is difficult if occlusion occurs due to the surrounding environment or other objects when generating a bounding box for an object of interest or detecting it via a center point. Furthermore, setting a center point using a bounding box of a specific area presents the difficulty of determining the extent of that range.

[0036] Meanwhile, since current CCTVs in South Korea are used for purposes such as nighttime surveillance, security, and abnormal situation detection, they utilize IR cameras equipped with infrared capabilities, and most CCTVs used in homes are also equipped with infrared functions.

[0037] Therefore, when considering existing CCTV infrastructure installed in homes, buildings, roads, alleys, etc., one can consider deep learning-based loitering detection technology that uses infrared images.

[0038] However, currently, there is a significant lack of research on anomaly detection and loitering detection using infrared images compared to the need for such methods, and captured RGB and infrared images have limitations in that they contain 2D information, requiring the detection of loitering to be done using limited information.

[0039] Accordingly, the present invention proposes a Frame by Frame Feature Matching-based loitering detection technology using a novel type of multimodal image information, which utilizes not only RGB and infrared images but also depth information visualization images from LiDAR sensors to realize robust loitering detection in unspecified environments such as occlusion, changes in the surrounding environment (night, rain, etc.), and changes in illumination.

[0040] Accordingly, the present invention proposes a Frame by Frame Feature Matching-based loitering detection technology utilizing multimodal image information. This technology involves extracting joint points using a deep learning algorithm that recognizes and represents human body parts, such as PoseNet, and then extracting multiple center points of the lines connecting these points. By finding corresponding points for the multiple center points per frame through Feature Matching and connecting the lines, movement patterns based on multiple center points can be identified. By utilizing the Feature Matching method for multiple center points, a specific threshold can be set, and loitering can be detected when the threshold is exceeded.

[0041] In other words, the present invention proposes a Frame by Frame Feature Matching-based loitering detection technology using novel multimodal image information, which enables robust loitering detection even in unspecified environments through a technique that simultaneously utilizes RGB images and infrared images captured by an IR camera (RGB Sensor, IR Sensor) to detect loitering, and enables loitering detection using complex information rather than fragmentary information by utilizing depth information visualization images extracted even from low-cost LiDAR sensors.

[0042] Figure 1 shows the configuration of a loitering detection system for implementing a Frame by Frame Feature Matching-based loitering detection technology using multimodal image information proposed in the present invention.

[0043] As illustrated in FIG. 1, a Frame by Frame Feature Matching-based loitering detection system (100) using multimodal image information according to one embodiment of the present invention may be configured to include an FPN operating unit (110) that operates the FPN (115) proposed in the present invention, a first loitering detection result generating unit (120), a second loitering detection result generating unit (130), and a decision control unit (140) that performs a final decision according to a decision algorithm (150).

[0044] The entire configuration or at least part of the Frame by Frame Feature Matching-based loitering detection system (100) using such multimodal image information may be implemented in the form of a hardware module or a software module, or in a combined form of a hardware module and a software module.

[0045] Here, a software module can be understood as an instruction executed by a processor that controls operations within a Frame by Frame Feature Matching-based loitering detection system (100) using, for example, multimodal image information, and such an instruction may have a form loaded in memory within the loitering detection system (100).

[0046] Ultimately, the Frame by Frame Feature Matching-based loitering detection system (100) using multimodal image information according to an embodiment of the present invention realizes the Frame by Frame Feature Matching-based loitering detection technology using multimodal image information proposed in the present invention through the aforementioned configuration.

[0047] Figure 2 is an example diagram showing the concept of deriving a final loitering detection result (final judgment) in the loitering detection model proposed in the present invention.

[0048] That is, in the present invention, by designing and implementing a loitering detection model consisting of an FPN (115), a first loitering detection result generation unit (120), a second loitering detection result generation unit (130), and a decision algorithm (150) within a Frame by Frame Feature Matching-based loitering detection system (100) using multimodal image information, the Frame by Frame Feature Matching-based loitering detection technology proposed in the present invention may be realized.

[0049] Therefore, the Frame by Frame Feature Matching-based loitering detection system (100) and the loitering detection model (100) using multimodal image information of the present invention can be said to be functionally corresponding to or identical to each other.

[0050] First, below, with reference to FIG. 1, each component of the Frame by Frame Feature Matching-based loitering detection system (100) using multimodal image information of the present invention will be described in more detail.

[0051] The FPN operation unit (110) operates the FPN (115), that is, the FPN (115) designed / proposed for more robust detection results in the present invention.

[0052] To explain in detail, the Feature Pyramid Network (FPN) is a technique for object detection that generates feature maps for each layer by rescaling a single image or video to various sizes via a bottom-up path using convolution layers, and then combining the feature maps by working downwards from the top layer along a top-down path.

[0053] In the present invention, an FPN (115) is proposed that can support more robust detection results by designing to generate feature information (feature map) for each layer for multiple images / videos for object detection, namely RGB images, infrared images, and depth information visualization images.

[0054] The FPN (115) proposed in the present invention is designed to have a structure that connects RGB images, infrared images, and depth information visualization images obtained from an RGB Sensor (10), an IR Sensor (20), and a LiDAR Sensor (30) in parallel.

[0055] And, FPN (115) can be designed to generate feature maps that are used in the first and second loitering detection result generation units (120, 130) described later by combining feature maps extracted from each layer for RGB images, infrared images, and depth information visualization images obtained through a parallel connection structure into a single set of feature maps that includes information of each image data type.

[0056] More specifically, FPN (115) may include an encoder designed for each data type of RGB image, infrared image and depth information visualization image, or may work with an encoder for each of these data types.

[0057] For example, encoders designed for each data type, such as RGB images, infrared images, and depth information visualization images, can extract feature maps of proportional size by rescaling the input images to various sizes according to a bottom-up path through a convolution layer.

[0058] In this case, the extraction technique used to extract feature information from the encoder can be selected from various existing techniques (e.g., ResNet), and a detailed explanation thereof will be omitted.

[0059] Accordingly, FPN (115) can receive feature information (feature map) extracted from each layer for each RGB image, infrared image, and depth information visualization image obtained through a parallel connection structure as multimodal image information from an encoder designed for each data type of the tactical method.

[0060] And, FPN (115) can combine multimodal image information (RGB image / infrared image / depth information visualization image feature map) for each layer into a single set of feature information containing information of each image data type, and generate feature information (feature map) according to the Top-down Pathway based on the result of the combination.

[0061] For example, FPN (115) can generate feature maps for each layer by combining multimodal image information (feature maps of RGB image / infrared image / depth information visualization image) into a single set of feature information containing information of each image data type, and then combining feature maps by going down from the top layer along the Top-down Pathway.

[0062] Figure 3 shows an example of the feature map generation flow of the FPN (115) described in the tactics.

[0063] As can be seen in Fig. 3, the FPN (115) can receive multimodal image information for each layer (RGB image / infrared image / feature map of depth information visualization image).

[0064] Specifically, in the case of Fig. 3, 4 Layers are assumed, x represents the feature map level of the FPN (115), and y is an index representing one of the RGB Sensor (10), IR Sensor (20), and LiDAR Sensor (30).

[0065] That is, C xy represents feature information (feature map) for an image obtained from a sensor of index y at the x layer or feature map level.

[0066] As can be seen in FIG. 3, the FPN (115) of the present invention provides multimodal image information for each layer, i.e., C 41 / C 42 / C 43 , C 31 / C 32 / C 33 , C 21 / C 22 / C 23 , C 11 / C 12 / C13 After concatenating each of the feature information sets, feature information (feature map, yellow / blue / red / green) can be generated for each of the 4 layers by combining each feature information (feature map) as you go down from the top layer (4->3->2->1) according to the Top-down Pathway.

[0067] As described above, the FPN (115) of the present invention is based on a parallel connection structure and is a multimodal image information of a loitering detection target (e.g., C 41 / C 42 / C 43 , C 31 / C 32 / C 33 , C 21 / C 22 / C 23 , C 11 / C 12 / C 13 By generating and transmitting feature information (feature map) using composite information that can be utilized in the first and second loitering detection result generation units (120, 130) described below, more robust detection results can be supported.

[0068] The first loitering detection result generation unit (120) is responsible for the function of detecting an object of interest with respect to the input multimodal image information and generating a first loitering detection result that determines whether the detected object is loitering.

[0069] More specifically, the first loitering detection result generation unit (120) can detect an object of interest based on feature information (feature map) generated / transmitted from the FPN (115) for the input multimodal image information, and generate a first loitering detection result that determines whether the detected object is loitering.

[0070] To explain a specific embodiment, the first wandering detection result generation unit (120) predicts a bounding box for the head and / or upper body of an object of interest based on feature information (feature map) generated / transmitted from the FPN (115).

[0071] And, the first wandering detection result generation unit (120) can extract a center point for the head using a Bounding Box predicted for the head of the object of interest, and can extract a center point for the upper body using a Bounding Box predicted for the upper body of the object of interest.

[0072] Accordingly, the first wandering detection result generation unit (120) can generate a first wandering detection result by determining that a frame identified through tracking the movement path of a center point for the extracted head and / or upper body is wandering, and determining that other frames not identified are not wandering.

[0073] FIG. 4 is an example diagram showing the process of determining whether an object of interest is wandering using the center point of the object of interest in the present invention. FIG. 4 describes the process by referring to the center point of the head of the object of interest for convenience of explanation.

[0074] With reference to FIG. 4 for a more detailed explanation, the first loitering detection result generation unit (120) determines whether there is a pre-set (generated) loitering boundary when a head center point is extracted / added (S10) as a result of detecting and predicting a tactical object of interest for a new sequence (frame) generated / transmitted from FPN (115) (S20).

[0075] The first wandering detection result generation unit (120) can determine that the object is wandering in a specific frame from which the head center point was extracted (S30 Yes) if there is a wandering boundary (S20 Yes) and the head center point extracted this time is included in the wandering boundary (S30 Yes).

[0076] The first loitering detection result generation unit (120) can determine that if the head center point extracted this time is not included in the loitering boundary (S30 No), the frame from which the head center point was extracted this time is not loitering (Normal).

[0077] Meanwhile, the first wandering detection result generation unit (120) determines whether the connection shape formed by connecting the head center point extracted this time and the previously extracted head center points satisfies the condition that it can be used as a wandering boundary when there is no wandering boundary (S20 No). (S40).

[0078] Here, the conditions that can be used as wandering boundaries may vary depending on the definition. However, as an example, the condition may be defined to be satisfied if the connection shape forms a circular shape, or it may be defined to be satisfied if the connection shape repeats in a certain shape even if it does not form a specific shape.

[0079] Accordingly, the first loitering detection result generation unit (120) may, when the condition that the connection shape can be used as a loitering boundary is satisfied (S40 Yes), set the current connection shape as a loitering boundary (S50), and determine that the frame from which the current head center point was extracted is not a loitering object (Normal).

[0080] Meanwhile, the first wandering detection result generation unit (120) may determine that the object does not wander when the connection shape does not satisfy the condition that it can be used as a wandering boundary (S40 No), and set the current connection shape as a wandering boundary (S50), and determine that the frame from which the current head center point was extracted is not wandering (Normal).

[0081] Next, the first wandering detection result generation unit (120) can determine that when a new head center point is extracted / added (S10), the movement path of the object of interest is tracked using the head center point of the object of interest, and if the set of head center points continuously repeats the wandering boundary from the frame where the first wandering boundary is drawn, the frames can be determined as wandering (Wandering).

[0082] Of course, the first wandering detection result generation unit (120) can track the movement path of the object of interest using the upper body center point of the object of interest in the same manner as in the tactical method, and if the set of upper body center points continuously repeats the wandering boundary from the frame where the first wandering boundary is drawn, the frames can be determined as wandering (Wandering).

[0083] Ultimately, the first wandering detection result generation unit (120) can generate two first wandering detection results determined using the head center point and upper body center point of the object of interest.

[0084] The first loitering detection result generation unit (120) can generate / provide a first loitering detection result for multimodal image information corresponding to loitering when deep learning is performed for the decision-making algorithm to be described later.

[0085] Here, "multimodal image information corresponding to wandering" means multimodal image information (feature map of RGB image / infrared image / depth information visualization image) for RGB image, infrared image, and depth information visualization image obtained from RGB Sensor (10), IR Sensor (20), and LiDAR Sensor (30) by photographing an object of interest that is wandering.

[0086] That is, the first loitering detection result generation unit (120) can support the training data necessary for the training of the decision-making algorithm (150) described later by generating a first loitering detection result based on the feature information (feature map) generated / transmitted from the FPN (115) with respect to the multimodal image information obtained by photographing a loitering object of interest.

[0087] Of course, when actually detecting loitering, the first loitering detection result generation unit (120) will generate a first loitering detection result based on feature information (feature map) generated / transmitted from the FPN (115) with respect to the multimodal image information of the loitering detection target.

[0088] The second loitering detection result generation unit (130) is responsible for generating a second loitering detection result by measuring an outlier of a re-implemented or predicted image frame for the input multimodal image information.

[0089] More specifically, the second loitering detection result generation unit (130) can reconstruct or predict an image frame based on feature information (feature map) generated / transmitted from the FPN (115) for the input multimodal image information, and can measure an outlier for the image frame thus reconstructed or predicted.

[0090] The second loitering detection result generation unit (130) can measure outliers for video frames only for multimodal video information corresponding to loitering when performing deep learning of the decision-making algorithm to be described later.

[0091] That is, the second loitering detection result generation unit (130) can support the training data required for the training of the decision algorithm (150) described later by reconstructing or predicting an image frame based on the feature information (feature map) generated / transmitted from the FPN (115) for multimodal image information obtained by photographing a loitering object of interest, and measuring outliers for this image frame to generate the result (second loitering detection result).

[0092] Of course, when actually detecting loitering, the second loitering detection result generation unit (130) will reconstruct or predict an image frame based on the feature information (feature map) generated / transmitted from the FPN (115) with respect to the multimodal image information of the loitering detection target, and measure an outlier for this image frame to generate a second loitering detection result.

[0093] The decision control unit (140) is a functional unit that performs a final decision according to the decision algorithm (150).

[0094] That is, the decision control unit (140) can finally determine whether the object of interest is wandering based on the first wandering detection result of the first wandering detection result generation unit (120) and the second wandering detection result of the second wandering detection result generation unit (130) through the previously learned decision algorithm (150).

[0095] Specifically, the decision algorithm (150) is an algorithm that learns criteria for finally determining loitering based on multimodal image information corresponding to loitering, that is, multimodal image information obtained by photographing an object of interest that is loitering, when designing the loitering detection model (100) of the present invention, and based on the first loitering detection result (head / upper body) and the second loitering detection result (measurement outlier) generated with respect to the multimodal image information corresponding to loitering.

[0096] Accordingly, according to one embodiment, the decision algorithm (150) can be generated through deep learning method training that causes the final judgment based on the first loitering detection result (head / upper body) and the second loitering detection result (measurement outlier) generated with respect to multimodal image information corresponding to loitering, that is, multimodal image information obtained by photographing an object of interest that is loitering, to converge to the loitering of the object.

[0097] As an example, the decision algorithm (150) can determine the threshold value to be used, as described below, by performing deep learning to ensure that the final judgment converges to the wandering of the object.

[0098] Accordingly, when actual loitering is detected, the decision control unit (140) can finally determine that among the frames determined to be loitering by the first loitering detection result based on the feature map generated / transmitted from the FPN (115) for the multimodal image information of the loitering detection target, the frames in which the outlier value predicted by the second loitering detection result is higher than the threshold value set through the tactical learning process are the final determination of the object loitering.

[0099] As described above, the decision control unit (140) can finally determine whether an object of interest is wandering based on a final decision criterion (e.g., threshold) learned through a decision algorithm (150) in a set of frames in which the object is judged to be wandering by a first wandering detection result and a set of frames in which an outlier is predicted by a second wandering detection result.

[0100] As described above, the Frame by Frame Feature Matching-based loitering detection system (100) using multimodal image information of the present invention realizes a new Frame by Frame Feature Matching-based loitering detection technology using multimodal image information that can improve the reliability and accuracy of loitering detection by implementing a specific technology that combines RGB images captured by an IR camera (RGB Sensor, IR Sensor), infrared images, and depth information visualization images extracted from a LiDAR Sensor, and performs loitering detection through a decision-making algorithm.

[0101] Meanwhile, referring to FIG. 2, in the wandering detection model (100) proposed in the present invention, multimodal image information consisting of RGB images, infrared images, and depth information visualization images obtained through an IR camera (RGB Sensor, IR Sensor) and a LiDAR Sensor passes through an FPN (115), and is transmitted to a Feature Matching based Wandering Detection Module as feature information (feature map) generated from the FPN (115) regarding the complex multimodal image information.

[0102] Here, the Feature Matching based Wandering Detection Module can be composed of each detection block, a Body Center Point based Wandering Detection Module, a Head Center Point based Wandering Detection Module, and a Frame based Wandering Detection Module.

[0103] Here, the Body Center Point based Wandering Detection Module and the Head Center Point based Wandering Detection Module correspond to the first wandering detection result generation unit (120), and the Frame based Wandering Detection Module corresponds to the second wandering detection result generation unit (130).

[0104] Accordingly, the detection results (first and second wandering detection results) from each detection block, namely the Body Center Point based Wandering Detection Module, the Head Center Point based Wandering Detection Module, and the Frame based Wandering Detection Module, are transmitted to the Decision Tree, and the Decision Tree can make a final determination of whether the object of interest is wandering based on the detection results (first and second wandering detection results) and the criteria for a final decision that has been learned in advance.

[0105] This Decision Tree may correspond to a decision algorithm (150).

[0106] In addition, as can be seen in FIG. 2, the Feature Matching based Wandering Detection Module may be designed to generate a skeleton with joint points for a person using a deep learning algorithm that generates a skeleton with joint points for feature information (feature map) generated from FPN (115) for complex multimodal image information, extract the center point of a bar connecting the joint points, identify the position change of the bar center point for each frame of the captured image through Feature Matching, and determine whether the object of interest is wandering by applying a threshold point for the range of motion.

[0107] For example, as illustrated in FIG. 7, through a deep learning algorithm, PoseNET Skeleton extraction based on human joint points and center point extraction of bars connecting the joint points can be performed, and multiple center points in the skeleton can be found and wandering detection can be performed through Feature Matching, thereby realizing a specific technical configuration that enables Frame by Frame Feature Matching based on multimodal image information.

[0108] Figure 5 is an example diagram showing the process of determining whether an object of interest is wandering using the center point of the skeleton in the present invention.

[0109] With reference to FIG. 5 for a more detailed explanation, the Feature Matching based Wandering Detection Module can determine whether the connection shape forms a specific section when the center points of the skeleton are connected as a result of detecting and predicting the object of interest generated / transmitted from the FPN (115).

[0110] Accordingly, the Feature Matching based Wandering Detection Module can set the current connection shape as a wandering boundary if the connection shape forms a specific section (Yes), but determine whether the current wandering is normal (Normal).

[0111] In addition, the Feature Matching based Wandering Detection Module can output an abnormal result as a result of determining whether wandering occurs if the connection shape connecting the center points of the skeleton continues to move within the wandering boundary even after that (Wandering).

[0112] Thus, by combining the advantages according to the data types of the RGB image, infrared image, and depth information visualization image in the present invention, the advantages can be described as overcoming low-light environments and enhancing object tracking through the simultaneous use of the RGB image and infrared image, and the advancement of loitering detection performance through the use of the depth information visualization image.

[0113] Deep learning models that perform loitering detection using only existing RGB images were unable to operate properly depending on the surrounding environment and required additional information.

[0114] However, according to the Frame by Frame Feature Matching-based loitering detection technology using multimodal image information of the present invention, by proposing a loitering detection model (100) of specific technology implementation that uses complex multimodal image information rather than fragmentary information, the performance of the deep learning model is greatly increased and the reliability of the loitering detection result is added as a result of inference using complex information is shown.

[0115] Accordingly, according to the present invention, a deep learning model, namely a loitering detection model (100), is designed to transmit depth information using a low-cost LiDAR sensor, enabling loitering detection even in situations such as camera obstruction, and to exhibit robust performance in all illumination levels and times of day through the simultaneous use of RGB images and infrared images, which demonstrates that it has more robust performance and reliability than existing loitering detection technology.

[0116] In addition, as the adoption of unmanned systems accelerates and reliability is required, the demand for deep learning models using multimodal images is increasing.

[0117] That is, the system (100) of the present invention, which detects loitering using multimodal image information of an IR camera (RGB Sensor, IR Sensor) and a LiDAR Sensor, satisfies the requirements of an intelligent system that reflects the latest trends, and can be redesigned into an intelligent system for various purposes, as additional performance enhancements such as an object of interest detector and a loitering detection module are possible through the attachment of additional sensors and the design of additional modules.

[0118] Ultimately, the present invention provides not only the effect of offering loitering detection results with more robust performance and reliability by robustly detecting loitering in unspecified environments—such as occlusion, changes in the surrounding environment (night, rain, etc.), or changes in lighting—compared to existing loitering detection technologies, without excessive cost increases, but also the effect of leading to the diversification of service models based on loitering detection technology by accelerating and universalizing the dissemination of loitering detection technology.

[0119] Hereinafter, with reference to FIG. 6, the operation flow of a Frame by Frame Feature Matching-based loitering detection method using multimodal image information according to an embodiment of the present invention will be described in detail.

[0120] The operation of the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention can be described by referring to a Frame by Frame Feature Matching-based loitering detection system (100) or a loitering detection model (100) using multimodal image information.

[0121] Hereinafter, for the convenience of explanation, the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention will be described by referring to the loitering detection model (100) as the subject of operation.

[0122] First, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, a decision algorithm (150) is generated through deep learning training, and an FPN (115) for more robust detection results is designed, and a loitering detection model (100) of the present invention can be generated / constructed (S100).

[0123] Subsequently, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, when actual loitering is detected, the loitering detection model (100) acquires an RGB image, an infrared image, and a depth information visualization image of a loitering detection target from an RGB Sensor (10), an IR Sensor (20), and a LiDAR Sensor (30) (S200), and can extract feature information (feature map) extracted from each layer by encoding each of the RGB image / infrared image / depth information visualization image (S300).

[0124] That is, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, the loitering detection model (100) can obtain feature information (feature map) extracted from each layer for RGB images, infrared images, and depth information visualization images as multimodal image information of the current loitering detection target (S300).

[0125] And, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, the loitering detection model (100), particularly the FPN (115), combines multimodal image information (feature map of RGB image / infrared image / depth information visualization image) for each layer into a single set of feature information including information of each image data type, and can generate feature information (feature map) according to a Top-down Pathway based on the result of the combination (S400, see FIG. 3).

[0126] Accordingly, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, the loitering detection model (100) detects an object of interest based on feature information (feature map) generated / transmitted from the FPN (115) in step S400, and can generate a first loitering detection result that determines whether the detected object is loitering (S500).

[0127] Specifically, the loitering detection model (100) can generate a first loitering detection result by predicting a bounding box for the head and / or upper body of an object of interest based on feature information (feature map) generated / transmitted from the FPN (115), extracting a center point for the head using the bounding box predicted for the head of the object of interest, extracting a center point for the upper body using the bounding box predicted for the upper body of the object of interest, and determining that a frame identified by tracking the movement path of the extracted center point for the head and / or upper body is a loitering frame.

[0128] Meanwhile, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, the loitering detection model (100) can reconstruct or predict an image frame based on feature information (feature map) generated / transmitted from the FPN (115) in step S400 and generate a second loitering detection result that measures an outlier for the image frame (S600).

[0129] Accordingly, according to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, the loitering detection model (100) can finally determine whether an object of interest is loitering based on the first loitering detection result and the second loitering detection result through the decision algorithm (150) (S700).

[0130] Specifically, the decision algorithm (150) is an algorithm that learns criteria for finally determining loitering based on the first loitering detection result (head / upper body) and the second loitering detection result (measurement outlier) generated from multimodal image information corresponding to loitering, that is, multimodal image information obtained by photographing an object of interest that is loitering, when designing the loitering detection model (100) of the present invention in the previously described S100 step.

[0131] Accordingly, according to one embodiment, the decision algorithm (150) can be generated through deep learning method training that causes the final judgment based on the first loitering detection result (head / upper body) and the second loitering detection result (measurement outlier) generated with respect to multimodal image information corresponding to loitering, that is, multimodal image information obtained by photographing an object of interest that is loitering, to converge to the loitering of the object.

[0132] As an example, the decision algorithm (150) can determine the threshold value to be used, as described below, by performing deep learning to ensure that the final judgment converges to the wandering of the object.

[0133] Accordingly, when actually detecting loitering, the loitering detection model (100) can finally determine that among the frames determined to be loitering by the first loitering detection result based on the feature map generated / transmitted from the FPN (115) for the multimodal image information of the loitering detection target, the frames in which the outlier value predicted by the second loitering detection result is higher than the threshold value set through the tactical learning process are the objects loitering (S700).

[0134] According to the Frame by Frame Feature Matching-based loitering detection method using multimodal image information of the present invention, if the current loitering detection procedure is not completed (S800 No), the loitering detection model (100) may repeat steps S300 and subsequent steps for the RGB image, infrared image, and depth information visualization image of the loitering detection target that are continuously acquired.

[0135] As explained above, according to the present invention, by implementing a specific technology that combines RGB images and infrared images captured by an IR camera (RGB Sensor, IR Sensor) and depth information visualization images extracted from a LiDAR Sensor, and performs loitering detection through a decision-making algorithm, a Frame by Frame Feature Matching-based loitering detection technology using new multimodal image information is realized, which can improve the reliability and accuracy of loitering detection.

[0136] Accordingly, the present invention provides not only the effect of providing loitering detection results with more robust performance and reliability by robustly detecting loitering in unspecified environments, such as occlusion, changes in the surrounding environment (night, rain, etc.), and changes in lighting, compared to existing loitering detection technologies, without excessive cost increase, but also the effect of leading to the diversification of service models based on loitering detection technology by accelerating and universalizing the dissemination of loitering detection technology.

[0137] The Frame by Frame Feature Matching-based loitering detection method using multimodal image information according to one embodiment of the present invention described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

[0138] Although the present invention has been described in detail with reference to preferred embodiments, the present invention is not limited to the above-described embodiments, and the technical concept of the present invention extends to the scope in which various modifications or alterations are possible by anyone with ordinary knowledge in the technical field to which the present invention belongs, without departing from the gist of the present invention as claimed in the following claims. Industrial applicability

[0139] According to the Frame by Frame Feature Matching-based loitering detection method and Frame by Frame Feature Matching-based loitering detection system using multimodal image information of the present invention, by realizing a specific technical configuration that combines multimodal image information obtained through an IR camera and a low-cost LiDAR sensor and performs loitering detection through a decision-making algorithm, the invention overcomes the limitations of existing technology. Consequently, it is an invention with industrial applicability, as it not only offers sufficient potential for the commercialization or business of the applied device rather than merely the use of related technology, but is also practically and clearly implementable.

[0140] No content

Claims

Claim 1 A method for detecting loitering based on Frame by Frame Feature Matching using multimodal image information, comprising: an input step of inputting multimodal image information of a loitering detection target into a loitering detection model that has been previously trained using multimodal image information composed of RGB images, infrared images, and depth information visualization images of an object; a step of generating feature information by processing the input multimodal image information through a Feature Pyramid Network (FPN) in the loitering detection model; and a step of detecting an object of interest using the feature information in the loitering detection model - wherein the loitering detection model includes a Feature Matching based Wandering Detection Module configured to generate a skeleton using human joint points through a deep learning algorithm on the feature information, extract multiple center points by extracting the center points of bars connecting the joint points, and identify positional changes of the multiple center points through Frame by Frame Feature Matching -; wherein in the Feature Matching based Wandering Detection Module of the loitering detection model, the loitering of the detected object A first loitering detection result generation step for generating a first loitering detection result by determining whether there is a loitering; a second loitering detection result generation step for generating a second loitering detection result by measuring an outlier of a re-implemented or predicted image frame for an image frame determined to be loitering by the first loitering detection result generation step in the feature-matching-based loitering detection module of the loitering detection model; and a final determination step for finally determining whether the object is loitering based on the second loitering detection result among the image frames determined to be loitering by the object according to the first loitering detection result in the loitering detection model;The first loitering detection result generation step comprises identifying positional changes of the plurality of center points on a frame-by-frame basis, determining whether a connection shape connecting the plurality of center points satisfies conditions for use as a loitering boundary, setting the connection shape as a loitering boundary if the connection shape satisfies the conditions, and determining that the object is loitering when the connection shape continues to move within the loitering boundary after the loitering boundary setting, thereby generating a first loitering detection result based on Frame by Frame Feature Matching using multimodal image information. Claim 2 A Frame by Frame Feature Matching-based loitering detection method using multimodal image information according to claim 1, wherein the loitering detection model includes the FPN designed to parallelly connect RGB images, infrared images, and depth information visualization images obtained from an RGB sensor, an IR sensor, and a LiDAR sensor, and to combine feature information extracted from each layer for each obtained RGB image, infrared image, and depth information visualization image into a single set of feature information including information of each image data type to generate feature information used in the first and second loitering detection result generation steps. Claim 3 In claim 2, the FPN in the loitering detection model receives feature information extracted from each layer for each of the acquired RGB image, infrared image, and depth information visualization image as multimodal image information, combines the multimodal image information for each layer into a single set of feature information including information of each image data type, and generates feature information according to a Top-down Pathway based on the combination result, characterized by a Frame by Frame Feature Matching-based loitering detection method using multimodal image information. Claim 4 A Frame by Frame Feature Matching-based loitering detection method using multimodal image information, wherein the condition usable as the loitering boundary includes a condition defined to be satisfied when the connection shape forms a circular shape, or a condition defined to be satisfied when the connection shape is repeated in a constant shape. Claim 5 delete Claim 6 A Frame by Frame Feature Matching-based loitering detection method using multimodal image information according to claim 1, wherein the final judgment step makes a final judgment through a decision algorithm, and the decision algorithm is an algorithm generated through deep learning training that causes a final judgment based on a first loitering detection result and a second loitering detection result generated for multimodal image information corresponding to loitering to converge to the loitering of an object. Claim 7 A Frame by Frame Feature Matching-based loitering detection method using multimodal image information, wherein, in claim 6, the final judgment step determines, through the decision-making algorithm, that among the frames determined to be loitering by the first loitering detection result, the frames in which the outlier predicted by the second loitering detection result is higher than the threshold value set through the learning process are finally determined to be object loitering. Claim 8 Combined with hardware, when multimodal image information of a loitering detection target is input, a step of processing the input multimodal image information to generate feature information; a step of detecting an object of interest using the feature information; a first loitering detection result generation step of determining whether the detected object is loitering and generating a first loitering detection result; a second loitering detection result generation step of measuring outliers of re-implemented or predicted image frames for image frames determined to be loitering by the first loitering detection result generation step and generating a second loitering detection result; The method is stored in a computer-readable recording medium to execute a final determination step for finally determining whether the object is wandering based on the second wandering detection result among video frames determined to be wandering by the object according to the first wandering detection result; the first wandering detection result generation step operates based on a feature matching based wandering detection module configured to generate a skeleton using human joint points through a deep learning algorithm on the feature information, extract multiple center points by extracting the center points of bars connecting the joint points, and identify positional changes of the multiple center points through Frame by Frame Feature Matching; the method identifies positional changes of the multiple center points frame by frame, determines whether the connecting shape connecting the multiple center points satisfies the condition for being usable as a wandering boundary, sets the connecting shape as a wandering boundary if the connecting shape satisfies the condition, and determines that the object is wandering if the connecting shape continues to move within the wandering boundary after the wandering boundary is set. A computer program that generates a first loitering detection result. Claim 9 A first loitering detection result generation unit that generates a first loitering detection result by using feature information generated by processing multimodal image information composed of RGB images, infrared images, and depth information visualization images that detect a loitering detection target, detecting an object of interest, and determining whether the detected object is loitering; and a second loitering detection result generation unit that generates a second loitering detection result by measuring outliers of re-implemented or predicted image frames for image frames determined by the first loitering detection result generation unit as loitering of the object. The multimodal system includes a decision control unit that makes a final determination of whether the object is wandering based on the second wandering detection result among video frames determined to be wandering by the object according to the first wandering detection result; the first wandering detection result generation unit operates based on a feature matching based wandering detection module configured to generate a skeleton using human joint points through a deep learning algorithm on the feature information, extract multiple center points by extracting the center points of bars connecting the joint points, and identify positional changes of the multiple center points through Frame by Frame Feature Matching; the module identifies positional changes of the multiple center points frame by frame, determines whether the connection shape connecting the multiple center points satisfies conditions for use as a wandering boundary, sets the connection shape as a wandering boundary if the connection shape satisfies the conditions, and if the connection shape continues to move within the wandering boundary after the wandering boundary is set, determines that the object is wandering and generates the first wandering detection result. Frame-by-Frame Feature Matching-based loitering detection system using image information. Claim 10 A Frame by Frame Feature Matching-based loitering detection system using multimodal image information according to claim 9, further comprising an FPN operating unit designed to operate a Feature Pyramid Network (FPN) that parallelly connects RGB images, infrared images, and depth information visualization images acquired from an RGB sensor, an IR sensor, and a LiDAR sensor, and combines feature information extracted from each layer for each acquired RGB image, infrared image, and depth information visualization image into a single set of feature information including information of each image data type to generate feature information utilized in the first and second loitering detection result generation units. Claim 11 A Frame by Frame Feature Matching-based loitering detection system using multimodal image information according to claim 10, wherein the FPN receives feature information extracted from each layer for each of the acquired RGB image, infrared image, and depth information visualization image as multimodal image information, combines the multimodal image information for each layer into a single set of feature information including information of each image data type, and generates feature information according to a Top-down Pathway based on the combination result. Claim 12 A Frame by Frame Feature Matching-based loitering detection system using multimodal image information, wherein the condition usable as the loitering boundary includes a condition defined to be satisfied when the connection shape forms a circular shape, or a condition defined to be satisfied when the connection shape is repeated in a constant shape. Claim 13 delete Claim 14 A Frame by Frame Feature Matching-based loitering detection system using multimodal image information, characterized in that, in claim 9, the decision control unit makes a final decision through a decision algorithm, and the decision algorithm is an algorithm generated through deep learning training that causes a final decision based on a first loitering detection result and a second loitering detection result generated for multimodal image information corresponding to loitering to converge to the loitering of an object. Claim 15 A Frame by Frame Feature Matching-based loitering detection system using multimodal image information, characterized in that, in claim 14, the decision control unit, through the decision algorithm, finally determines as object loitering a frame in which the outlier value predicted by the second loitering detection result is higher than the threshold value set through the learning process among the frames determined as object loitering by the first loitering detection result. Claim 16 delete

Images