Method and apparatus for recognizing event by using object detection and optical flow

Abstract

Disclosed are a method and apparatus for recognizing an event by using object detection and optical flow. According to an aspect of the present disclosure, a computer-implemented method for recognizing the occurrence of a target event comprises the steps of: generating one or more optical flow vectors by estimating an optical flow between a current frame and a previous frame; on the basis of whether an object is detected in the current frame, determining at least a partial area of the current frame as a target area; on the basis of the one or more optical flow vectors and the target area, calculating one or more evaluation scores indicating whether there is a movement corresponding to the target event in the current frame; and on the basis of the one or more evaluation scores, determining whether the target event has occurred.

Description

Event recognition method and device using object detection and optical flow

[0001] The present disclosure relates to a method and device for object detection and event recognition using optical flow.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] With recent advancements in IT, research on recognizing object-related events from camera footage has been actively conducted. However, abnormal situations such as falling, slipping, and collapsing are extremely challenging due to their short duration and shorter recognition time compared to other situations. Typically, rather than recognizing the event itself, the occurrence of the event is estimated based on the object's posture after the event. However, this approach suffers from the problem of inaccurate event recognition when the target object is obscured by another object after the event. Furthermore, existing technologies cannot accurately distinguish between cases where the object is moving rapidly and cases where the event is triggered by an abnormal situation. Furthermore, event recognition accuracy varies depending on the size of the object in the video, which is a limitation.

[0004] The present disclosure aims to provide a method and device capable of accurately recognizing the occurrence of an event due to an abnormal situation by combining object detection and optical flow analysis.

[0005] The present disclosure aims to provide a method and device capable of recognizing the occurrence of an event even when an object is not detected in an image at a specific moment or over an entire time period through optical flow analysis based on a virtual bounding box and / or pre-specified grid areas.

[0006] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0007] According to one aspect of the present disclosure, a computer-implemented method for recognizing the occurrence of a target event is provided, comprising: a step of generating one or more optical flow vectors by estimating optical flow between a current frame and a previous frame; a step of determining at least a portion of a current frame as a target region based on whether an object is detected in the current frame; a step of calculating one or more evaluation scores indicating whether there is a movement corresponding to the target event in the current frame based on the one or more optical flow vectors and the target region; and a step of determining whether the target event occurs based on the one or more evaluation scores.

[0008] According to another aspect of the present disclosure, there is provided an event recognition device, comprising: a memory storing instructions; and at least one processor, wherein the at least one processor executes the instructions to estimate an optical flow between a current frame and a previous frame, thereby generating one or more optical flow vectors; determining at least a portion of the current frame as a target area based on whether an object is detected in the current frame; calculating one or more evaluation scores indicating whether there is a movement corresponding to a target event in the current frame based on the one or more optical flow vectors and the target area; and determining whether the target event has occurred based on the one or more evaluation scores.

[0009] According to another aspect of the present disclosure, there is provided a computer-readable, non-transitory recording medium having stored thereon instructions, which, when executed by the computer, cause the computer to perform the steps of: estimating optical flow between a current frame and a previous frame to generate one or more optical flow vectors; determining at least a portion of the current frame as a target region based on whether an object is detected in the current frame; calculating one or more evaluation scores indicating whether there is a movement corresponding to the target event in the current frame based on the one or more optical flow vectors and the target region; and determining whether the target event has occurred based on the one or more evaluation scores.

[0010] According to an embodiment of the present disclosure, by combining object detection and optical flow analysis, it is possible to accurately recognize the occurrence of an event due to an abnormal situation.

[0011] According to an embodiment of the present disclosure, an event occurrence can be recognized even when an object is not detected in an image at a specific moment or over an entire time interval through optical flow analysis based on a virtual bounding box and / or pre-specified grid areas.

[0012] According to an embodiment of the present disclosure, by recognizing the occurrence of an event itself rather than the state of the object after the occurrence of the event, an event alarm can be generated normally even if the object is covered by another object after the occurrence of the event.

[0013] According to an embodiment of the present disclosure, by identifying event candidates based on various conditions and / or excluding identified event candidates, it is possible to distinguish between cases where an object moves quickly and cases where an event occurs due to an abnormal situation.

[0014] According to an embodiment of the present disclosure, the influence of the size of an object on the accuracy of event recognition can be compensated by performing normalization on optical flow vectors based on the area of ​​the region where the object is detected in the image.

[0015] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0016] FIG. 1 is a block diagram schematically showing an event recognition device according to one embodiment of the present disclosure.

[0017] FIG. 2 is a diagram illustrating a confidence score of an object tracked in multiple frames according to one embodiment of the present disclosure.

[0018] FIG. 3A and FIG. 3B are exemplary diagrams referenced to explain an operation of calculating an evaluation score using a bounding box according to various embodiments of the present disclosure.

[0019] FIGS. 4A and 4B are exemplary diagrams referenced to explain an operation of calculating an evaluation score using optical flow according to various embodiments of the present disclosure.

[0020] FIGS. 5A to 5C are exemplary diagrams referenced to explain an operation for determining the occurrence of an event according to various embodiments of the present disclosure.

[0021] FIGS. 6A to 6C are exemplary diagrams referenced to explain an operation of recognizing an event using a posture direction vector according to one embodiment of the present disclosure.

[0022] FIG. 7a and FIG. 7b are flowcharts illustrating a process in which an event recognition device according to one embodiment of the present disclosure processes consecutive frames.

[0023] Figure 8 is a flowchart illustrating an event recognition method according to one embodiment of the present disclosure.

[0024] FIG. 9 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0025] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0026] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0027] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0028] In this disclosure, the term “image” may be used to encompass both still images and moving images (i.e., video).

[0029] FIG. 1 is a block diagram schematically showing an event recognition device according to one embodiment of the present disclosure.

[0030] The event recognition device (10) can recognize the occurrence of a target event from an image by utilizing the results of optical flow estimation and / or object detection. Here, the target event may indicate an abnormal situation that occurs instantaneously, such as an object falling, slipping, or collapsing. The types of target events are not limited to the examples described above, and may include other types of abnormal situations in which the position and / or posture of an object changes in a short period of time.

[0031] As illustrated in FIG. 1, the event recognition device (10) may include all or part of an optical flow estimation module (100), an object detection module (110), an object tracking module (120), a posture estimation module (130), a candidate identification module (140), and an event judgment module (150). Not all blocks illustrated in FIG. 1 are essential components, and some blocks may be added, changed, or deleted in other embodiments. The components illustrated in FIG. 1 may be implemented as one or more software modules or components installed on one or more computing devices at one or more locations. In some examples, one or more computing devices may be dedicated to a specific component. In other examples, multiple components may be executed on the same computing device (or computing devices).

[0032] The optical flow estimation module (100) can estimate optical flow between multiple frames and generate one or more optical flow vectors. The optical flow vector is data representing the motion of pixels between consecutive frames and can be composed of a component for the horizontal direction and a component for the vertical direction of the image. Assuming that the current frame is the nth frame, the optical flow estimation module (100) can estimate optical flow from the (n-1)th frame to the nth frame and generate one or more optical flow vectors. When the width and height of the frame are W and H, respectively, the optical flow estimation module (100) can generate W×H optical flow vectors, but is not limited thereto.

[0033] In some examples, the optical flow estimation module (100) may generate optical flow vectors for some of the pixels constituting the frame. For example, the optical flow estimation module (100) may perform optical flow estimation only for pixels within a pre-specified region of interest (ROI). As another example, the optical flow estimation module (100) may estimate an area where motion has occurred through a separate motion estimation algorithm and perform optical flow estimation only for the estimated area. At this time, an algorithm with a relatively small amount of calculation compared to the optical flow estimation algorithm, such as a pixel difference algorithm, may be applied as the motion estimation algorithm, but is not limited to these examples.

[0034] The optical flow estimation module (100) can estimate optical flow between adjacent frames using various rule-based optical flow estimation algorithms and / or learning-based optical flow estimation algorithms, and the present disclosure does not limit this to a specific method.

[0035] The object detection module (110) can detect an object of a predefined type from an input frame. Here, the object of the predefined type may be, for example, a person. The object detection module (110) can identify each object as a separate bounding box. The object detection module (110) can output a confidence score indicating the probability that an object of the predefined type exists in the corresponding bounding box, along with the location and / or size of the bounding box.

[0036] The object detection module (110) can detect objects from a single frame using various rule-based and / or learning-based object detection algorithms. For example, the object detection module (110) can utilize an artificial intelligence model trained to predict the coordinates of a bounding box surrounding a person, but is not limited to such examples.

[0037] The object tracking module (120) can track objects appearing in an image between multiple frames and identify objects having the same identity. For example, among the bounding boxes detected in multiple frames, the bounding boxes surrounding objects having the same identity can be assigned to the same track. The object tracking module (120) can utilize various object tracking algorithms to assign an object appearing in a specific frame to a certain track and manage the life cycle of the track, and the present disclosure does not limit this in any particular manner.

[0038] When there is an object tracked in multiple frames, i.e., when there are bounding boxes generated in different frames and assigned to the same track, the object tracking module (120) can identify whether the tracked object is a true positive object or a false positive object based on the confidence scores of the bounding boxes assigned to the corresponding track.

[0039] FIG. 2 is a diagram illustrating a confidence score of an object tracked in multiple frames according to one embodiment of the present disclosure.

[0040] Referring to FIG. 2, when a target event occurs, the confidence score of the object detection result may drop sharply due to the object's unusual posture and / or motion blur appearing in the image. In this case, if the tracking target is selected based on a high-level criterion (e.g., 0.5) that is commonly used in general image recognition, it becomes impossible to track the object at the time of (or after) the target event occurrence. In order to enable object tracking in such a situation, the object tracking module (120) may select all objects with a confidence score higher than a relatively low criterion (e.g., 0.05) as tracking targets. However, if such a low criterion is used, the discrimination ability for object detection may decrease, which may result in false alarms due to false detection objects.

[0041] To prevent such problems, the object tracking module (120) can analyze the trend of the confidence score over time to identify whether the tracked object is a true positive or a false positive. For example, if the confidence score of the bounding box generated in the current frame is less than a predetermined threshold (TH), the object tracking module (120) can calculate the average of the confidence scores of the past bounding boxes assigned to the same track as the bounding box. The threshold (TH) applied here may have a higher value than the reference value applied to the selection of the tracking target. If the calculated average is greater than or equal to the threshold (TH), the object tracking module (120) can determine that the detected object is valid, i.e., a true positive. On the other hand, if the calculated average is less than the threshold, the object detection module (110) can determine that the detected object is a false positive and may not perform analysis on whether a target event occurs for the object.

[0042] The candidate identification module (140) can identify a specific frame and / or an object detected in the frame as an event candidate by combining the optical flow vectors estimated from the image and the object tracking results.

[0043] The candidate identification module (140) can produce one or more evaluation scores indicating whether there is a movement corresponding to a target event between the current frame and the previous frame.

[0044] For example, if an object tracked in one or more frames preceding the current frame is also detected in the current frame, the candidate identification module (140) can calculate an evaluation score based on the amount of change in the bounding box over time.

[0045] FIG. 3A and FIG. 3B are exemplary diagrams referenced to explain an operation of calculating an evaluation score using a bounding box according to various embodiments of the present disclosure.

[0046] Referring to FIG. 3a, the candidate identification module (140) generates a current bounding box (BBOX) generated in the current frame. n ) and previous bounding boxes (BBOX) generated from previous frames over a preset period (e.g., 1 second). n-k ~ BBOX n-1 ) can be compared to determine the first evaluation score. Here, the bounding boxes (BBOX n-k ~ BBOX n ) may be bounding boxes surrounding objects with identity, i.e., assigned to the same track.

[0047] The first evaluation score is the current bounding box (BBOX) n ) aspect ratio and previous bounding boxes (BBOX n-k ~ BBOX n-1 ) can have a larger value as the difference between the aspect ratios of the two images increases. For example, the current bounding box (BBOX) n ) aspect ratio and previous bounding boxes (BBOXn-k ~ BBOX n-1 ) can be used as the first evaluation score. Here, the average aspect ratio is preferably the difference between the previous bounding boxes (BBOX n-k ~ BBOX n-1 ) may be the arithmetic mean of the aspect ratios. At this time, the first evaluation score calculated for the nth frame can be expressed as in mathematical expression 1.

[0048]

[0049] Here, AR n represents the aspect ratio of the bounding box generated in the nth frame, T represents the set of indices of frames where the bounding box assigned to a specific track exists, and I {·} is an indicator function that outputs 1 when the condition is true, and 0 otherwise.

[0050] Referring to FIG. 3b, the candidate identification module (140) generates a current bounding box (BBOX) generated in the current frame. n ) and the previous bounding box (BBOX) generated in the previous frame n-1 ) can be determined based on the overlap rate between the bounding boxes (BBOX n-1 and BBOX n ) may be bounding boxes surrounding objects with identity, i.e., assigned to the same track. In some examples, if an object was not detected in the previous frame, the previous bounding box (BBOX n-1 ) Instead, the last detected bounding box before the current frame within the same track may be used.

[0051] The second evaluation score is the current bounding box (BBOX) n ) and the previous bounding box (BBOX) n-1 ) can have a larger value as the overlap rate between them increases. For example, the current bounding box (BBOX n) and the previous bounding box (BBOX) n-1 ) can be used as the second evaluation score. At this time, the second evaluation score calculated for the nth frame can be expressed as in mathematical expression 2.

[0052]

[0053] Here, BBOX p represents the bounding box detected immediately before the current bounding box among the bounding boxes assigned to a specific track. Meanwhile, the present disclosure assumes that IoU is used as a metric to express the overlap rate between bounding boxes, but the present disclosure is not limited thereto, and various other metrics may be used.

[0054] The candidate identification module (140) determines whether the first evaluation score and / or the second evaluation score are within a preset numerical range, and then determines whether the current bounding box (BBOX) is within the preset numerical range. n ) can be identified as an event candidate (hereinafter, current object). At this time, the numerical ranges can be separately provided for the first evaluation score and the second evaluation score. For example, the candidate identification module (140) can identify the current object as an event candidate when the first evaluation score is 0.4 or higher. As another example, the candidate identification module (140) can identify the object as an event candidate when the second evaluation score is 0.4 or lower. In order to increase the identification accuracy, the candidate identification module (140) can also use a combination of the first evaluation score and the second evaluation score. For example, the candidate identification module (140) can identify the current object as an event candidate when the first evaluation score is 0.4 or higher and the second evaluation score is 0.9 or lower. As another example, the candidate identification module (140) can identify the current object as an event candidate when the first evaluation score is 0.05 or higher and the second evaluation score is 0.4 or lower. Identifying the current object as an event candidate is done by determining the current bounding box (BBOX). n) may include identifying the track to which the event is assigned as an event candidate.

[0055] The candidate identification module (140) can calculate a third evaluation score using estimated optical flow vectors for at least a portion of the current frame. At this time, the target region for calculating the third evaluation score may be determined differently depending on whether an object is tracked in one or more frames preceding the current frame and / or whether a tracked object is detected in the current frame.

[0056] FIGS. 4A and 4B are exemplary diagrams referenced to explain an operation of calculating an evaluation score using optical flow according to various embodiments of the present disclosure.

[0057] Figure 4a shows an example of setting a target area based on a bounding box generated from one or more frames.

[0058] For example, if an object tracked in the preceding frame(s) is also detected in the current frame, i.e. the current bounding box (BBOX) generated in the current frame n ) is assigned to the track and the track is maintained, the candidate identification module (140) is currently bounding box (BBOX n ) can be determined as the target area.

[0059] As another example, if an object tracked in the preceding frame(s) is not detected in the current frame, for example, if the track is lost because the object is not detected in the current frame, the candidate identification module (140) calculates the last immediately preceding detection bounding box (BBOX) assigned to the track. p ), the target area can be determined. The candidate identification module (140) determines the target area based on the immediately preceding detection bounding box (BBOX p ) can be used to estimate the virtual bounding box (vBBOX) for the current frame and determine it as the target area. The virtual bounding box (vBBOX) is the previously detected bounding box (BBOX p) can have the same focus as the previous detection bounding box (BBOX). The width and height of the virtual bounding box (vBBOX) are each equal to the width and height of the previous detection bounding box (BBOX p ) can be determined based on the width and height of the virtual bounding box (vBBOX). For example, the width and height of the virtual bounding box (vBBOX) can be determined based on the width and height of the immediately preceding detected bounding box (BBOX p ) can be set as a multiple of the maximum of the width and height. At this time, different multiples can be applied to calculate the width and height of the virtual bounding box (vBBOX). For example, the multiple can be set considering the aspect ratio of the image, but is not limited to this example.

[0060] Meanwhile, cases where a track is lost can be broadly divided into cases where an object moves out of the surveillance area and does not appear in the image (IMG) (i.e., true negative), and cases where an object actually appears in the image (IMG) but detection fails (i.e., false negative). The candidate identification module (140) detects the immediately preceding detection bounding box (BBOX) p ) can be used to determine whether the loss of a track is due to detection or non-detection. For example, the previous detection bounding box (BBOX) p ) is located within a preset outer region (BRD), the candidate identification module (140) can determine that the loss of the track is due to reconnaissance. The outer region (BRD) can be set to have a certain ratio in the width and / or height direction of the image (IMG), but is not limited to this example. If the loss of the track is determined to be due to reconnaissance, the candidate identification module (140) can exclude the track from the event candidate selection target. On the other hand, if the loss of the track is determined to be due to a false detection, the candidate identification module (140) can exclude the track from the event candidate selection target as described above. p ) can be used to determine the virtual bounding box (vBBOX) for the current frame.

[0061] The candidate identification module (140) selects the target area (i.e., the current bounding box (BBOX)) from among the optical flow vectors estimated between the current frame and the previous frame. n ) or virtual bounding box (vBBOX)) can be selected, and the third evaluation score can be determined using the selected optical flow vectors.

[0062] The candidate identification module (140) can calculate a third evaluation score based on the average of the components projected from each selected optical flow vector onto a predetermined reference vector. Here, the reference vector can be defined based on the direction in which motion mainly occurs when a target event occurs. For example, if the target event is a falling event, the reference vector can be defined parallel to the vertical axis of the image. In this case, the third evaluation score generated for the nth frame can be expressed as in mathematical expression 3.

[0063]

[0064] Here, R represents the target area, and N R represents the number of optical flow vectors calculated within the target area, and OF i,y represents the y-axis component of the ith optical flow vector within the target area.

[0065] An object that appears large in the image may have a large third evaluation score even if its actual movement is small. To compensate for the effect of the size of such an object, the candidate identification module (140) may normalize the optical flow vector or the component projected onto the reference vector based on the area of ​​the target area. For example, the candidate identification module (140) may normalize the current bounding box (BBOX) as in Equation 4. n ) can be normalized based on the area of ​​the optical flow vector.

[0066]

[0067] Meanwhile, the direction of the reference vector is not limited to the examples described above, and can be defined in various ways considering the type of target event and / or the installation environment of the camera.

[0068] Current bounding box (BBOX) n ) or a third evaluation score calculated for a virtual bounding box (vBBOX) is greater than or equal to a predetermined threshold, the candidate identification module (140) can identify an object corresponding to the bounding box as an event candidate. For example, the candidate identification module (140) can identify an object corresponding to the current bounding box (BBOX) as an event candidate. n ) is the track to which the virtual bounding box (vBBOX) is assigned, or the immediately preceding detected bounding box (BBOX) that serves as the basis for generating the virtual bounding box (vBBOX). p ) can identify the track to which it is assigned as an event candidate.

[0069] Figure 4b shows an example of setting a target area based on multiple grid areas.

[0070] If no object is detected in the current frame, the candidate identification module (140) can determine each of a plurality of grid areas (G1 to G20) that are divided into the image (IMG) according to preset criteria as a target area. That is, the candidate identification module (140) can calculate a third evaluation score for each grid area (G1 to G20). The candidate identification module (140) can identify a grid area whose third evaluation score is equal to or greater than a first threshold as a valid grid area. The candidate identification module (140) can identify the current frame as an event candidate when the number of valid grid areas identified for the current frame is greater than a preset second threshold. In this case, the number of valid grid areas identified for the n-th frame can be expressed as in mathematical expression 5.

[0071]

[0072] Here, OF_Score n,Girepresents the third evaluation score calculated for the i-th grid of the n-th frame, TH1 represents the first threshold value compared with the third evaluation score, and N G represents the number of grid areas, and I {·} is an instruction function that outputs 1 when the condition is true, and 0 otherwise.

[0073] Meanwhile, in Fig. 4b, an example of dividing the entire area of ​​the image (IMG) into multiple grid areas is shown, but in another example, a part of the image (IMG) may be divided into multiple grid areas. For example, if an object is not detected in the current frame and the track is lost, the candidate identification module (140) detects the previous detection bounding box (BBOX) p ) can be used to generate a virtual bounding box (vBBOX), and the virtual bounding box (vBBOX) can be divided into multiple grid areas. If the number of valid grid areas identified for the virtual bounding box (vBBOX) is a preset second threshold value, the candidate identification module can identify an object or the current frame corresponding to the virtual bounding box (vBBOX) as an event candidate.

[0074] The event judgment module (150) can additionally observe subsequent frames after the point in time at which the event candidate is identified to determine whether a target event has occurred. The event judgment module (150) can exclude the identified event candidate from the analysis target if the object and / or optical flow tracked in the subsequent frames satisfies a predetermined filtering condition. That is, the event judgment module (150) can verify whether the identified event candidate corresponds to the target event. If the event candidate is not excluded until the observation of a preset number of subsequent frames is completed, the event judgment module (150) can determine that the target event has occurred and generate an event alarm.

[0075] FIGS. 5A to 5C are exemplary diagrams referenced to explain an operation for determining the occurrence of an event according to various embodiments of the present disclosure.

[0076] Referring to FIG. 5A, the event judgment module (150) can determine whether to exclude an event candidate based on the movement trajectory of the object identified as the event candidate. If the object identified as the event candidate moves stably in subsequent frames, the event judgment module (150) can exclude the event candidate from the event alarm generation target. For example, the event judgment module (150) may exclude bounding boxes (BBOX) assigned to the track prior to candidate event identification. n-4 Inland BBOX n-1 ) and the bounding boxes (BBOX) assigned to the track after candidate event identification. n+14 Inland BBOX n+3 ) can be compared. If the trajectories before and after candidate event identification have similar directions, the event judgment module (150) can determine that the candidate event identification was performed incorrectly.

[0077] Referring to FIG. 5b, the event judgment module (150) can determine whether to exclude an event candidate based on pre-specified regions of interest (ROI1 and ROI2). The event judgment module (150) can compare the location of a bounding box corresponding to an object identified as a candidate event with the regions of interest (ROI1 and ROI2). For example, if an object detected at a point far from the region of interest (ROI1) is identified as an event candidate, the event judgment module (150) can exclude the event candidate from the event alarm generation target. As another example, if the object does not approach the region of interest (ROI2) in subsequent frames after the event candidate is identified, the event judgment module (150) can exclude the event candidate from the event alarm generation target. To this end, a first region of interest (ROI1) may be set for an area where a target event is likely to occur, and / or a second region of interest (ROI2) may be set for an area where an object is predicted to reach after the target event occurs. For example, in the case of a fall event, an area with a height difference, such as a staircase, railing, or ladder, may be set as a first region of interest (ROI1), and an expected drop area where the object is expected to reach after a fall may be set as a second region of interest (ROI2).

[0078] Referring to FIG. 5c, the event judgment module (150) can determine whether to exclude an event candidate based on the number of valid grid areas identified in a plurality of frames. The event judgment module (150) can calculate a third evaluation score for each grid for a preset number of frames following the frame identified as an event candidate, and identify the number of valid grids.

[0079] The event judgment module (150) can limit the minimum value of the number (V1) of valid grids identified in the first number (N1) of frames. For example, if the number (V1) of valid grids identified in the first number (N1) of frames is less than a predetermined lower limit threshold, the event candidate can be excluded from the event alarm generation target. This is to exclude from the event generation target a situation in which a strong optical flow is estimated for a short moment only in one frame and / or one grid area, in order to minimize the possibility of false alarms. At this time, if event judgment is performed simply based on the sum of the third evaluation scores calculated for each grid area, the value of noise is also added to the sum, so the valid grid area is first filtered based on the third evaluation score.

[0080] Additionally or alternatively, the event judgment module (150) may limit the maximum value of the number (V2) of valid grids identified in the second number (N2) of frames. Here, the second number (N2) may have a larger value than the first number (N1). For example, if the number (V2) of valid grids identified in the second number (N2) of frames exceeds a predetermined upper limit threshold, the event candidate may be excluded from the event alarm generation target. This is to exclude a situation in which a large optical flow is estimated for a long time from the event generation target, considering the characteristic that the target event occurs in a short moment. Such a situation may mainly occur when an object moves at a high speed in a specific direction.

[0081] In some examples, the event judgment module (150) may determine whether to exclude an event candidate based on the trend of the confidence score over time as described above in FIG. 2. For example, the event judgment module (150) may track an object identified as an event candidate for a predetermined number of subsequent frames, and then compare the confidence score of each of the bounding boxes assigned to its track with a threshold (TH). If the confidence scores of all the bounding boxes assigned to the track are less than the threshold (TH), the event judgment module (150) may exclude the event candidate from the event alarm generation target.

[0082] In some examples, to improve the accuracy of target event recognition, a pose direction vector representing the object's pose may be additionally utilized for event candidate identification and / or final event determination. To this end, the pose estimation module (130) may estimate the positions of key points corresponding to the object from the image and generate a pose direction vector using the key points. Key points may also be referred to as landmarks or joints.

[0083] FIGS. 6A to 6C are exemplary diagrams referenced to explain an operation of recognizing an event using a posture direction vector according to one embodiment of the present disclosure.

[0084] Referring to FIG. 6A, the pose estimation module (130) can estimate the positions of key points (J0 to J14) corresponding to one or more parts of an object, respectively. The parts and the number of key points from which the pose estimation module (130) extracts key points may vary depending on the implementation example. For example, in FIG. 6A, only one key point (J0) is extracted from the head part, but the pose estimation module (130) according to another example may extract multiple key points corresponding to detailed parts of the head (e.g., eyes, nose, ears, etc.). Table 1 shows body parts corresponding to the key points (J0 to J14) illustrated in FIG. 6A.

[0085] Key Point AreasKey Point AreasJ0Center of HeadJ1NeckJ8Lower TorsoJ2Right ShoulderJ9Right HipJ3Right ElbowJ10Right KneeJ4Right WristJ11Right AnkleJ5Left ShoulderJ12Left HipJ6Left ElbowJ13Left KneeJ7Left WristJ14Left Ankle

[0086] The pose estimation module (130) can estimate the coordinates of one or more key points by applying a pose estimation algorithm to the entire image or a portion of the image where an object is detected. For example, the pose estimation module (130) can extract key points for individual objects by applying a pose estimation algorithm to each bounding box generated by the object detection module (110). As another example, the pose estimation module (130) can extract all possible key points by applying a pose estimation algorithm to the entire image. At this time, the object detection module (110) can identify individual objects by connecting at least some of the extracted key points according to a predetermined key point connecting algorithm. Here, pose estimation and / or key point connecting can be implemented by various rule-based or learning-based algorithms, and the present disclosure does not limit them in a specific manner.

[0087] Referring to FIG. 6b, the posture estimation module (130) can determine a posture direction vector (PV) representing the posture of the object based on one or more extracted key points. The posture estimation module (130) can generate the posture direction vector (PV) using one or more of the key points (J0 to J7) corresponding to the upper body and one or more of the key points (J8 to J14) corresponding to the lower body. For example, the posture direction vector (PV) can be a vector that points from the midpoint (C1) of the right pelvic keypoint (J9) and the left pelvic keypoint (J12) to the midpoint (C1) of the right shoulder keypoint (J2) and the left shoulder keypoint (J5). At this time, the angle (θ) of the posture direction vector (PV) can be measured counterclockwise with respect to the horizontal axis (X) of the image.

[0088] Referring to FIG. 6c, an event candidate can be identified and / or a target event can be determined based on the amount of change in the angle (θ) of the posture direction vector (PV) between multiple frames.

[0089] For example, the candidate identification module (140) can calculate a fourth evaluation score by comparing the angle measured at the direction of the object in the current frame with the angle measured at the direction of the object in one or more preceding frames. The fourth evaluation score can have a larger value as the difference between the angle measured in the preceding frames and the angle measured in the current frame increases. The candidate identification module (140) can identify the object as an event candidate when the fourth evaluation score is equal to or greater than a predetermined threshold.

[0090] Additionally or alternatively, the event determination module (150) may determine whether a target event has occurred by analyzing a change trend in the angle (θ) of the posture direction vector (PV) for the entire track after tracking an object identified as an event candidate for a predetermined number of subsequent frames. For example, the event determination module (150) may determine that a target event has occurred if a posture change greater than a predetermined threshold appears between at least some of the frames in which an object identified as an event candidate is detected.

[0091] FIG. 7a and FIG. 7b are flowcharts illustrating a process in which an event recognition device according to one embodiment of the present disclosure processes consecutive frames.

[0092] The event recognition device (10) can receive the current frame (S700).

[0093] The event recognition device (10) can detect an object in the current frame and assign the detected object to a track (S702).

[0094] The event recognition device (10) can estimate the optical flow from the previous frame to the current frame and generate optical flow vectors (S704).

[0095] The event recognition device (10) can check whether there is a pre-identified event candidate (S706). If there is an object detected in step S702, it can be checked whether the object has the same identifier as the object identified as the event candidate.

[0096] If there are no pre-identified event candidates, the event recognition device (10) can check whether an object has been detected in the current frame (S710). If an object has been detected in the current frame, the event recognition device (10) can check whether the object has been tracked in preceding frames (S712). In other words, the event recognition device (10) can check whether a track generated in any preceding frame is maintained in the current frame.

[0097] If an object tracked in preceding frames is detected in the current frame, the event recognition device (10) can calculate an evaluation score using the current bounding box generated for the object in the current frame (S714). For example, the event recognition device (10) can calculate the first to third evaluation scores using the current bounding box. Optionally, the event recognition device (10) can also calculate a fourth evaluation score by comparing the angle of the posture direction vector generated in the preceding frames with the angle of the posture direction vector generated in the current frame.

[0098] If an object is not detected in the current frame, the event recognition device (10) can check whether an object being tracked in preceding frames exists. That is, the event recognition device (10) can check whether the track has been lost because an object is not detected in the current frame (S720). In this case, if the maximum age period has elapsed since the track was lost, the object being tracked can be considered to not exist.

[0099] If there is an object being tracked, i.e., a lost track, the event recognition device (10) can create a virtual bounding box using the bounding box last assigned to the track and calculate an evaluation score based on this (S722). For example, the event recognition device (10) can calculate a third evaluation score for the entire virtual bounding box. As another example, the event recognition device (10) can divide the virtual bounding box into a plurality of grid areas and calculate a third evaluation score for each grid area.

[0100] If there is no object being tracked, the event recognition device (10) can divide the current frame into multiple grid areas and calculate an evaluation score for each grid area (S730). For example, the event recognition device (10) can calculate a third evaluation score for each grid area.

[0101] The event recognition device (10) can verify whether the calculated evaluation score(s) satisfy predetermined candidate identification conditions (S740). The candidate identification conditions may be set individually for each evaluation score or may be set for a combination of multiple evaluation scores. For example, the candidate identification conditions may include that the first evaluation score is equal to or greater than a first threshold and the second evaluation score is equal to or less than the second threshold. As another example, the candidate identification conditions may include that the third evaluation score is equal to or greater than the third threshold. As yet another example, the candidate identification conditions may include that the number of valid grid areas in which the third evaluation score is equal to or greater than a fourth threshold is equal to or greater than a fifth threshold.

[0102] The event recognition device (10) can add the current frame and / or an object detected in the current frame to the event candidates if any one of one or more candidate identification conditions is satisfied (S742).

[0103] If there is a pre-identified event candidate, the event recognition device (10) can check whether the current frame and / or the object detected in the current frame satisfies a predetermined event filtering condition (S750). The event filtering condition may be applied differently depending on whether the event candidate was identified based on a bounding box or a grid area. For example, if a specific object is identified as an event candidate based on a bounding box, the event filtering condition may include that the object has a stable movement trajectory in subsequent frames, that all bounding boxes detected for the object have a low level of confidence score, and / or that the object does not approach a preset region of interest. As another example, if a specific frame is identified as an event candidate based on a grid area, the event filtering condition may include that the number of valid grids identified in a first number of subsequent frames is less than a predetermined lower threshold and / or that the number of valid grids identified in a second number of subsequent frames is greater than a predetermined upper threshold.

[0104] The event recognition device (10) can exclude the corresponding object and / or frame from the event candidates if any one of one or more event filtering conditions is satisfied (S752).

[0105] After event identification, the event recognition device (10) can check whether a preset number (N) of subsequent frames have elapsed (S760 and S762). If the event candidate is not excluded until observation of the preset number (N) of subsequent frames is completed, the event recognition device (10) can generate an alarm for the target event (S764).

[0106] Figure 8 is a flowchart illustrating an event recognition method according to one embodiment of the present disclosure.

[0107] The event recognition device (10) estimates the optical flow between the current frame and the previous frame and generates one or more optical flow vectors (S800).

[0108] The event recognition device (10) determines at least a portion of the current frame as a target area based on whether an object is detected in the current frame (S820). For example, if an object is detected in the current frame, the event recognition device (10) may determine a bounding box surrounding the object as the target area. As another example, if an object is not detected in the current frame, the event recognition device (10) may determine each of a plurality of grid areas divided into the current frame according to preset criteria as a target area. The event recognition device (10) may determine a different area as the target area depending on whether there is a track tracked in one or more frames preceding the current frame and / or whether a tracked object is detected in the current frame. For example, if an object tracked in the preceding frame(s) is not detected in the current frame, the event recognition device (10) may determine an area including at least one bounding box generated in the preceding frames and the periphery of the bounding box as the target area.

[0109] The event recognition device (10) calculates one or more evaluation scores indicating whether there is a movement corresponding to a target event between a current frame and a previous frame based on one or more optical flow vectors and a target area (S840). The one or more evaluation scores may include a first evaluation score calculated based on a difference between an aspect ratio of a first bounding box generated for a specific object in the current frame and an average aspect ratio of one or more second bounding boxes generated for the object in the preceding frame(s). Additionally or alternatively, the one or more evaluation scores may include a second evaluation score calculated based on an overlap rate between a first bounding box generated for a specific object in the current frame and one or more second bounding boxes generated for the object in the preceding frame(s). Additionally or alternatively, the one or more evaluation scores may include a third evaluation score calculated based on an average of components obtained by projecting each optical flow vector corresponding to the target area onto a predetermined reference vector. In some examples, each optical flow vector or its projected component onto a reference vector may be normalized based on the area of ​​the target region. Additionally or alternatively, one or more evaluation scores may include a fourth evaluation score calculated based on the temporal change in a posture direction vector representing the pose of the object. To this end, the event recognition device (10) may extract one or more keypoints for the object from the current frame and generate a posture direction vector using the extracted keypoints.

[0110] The event recognition device (10) determines whether the target event has occurred based on one or more evaluation scores (S860). The event recognition device (10) can identify an event candidate based on one or more evaluation scores. For example, if any one of the first to fourth evaluation scores has a value higher than a preset threshold, the event recognition device (10) can add the current frame and / or an object detected in the current frame to the event candidate. The event recognition device (10) can verify the event candidate by additionally observing optical flow vectors or objects in frames subsequent to the current frame. For example, the event recognition device (10) can verify whether the event candidate corresponds to the target event based on evaluation scores (e.g., the third evaluation score) for optical flow vectors estimated in frames subsequent to the current frame. Additionally or alternatively, the event recognition device (10) can verify whether the event candidate corresponds to the target event based on a comparison between the object's movement trajectory in frames preceding the current frame and the object's movement trajectory in frames subsequent to the current frame. Additionally or alternatively, the event recognition device (10) can verify whether the event candidate corresponds to the target event based on a confidence score of one or more bounding boxes generated for the object. Additionally or alternatively, the event recognition device (10) can verify whether the event candidate corresponds to the target event based on whether the object approaches a preset region of interest in frames subsequent to the current frame.

[0111] FIG. 9 is a block diagram schematically illustrating an exemplary computing device that can be used to implement the devices and methods described in the present disclosure.

[0112] The computing device (90) may include some or all of a memory (900), a processor (920), storage (940), an input / output interface (960), and a communication interface (980). The computing device (90) may structurally and / or functionally include at least a portion of the event recognition device (10). The computing device (90) may be a stationary computing device such as a desktop computer, a server, etc., as well as a mobile computing device such as a laptop computer, a smart phone, etc. The computing device (90) may be implemented with any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (90) may include a graphic processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0113] The memory (900) may store a program that causes the processor (920) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (920), and the above-described method or operation may be performed by executing the plurality of instructions by the processor (920). The memory (900) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in a plurality of memories. When the memory (900) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (900) may include at least one of a volatile memory and a non-volatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the non-volatile memory includes a flash memory.

[0114] The processor (920) may include at least one core capable of executing at least one instruction. The processor (920) may execute instructions stored in the memory (900). The processor (920) may be a single processor or multiple processors.

[0115] Storage (940) maintains stored data even when power supplied to the computing device (90) is cut off. For example, storage (940) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk. A program stored in storage (940) may be loaded into memory (900) before being executed by processor (920). Storage (940) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (900). Storage (940) may store data to be processed by processor (920) and / or data processed by processor (920).

[0116] The input / output interface (960) may provide an interface with an input device such as a keyboard, mouse, etc. and / or an output device such as a display device, printer, etc. A user may trigger the execution of a program by the processor (920) through an input device and / or check the processing result of the processor (920) through an output device.

[0117] The communication interface (980) may provide access to an external network. The computing device (90) may communicate with other devices (e.g., a camera) via the communication interface (980).

[0118] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0119] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0120] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0121] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0122] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0123]

[0124] CROSS-REFERENCE TO RELATED APPLICATION

[0125] This patent application claims priority to Korean Patent Application No. 10-2024-0102587, filed on August 1, 2024, which is incorporated herein by reference in its entirety.

Claims

1. A computer implementation method for recognizing the occurrence of a target event, A process of generating one or more optical flow vectors by estimating the optical flow between the current frame and the previous frame; A process of determining at least a portion of the current frame as a target region based on whether an object is detected in the current frame; A process of calculating one or more evaluation scores indicating whether there is a movement corresponding to the target event in the current frame based on the one or more optical flow vectors and the target area; and A process for determining whether the target event has occurred based on one or more of the above evaluation scores. A computer-implemented method comprising:

2. In paragraph 1, The target area is determined based on the first bounding box of the object generated in the current frame, A computer-implemented method, wherein the one or more evaluation scores include a first evaluation score calculated based on a difference between an aspect ratio of the first bounding box and an average aspect ratio of one or more second bounding boxes generated for the object in one or more frames preceding the current frame.

3. In paragraph 1, The target area is determined based on the first bounding box of the object generated in the current frame, A computer-implemented method, wherein the one or more evaluation scores include a second evaluation score calculated based on an overlap rate between the first bounding box and one or more second bounding boxes generated for the object in one or more frames preceding the current frame.

4. In paragraph 1, One or more of the above evaluation scores, A computer-implemented method, comprising a third evaluation score calculated based on the average of components projected from each optical flow vector corresponding to the target area onto a predetermined reference vector.

5. In paragraph 4, The target area is determined based on the bounding box of the object created in the current frame or a frame preceding the current frame, A computer-implemented method, wherein the above-described calculating process includes a process of normalizing each optical flow vector or a component projected onto the reference vector based on the area of ​​the target region.

6. In paragraph 1, The above decision process is, A computer-implemented method for determining a different region as the target region based on whether there is a tracked object in one or more frames preceding the current frame and whether the tracked object is detected in the current frame.

7. In paragraph 1, The above decision process is, A computer-implemented method for determining an area including at least one bounding box generated in the preceding one or more frames and the periphery of the at least one bounding box as the target area, when an object tracked in one or more frames preceding the current frame is not detected in the current frame.

8. In paragraph 1, The above decision process is, A computer-implemented method for determining each of a plurality of grid areas divided into the current frame according to preset criteria as the target area when the object is not detected in the current frame.

9. In paragraph 1, The above judgment process is, A process of identifying event candidates based on one or more of the above evaluation scores; and A process of verifying the event candidate by observing the optical flow vectors or the object in frames subsequent to the current frame. A computer-implemented method comprising:

10. In paragraph 9, The above verification process is, A computer-implemented method for verifying whether the event candidate corresponds to the target event based on an evaluation score calculated based on the optical flow vectors estimated in frames subsequent to the current frame.

11. In paragraph 9, The above verification process is, A computer-implemented method for verifying whether the event candidate corresponds to the target event based on a comparison between the movement trajectory of the object in frames prior to the current frame and the movement trajectory of the object in frames subsequent to the current frame.

12. In paragraph 9, The above verification process is, A computer-implemented method for verifying whether the event candidate corresponds to the target event based on the confidence score of one or more bounding boxes generated for the object.

13. In paragraph 9, The above verification process is, A computer-implemented method for verifying whether the event candidate corresponds to the target event based on whether the object approaches a preset region of interest in frames subsequent to the current frame.

14. In paragraph 1, A process of extracting one or more keypoints for the object from the current frame; and Further comprising a process of generating a direction vector expressing the pose of the object using one or more of the above key points, A computer-implemented method, wherein the one or more evaluation scores include a fourth evaluation score calculated based on the amount of change in the direction vector over time.

15. A memory for storing instructions; and at least one processor, At least one processor executes the instructions, Estimate the optical flow between the current frame and the previous frame, and generate one or more optical flow vectors, Based on whether an object is detected in the current frame, at least a portion of the current frame is determined as a target region, Based on the one or more optical flow vectors and the target area, one or more evaluation scores indicating whether there is a movement corresponding to a target event in the current frame are calculated, An event recognition device that determines whether the target event has occurred based on one or more of the evaluation scores.

16. A non-transitory computer-readable storage medium having stored thereon a command, wherein the command, when executed by the computer, causes the computer to: A process of generating one or more optical flow vectors by estimating the optical flow between the current frame and the previous frame; A process of determining at least a portion of the current frame as a target region based on whether an object is detected in the current frame; A process of calculating one or more evaluation scores indicating whether there is a movement corresponding to the target event in the current frame based on the one or more optical flow vectors and the target area; and A process for determining whether a target event has occurred based on one or more of the above evaluation scores. A non-transitory computer-readable storage medium that enables execution.

Images