Information identification system and method thereof
Through the multi-dimensional information identification system of the Internet of Things and big data technology, the abnormal behaviors in the security inspection area are tracked and detected in real time, and the problem that the existing security inspection system cannot effectively detect non-metallic hazardous items and contraband in the body is solved, improving security inspection efficiency and safety sex.
Patent Information
- Application Number
- CN201910985868.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-10-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-10-16
AI Technical Summary
The existing security inspection system is inefficient and cannot effectively detect non-metal hazardous items and entrained contraband in the body, resulting in safety risks and traffic congestion, and the person under inspection has poor experience.
Adopt a multi-dimensional information identification system based on the Internet of Things and big data, collect video streams and other information through multi-dimensional information collection equipment, track objects in real time, detect abnormal behaviors such as bars, item legacy and personnel retention, and issue alarms based on the detection results. .
It has achieved efficient, accurate and insensitive identification of dangerous behaviors in dense areas such as public transportation, improved security inspection efficiency and safety, and reduced blockage and disgust of the inspected person.
Smart Images

Figure CN112668377B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of identification, and more specifically, to a multi-dimensional information dangerous behavior identification system and method based on the Internet of Things and big data technology. Background Art
[0002] Public security is related to the stability of the country and society, and directly affects the safety of people's lives and property. In order to protect personal and public safety, comprehensive security inspection has gradually become an inevitable trend. However, due to the low efficiency of traditional security inspection methods, limited ability to detect dangerous information, and poor experience of the inspected objects, the existing security inspection system can no longer meet the needs of comprehensive security inspection, especially during peak hours, which will cause serious congestion and affect traffic efficiency. For example, traditional security inspection gates can only detect metal objects carried by the person, and cannot detect non-metallic dangerous objects such as explosive reagents and contraband carried in the body. In addition, methods such as hand touch and visual inspection are even more inefficient and will cause disgust to the inspected person.
[0003] At present, the main dangerous behaviors in the security inspection field include: the person being inspected carrying contraband on the body; carrying contraband in the items carried; the person being inspected being contaminated with toxic or explosive chemical reagents; passing uninspected items across fences or security inspection machines (referred to as passing items across fences); leaving objects behind, fighting, illegal detention, running in the opposite direction, etc. in (designated) public areas. Summary of the invention
[0004] The purpose of this application is to solve the above-mentioned existing problems.
[0005] In a first aspect of the present application, an information identification system is provided, which may include: an information acquisition module, configured to include a multi-dimensional information acquisition device to acquire multi-dimensional information about an object; an information processing module, configured to detect whether there is abnormal information about the object based on the information acquired by the information acquisition module; a storage module, configured to store the information acquired by the information acquisition module and various data generated by the detection by the information processing module; and a display and reminder module, configured to display abnormal information and issue an alarm.
[0006] According to the first aspect, the information processing module may also be configured to: when multiple abnormal information is detected simultaneously, determine the priority of the multiple abnormal information, and send the abnormal information to the display and reminder module according to the priority.
[0007] According to the first aspect, the information processing module can also be configured to: extract the bounding box and features about the object from the video stream collected by the information acquisition module, and perform the following operations based on at least one of the bounding box and the features: real-time tracking or searching for the object, detecting whether the object is passing objects through the fence, and whether there are any objects left behind or people stranded in the area of interest.
[0008] According to the first aspect, the information processing module can also be configured to perform the following operations for real-time tracking or searching: extracting multiple bounding boxes from a first frame in a video stream, and extracting multiple bounding boxes from a second frame in the video stream; pairing each of the multiple bounding boxes extracted from the first frame with each of the bounding boxes extracted from the second frame, thereby forming a pairing matrix including multiple pairing items; and calculating the value of each of the multiple pairing items in the pairing matrix, and taking the item with the optimal value as the target item, wherein the target item refers to the item formed by pairing rectangular bounding boxes for the same object.
[0009] According to the first aspect, the information processing module can also be configured to: calculate the intersection-and-union ratio between paired bounding boxes in each of multiple paired items; calculate the similarity between the features in the paired bounding boxes in each paired item; and select the paired item with the largest intersection-and-union ratio between the bounding boxes and the largest similarity between the features in the bounding boxes as the target item.
[0010] According to the first aspect, the information processing module can also be configured to perform the following operations to detect whether there is a barrier-passing action: detecting whether there are objects in the region of interest on both sides of the position of interest based on the bounding box; when the object completely enters the region of interest, detecting the key points of the object and determining the distance between the key points at the wrist; and when the distance between the key points at the wrist is less than or equal to a first threshold, determining that there is a barrier-passing action between the objects.
[0011] According to the first aspect, the information processing module can also be configured to: extract the bounding box of the object within the region of interest on both sides of the position of interest, and calculate the area ratio between the bounding box and the region of interest; and when the area ratio is greater than or equal to a second threshold, determine that the object has completely entered the region of interest.
[0012] According to the first aspect, the information processing module can also be configured to perform the following operations to detect whether there are any items left behind or people stranded in the area of interest: detecting whether there are any static objects in the area of interest; detecting the residence time of the static object in the area of interest; based on the characteristics, detecting whether the static object is a person or an object; and in the case where the static object is a person and the residence time is greater than or equal to a threshold, there are people stranded in the area of interest; or in the case where the static object is an object, detecting whether there are people within a threshold distance from the object, and in the case where there are no people within the threshold distance from the object and the residence time is greater than or equal to the threshold, there are items left behind in the area of interest.
[0013] In a second aspect of the present application, an information identification method is provided, which may include: using a multidimensional information acquisition device to acquire multidimensional information about an object; based on the acquired multidimensional information, detecting whether there is abnormal information about the object; storing the acquired multidimensional information and various data generated when performing the detection; and displaying the abnormal information and issuing an alarm.
[0014] According to the second aspect, the method may further include: in a case where a plurality of abnormal information is detected simultaneously, determining priorities of the plurality of abnormal information, thereby displaying the abnormal information according to the priorities.
[0015] According to the second aspect, the detection of whether there is abnormal information for the object based on the collected multi-dimensional information may include: extracting a bounding box and features about the object from the video stream collected by the information acquisition module, and performing the following operations based on at least one of the bounding box and the features: real-time tracking or searching for the object, detecting whether the object has a barrier-passing action, and detecting whether there are any objects left behind or people stranded in the area of interest.
[0016] According to the second aspect, the real-time tracking or searching of the object may include: extracting multiple bounding boxes from a first frame in a video stream, and extracting multiple bounding boxes from a second frame in the video stream; pairing each of the multiple bounding boxes extracted from the first frame with each of the bounding boxes extracted from the second frame, thereby forming a pairing matrix including multiple pairing items; and calculating the value of each of the multiple pairing items in the pairing matrix, and taking the item with the optimal value as the target item, wherein the target item refers to the item formed by pairing the rectangular bounding boxes of the same object.
[0017] According to the second aspect, calculating the value of each of the multiple pairing items in the pairing matrix and taking the item with the best value as the target item may include: calculating the intersection-over-union ratio between the paired bounding boxes in each of the multiple pairing items; calculating the similarity between the features in the paired bounding boxes in each pairing item; and selecting the pairing item with the largest intersection-over-union ratio between the bounding boxes and the largest similarity between the features in the bounding boxes as the target item.
[0018] According to the second aspect, the detection of whether there is a barrier-passing action between objects may include: detecting whether there are objects in the region of interest on both sides of the position of interest based on a bounding box; when the object completely enters the region of interest, detecting the key points of the object to determine the distance between the key points; and when the distance between the key points is less than or equal to a first threshold, determining that there is a barrier-passing action between the objects.
[0019] According to the second aspect, detecting whether there are objects in the regions of interest on both sides of the position of interest based on the bounding box may include: extracting the bounding boxes of the objects in the regions of interest on both sides of the position of interest, and calculating the area ratio between the bounding boxes and the regions of interest; and determining that the object has completely entered the region of interest when the area ratio is greater than or equal to a second threshold.
[0020] According to the second aspect, the detection of whether there are any items left behind or people stranded in the area of interest may include: detecting whether there are any static objects in the area of interest; based on the features, detecting whether the static object is an object or an object, and detecting the residence time of the static object in the area of interest; and in the case where the static object is an object and the residence time is greater than or equal to a threshold, there is an object stranded in the area of interest; or in the case where the static object is an object, detecting whether there is an object within a threshold distance from the object, and in the case where there is no object within the threshold distance from the object and the residence time is greater than or equal to the threshold, there are items left behind in the area of interest.
[0021] According to a third aspect of the present application, a non-transitory computer-readable medium is provided, comprising program code instructions recorded thereon and capable of implementing the method according to the second aspect.
[0022] Through the information identification system, information identification method and non-temporary computer-readable medium provided according to the above-mentioned various aspects of the present application, it is possible to comprehensively, efficiently, accurately and imperceptibly identify dangerous behaviors based on multi-dimensional information in public transportation, major events, key areas, crowded areas and other areas requiring security checks. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A block diagram of an information identification system according to an example embodiment is shown in FIG.
[0024] Figure 2 An example block diagram of an information acquisition module according to an example embodiment is shown in FIG.
[0025] Figure 3 A block diagram of an information processing module according to an example embodiment is shown in FIG.
[0026] Figure 4 A flow chart of a process for pedestrian tracking according to an example embodiment is shown in FIG.
[0027] Figure 5 A flow chart of a process for stall object detection according to an example embodiment is shown in FIG.
[0028] Figure 6 A flowchart of processing of illegal stay of persons / leftover items according to an example embodiment is shown in FIG. DETAILED DESCRIPTION
[0029] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not have to be used to implement the present invention. In other examples, in order to avoid confusing the present invention, known circuits, materials or methods are not specifically described.
[0030] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any appropriate combinations and / or subcombinations.
[0031] It should be understood that when an element is said to be "coupled to" or "connected to" another element, it can be directly coupled or connected to the other element or there may be intermediate elements. On the contrary, when an element is said to be "directly coupled to" or "directly connected to" another element, there are no intermediate elements.
[0032] Furthermore, as used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.
[0033] It will be understood that a noun corresponding to a term in the singular form may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B or C" may include all possible combinations of items listed together with a corresponding one of the plurality of phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component and do not limit the components in other respects (e.g., importance or order).
[0034] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).
[0035] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or replacement forms for the corresponding embodiments. Unless otherwise clearly defined herein, all terms will be given their broadest possible interpretation, including the meanings implied in the specification and the meanings understood by those skilled in the art and / or defined in dictionaries, papers, etc.
[0036] In addition, it should be understood by those skilled in the art that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale. For the description of the drawings, similar reference numerals may be used to refer to similar or related elements. The present disclosure will be described exemplarily with reference to the drawings below.
[0037] In order to meet the needs of comprehensive security inspection and accurate identification of abnormal information, the present application provides an information identification system and method that can efficiently, accurately and imperceptibly identify abnormal information.
[0038] Figure 1 A block diagram of an information identification system 1 according to an exemplary embodiment is shown in FIG. As shown in the figure, the information identification system 1 may include an information acquisition module 100 , an information processing module 101 , a storage module 102 , and a display and reminder module 103 .
[0039] The information acquisition module 100 may be configured to acquire information of the inspected object and / or the object of interest. In an example embodiment, the information acquisition module 100 may include a multi-dimensional information acquisition device. The information acquisition module 100 may be one or more of a camera, a terahertz / millimeter wave security inspection instrument, an article machine, an infrared camera, an electronic nose, a depth camera, etc., but is not limited thereto.
[0040] In an example embodiment, the information acquisition module 100 may acquire one or more of video information, image information, motion information, face information, gait information, skeleton information, iris information, key point information (e.g., wrist, elbow, etc.) of the inspected object and / or the object of interest, but is not limited thereto.
[0041] The configuration and functions of the information acquisition module 100 will be described in detail in the following description.
[0042] The information processing module 101 may be configured to receive the collected information about the inspected object and / or the object of interest from the information collecting module 100 , and identify whether there is abnormal information based on the received information.
[0043] Abnormal information may include, but is not limited to, the inspected object carrying prohibited items, the inspected object being contaminated with drugs or explosive fuel, the object crossing the fence to deliver items, people illegally staying, items left behind, the object having abnormal behavior, etc.
[0044] The information processing module 101 may be configured to enable multiple security inspection devices to be linked by comprehensively processing and judging abnormal information.
[0045] The information processing module 101 may include multiple functional modules. The configuration and functions of the information processing module 101 will be described in detail in the following description.
[0046] The storage module 102 may be configured to store any information and data generated when the information recognition system 1 performs information recognition processing. For example, the storage module 102 may store any information about the inspected object and / or the object of interest obtained by the information acquisition module 100, various data generated when the information processing module 101 performs abnormal information recognition, etc., but is not limited thereto.
[0047] The display and reminder module 103 can be configured to display the abnormal information identified by the information processing module 101 and issue an alarm.
[0048] In one example, the display and reminder module 103 may be configured to display the object causing the abnormal information, etc.
[0049] In one example, the display and reminder module 103 may be configured to display the time and location of occurrence of the abnormal information.
[0050] In an exemplary embodiment, the display and reminder module 103 may include a display device and an alarm device. The display device may be, for example, any device capable of providing a display function, such as a computer, a tablet computer, a personal digital assistant, any wearable device (e.g., an electronic watch, electronic glasses, etc.), etc., but is not limited thereto.
[0051] The alarm device may be any device capable of providing alarm information, such as an acoustic alarm device (e.g., a bell), an optical alarm device (e.g., a flashing light), a tactile alarm device (e.g., a vibration device), etc., but is not limited thereto.
[0052] The configuration and functions of the display and reminder module 103 are described in the following description.
[0053] Figure 2An example block diagram of the information acquisition module 100 according to an example embodiment is shown in FIG. Figure 2 The configuration of the information acquisition module 100 shown in FIG. 1 is merely illustrative for the purpose of convenience of explanation, and those skilled in the art should understand that the configuration of the information acquisition module 100 is not limited thereto.
[0054] As shown in the figure, the information collection module 100 may, for example, include a terahertz / millimeter wave device 1000 for security inspection in the security inspection area, an electronic nose 1001, an item machine 1002, and a device (e.g., a camera) 1003 for video collection in object gathering areas (e.g., waiting halls, terminal halls, etc.) and security inspection areas.
[0055] In an example embodiment, the terahertz / millimeter wave device 1000 may be configured to perform perspective photography of the inspected object to check whether the inspected object has any contraband hidden on it.
[0056] In an example embodiment, the electronic nose 1001 may be configured to acquire the odor of the subject to be inspected, so as to analyze whether the subject to be inspected has been in contact with drugs, explosives, etc.
[0057] In an example embodiment, the item machine 1002 may be configured to obtain information about items in the luggage for use in detecting whether contraband items are hidden in the luggage, etc.
[0058] In an example embodiment, the video capture device 1003 may be configured to capture video about objects in the gathering area.
[0059] The number of various devices included in the information collection module 100 may be one or more. In addition, according to actual needs, the information collection module 100 may also include other modules in addition to the above modules.
[0060] Figure 3 A block diagram of the information processing module 101 according to an example embodiment is shown in FIG. Figure 3 The configuration of the information processing module 101 shown in FIG. 1 is merely illustrative for the purpose of convenience of explanation, and those skilled in the art should understand that the configuration of the information processing module 101 is not limited thereto.
[0061] As shown in the figure, the information processing module 101 may include a video / image analysis module 1010 and a comprehensive analysis module 1011 .
[0062] The video / image analysis module 1010 can be configured to implement functions such as object tracking, object searching, identification of objects handed over through barriers, and identification of illegal stays of pedestrians and objects left behind based on videos / images acquired from video acquisition devices, terahertz / millimeter wave devices and / or object machines.
[0063] For ease of explanation, the following embodiments are taken as examples to describe the functions implemented in the video / image analysis module 1010, such as object tracking, object search, barrier object passing recognition, and illegal stay of pedestrians and objects left behind.
[0064] Object tracking
[0065] The general principle of real-time object tracking is: first, obtain the rectangular bounding box of the object in different image frames of the video stream, and extract the object features in the rectangular bounding box; then, exclude some rectangular bounding boxes by calculating the overlap ratio of the rectangular bounding boxes, and then, through feature comparison, determine that the objects with the most similar features are the same objects.
[0066] Figure 4 A flowchart of a process for pedestrian tracking according to an example embodiment is shown in FIG. Figure 4 Give a description.
[0067] Specifically, when performing real-time object tracking, at step 41, the video / image analysis module 1010 obtains a video stream F={f i}, i = 1: T, where T represents time, f i Represents each frame of the image.
[0068] Afterwards, the video / image analysis module 1010 analyzes the video stream to track the pedestrian. Specifically, at step 42, for each frame of image f i , using the detection algorithm based on deep convolutional network, the video frame is taken as the network input, and the rectangular bounding box of the pedestrian in the frame image is output, denoted as pbox = [x, y, w, h], where (x, y) represents the upper left corner vertex of the marked rectangular box, and (w, h) represents the width and height of the marked rectangular box.
[0069] At step 43, a series of pedestrian rectangular boxes obtained by detecting the current frame are paired with the rectangular boxes obtained by detecting the previous frame. The pairing weights the pedestrian re-identification result (reid_cost) and the overlap ratio of the pedestrian rectangular box (box_overlap_cost) to obtain the pairing matrix C pre*cur , where pre*cur represents the matrix dimension, pre is the number of pboxes obtained by detecting the previous frame, and cur represents the number of pboxes obtained by detecting the current frame.
[0070] The rectangular pairing matrix can be, for example:
[0071]
[0072] where c 1*1Indicates that the first rectangular bounding box in the previous frame is paired with the first rectangular bounding box in the current frame, c 1*cur Indicates that the first rectangular bounding box in the previous frame is paired with the cur-th rectangular bounding box in the current frame, c pre*1 Indicates that the pre-th rectangular bounding box in the previous frame is paired with the first rectangular bounding box in the current frame, C pre*cur It means that the pre-th rectangular bounding box in the previous frame is paired with the cur-th rectangular bounding box in the current frame, and so on. Thus, pre*cur rectangular bounding box pairs can be formed.
[0073] At step 44, the value of each item in the pairing matrix is calculated, and the item with the smallest value is selected as the target item.
[0074] Pairing Matrix C pre*cur Each item in (c 1*1 , ..., c 1*cur , ..., c pre*1 , ...c pre*cur ) can be expressed as:
[0075] C=w1(box_overlap_cos t)+w2(reid_cos t), w1, w2 are weighting coefficients.
[0076] Among them, (box_overlap_cost) represents the degree of overlap between pedestrian rectangular boxes, and (reid_cost) represents the similarity between pedestrian features within the pedestrian rectangular box.
[0077] In order to determine (reid_cost), a pedestrian re-identification model based on a deep convolutional network is used to extract pedestrian features in pbox (in, Indicates the features of the object extracted from the pth rectangular bounding box, and N indicates the number of bounding boxes). Then, based on the extracted features, the Euclidean distance between the corresponding features in the pbox in the previous and next frames is calculated as the reid_cost, that is:
[0078]
[0079] Based on the above description, it can be known that when (box_overlap_cost) is the largest and (reid_cost) is the smallest, it can be determined that the objects with the largest feature similarity in the rectangular bounding box with the largest overlap ratio are the same person.
[0080] To facilitate determination, C = w1(box_overlap_cos t) + w2(reid_cos t) is modified to Therefore, the project with the smallest C' value is determined as the target project, and the objects included in the target project are the same object.
[0081] In this way, real-time pedestrian tracking is achieved, and the returned result is marked as tracker = [id, x, y, w, h], where id represents the unique identifier of the pedestrian, through which the object can be located.
[0082] The above description is about real-time tracking of objects between consecutive frames. In some cases, this principle can also be used to realize personnel search between non-consecutive frames (for example, frames separated by a period of time).
[0083] In one example, when a fight has occurred at a gathering place and the fight has ended, in order to find the people involved in the fight, the object tracking function can be used to find the people involved in the fight. The principle is as follows:
[0084] Based on the fight video collected by the video / image acquisition device, a detection algorithm based on a deep convolutional network is used to obtain the rectangular bounding box pbox = [x, y, w, h] of the people involved in the fight, where (x, y) represents the upper left corner vertex of the marked rectangular box, and (w, h) represents the width and height of the marked rectangular box.
[0085] Video / image acquisition equipment is used to continuously collect videos from various places in the gathering area. For each frame of the image, a detection algorithm based on a deep convolutional network is also used to obtain the rectangular bounding box of the object in the image. After that, each rectangular bounding box in the frame is paired with the rectangular bounding boxes of the people involved in the fight in the fight video image frame to obtain a pairing matrix C". According to the above description, the calculation formula for the values of each item in the pairing matrix C" can be transformed into:
[0086] w1, w2 are weighting coefficients.
[0087] Among them, (box_overlap_cost) represents the degree of overlap between pedestrian rectangular boxes, and in, (reid_cost) represents the similarity between pedestrians in the pedestrian rectangle.
[0088] The pedestrian re-identification model based on the deep convolutional network is used to extract the features of the objects in each rectangular bounding box pbox (for example, facial features, gait features, body features, iris features, etc.). Based on the extracted features, the Euclidean distance between the corresponding features in the rectangular bounding box pbox is calculated:
[0089]
[0090] In addition, the intersection-over-union ratios between the rectangular bounding boxes in the current frame and the rectangular bounding boxes of the persons participating in the fight in the fight video image frame are also calculated.
[0091] Based on this, when the C" value is the smallest, it can be indicated that the item with the smallest value in the pairing matrix is the target item, that is, the objects in the two rectangular bounding boxes of the target item are the same person.
[0092] At this point, the pedestrians involved in the fight can be tracked.
[0093] ·Detection of objects passing through barriers
[0094] The rough principle of fence-passing detection is: based on the region of interest (R) (e.g., the area on both sides of the fence), the location of interest (e.g., the fence L), and the pedestrian rectangular bounding box pbox(i), the area ratio of pedestrians on both sides of the location of interest entering the region of interest is calculated:
[0095]
[0096] If area_ratio is greater than the threshold, it means that pedestrians on both sides of the location of interest have entered the area of interest. At this time, by detecting the distance between the key points of the pedestrians and the distance between the key points at the wrist of the human body, it is determined whether there is a barrier between pedestrians.
[0097] Figure 5 A flow chart of the process of detecting a handover between fences according to an example embodiment is shown in FIG. At step 51 , a region of interest R (eg, a distance range of 0.5 from the fence, etc.) is first selected.
[0098] At step 52, it is detected whether there are pedestrians in the region of interest. Specifically, for each frame, for each region of interest on both sides of the position of interest, it is determined whether there are pedestrians in the region of interest.
[0099] At step 53, if it is determined that area_ratio is greater than or equal to the first threshold ("yes" in step 53), it is determined that pedestrians have entered the regions of interest on both sides of the location of interest. If area_ratio is less than the first threshold ("no" in step 53), it is determined that no pedestrians have entered the regions of interest on both sides of the location of interest or only one side has entered.
[0100] When no pedestrians enter the regions of interest on both sides of the position of interest or when pedestrians enter only one region (“Yes” in step 53 ), the process returns to step 52 again.
[0101] When pedestrians enter the regions of interest on both sides of the position of interest ("Yes" in step 53), the process proceeds to step 54 to estimate the postures of the pedestrians entering the regions on both sides of the position of interest.
[0102] At step 55, if the distance between the key points at the wrists of the pedestrians on both sides of the location of interest is greater than the second threshold value ("No" at step 55), it is determined that there is no barrier-passing behavior between the pedestrians entering the regions of interest on both sides of the location of interest, and the process returns to step 52. If the distance between the key points of the pedestrians on both sides of the location of interest is less than or equal to the second threshold value ("Yes" at step 55), the process proceeds to step 56, and it is determined that there is a barrier-passing behavior between the pedestrians entering the regions of interest on both sides of the location of interest.
[0103] Alternatively, when identifying whether pedestrians and / or objects enter the regions of interest on both sides of the position of interest, it is also possible to analyze whether the features are human features or object features based on the features for the pedestrians and / or objects.
[0104] Detect illegal stay of pedestrians and items left behind
[0105] The rough principle of detecting pedestrians illegally staying or leaving items behind is: first determine whether the static object is a person or an item, then determine whether the person or item meets the stay time, and thus determine whether the person or item is illegally staying / left behind.
[0106] Figure 6 A flowchart of processing of illegal stay of persons / leftover items according to an example embodiment is shown in FIG.
[0107] At step 61, an area of interest R is selected. For example, a corner in a terminal hall may be selected.
[0108] At step 62, it is detected whether a certain object in the region of interest is a static object. For example, a Gaussian mixture model is used to model the background, and a frame difference method is used to detect static objects based on the acquired background information and the initial background.
[0109] At step 63, the type of the static object is detected, that is, whether the static object is a person or an object. For example, based on the above description, the rectangular bounding box pbox of the static object can be first obtained, and then the features of the static object can be extracted from the rectangular bounding box pbox. If one (or some) features of the static object (for example, face features, gait features, body features, iris features, etc.) can be extracted, it can be determined that the static object is a person. Otherwise, it can be determined that the static object is an object.
[0110] When it is determined that the static object is a person ("Yes" at step 64), the process proceeds to step 65 to determine whether the static person is the same person. For example, a rectangular bounding box pbox of the static person can be extracted in each frame image, and then the features of the static person (e.g., facial features, gait features, body features, iris features, etc.) can be extracted in the rectangular bounding box pbox. By comparing the extracted features, it is determined whether the static person is the same person.
[0111] In the case where the static person is not the same person ("No" at step 65), the process returns to step 62 again to detect a static object.
[0112] In the case where the static person is the same person ("Yes" at step 65), the process proceeds to step 66 to determine whether the static person's stay time in the region of interest is greater than or equal to a threshold. In the case where the static person's stay time is greater than the threshold, the process proceeds to step 67 to determine that the static person is an illegal stayer.
[0113] In the case where the static object is an article ("No" at step 64), the process proceeds to step 68 to determine whether the static articles are the same article and there are no people around the article. For example, it can be determined whether they are the same article by extracting and comparing the features of the static articles. For example, it can be determined whether there are people around the article by judging whether the features of a person can be extracted around the article.
[0114] In the case that the static object is not the same object and / or there is a person around the object (“No” in step 68 ), the process returns to step 62 again to re-detect the static object.
[0115] In the case where the static object is the same object and there is no person around the object (“Yes” in step 68 ), the process proceeds to step 69 to determine whether the residence time of the object is greater than or equal to a threshold value.
[0116] When the residence time of the item is less than the threshold value (“No” at step 69 ), the process returns to step 62 again.
[0117] In the case where the retention time of the item is greater than or equal to the threshold value (“Yes” at step 69 ), the process proceeds to step 610 , where it is determined that the item is a left-behind item.
[0118] Although in the above description, the residence time judgment (step 66 and step 69) is described after judging whether the static object pair is the same person or whether the static objects are the same object and there is no one around them (step 65 and step 68), those skilled in the art should understand that the residence time judgment (step 66 and step 69) can be performed before judging whether the static object pair is the same person or whether the static objects are the same object and there is no one around them (step 65 and step 68).
[0119] The above is a description of the functions of real-time tracking of personnel, searching for a certain person, detecting objects handed over by barriers, illegal stay of personnel and / or objects left behind, etc., which are implemented by the video / image analysis module 1010 based on the video / image provided by the information acquisition module 100. It should be understood by those skilled in the art that the various operations described above are not restrictive. Any operation that conforms to the concept of the present invention is included within the scope of the present invention.
[0120] Reference again Figure 3 The video / image analysis module 1010 can also be configured to detect whether the inspected object is carrying / contaminated with prohibited items and detect the category of the prohibited items based on the video / image obtained from the video acquisition device, the terahertz / millimeter wave device and / or the item machine, and the odor information obtained from the electronic nose.
[0121] The description of how to detect whether the inspected object is carrying / contaminated with prohibited items and how to detect the type of prohibited items will not be described in detail here.
[0122] Reference again Figure 3 The comprehensive analysis module 1011 may be configured to perform comprehensive analysis based on the analysis results obtained by the video / image analysis module 1010 and the model pre-stored in the comprehensive analysis module 1011 .
[0123] In an exemplary embodiment, comprehensive analysis may refer to, for example, but not limited to, one or more of the following items: analyzing whether the person or item that causes the abnormal information (for example, carrying explosives, passing objects through barriers, illegal stay, etc.) has previously performed other dangerous behaviors; in the case of multiple abnormal information, determining the priority of each abnormal information; analyzing whether the abnormal information has appeared before; analyzing whether the person who caused the abnormal information has previously caused other abnormal information, etc.
[0124] In one example, when the video / image analysis module 1010 simultaneously obtains dangerous information that object A at the security checkpoint carries contraband and that there is a fight in the terminal hall based on the video and / or image sent by the information collection module 100, the comprehensive analysis module 1011 can analyze that the security level of object A carrying contraband is higher than the security level of dangerous information that there is a fight in the terminal hall based on the model pre-stored therein. Therefore, the comprehensive analysis module 1011 can preferentially send the information about object A carrying contraband to the display and reminder module 103, and then send the information about the fight in the terminal hall to the display and reminder module 103.
[0125] Alternatively, when the display and reminder modules 103 are arranged in multiple locations, when the video / image analysis module 1010 simultaneously draws dangerous information such as a fight in the terminal hall or a knife attack in the terminal hall based on the video and / or image sent by the information collection module 100, the comprehensive analysis module 1011 can send each information to the display and reminder module 103 at or near the incident site.
[0126] In one example, the comprehensive analysis module 1011 may compare the danger information generated by a certain object analyzed by the video / image analysis module 1010 with the danger information previously generated by the object stored in the storage module 102 to determine whether the danger information is the same as the previous danger information. In the case where the danger information is the same, the comprehensive analysis module 1011 may no longer repeatedly send the alarm information to the display and reminder module 103 to avoid unnecessary interference with other information.
[0127] Although the various steps are described above according to the order shown in the drawings, those skilled in the art should understand that the various steps may be performed in a different order, or the embodiments of the present invention may be implemented without one or more of the above steps.
[0128] It can be understood from the foregoing that the electronic components of one or more systems or devices may include, but are not limited to, at least one processing unit, a memory, and a communication bus or communication device that couples the various components including the memory to the processing unit. The system or device may include or may access various device-readable media. The system memory may include device-readable storage media in the form of volatile and / or non-volatile memory (e.g., read-only memory (ROM) and / or random access memory (RAM)). By way of example and not limitation, the system memory may also include an operating system, application programs, other program modules, and program data.
[0129] Embodiments may be implemented as systems, methods, or program products. Thus, embodiments may take the form of all-hardware embodiments or embodiments that include software (including firmware, resident software, microcode, etc.), which may be collectively referred to herein as "circuits," "modules," or "systems." In addition, embodiments may take the form of a program product embodied in at least one device-readable medium having device-readable program code embodied thereon.
[0130] Combinations of device-readable storage media may be used. In the context of this document, a device-readable storage medium ("storage medium") may be any tangible, non-signal medium that may contain or store a program consisting of program code configured to be used by or in conjunction with an instruction execution system, apparatus, or device. For purposes of this disclosure, a storage medium or device should be interpreted as being non-transitory, i.e., not including signal or propagation media.
[0131] The present disclosure is presented for purposes of illustration and description, but is not intended to be exhaustive or limiting. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to illustrate the principles and practical applications, and to enable those of ordinary skill in the art to understand the various embodiments of the present disclosure with various modifications suitable for the intended specific use.
Claims
1. An information identification system, comprising: An information collection module is configured to include a plurality of information collection devices to collect multi-dimensional information about the object; an information processing module, configured to detect whether there is abnormal information for the object based on the information collected by the information collection module; a storage module configured to store the information collected by the information collection module and various data generated by the detection by the information processing module; and The display and reminder module is configured to display abnormal information and issue an alarm. Wherein, the information processing module is further configured to: Extracting a rectangular bounding box and features about the object from the video stream collected by the information collection module; Detecting whether there are objects in the regions of interest on both sides of the position of interest based on the bounding box; In the case where the object completely enters the region of interest, detecting key points at the wrist of the object to determine distances between the key points; and When the distance between the key points is less than or equal to the first threshold, it is determined that there is a barrier-passing action between the objects.
2. The information recognition system according to claim 1, wherein: The information processing module is further configured to: when a plurality of abnormal information is detected at the same time, determine the priority of the plurality of abnormal information, so as to send the abnormal information to the display and reminder module according to the priority.
3. The information recognition system according to claim 1, wherein: The information processing module is further configured to perform the following operations based on at least one of the bounding box and the feature: real-time tracking or searching for an object, and whether there are any items left behind or people stranded in an area of interest.
4. The information recognition system according to claim 3, wherein: The information processing module is also configured to perform the following operations to perform real-time tracking or searching: extracting a plurality of bounding boxes from a first frame in the video stream, and extracting a plurality of bounding boxes from a second frame in the video stream; Pairing each of the plurality of bounding boxes extracted from the first frame with each of the bounding boxes extracted from the second frame, respectively, to form a pairing matrix including a plurality of pairing items; as well as The value of each item in the plurality of paired items in the pairing matrix is calculated, and the item with the best value is used as the target item, wherein the target item refers to an item formed by pairing rectangular bounding boxes of the same object.
5. The information recognition system according to claim 4, wherein: The information processing module is further configured to: calculating an intersection-over-union ratio between paired bounding boxes in each of the plurality of paired items; Calculate the similarity between the features in the paired bounding boxes in each paired item; as well as The paired item with the largest intersection-over-union ratio between bounding boxes and the largest similarity between features in the bounding boxes is selected as the target item.
6. The information identification system according to claim 3, wherein the information processing module is further configured to: extracting bounding boxes of objects within a region of interest on both sides of the position of interest, and calculating an area ratio between the bounding boxes and the region of interest; and When the area ratio is greater than or equal to a second threshold, it is determined that the object has completely entered the region of interest.
7. The information recognition system according to claim 3, wherein the information processing module is further configured to perform the following operations to detect whether there are any objects left behind or people stranded in the area of interest: Detect whether there are static objects in the area of interest; detecting the residence time of the static object in the region of interest; Based on the features, detecting whether the static object is a person or an object; as well as In the case where the static object is a person and the retention time is greater than or equal to a threshold, there is a person retained in the region of interest; or When the static object is an article, it is detected whether there is a person within a threshold distance from the article. When there is no person within the threshold distance from the article and the residence time is greater than or equal to the threshold, there is an article left in the area of interest.
8. An information identification method, comprising: Collecting multi-dimensional information about the object using a multi-dimensional information collection device; Based on the collected multi-dimensional information, detecting whether there is abnormal information for the object; Storing the collected multi-dimensional information and various data generated when performing the detection; as well as Display abnormal information and issue alarms. Wherein, the information identification method further includes: Extracting a rectangular bounding box and features about the object from the video stream collected by the information collection module; Detecting whether there are objects in the regions of interest on both sides of the position of interest based on the bounding box; In the case where the object completely enters the region of interest, detecting key points at the wrist of the object to determine distances between the key points; and When the distance between the key points is less than or equal to the first threshold, it is determined that there is a barrier-passing action between the objects.
9. The information identification method according to claim 8, wherein: The method further includes: in a case where a plurality of abnormal information is detected simultaneously, determining priorities of the plurality of abnormal information, so as to display the abnormal information according to the priorities.
10. The information identification method according to claim 9, wherein: The detecting whether there is abnormal information for the object based on the collected multi-dimensional information includes: performing the following operations based on at least one of the bounding box and the feature: real-time tracking or searching for the object, and detecting whether there are objects left behind or people stranded in the area of interest.
11. The information identification method according to claim 10, wherein: The real-time tracking or searching of the object includes: extracting a plurality of bounding boxes from a first frame in the video stream, and extracting a plurality of bounding boxes from a second frame in the video stream; Pairing each of the plurality of bounding boxes extracted from the first frame with each of the bounding boxes extracted from the second frame, respectively, to form a pairing matrix including a plurality of pairing items; and The value of each item in the plurality of paired items in the pairing matrix is calculated, and the item with the best value is used as the target item, wherein the target item refers to an item formed by pairing rectangular bounding boxes of the same object.
12. The information identification method according to claim 11, wherein: The calculating the value of each item in the plurality of paired items in the paired matrix and taking the item with the best value as the target item comprises: calculating an intersection-over-union ratio between paired bounding boxes in each of the plurality of paired items; Calculating the similarity between the features in the paired bounding boxes in each paired item; and The paired item with the largest intersection-over-union ratio between bounding boxes and the largest similarity between features in the bounding boxes is selected as the target item.
13. The information recognition method according to claim 8, wherein the detecting whether there are objects in the regions of interest on both sides of the position of interest based on the bounding box comprises: extracting a bounding box of an object within a region of interest on both sides of the position of interest, and calculating an area ratio between the bounding box and the region of interest; as well as When the area ratio is greater than or equal to a second threshold, it is determined that the object has completely entered the region of interest.
14. The information recognition method according to claim 10, wherein the step of detecting whether there are any items left behind or people stranded in the region of interest comprises: Detect whether there are static objects in the area of interest; Based on the features, detecting whether the static object is an object or an article, and analyzing whether the static object; detecting the residence time of the static object in the region of interest; as well as In the case where the static object is an object and the retention time is greater than or equal to a threshold, an object is retained in the region of interest; or When the static object is an article, it is detected whether there is an object within a threshold distance from the article. When there is no object within the threshold distance from the article and the residence time is greater than or equal to the threshold, there is an article left in the area of interest.
15. A non-transitory computer-readable medium comprising program code instructions recorded thereon and capable of implementing the method according to any one of claims 8 to 14.
