Anti-tailgating detection method based on event camera and RGB camera

Abstract

The present invention discloses an anti-tailing detection method based on an event camera and an RGB camera, comprising: (A) collecting an event stream in a preset channel area by an event camera, and collecting a frame image in the preset channel area by an RGB camera; (B) preprocessing the event stream and the frame image, inputting them into a neural network model for training to obtain a detection model, converting the event stream into an event surface, and obtaining the event surface by calculating an event count graph in a time bin with a length of around a timestamp of the frame; (C) performing pedestrian detection on the two input images by the detection model to obtain the number of pedestrians in the current image; (D) screening from a final regression matrix M to find a pedestrian set P whose score is greater than a threshold and whose pedestrian detection box vector is within a preset range, and determining whether there is a tailing behavior by judging the number of the pedestrian set P.

Description

Technical Field

[0001] The present invention relates to the technical field of anti-tailgating, and in particular to an anti-tailgating detection method based on an event camera and an RGB camera. Background Art

[0002] There are two types of existing anti-tailgating detection. The first is to use infrared technology for detection, which has the problems of high false alarms, many missed alarms, and no visual evidence. The second is to use deep learning algorithms to detect pedestrians in video images obtained by RGB cameras, determine the number of pedestrians in the preset channel area, and thus determine whether the pedestrian is being followed. When applied to moving objects, the motion causes blurred imaging, which is prone to false detection, thus affecting the final judgment result. In short, the problems of existing technologies include: RGB cameras will produce blurred imaging for moving objects, and have poor image data collection capabilities under extreme lighting, which affects the image detection effect and is prone to false detection; in application scenarios where there is partial occlusion of pedestrians, the imaging is affected, resulting in missed detection. Summary of the invention

[0003] One object of the present invention is to provide an anti-tailing detection method based on an event camera and an RGB camera. Through the event camera and the RGB camera, the RGB camera records the full image in a preset channel area, and the event camera records the pixel points with "changes" in brightness in the preset channel area. It has the key characteristics of high dynamic range and low latency, can capture environmental information and generate images faster than an RGB camera, and will not be affected by motion blur. It can also record complete visual information of the scene behind the occlusion, which can improve the image data acquisition capability and the yield of anti-tailing detection.

[0004] Other advantages and features of the present invention will become more apparent from the following detailed description and may be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0005] According to one aspect of the present invention, an anti-tailgating detection method based on an event camera and an RGB camera of the present invention, which can achieve the aforementioned objects and other objects and advantages, comprises the following steps:

[0006] (A): The event camera collects the event stream in the preset channel area, and the RGB camera collects the frame image in the preset channel area;

[0007] (B): Preprocess the event stream and frame image, input them into the neural network model for training, obtain the detection model, and convert the event stream into an event surface , by calculating the length of the frame timestamp around The event count graph in the time bin is used to obtain the event surface ;

[0008] (C): Pedestrian detection is performed on the two input images through the detection model to obtain the number of pedestrians in the current image. The event stream of the output event camera is encoded through the pulse neural network, and feature extraction is performed through the convolutional neural network to obtain the event feature F of the event camera. e For the image frame data of the RGB camera, first compare it with the event surface Connect them, and then use the convolutional neural network to extract features to obtain image features F i , the event features and RGB image features are fused to obtain the final feature fusion map F, the final feature fusion map F is used as the input of the prediction head, and the final regression matrix M is output. The final regression matrix M is the pedestrian detection box vector;

[0009] (D): Filter from the final regression matrix M to find the pedestrian set P whose scores are greater than the threshold and whose pedestrian detection box vectors are within the preset range. Determine whether there is tailing behavior by judging the number of pedestrians in the set P. If the number of pedestrians in the set P is greater than 1, it is judged that there is tailing. If the number of pedestrians in the set P is equal to 1, it is judged that there is no tailing.

[0010] According to one embodiment of the present invention, the method further comprises step (E): voting is performed based on the results of multiple judgments, wherein the voting rule is to repeat steps (A) to (D) multiple times, and then calculate the average value of the probability, set a threshold, and if the average value is greater than the threshold, it is judged that there is a tailing; if the average value is equal to the threshold, it is judged that there is no tailing.

[0011] According to an embodiment of the present invention, the step (A) includes a sub-step: the anti-tailgating detection is used for a preset channel to verify whether there is a tailgating phenomenon among pedestrians in the preset channel area.

[0012] According to one embodiment of the present invention, step (B) includes sub-steps: the event stream is in the form of a list of four-tuples (x, y, t, p), wherein x and y represent the plane encoding of the point in the infrared light dot matrix, t represents time, and p represents the vertical movement amplitude of the point.

[0013] According to an embodiment of the present invention, step (B) includes the sub-steps of encoding and denoising the event stream through a pulse neural network, the image data acquired by the event camera contains rich edge information, and the event stream is converted into an event surface. .

[0014] According to one embodiment of the present invention, step (B) comprises sub-steps: the event surface formula is , where T k is the timestamp of the kth frame, , t jrepresents the event stamp of the jth frame, t j The range is , is the event quantization function (the pixel with a positive event outputs 1, otherwise it is 0), x and y represent the plane encoding of the midpoint of the infrared light dot matrix of the kth frame, x j ,y j Represents the plane encoding of the midpoint of the infrared light dot matrix in the jth frame.

[0015] According to an embodiment of the present invention, the step (C) comprises the sub-steps of: event feature F of the event camera e , the formula is , where E s is the event stream obtained by the event camera, CNN is a convolutional neural network, SNN is a spike neural network, and for the image frame data of the RGB camera, first compare it with the event surface Connect them, and then use the convolutional neural network to extract features to obtain image features F i , the formula is , where I f is the image frame data of the RGB camera, Cat is the connection operation, The event surface is converted into an event stream. The event features and RGB image features are fused to obtain the final feature fusion graph F. The formula is: , where Conv is the convolution operation, F i is the image feature, F e Characteristics of the event.

[0016] According to one embodiment of the present invention, the step (D) comprises sub-steps: the formula is , where Is_Tailgating represents whether there is tailgating, len(P) is the summation function of the pedestrian set P, when len(P)>1, Is_Tailgating=True represents tailgating, when len(P)=1, Is_Tailgating=False represents no tailgating.

[0017] The beneficial effects of the present invention are: through the cooperation of the event camera and the RGB camera, the full-frame image in the preset channel area is recorded by the RGB camera, the shooting effect of the moving object is improved by the event camera, and the blurred image caused by motion and occlusion is minimized, thereby improving the judgment accuracy of the anti-tailgating scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of an anti-tailgating detection method based on an event camera and an RGB camera according to an embodiment of the present invention.

[0019] Figure 2It is an algorithm flow chart of the anti-tailgating detection method based on event camera and RGB camera according to the above embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0021] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0022] It is understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number. In addition, in the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more than two.

[0023] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0025] like Figures 1 to 2 As shown, an anti-tailing detection method based on an event camera and an RGB camera according to an embodiment of the present invention is disclosed. The anti-tailing detection method based on an event camera and an RGB camera records the full-frame image in a preset channel area through the cooperation of the event camera and the RGB camera, improves the shooting effect of moving objects through the event camera, and minimizes the blurred image caused by motion and occlusion, thereby improving the judgment accuracy of the anti-tailing scene. The anti-tailing detection method based on an event camera and an RGB camera includes the following steps:

[0026] 1) The event camera is used to collect event streams in the preset channel area, and the RGB camera is used to collect frame images in the preset channel area. In detail, in specific application scenarios, anti-tailgating detection is mainly used in the preset channel to verify whether there is a tailgating phenomenon in the channel area. The RGB camera outputs pictures frame by frame at a fixed frame rate and finally forms a video stream. The event camera records the changes of a scene. For a single pixel, the pixel will only be output when the received light intensity changes;

[0027] 2) Preprocess the event stream and frame image, input them into the neural network model for training, and obtain the detection model. In detail, for the event stream, it is in the form of a list of four tuples (x, y, t, p), where x and y represent the plane encoding of the point in the infrared light dot matrix, t represents time, and p represents the vertical movement amplitude of the point. The four tuple (x, y, t, p) list cannot be directly processed using a convolutional neural network, and the original event stream captured by the event camera contains a lot of noise. Therefore, the present invention uses a pulse neural network to encode and denoise the event stream. The pulse neuron of the pulse neural network takes the pulse sequence as input. Each input pulse will cause a change in the hidden state of the neuron. When the hidden state of the pulse neuron exceeds a certain threshold, the neuron will trigger an output pulse. Therefore, the pulse neural network can naturally encode asynchronous event streams. At the same time, due to the principle of output pulse triggering, discrete noise events cannot cause a sufficiently large change in the hidden state of the pulse neuron to trigger an output pulse. Therefore, discrete noise events will be suppressed. In detail, the image data acquired by the event camera contains rich edge information, and the event stream is converted into an event surface. , by calculating the length of the frame timestamp around The event count graph in the time bin is obtained, and the formula is , where T k is the timestamp of the kth frame, , is the length of time selected for the video. The value range of k is the video frame rate multiplied by Values ​​within t j represents the event stamp of the jth frame, t j The range is , is the event quantization function (the pixel with a positive event outputs 1, otherwise it is 0), x and y represent the plane encoding of the midpoint of the infrared light dot matrix of the kth frame, x j ,y j Represents the plane code of the midpoint of the infrared light dot matrix of the jth frame;

[0028] 3) Pedestrian detection is performed on the two input images through the detection model to obtain the number of pedestrians in the current image. Specifically, the event stream of the event camera output in step 2) is encoded through the pulse neural network, and feature extraction is performed through the convolutional neural network to obtain the event feature F of the event camera. e , the formula is , where E s is the event stream obtained by the event camera, CNN is the convolutional neural network, SNN is the pulse neural network,

[0029] For the image frame data of the RGB camera, first compare it with the event surface Connect them, and then use the convolutional neural network to extract features to obtain image features F i , the formula is , where I f is the image frame data of the RGB camera, Cat is the connection operation, is the event surface of event stream transformation,

[0030] It is worth noting that the event stream only records the pixels with "changes" in brightness in the scene within the channel area, while the RGB image contains visual information. In order to make the acquired image features rich enough, the present invention fuses the event features and the RGB image features to obtain the final feature fusion graph F, which is formulated as follows: , where Conv is the convolution operation, F i is the image feature, F e For event characteristics,

[0031] Furthermore, considering that the event stream features are relatively sparse, while the RGB image features are relatively dense, directly connecting the two modal features cannot fully fuse. Furthermore, a convolution operation Conv is added after the connection to better fuse the two features. The final feature fusion map F is used as the input of the prediction head, and the final regression matrix M is output. The final regression matrix M is the pedestrian detection box vector.

[0032] 4) By judging whether the number of pedestrians in the set channel area meets the preset conditions, it is judged whether there is a pedestrian tailing. Specifically, the final regression matrix M is screened to find the pedestrian set P whose score is greater than the threshold and whose pedestrian detection box vector is within the preset range. The number of pedestrians in the set P is judged to determine whether there is a tailing behavior. If the number of pedestrians in the set P is greater than 1, it is judged that there is a tailing behavior. If the number of pedestrians in the set P is equal to 1, it is judged that there is no tailing behavior. , where Is_Tailgating represents whether there is tailgating, len(P) is the summation function of the pedestrian set P, when len(P)>1, Is_Tailgating=True represents tailgating, when len(P)=1, Is_Tailgating=False represents no tailgating;

[0033] 5) Voting is done based on the results of multiple judgments. The voting rule is to repeat steps 1) to 4) multiple times, preferably 5 times, to obtain 5 probability values ​​[z1, z2, z3, z4, z5], and then calculate the average probability. , where i is any of (1, 2, 3, 4, 5). According to the threshold 1, if If it is greater than 1, it is considered to be trailing. If it is equal to 1, it is judged that there is no tailing, so that the judgment result of the anti-tailing detection method based on event camera and RGB camera is more accurate.

Claims

1. An anti-tailgating detection method based on event camera and RGB camera, characterized in that: The following steps are involved: (A): The event camera collects the event stream in the preset channel area, and the RGB camera collects the frame image in the preset channel area; (B): Preprocess the event stream and frame image, input them into the neural network model for training, obtain the detection model, and convert the event stream into an event surface , by calculating the length of the frame timestamp around The event count graph in the time bin is used to obtain the event surface ; (C): Pedestrian detection is performed on the two input images through the detection model to obtain the number of pedestrians in the current image: the event stream of the output event camera is encoded through the pulse neural network, and feature extraction is performed through the convolutional neural network to obtain the event feature F of the event camera. e For the image frame data of the RGB camera, first compare it with the event surface Connect them, and then use the convolutional neural network to extract features to obtain image features F i , the event features and RGB image features are fused to obtain the final feature fusion map F, the final feature fusion map F is used as the input of the prediction head, and the final regression matrix M is output. The final regression matrix M is the pedestrian detection box vector; (D): Filter from the final regression matrix M to find the pedestrian set P whose scores are greater than the threshold and whose pedestrian detection box vectors are within the preset range. Determine whether there is tailing behavior by judging the number of pedestrians in the set P. If the number of pedestrians in the set P is greater than 1, it is judged that there is tailing. If the number of pedestrians in the set P is equal to 1, it is judged that there is no tailing.

2. The anti-tailgating detection method according to claim 1, characterized in that: The method further includes step (E): voting based on the results of multiple judgments. The voting rule is to repeat steps (A) to (D) multiple times, calculate the average value of the probability, set a threshold, and if the average value is greater than the threshold, it is judged that there is a tailing; if the average value is equal to the threshold, it is judged that there is no tailing.

3. The anti-tailgating detection method according to claim 2, characterized in that: The step (A) includes sub-steps: the anti-tailgating detection is used for a preset channel to verify whether there is a tailgating phenomenon among pedestrians in the preset channel area.

4. The anti-tailgating detection method according to claim 3, characterized in that: The step (B) includes sub-steps: the event stream is in the form of a list of four-tuples (x, y, t, p), wherein x and y represent the plane encoding of the point in the infrared light dot matrix, t represents time, and p represents the vertical movement amplitude of the point.

5. The anti-tailgating detection method according to claim 4, characterized in that: The step (B) includes the following sub-steps: encoding and denoising the event stream by a pulse neural network. The image data acquired by the event camera contains rich edge information. The event stream is converted into an event surface. .

6. The anti-tailgating detection method according to claim 5, characterized in that: The step (B) includes sub-steps: the event surface formula is , Among them, T k is the timestamp of the kth frame, , t j represents the event stamp of the jth frame, t j The range is , is the event quantization function (the pixel with a positive event outputs 1, otherwise it is 0), x and y represent the plane encoding of the midpoint of the infrared light dot matrix of the kth frame, x j ,y j Represents the plane encoding of the midpoint of the infrared light dot matrix in the jth frame.

7. The anti-tailgating detection method according to claim 1, characterized in that: The step (C) comprises the sub-steps of: the event feature F of the event camera e , the formula is , where E S is the event stream obtained by the event camera, CNN is the convolutional neural network, SNN is the pulse neural network, For the image frame data of the RGB camera, first compare it with the event surface Connect them, and then use the convolutional neural network to extract features to obtain image features F i , the formula is , where I f is the image frame data of the RGB camera, Cat is the connection operation, The event surface is converted into an event stream. The event features and RGB image features are fused to obtain the final feature fusion graph F. The formula is: Among them, Conv is the convolution operation, F i is the image feature, F e Characteristics of the event.

8. The anti-tailgating detection method according to claim 1, characterized in that: The step (D) comprises the sub-steps: the formula is , where Is_Tailgating represents whether there is tailgating, len(P) is the summation function of the pedestrian set P, when len(P)>1, Is_Tailgating=True represents tailgating, when len(P)=1, Is_Tailgating=False represents no tailgating.

Images