Behavior detection method, device, electronic device and computer-readable storage medium

Abstract

The present application provides a behavior detection method, device, electronic device and computer-readable storage medium. The method includes: extracting multiple image frames from a surveillance video in chronological order; inputting the multiple image frames into a pedestrian detection model, detecting by the pedestrian detection model whether there is a pedestrian in each of the multiple image frames, and generating a pedestrian border in the image frame where the pedestrian exists; when there is a pedestrian in at least two of the multiple image frames, determining the displacement pixel distance of the pedestrian in the adjacent image frames by a preset target tracking algorithm; when the displacement pixel distance is greater than or equal to a preset threshold corresponding to the size of the pedestrian border, inputting the image area where the pedestrian border exists in the multiple image frames into a posture detection model, and obtaining a detection result output by the posture detection model that characterizes whether the pedestrian has a running behavior, which is conducive to improving the accuracy and reliability of behavior detection.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a behavior detection method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In some specific scenarios, there is a need to detect people running. For example, if people run in some parks, it often indicates that an emergency may occur. Timely detection of abnormal running behavior can help supervisors quickly locate abnormal event status, troubleshoot dangerous factors on the scene, and protect the lives and property of others. At present, manual inspections or traditional machine learning methods are usually used to detect people running. Park video surveillance systems often need to monitor dozens or hundreds of videos at the same time. Due to the limitations of human vision, manual inspections often miss things, and this method has the disadvantages of low efficiency and high labor costs. When using traditional machine learning methods to detect running, due to the complexity and variability of monitoring scenes, traditional machine learning methods have low generalization capabilities and are prone to missed detections and false alarms. Summary of the invention

[0003] The purpose of the present application is to provide a behavior detection method, device, electronic device and computer-readable storage medium, which can improve the problem of missed detection and false alarm in the detection of running behavior in complex monitoring scenarios.

[0004] In order to achieve the above objectives, the technical solutions provided by this application include:

[0005] In a first aspect, a behavior detection method is provided, the method comprising: extracting a plurality of image frames from a surveillance video in chronological order; inputting the plurality of image frames into a pedestrian detection model, detecting by the pedestrian detection model whether a pedestrian exists in each of the plurality of image frames, and generating a pedestrian bounding box in the image frame where a pedestrian exists; when a pedestrian exists in at least two of the plurality of image frames, determining a displacement pixel distance of the pedestrian in adjacent image frames by a preset target tracking algorithm; when the displacement pixel distance is greater than or equal to a preset threshold value corresponding to the size of the pedestrian bounding box, inputting an image region of the plurality of image frames where the pedestrian bounding box exists into a posture detection model, and obtaining a detection result output by the posture detection model that characterizes whether the pedestrian has a running behavior.

[0006] In the above-mentioned implementation mode, a pedestrian detection model, a target tracking algorithm and a posture detection model are combined to detect image frames in the surveillance video. After the pedestrian detection model determines that there are pedestrians in the image frames, the target tracking algorithm is used to preliminarily determine whether the pedestrian is running, and the image area where the running behavior exists is input into the posture detection model. The posture detection model further determines whether the pedestrian is running. In this way, the accuracy and reliability of the detection can be improved.

[0007] In combination with the first aspect, in some optional implementations, extracting multiple image frames from the surveillance video in chronological order includes: sequentially extracting the multiple image frames from the surveillance video at intervals of a specified number of frames.

[0008] In the above-mentioned implementation, extracting image frames at intervals is beneficial to reducing the number of images to be detected and alleviating processing pressure.

[0009] In combination with the first aspect, in some optional embodiments, the pedestrian detection model includes a backbone network model and a target detection model, and the multiple image frames are image frames after normalizing the pixel values ​​of each image frame; the multiple image frames are input into the pedestrian detection model, and the pedestrian detection model detects whether there are pedestrians in each of the multiple image frames, and generates pedestrian bounding boxes in the image frames where pedestrians exist, including: inputting the normalized image frames into the backbone network model, and the backbone network model extracts features from the normalized image frames to obtain a depth feature map; inputting the depth feature map into the target detection model, and the target detection model determines whether there are pedestrians in the multiple image frames through the depth feature map; when there are pedestrians in the multiple image frames, generating pedestrian bounding boxes based on the center points and sizes of the pedestrians in the multiple image frames.

[0010] In combination with the first aspect, in some optional embodiments, the displacement pixel distance of pedestrians in adjacent image frames is determined by a preset target tracking algorithm, including: determining the motion trajectory of each pedestrian by the preset target tracking algorithm and the pedestrian bounding boxes in the multiple image frames; determining the displacement pixel distance of each pedestrian in adjacent image frames based on the motion trajectory, wherein the adjacent image frames are two image frames that are adjacent in position in the multiple images and both have pedestrians.

[0011] In combination with the first aspect, in some optional embodiments, the image area in the multiple image frames where the pedestrian border exists is input into a posture detection model, and a detection result output by the posture detection model that characterizes whether the pedestrian is running is obtained, including: inputting the image area in the multiple image frames where the pedestrian border exists into the posture detection model; when the posture detection model detects that the same pedestrian is identified as running in more than a specified number of consecutive image frames, the posture detection model outputs a detection result that characterizes the running behavior of the pedestrian in the time period corresponding to the multiple image frames.

[0012] In combination with the first aspect, in some optional implementations, the method further includes: when the detection result indicates that there is pedestrian running behavior in the multiple image frames, extracting a video clip corresponding to the running behavior from the surveillance video for storage.

[0013] In the above-mentioned implementation, by storing the extracted video clips with running behavior in chronological order, it is convenient to directly perform a traceability query later to quickly find the video clips with running behavior.

[0014] In combination with the first aspect, in some optional implementations, the pedestrian detection model is a trained and tested CenterNet network model, and the posture detection model is a trained and tested ResNet18 network model.

[0015] In the second aspect, the present application also provides a behavior detection device, which includes: an extraction unit, which is used to extract multiple image frames from a surveillance video in chronological order; a first detection unit, which is used to input the multiple image frames into a pedestrian detection model, and the pedestrian detection model detects whether there is a pedestrian in each of the multiple image frames, and generates a pedestrian border in the image frame where a pedestrian exists; a determination unit, which is used to determine the displacement pixel distance of the pedestrian in adjacent image frames by a preset target tracking algorithm when there are pedestrians in at least two of the multiple image frames; and a second detection unit, which is used to input the image area where the pedestrian border exists in the multiple image frames into a posture detection model when the displacement pixel distance is greater than or equal to a preset threshold corresponding to the size of the pedestrian border, to obtain a detection result output by the posture detection model that characterizes whether the pedestrian has a running behavior.

[0016] In a third aspect, the present application also provides an electronic device, comprising a processor and a memory coupled to each other, wherein the memory stores a computer program, and when the computer program is executed by the processor, the electronic device executes the above-mentioned behavior detection method.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed on a computer, the computer executes the above-mentioned behavior detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a behavior detection method provided in an embodiment of the present application.

[0020] Figure 2 A schematic diagram of the network structure of the CenterNet network model provided in an embodiment of the present application.

[0021] Figure 3 A schematic diagram of the network structure of the ResNet18 network model provided in an embodiment of the present application.

[0022] Figure 4 A block diagram of a behavior detection device provided in an embodiment of the present application.

[0023] Icon: 200 - behavior detection device; 210 - extraction unit; 220 - first detection unit; 230 - determination unit; 240 - second detection unit. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. It should be noted that the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0025] The present application provides an electronic device that can be used to detect whether there are pedestrians in a surveillance video, and when there are pedestrians, detect whether the pedestrians in the surveillance video are running, which can improve the accuracy and reliability of detection.

[0026] The electronic device may include a processing module and a storage module. The storage module stores a computer program, and when the computer program is executed by the processing module, the electronic device can perform each step in the following behavior detection method.

[0027] The electronic device may be, but is not limited to, a personal computer, a server, and other devices, which are not specifically limited here.

[0028] Please refer to Figure 1 The present application also provides a behavior detection method, which can be applied to the above electronic device, and each step of the method is executed or implemented by the electronic device. The method may include the following steps:

[0029] Step 110, extracting a plurality of image frames from the surveillance video in time sequence;

[0030] Step 120, inputting the multiple image frames into a pedestrian detection model, and the pedestrian detection model detects whether there is a pedestrian in each of the multiple image frames, and generates a pedestrian bounding box in the image frame where a pedestrian exists;

[0031] Step 130, when there is a pedestrian in at least two of the multiple image frames, determining the displacement pixel distance of the pedestrian in adjacent image frames by using a preset target tracking algorithm;

[0032] Step 140, when the displacement pixel distance is greater than or equal to a preset threshold corresponding to the size of the pedestrian border, the image area containing the pedestrian border in the multiple image frames is input into a posture detection model to obtain a detection result output by the posture detection model indicating whether the pedestrian is running.

[0033] In the above-mentioned implementation mode, a pedestrian detection model, a target tracking algorithm and a posture detection model are combined to detect image frames in the surveillance video. After the pedestrian detection model determines that there are pedestrians in the image frames, the target tracking algorithm is used to preliminarily determine whether the pedestrian is running, and the image area where the running behavior exists is input into the posture detection model. The posture detection model further determines whether the pedestrian is running. In this way, the accuracy and reliability of the detection can be improved.

[0034] The following is a detailed description of each step of the method:

[0035] In step 110, the surveillance video used to extract the image frame may be a surveillance video shot in real time, or a surveillance video stored historically. The real-time surveillance video shot by the surveillance camera may be transmitted to the electronic device for the electronic device to extract the image frame. For example, the electronic device may extract multiple image frames from the surveillance video shot in real time or the surveillance video stored historically in chronological order.

[0036] In this embodiment, extracting multiple image frames in time sequence may mean that the electronic device extracts image frames one by one or extracts image frames at intervals according to the time sequence of image frames in the surveillance video to obtain multiple image frames.

[0037] For example, step 110 may include: sequentially extracting the plurality of image frames from the surveillance video at intervals of a specified number of frames, wherein the specified number of frames of the interval may be flexibly determined according to actual conditions, for example, the specified number of frames may be 4, 5, 6, etc. frames.

[0038] In step 120, the pedestrian detection model is a trained neural network model that can be used to detect whether there are pedestrians in an image. For example, the pedestrian detection model is a trained and tested CenterNet network model.

[0039] Exemplarily, the CenterNet network model includes a backbone network model and a target detection model, and the multiple image frames are image frames after normalizing the pixel values ​​of each image frame. Step 120 may include:

[0040] Inputting the normalized image frame into the backbone network model, and extracting features from the normalized image frame by the backbone network model to obtain a depth feature map;

[0041] Inputting the depth feature map into the target detection model, and the target detection model determines whether there is a pedestrian in the multiple image frames through the depth feature map;

[0042] When there are pedestrians in the multiple image frames, the target detection model generates a bounding box of the pedestrian according to the center point and size of the pedestrian in the multiple image frames.

[0043] In this embodiment, the normalized image frame refers to: adjusting the image frame of the surveillance video to an image frame of a specified size, and then dividing all pixel values ​​of the resized image frame by 255 (normalizing the pixel values ​​of the image frame from a range of 0-255 to a range of 0-1), thereby obtaining the normalized image frame. The specified size can be determined according to actual conditions, for example, it can be an image of 640×640 pixels.

[0044] The backbone network model may be a ResNet18-FPN network model, which is used to extract deep features of an image and obtain a deep feature map.

[0045] The object detection module includes two branch modules. The first branch module is used to predict the center point position of the target object (such as a pedestrian) based on the deep feature map of the image frame extracted by the backbone network model. The second branch module is used to predict the size of the target object (such as a pedestrian) based on the deep feature map and generate the pedestrian bounding box based on the size of the pedestrian.

[0046] In step 130, the displacement pixel distance of the pedestrian in adjacent image frames is determined by a preset target tracking algorithm, including:

[0047] Determine the motion trajectory of each pedestrian by using the preset target tracking algorithm and the pedestrian bounding boxes in the multiple image frames;

[0048] The displacement pixel distance of each pedestrian in adjacent image frames is determined according to the motion trajectory, wherein the adjacent image frames are two image frames in the multiple images that are adjacent in position and both have pedestrians.

[0049] The preset target tracking algorithm may be a SORT (Simple Online And Realtime Tracking) algorithm based on Multi Object Tracking (MOT), which may be used to detect the motion trajectory of each pedestrian in the surveillance video. In the surveillance video, the motion trajectory may be determined based on the position of the center point of the pedestrian on the image frame at different times, and each pedestrian corresponds to a motion trajectory.

[0050] When determining the displacement pixel distance, the position coordinates of the pedestrian center point on the monitoring video display screen can be calculated in the same coordinate system, and then based on the position coordinates of the same pedestrian on the image frame at different times, the corresponding displacement pixel distances at different times can be calculated.

[0051] Generally speaking, pedestrians run faster than they walk, and the displacement pixel distance corresponding to pedestrian running is larger. In the surveillance video, the closer the same pedestrian is to the surveillance camera, the larger the size of the pedestrian is. In order to preliminarily filter out image frames with non-running behavior, if the displacement pixel distance is less than the preset threshold corresponding to the size of the pedestrian's border, this indicates that the pedestrian is not running, and there is no need to perform running detection on the pedestrian. If the displacement pixel distance is greater than or equal to the preset threshold corresponding to the size of the pedestrian's border, the image area with the pedestrian border in multiple image frames is input into the posture detection model, and the posture detection model performs running behavior detection on the pedestrian.

[0052] The preset threshold corresponding to the size of the pedestrian border is usually a small value, which can be flexibly determined according to the actual situation, to ensure that when the distance between the pedestrian and the surveillance camera changes, the relationship between the displacement pixel distance and the preset threshold can still be used to preliminarily determine whether the pedestrian may be running. For example, if the displacement pixel distance is greater than or equal to the preset threshold, it is determined that the pedestrian may be running; if the displacement pixel distance is less than the preset threshold, it is determined that the pedestrian is not running.

[0053] In step 140, when the displacement pixel distance is greater than or equal to a preset threshold value corresponding to the size of the pedestrian frame, it indicates that the pedestrian in the multiple image frames may be running, and the image area with the pedestrian frame in the multiple image frames needs to be input into the posture detection model for further detection. Among them, only the image area of ​​the pedestrian frame is input into the posture detection model, which is conducive to the posture detection model to detect the posture of the pedestrian, without the need to detect other non-pedestrian areas in the image, which is conducive to improving the efficiency and accuracy of the detection.

[0054] In this embodiment, step 140 may include:

[0055] Inputting the image area containing the pedestrian border in the plurality of image frames into a posture detection model;

[0056] When the posture detection model detects that the same pedestrian is identified as running in more than a specified number of consecutive image frames, the posture detection model outputs a detection result indicating that the pedestrian is running in the time period corresponding to the multiple image frames.

[0057] In this embodiment, the posture detection model can detect the posture of the pedestrian in each image frame to determine whether the pedestrian in the image frame is in a running posture. If multiple consecutive image frames are all identified as running behaviors, it is determined that the pedestrian is in a running behavior during the period of time. The specified number can be flexibly determined according to actual conditions, and can be 3, 4, 5, etc., which is conducive to improving the accuracy of running behavior detection.

[0058] In this embodiment, the method may further include: when the detection result indicates that there is a running behavior of a pedestrian in the multiple image frames, extracting a video clip corresponding to the running behavior from the surveillance video for storage.

[0059] In this embodiment, the electronic device can extract video clips with running behavior detected in the surveillance video, and then store the extracted video clips with running behavior in chronological order. Each video clip can be used as a video file to facilitate subsequent direct traceability query.

[0060] The video clips may be stored locally on the electronic device or in a cloud server, and the manner of storing the video clips is not specifically limited here.

[0061] In this embodiment, the electronic device can simultaneously perform pedestrian detection and pedestrian running behavior detection on multiple surveillance videos. The detection method for each surveillance video can refer to the above steps 110 to 140, which will not be repeated here.

[0062] To facilitate understanding of the implementation process of the behavior detection method, the following example will illustrate the implementation process of the method:

[0063] For example, the pedestrian detection model is a CenterNet network model, and the backbone network model in the CenterNet network model adopts a ResNet18-FPN network model. The basic network structure of ResNet18 can be referenced Figure 2 Among them, in the backbone network model, the network structure behind C4 can be omitted.

[0064] For example, in the backbone network model, given an input image I input ∈R H×W×3 , R H×W×3 The representation of the input image is H×W×3, where H and W refer to the height and width of the image frame respectively, and the number "3" represents the number of channels of the image. First, after the calculation of the ResNet18 basic network, three feature maps of different scales are output. Then, through the processing of the Feature Pyramid Net (FPN) network structure, the three feature map information of C2, C3, and C4 are fused to output the deep feature map of the image.

[0065] In the target detection module, the first branch module is used to predict the center point position of the target object, including two convolution layers (Conv1_1 convolution kernel size is 3×3, output channel is 256; Conv1_2 convolution kernel size is 1×1, output channel is K, K represents the total number of target categories) and 1 Sigmoid function layer (to ensure that the output value range is 0 to 1); the second branch module is used to predict the size of the target, including two convolution layers (Conv2_1 convolution kernel size is 3×3, output channel is 256; Conv2_2 convolution kernel size is 1×1, output channel is 2, corresponding to the width and height of the target respectively) and 1 Exp function layer (to ensure that the output value is positive). Input the depth feature map P2, after calculation by the target detection module, the target center point feature map is output and target size feature map Remember F hm Each element in is ρ x，y，k , F size Each element in is S x，y，i ,in k={0, 1,..., K-1}, i={0, 1}. Then ρ x，y，k It represents the probability that the position point (x, y) is predicted to be the center point of the k-th target, S x，y，0 and S x，y，1 They represent the length and width of the target corresponding to the position point (x, y) respectively.

[0066] When performing pedestrian detection, the electronic device can decode the surveillance video stream data into a surveillance video, and extract an image frame from the surveillance video every fixed frame number 5. The extracted multiple image frames form an image sequence and serve as the image input to the CenterNet network model. When the CenterNet network model performs pedestrian detection, the length W and width H of the input image are both set to 640, and the total number of categories K is set to 1, that is, the target only includes one category of pedestrians.

[0067] The image frame extracted from the surveillance video is used as the original image First, the original image is resized to a 640×640 image, and then all pixel values ​​of the resized image are divided by 255 (normalizing the pixel values ​​from the range of 0-255 to the range of 0-1) to obtain the normalized image I input ∈R 640×640×3 .

[0068] Then, the preprocessed image I input Input into the above CenterNet network model, and output the pedestrian center point feature map F hm and pedestrian size feature map F size , then for F hm and F size Post-processing is performed to extract the border positions of pedestrians in the image. The specific processing process is as follows:

[0069] Step A, Extraction F hm The peak point in F hm Perform a maximum pooling operation (with parameters kernel=3×3, stride=1×1) to obtain a feature map F' of the same size hm ; Then compare F hm and F' hm Each element value in the two feature maps records the subscript position of the elements whose element values ​​are not equal; finally, F hm The element values ​​corresponding to these marked positions are set to 0, F hm The non-zero elements are the peak points that need to be extracted.

[0070] Step B, Extraction of F hm The subscripts and values ​​of all elements whose values ​​are greater than 0.5 are denoted as (x, y) and ρ respectively. x，y Each extracted element corresponds to the center point of a detected pedestrian, and the set of all detected pedestrian center points is recorded as Where (x i ,y i ) represents the coordinates of the center point corresponding to the i-th pedestrian on the feature map, represents the confidence of the i-th pedestrian, and M represents the total number of pedestrians detected.

[0071] Step C: based on the pedestrian center point coordinate set extracted in step B From F size Extract the element value at the corresponding position point, recorded as in Represents the width and height of the i-th pedestrian. The element value can be understood as the value of the original pixel on the feature map after the above corresponding processing (such as normalization).

[0072] Step D: According to the pedestrian center point and pedestrian width and height extracted in steps B and C, generate the pedestrian border on the feature map, and map the pedestrian border to the input image to obtain the pedestrian border set on the input image, which is recorded as in, and Respectively represent the coordinates of the upper left corner and lower right corner of the i-th pedestrian in the input image, and then measure the image I t The set of pedestrian bounding boxes P in t .

[0073] Then, the multi-target tracking SORT algorithm is used to convert the pedestrian bounding box set P detected at the current moment into t The pedestrian bounding box set P detected at the last moment t-1 Matching and association are performed to further obtain the motion trajectory of each pedestrian in the image.

[0074] Then, according to the motion trajectory of each pedestrian, the displacement pixel of the pedestrian from the previous moment to the current moment is calculated. and Respectively represent the coordinates of the upper left corner of the pedestrian in the image at the previous moment and the current moment. t If the value is greater than the set threshold, the posture detection model is further used to identify the pedestrian’s posture S at the current moment. t , marking the pedestrian’s posture at the current moment as running; otherwise marking the pedestrian’s posture at the current moment as non-running.

[0075] The posture detection model uses the ResNet18 network model. The structure and parameters of the model are as follows Figure 3 As shown in the figure, there are a total of 18 convolutional layers, 1 global average pooling layer and 1 Softmax function layer (to ensure that the output value range is 0 to 1). When using the ResNet18 network model for image classification and recognition, the input image I input ∈R HxW×3(W, H, and 3 represent the length, width, and number of channels of the image, respectively). After a series of neural network operations (18 convolutional layers, 1 global average pooling layer, and 1 Softmax function layer), the probability vector ρ∈R representing the category to which the image belongs can be output. 1×K (K represents the total number of categories). The length W and width H of the input image are set to 128 and 192 respectively, and the total number of categories K is set to 2, that is, people have two postures: running and non-running. When performing running posture recognition, the implementation process can be as follows:

[0076] Step 1: Image preprocessing. Given the original image First, according to the detected pedestrian bounding box (x 1 ,y 1 , x 2 ,y 2 )((x 1 ,y 1 ) and (x 2 ,y 2 ) represent the pedestrian in image I s The human body image is cropped from the original image using the coordinates of the upper left corner and the lower right corner; the cropped human body image is then adjusted to an image of 192×128 size; finally, all pixel values ​​of the adjusted image are divided by 255 (the pixel values ​​are normalized from the range of 0-255 to the range of 0-1) to obtain the normalized human body image I person ∈R 192×128×3 .

[0077] Step 2: Network inference. The preprocessed human image I person Input it into the above ResNet18 network model and calculate the probability vector ρ = [ρ bp , ρ unbp ],ρ bp and ρ unbp Represent the probability of a person being in a running and non-running posture respectively. bp >ρ unbp , it means that the person is currently in a running posture, otherwise it is a non-running posture.

[0078] In order to improve the accuracy of running behavior detection, the detection results of multiple frames are used to finally determine whether the person being tested has running behavior. That is, if a pedestrian has a running behavior at three consecutive moments [S t-2 , S t-1 , S t ] are all identified as running, it means that the person is running at the current moment, otherwise not.

[0079] Based on the above design, the behavior detection method can realize the automatic detection of running behavior of people in the park video surveillance system, reduce labor costs, and improve the monitoring efficiency and automation of the supervision system. In addition, in the behavior detection method, the pedestrian detection model, the preset target tracking algorithm and the posture detection model are combined, which has good robustness and can maintain high detection and recognition accuracy and real-time performance in various complex scenes such as blur and noise, and the method is easy to promote and apply.

[0080] Please refer to Figure 4 , the embodiment of the present application also provides a behavior detection device 200, which can be applied to the above-mentioned electronic device to execute each step in the method. The behavior detection device 200 includes at least one software function module that can be stored in the storage module in the form of software or firmware (Firmware) or solidified in the operating system (OS) of the electronic device. The processing module is used to execute the executable module stored in the storage module, such as the software function module and computer program included in the behavior detection device 200.

[0081] The behavior detection device 200 may include an extraction unit 210, a first detection unit 220, a determination unit 230, and a second detection unit 240. The functions of each unit may be as follows:

[0082] An extraction unit 210, configured to extract a plurality of image frames from the surveillance video in time sequence;

[0083] A first detection unit 220, configured to input the plurality of image frames into a pedestrian detection model, and to detect by the pedestrian detection model whether a pedestrian exists in each of the plurality of image frames, and to generate a pedestrian bounding box in an image frame where a pedestrian exists;

[0084] A determination unit 230, configured to determine a displacement pixel distance of the pedestrian in adjacent image frames by using a preset target tracking algorithm when there is a pedestrian in at least two of the multiple image frames;

[0085] The second detection unit 240 is used to input the image area containing the pedestrian border in the multiple image frames into the posture detection model when the displacement pixel distance is greater than or equal to a preset threshold corresponding to the size of the pedestrian border, and obtain a detection result output by the posture detection model that characterizes whether the pedestrian is running.

[0086] Optionally, the extraction unit 210 is used to sequentially extract the multiple image frames at intervals of a specified number of frames from the surveillance video.

[0087] Optionally, the pedestrian detection model includes a backbone network model and a target detection model, and the multiple image frames are image frames obtained by normalizing the pixel values ​​of each image frame. The first detection unit 220 is also used to: input the normalized image frames into the backbone network model, and the backbone network model performs feature extraction on the normalized image frames to obtain a depth feature map; input the depth feature map into the target detection model, and the target detection model determines whether there are pedestrians in the multiple image frames through the depth feature map; when there are pedestrians in the multiple image frames, the target detection model generates a border of the pedestrian based on the center point and size of the pedestrian in the multiple image frames.

[0088] Optionally, the determination unit 230 is further used to: determine the motion trajectory of each pedestrian through the preset target tracking algorithm and the pedestrian bounding boxes in the multiple image frames; and determine the displacement pixel distance of each pedestrian in adjacent image frames based on the motion trajectory, wherein the adjacent image frames are two image frames that are adjacent in position in the multiple images and both have pedestrians.

[0089] Optionally, the second detection unit 240 is also used to: input the image area where the pedestrian border exists in the multiple image frames into a posture detection model; when the posture detection model detects that the same pedestrian is identified as running in more than a specified number of consecutive image frames, the posture detection model outputs a detection result representing that the pedestrian is running in the time period corresponding to the multiple image frames.

[0090] Optionally, the behavior detection device 200 may further include a storage unit for extracting and storing a video clip corresponding to the running behavior from the surveillance video when the detection result indicates that there is a running behavior of a pedestrian in the multiple image frames.

[0091] In this embodiment, the processing module can be an integrated circuit chip with signal processing capabilities. The above-mentioned processing module can be a general-purpose processor. For example, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of this application.

[0092] The storage module may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the storage module may be used to store images to be detected, etc. Of course, the storage module may also be used to store a program, and the processing module executes the program after receiving the execution instruction.

[0093] The communication module is used to establish a communication connection between the electronic device and other devices through the network, and to send and receive data through the network.

[0094] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process of each step in the aforementioned method, and will not be elaborated here.

[0095] The embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is run on a computer, the computer executes the behavior detection method described in the above embodiment.

[0096] Through the description of the above implementation methods, technical personnel in this field can clearly understand that the present application can be implemented by hardware, and can also be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0097] To summarize, in this scheme, a pedestrian detection model, a target tracking algorithm and a posture detection model are combined to detect image frames in surveillance videos. After the pedestrian detection model determines that there are pedestrians in the image frames, the target tracking algorithm is used to preliminarily determine whether the pedestrian is running, and the image area with running behavior is input into the posture detection model, which further determines whether the pedestrian is running. In this way, the accuracy and reliability of detection can be improved.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed device, system and method can also be implemented in other ways. The above-described device, system and method embodiments are merely schematic, for example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0099] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A behavior detection method, characterized in that: The method comprises: Extracting multiple image frames from the surveillance video in time sequence; Inputting the multiple image frames into a pedestrian detection model, the pedestrian detection model detecting whether there is a pedestrian in each of the multiple image frames, and generating a pedestrian bounding box in the image frame where a pedestrian exists; When there is a pedestrian in at least two of the multiple image frames, determining the displacement pixel distance of the pedestrian in adjacent image frames by a preset target tracking algorithm; When the displacement pixel distance is greater than or equal to a preset threshold value corresponding to the size of the pedestrian frame, the image region in the plurality of image frames where the pedestrian frame exists is input into a posture detection model, and a detection result output by the posture detection model indicating whether the pedestrian is running is obtained; The pedestrian detection model includes a backbone network model and a target detection model, and the multiple image frames are image frames obtained by normalizing the pixel values ​​of each image frame; The method includes inputting the plurality of image frames into a pedestrian detection model, detecting by the pedestrian detection model whether a pedestrian exists in each of the plurality of image frames, and generating a pedestrian bounding box in the image frame where a pedestrian exists, including: Inputting the normalized image frame into the backbone network model, and performing feature extraction on the normalized image frame by the backbone network model to obtain a depth feature map; Inputting the depth feature map into the target detection model, and the target detection model determines whether there is a pedestrian in the multiple image frames through the depth feature map; When there are pedestrians in the multiple image frames, the object detection model generates a bounding box of the pedestrian according to the center point and size of the pedestrian in the multiple image frames.

2. The method according to claim 1, characterized in that Extract multiple image frames from the surveillance video in time sequence, including: The multiple image frames are sequentially extracted from the surveillance video at intervals of a specified number of frames.

3. The method according to claim 1, characterized in that The displacement pixel distance of pedestrians in adjacent image frames is determined by a preset target tracking algorithm, including: Determine the motion trajectory of each pedestrian by using the preset target tracking algorithm and the pedestrian bounding boxes in the multiple image frames; The displacement pixel distance of each pedestrian in adjacent image frames is determined according to the motion trajectory, wherein the adjacent image frames are two image frames in the multiple images that are adjacent in position and both have pedestrians.

4. The method according to claim 1, characterized in that Inputting the image region containing the pedestrian border in the plurality of image frames into a posture detection model, and obtaining a detection result output by the posture detection model indicating whether the pedestrian is running, including: Inputting the image area containing the pedestrian border in the plurality of image frames into a posture detection model; When the posture detection model detects that the same pedestrian is identified as running in more than a specified number of consecutive image frames, the posture detection model outputs a detection result indicating that the pedestrian is running in the time period corresponding to the multiple image frames.

5. The method according to claim 1, characterized in that The method further comprises: When the detection result indicates that there is a running behavior of a pedestrian in the multiple image frames, a video segment corresponding to the running behavior is extracted from the monitoring video and stored.

6. The method according to claim 1, characterized in that The pedestrian detection model is a trained and tested CenterNet network model, and the posture detection model is a trained and tested ResNet18 network model.

7. A behavior detection device, characterized in that: The device comprises: An extraction unit, used to extract multiple image frames from the surveillance video in time sequence; A first detection unit, configured to input the plurality of image frames into a pedestrian detection model, and to detect by the pedestrian detection model whether a pedestrian exists in each of the plurality of image frames, and to generate a pedestrian bounding box in the image frame where a pedestrian exists; a determining unit, configured to determine, when a pedestrian exists in at least two of the plurality of image frames, a displacement pixel distance of the pedestrian in adjacent image frames by using a preset target tracking algorithm; a second detection unit, configured to input, when the displacement pixel distance is greater than or equal to a preset threshold value corresponding to the size of the pedestrian frame, an image region in the plurality of image frames where the pedestrian frame exists into a posture detection model, and obtain a detection result output by the posture detection model indicating whether the pedestrian has a running behavior; The pedestrian detection model includes a backbone network model and a target detection model, and the multiple image frames are image frames obtained by normalizing the pixel values ​​of each image frame; The first detection unit is specifically used to: input the normalized image frame into the backbone network model, and the backbone network model performs feature extraction on the normalized image frame to obtain a depth feature map; input the depth feature map into the target detection model, and the target detection model determines whether there are pedestrians in the multiple image frames through the depth feature map; when there are pedestrians in the multiple image frames, the target detection model generates the pedestrian's bounding box according to the center point and size of the pedestrian in the multiple image frames.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory coupled to each other, wherein the memory stores a computer program. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images