Pedestrian action detection method and device based on monitoring video

Abstract

The invention relates to the technical field of image processing, and discloses a pedestrian action detection method and device based on a monitoring video, and the method comprises the steps: obtaining a to-be-detected scene data set which comprises at least one video clip, and each video clip comprises at least one frame of image; and detecting each pedestrian in the video clip by using a preset pedestrian action detection model, and generating the position and action category of each pedestrian. According to the method provided by the invention, the position and the action category of the pedestrian can be accurately detected by presetting the pedestrian action detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for detecting pedestrian movements based on surveillance videos. Background Art

[0002] In recent years, with the continuous acceleration of urbanization, the number of vehicles has gradually increased. However, due to the serious shortage of parking spaces, problems such as difficulty in parking and traffic congestion have arisen. In order to alleviate this problem, relevant departments have proposed some solutions, such as setting up parking spaces on the roadside to alleviate the problem of insufficient parking spaces. At the same time, in order to standardize the management of roadside parking, high-position video technology has developed. By installing high-position video cameras on the roadside, vehicle parking can be judged and managed, which has a positive effect on urban traffic management.

[0003] At present, based on the existing high-position video surveillance equipment, the obtained monitoring data and the current visual processing algorithms, it is not only possible to realize the charging and management of roadside parking spaces, but also to identify vehicle violations, duplicate license plate vehicles, and restricted vehicles, etc., which can more effectively use high-position video technology to promote the sound development of urban transportation. In addition, based on the above equipment, data and algorithm technology, it is also possible to detect and identify the actions of pedestrians on the roadside, and timely detect and identify pedestrian actions that may have dangerous behaviors, disrupt public security, and suddenly fall, so as to help intelligent traffic management and public security. Therefore, when detecting pedestrian actions, two-dimensional images are mainly used for action prediction, and there is a problem that the timing information in the video is not used. Because action recognition relies on single-frame images, action ambiguity is prone to occur. Summary of the invention

[0004] In view of this, the present invention provides a method and device for detecting pedestrian motion based on surveillance video, so as to solve the problem of motion ambiguity caused by relying on single-frame images for motion recognition when detecting pedestrian motion.

[0005] In a first aspect, the present invention provides a method for detecting pedestrian movements based on surveillance video, the method comprising:

[0006] Acquire a scene data set to be detected, wherein the data set includes at least one video clip, and each video clip includes at least one frame of image;

[0007] The preset pedestrian action detection model is used to detect each pedestrian in the video clip and generate the position and action category of each pedestrian.

[0008] In an optional implementation, the preset pedestrian motion detection model includes: a preset dual-branch feature extraction part, a preset feature decoupling part, a preset feature fusion part, and a preset pedestrian motion detection part.

[0009] In an optional implementation, the preset dual-branch feature extraction part includes: a two-dimensional backbone network and a three-dimensional backbone network, wherein:

[0010] The 2D backbone network is used to model the spatial dimension;

[0011] The 3D backbone network is used to extract spatiotemporal feature information.

[0012] In an optional implementation, the preset feature decoupling part is used to decouple features of different scales output by the two-dimensional backbone network, and separate feature images of different scales into regression and classification branches.

[0013] In an optional embodiment, the preset feature fusion part is used to perform feature fusion on the two-dimensional branch and the three-dimensional branch, and to splice the two-dimensional and three-dimensional feature maps of the same scale in the channel dimension, wherein the two-dimensional branch is the output of the two-dimensional backbone network, and the three-dimensional branch is the output of the three-dimensional backbone network.

[0014] In an optional embodiment, the preset pedestrian action detection part includes: performing regression and classification at different scales according to the fusion features of two-dimensional feature maps and three-dimensional feature maps of different scales, completing the regression of pedestrian bounding boxes and the classification of pedestrian actions, and generating the position and action category of each pedestrian in the video clip.

[0015] In an optional embodiment, when a first preset condition is met, the regression of the pedestrian's bounding box and the classification of the pedestrian's actions are completed, wherein the first preset condition includes: the number of classifications of the pedestrian's bounding box and pedestrian action in the preset pedestrian action detection part reaches a preset value, and / or the classification loss function of the pedestrian action classification meets the second preset condition, and / or the regression loss function of the pedestrian detection box meets the third preset condition.

[0016] The method and device for detecting pedestrian actions based on surveillance video provided by the present invention obtain a scene data set to be detected, the data set includes at least one video clip, and each video clip includes at least one frame of image; each pedestrian in the video clip is detected using a preset pedestrian action detection model to generate the position and action category of each pedestrian. The method provided by the present invention can accurately detect the position and action category of pedestrians by presetting the pedestrian action detection model.

[0017] In a second aspect, the present invention provides a pedestrian motion detection device based on surveillance video, the device comprising:

[0018] A data collection and preprocessing module is used to collect the data set to be processed, preprocess the data set to be processed, and store the preprocessing results in a preset time series medical knowledge base;

[0019] A graph community construction module, used to construct a graph community of a preset medical knowledge base according to a preset time-series medical knowledge base;

[0020] Hidden Markov model application module, used to perform time series inference at different time points on the graph community of the preset medical knowledge base based on the hidden Markov model;

[0021] An emission probability matrix construction module is used to construct an emission probability matrix by creating a preset retrieval system and setting an emission probability model;

[0022] The article writing module is used to select article fragments according to the emission probability matrix, write and modify the article according to the preset model, and perform manual review and revision to complete the writing of the medical article.

[0023] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the pedestrian motion detection method based on surveillance video of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the pedestrian motion detection method based on surveillance video of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 is a flow chart of a method for detecting pedestrian movements based on surveillance video according to an embodiment of the present invention;

[0027] Figure 2 is a structural block diagram of a pedestrian motion detection device based on surveillance video according to an embodiment of the present invention;

[0028] Figure 3 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] According to an embodiment of the present invention, an embodiment of a pedestrian motion detection method based on surveillance video is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] In this embodiment, a method for detecting pedestrian movements based on surveillance video is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a method for detecting pedestrian movements based on surveillance video according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0032] Step S101 : obtaining a scene data set to be detected, wherein the data set includes at least one video clip, and each video clip includes at least one frame of image.

[0033] In an embodiment of the present invention, the scene data set to be detected is captured by high-position video surveillance, and the roadside scene data set based on high-position video surveillance includes at least one video segment, and each video segment includes at least one frame of image. This is only an example and is not limited to this. In actual applications, the corresponding number of video segments and image frames are selected according to actual conditions.

[0034] Specifically, the selection of video clips includes different action categories of different pedestrians at different times and places, enriching the diversity of video clips. The actions include: walking, running, jumping, squatting, waving, jumping while waving, hopping on one foot, walking sideways, etc. There is no restriction here, and the corresponding actions are selected according to the actual situation.

[0035] Specifically, the two-dimensional target detection frame of each pedestrian in each image in each video clip and the action category of the pedestrian are annotated respectively. The two-dimensional target detection frame annotation of the pedestrian refers to the coordinates of the upper left corner and the lower right corner of the two-dimensional rectangular box describing the pedestrian.

[0036] Step S102: Detect each pedestrian in the video clip using a preset pedestrian action detection model to generate a position and action category of each pedestrian.

[0037] In an embodiment of the present invention, the preset pedestrian motion detection model includes: a preset dual-branch feature extraction part, a preset feature decoupling part, a preset feature fusion part, and a preset pedestrian motion detection part.

[0038] In the embodiment of the present invention, a dual-branch feature extraction part is preset, including two modules, namely a two-dimensional backbone network for spatial dimension modeling and a three-dimensional backbone network for spatiotemporal feature extraction.

[0039] Specifically, the two-dimensional backbone network used for spatial dimension modeling takes as input the last frame image of a video clip, and the input image size is C*H*W, where H and W represent the height and width of the image respectively, and C represents the number of channels. In this case, C is 3, indicating an RGB three-channel image.

[0040] Specifically, the two-dimensional backbone network adopts a network architecture based on YOLO, including a series of operations such as convolution, normalization, and activation functions. This is only an example and is not limited to this. In practical applications, the corresponding operations are selected according to the actual situation. The pyramid network structure is used in the YOLO network structure to make full use of multi-scale information. Therefore, the output includes three parts, which are outputs of different scales, specifically expressed as:

[0041] F lv1 :

[0042] F lv2 :

[0043] F lv3 :

[0044] Specifically, the three-dimensional backbone network is used to extract spatiotemporal feature information. The input is a video clip with a size of T*C*H*W, where H and W represent the height and width of the image respectively, C represents the number of channels, and C is 3 at this time, indicating an RGB three-channel image, and T represents the number of frames of the video clip.

[0045] Specifically, the three-dimensional backbone network adopts a commonly used convolutional neural network, including but not limited to convolutional neural networks using structures such as ResNet, Mobile Net, and VGG, and the output is an image feature representation of each video frame in the video clip, and the output is represented by F 3D :

[0046] In an embodiment of the present invention, the preset feature decoupling part refers to feature decoupling of features of different scales output by the two-dimensional backbone network. Usually, the same feature map is used for subsequent regression and classification tasks, but this will affect the training accuracy of the regression and classification branches. Therefore, feature decoupling is used to separate feature maps of different scales and send them to the regression and classification branches.

[0047] In the embodiment of the present invention, the preset feature fusion part is to fuse the features of the two-dimensional and three-dimensional branches, and the input is expressed as:

[0048] F lv1 :

[0049] F lv2 :

[0050] F lv3 :

[0051] F 3D :

[0052] Specifically, for the three-dimensional branch, the dimension is first compressed by using the squeeze function to align it with the input dimension of the two-dimensional branch; it is represented by F 3D : Then, the three-dimensional branch F 3D Perform upsampling operations and align them with feature maps of different scales of the two-dimensional branch respectively; finally, concatenate the two-dimensional and three-dimensional feature maps of the same scale in the channel dimension, expressed as:

[0053] F concat1 :

[0054] F concat2 :

[0055] F concat3 :

[0056] In the embodiment of the present invention, the preset pedestrian action detection part includes regressing the pedestrian's bounding box and classifying the pedestrian's action. According to the two-dimensional and three-dimensional fusion features at different scales, regression and classification tasks are performed at three different scales respectively. The outputs of the three scales are expressed as follows:

[0057] First scale: classification return

[0058] Second Scale: Classification return

[0059] The third scale: classification return

[0060] Among them, num_classes represents the number of categories of pedestrian actions, K represents the number of frames of the short video, and 4 represents the coordinates describing the two-dimensional target detection [x, y, w, h], which represent the center point coordinates and width and height of the two-dimensional box respectively.

[0061] In an embodiment of the present invention, when the first preset condition is met, the regression of the pedestrian's bounding box and the classification of the pedestrian's action are completed, wherein the first preset condition includes: the number of classifications of the pedestrian's bounding box and pedestrian action in the preset pedestrian action detection part reaches a preset value, and / or, the classification loss function of the pedestrian action classification satisfies the second preset condition, and / or, the regression loss function of the pedestrian detection box satisfies the third preset condition. This is only an example and is not limited to this. In actual applications, the corresponding preset conditions are selected according to actual conditions. When the second preset condition and the third preset condition can both meet the preset values, the preset values ​​are not limited here.

[0062] Specifically, the loss function consists of two parts: the classification loss function of pedestrian action classification and the regression loss function of pedestrian detection box.

[0063] Specifically, we first define the matching strategy between the labeled true value and the predicted value during the model training process, and use similarity to measure the predicted value and the true value, including classification and detection box, which is specifically expressed as:

[0064] metric=cls_error α box_error β

[0065] cls_error=BCE(cls pred ,cls truth )

[0066] box_error=CIoU(box pred ,box truth )

[0067] Among them, Metric represents the prediction result evaluation index, Cls-error represents the classification loss value, Box-error represents the detection box loss value, BCE represents the binary cross entropy function, CIoU represents the CIoU loss function; cls pred ,cls pred Represents the categories of prediction and true value, box pred ,box truthThey represent the detection boxes of the prediction and true value respectively, α and β are parameters, and the corresponding values ​​are selected according to the actual situation.

[0068] Specifically, for each truth value, defined as d truth ∈T, T represents all true value labeled data, for each predicted value, it is defined as d pred ∈P, P represents all prediction data.

[0069] Specifically, for each true value, it is matched with the topk predicted values ​​with the highest metric value, but each predicted value can only match one true value at most; if a predicted value and multiple true values ​​all get the highest metric value, the CIoU value is calculated, and the predicted value with a higher CIoU value is matched with the true value; the calculation of the matching probability is expressed as follows:

[0070]

[0071] B(d i )=max CIoU(d j ,d i ),d j ∈M(d i )

[0072] C(d i )=max metric(d j ,d i ),d j ∈M(d i )

[0073] Among them, P i Represents the matching probability between the true value and the predicted value; M(d i ) represents the set of all predicted values ​​that match the true value di; B(P i ) represents all the values ​​with the truth value d i In the set of predicted values ​​matched, i Calculate CIou, the maximum CIou value, CIou represents Complete-IoU, a type of intersection-over-union ratio; C(P i ) represents all the values ​​with the truth value d i In the set of predicted values ​​matched, i Calculate the metric and the maximum metric value;

[0074] Specifically, the loss function is expressed as:

[0075] L=L cls +L box

[0076] Specifically, for the action classification loss function L cls, using the binary cross entropy loss function, for the regression loss function L box , using CIoU loss function.

[0077] In an embodiment of the present invention, a video clip of a roadside scene monitoring video is input, and the position and action category of each pedestrian in the short video clip are obtained as output.

[0078] The method for detecting pedestrian actions based on surveillance video provided in this embodiment obtains a scene data set to be detected, the data set includes at least one video clip, and each video clip includes at least one frame of image; each pedestrian in the video clip is detected using a preset pedestrian action detection model to generate the position and action category of each pedestrian. The method provided by the present invention can accurately detect the position and action category of pedestrians by presetting the pedestrian action detection model.

[0079] The pedestrian motion detection method based on surveillance video provided in the present embodiment realizes the detection and recognition of the motions of roadside pedestrians by utilizing roadside surveillance video data. The detection is to obtain the position of the pedestrian performing the action by means of a two-dimensional detection frame, and the recognition is to output the category of the pedestrian's motion. By performing motion detection and recognition on pedestrians based on roadside video surveillance, when dangerous motions such as falls or other sudden unsafe motions are found, warnings can be promptly sent to relevant departments, and then the relevant behaviors can be processed, thereby realizing intelligent traffic management and social security management, etc., and also avoiding the situation where dangerous situations cannot be discovered in time and thus tragedy occurs in special sections with less traffic late at night.

[0080] In this embodiment, a pedestrian motion detection device based on surveillance video is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0081] This embodiment provides a pedestrian motion detection device based on surveillance video, such as Figure 2 As shown, including:

[0082] A data acquisition module 201 is used to acquire a scene data set to be detected, wherein the data set includes at least one video clip, and each video clip includes at least one frame of image;

[0083] The image shadow removal module 202 is used to detect each pedestrian in the video clip using a preset pedestrian action detection model to generate a position and action category of each pedestrian.

[0084] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0085] The pedestrian motion detection device based on surveillance video in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0086] The embodiment of the present invention also provides a computer device having the above Figure 3 The pedestrian motion detection device based on surveillance video is shown.

[0087] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 3 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 3 A processor 10 is taken as an example.

[0088] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0089] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0090] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0091] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0092] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0093] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0094] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A pedestrian motion detection method based on surveillance video, characterized in that: The method comprises: Acquire a scene data set to be detected, wherein the data set includes at least one video clip, and each video clip includes at least one frame of image; The preset pedestrian action detection model is used to detect each pedestrian in the video clip and generate the position and action category of each pedestrian.

2. The method according to claim 1, characterized in that The preset pedestrian action detection model includes: a preset dual-branch feature extraction part, a preset feature decoupling part, a preset feature fusion part, and a preset pedestrian action detection part.

3. The method according to claim 2, characterized in that The preset dual-branch feature extraction part includes: a two-dimensional backbone network and a three-dimensional backbone network, wherein: The 2D backbone network is used to model the spatial dimension; The 3D backbone network is used to extract spatiotemporal feature information.

4. The method according to claim 3, characterized in that The preset feature decoupling part is used to decouple the features of different scales output by the two-dimensional backbone network, and divide the feature images of different scales into regression and classification branches.

5. The method according to claim 4, characterized in that The preset feature fusion part is used to fuse the features of the two-dimensional branch and the three-dimensional branch, and to splice the two-dimensional and three-dimensional feature maps of the same scale in the channel dimension, where the two-dimensional branch is the output of the two-dimensional backbone network and the three-dimensional branch is the output of the three-dimensional backbone network.

6. The method according to claim 5, characterized in that The preset pedestrian action detection part includes: according to the fusion features of two-dimensional feature maps and three-dimensional feature maps of different scales, regression and classification are performed at different scales respectively, the pedestrian bounding box is regressed and the pedestrian action is classified, and the position and action category of each pedestrian in the video clip are generated.

7. The method according to claim 6, characterized in that When the first preset condition is met, the regression of the pedestrian's bounding box and the classification of the pedestrian's actions are completed, wherein the first preset condition includes: the number of classifications of the pedestrian's bounding box and pedestrian action in the preset pedestrian action detection part reaches a preset value, and / or the classification loss function of the pedestrian action classification meets the second preset condition, and / or the regression loss function of the pedestrian detection box meets the third preset condition.

8. A pedestrian motion detection device based on surveillance video, characterized in that: The device comprises: A data acquisition module, used to acquire a scene data set to be detected, wherein the data set includes at least one video clip, and each video clip includes at least one frame of image; The detection module is used to detect each pedestrian in the video clip using a preset pedestrian action detection model and generate the position and action category of each pedestrian.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the pedestrian motion detection method based on surveillance video according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the pedestrian motion detection method based on surveillance video according to any one of claims 1 to 7.

Images