Behavior detection method, electronic device and computer-readable storage medium

Abstract

The present application provides an activity detection method, comprising: acquiring a plurality of frames of video image frame data from a video stream; and detecting an activity of a passerby in the video stream based on the plurality of frames of video image frame data, wherein the step of detecting an activity of a passerby in the video stream based on the plurality of frames of video image frame data includes at least the steps of inputting the plurality of frames of video image frame data into a two-dimensional convolutional neural network, and recognizing an activity of a passerby in the video stream based on a time-series association relationship between the plurality of frames of video image frame data and the plurality of frames of video image frame data. The present application further provides an electronic device and a computer-readable storage medium.

Description

[Technical field]

[0001] This application claims priority to Chinese Patent Application No. 202110459730.8 filed on April 27, 2021, the contents of which are incorporated herein by reference.

[0002] The present application relates to the field of image recognition, and in particular to a behavior detection method, an electronic device and a computer-readable storage medium. [Background technology]

[0003] Intelligent video surveillance is based on computer vision technology and can intelligently analyze video data. It has been widely applied to security and intelligent traffic, greatly improving the response speed of security and saving manpower. Passersby are the main target of intelligent video surveillance, and the detection and recognition of various behaviors of passersby (such as abnormal behavior) is one of the important needs in the security field. Summary of the Invention [Problem to be solved by the invention]

[0004] In some related technologies, intelligent video surveillance uses intelligent video analysis technology to detect and recognize various behaviors of passers-by from a huge amount of surveillance video, which can provide important references for public safety emergency management and contribute to reducing the harm of public safety emergencies. However, the related technologies for detecting and recognizing the behaviors of passers-by cannot meet the deployment needs of some actual application scenarios. [Means for solving the problem]

[0005] In a first aspect, the present embodiment comprises: obtaining a plurality of frames of video image frame data from a video stream; detecting passerby actions in the video stream based on a plurality of frames of the video image frame data; The step of detecting a passerby's behavior in the video stream based on a plurality of frames of the video image frame data includes: The behavior detection method includes at least the steps of inputting the video image frame data of a plurality of frames into a two-dimensional convolutional neural network, and recognizing behaviors of passersby in the video stream based on a time-series association relationship between the video image frame data of the plurality of frames and the video image frame data of the plurality of frames.

[0006] In a second aspect, the present embodiment comprises: one or more processors; A memory in which one or more computer programs are stored, the one or more computer programs being executed by the one or more processors to cause the one or more processors to implement the behavior detection method provided by the embodiments of the present application in the first aspect; and one or more I / O interfaces coupled between the processor and the memory and arranged to enable the processor and the memory to exchange information.

[0007] In a third aspect, the present embodiment comprises: There is provided a computer-readable storage medium having a computer program stored therein, the computer program implementing the behavior detection method provided by the embodiments of the present application in a first aspect when executed by a processor. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a flowchart of a behavior detection method according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a flowchart showing some steps in the behavior detection method according to the embodiment of the present invention. [Diagram 3]FIG. 3 is a flowchart showing some steps in the behavior detection method according to the embodiment of the present invention. [Figure 4] FIG. 4 is a flow chart of some steps in the behavior detection method according to the embodiment of the present invention. [Diagram 5] FIG. 5 is a flowchart of some steps in the behavior detection method according to an embodiment of the present invention. [Figure 6] FIG. 6 is a flow chart of some steps in the behavior detection method according to an embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart showing some steps in a behavior detection method according to an embodiment of the present invention. [Figure 8] FIG. 8 is a flow chart of some steps in the behavior detection method according to the embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart of some steps in a behavior detection method according to an embodiment of the present invention. [Figure 10] FIG. 10 is a flow chart of some steps in the behavior detection method according to the present embodiment. [Figure 11] FIG. 11 is a block diagram showing the schematic configuration of an electronic device according to an embodiment of the present invention. [Figure 12] FIG. 12 is a schematic block diagram of a computer-readable medium according to an embodiment of the present invention. [Figure 13] FIG. 13 is a schematic diagram of an example behavior detection device and system configuration in an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In order to enable those skilled in the art to better understand the technical solution of the present application, the following detailed description of the behavior detection method, electronic device and computer-readable storage medium provided in the present application is provided in combination with the accompanying drawings.

[0010] Exemplary embodiments are described more fully below with reference to the drawings, which may be embodied in different forms and should not be construed as being limited to the embodiments described herein, but rather are provided in order to make the present application thorough and complete and to fully convey the scope of the present application to those skilled in the art.

[0011] Unless contradictory, each embodiment and each feature in each embodiment of the present application can be combined with each other.

[0012] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0013] The terms used herein are used only to describe certain embodiments and are not intended to limit the present application. As used herein, the singular terms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. Furthermore, when used herein, the terms "comprising" and / or "consisting of" refer to the presence of certain features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof.

[0014] Unless otherwise specified, the meaning of all terms (including technical and scientific terms) used herein is the same as that commonly understood by those skilled in the art. Furthermore, these terms defined in common dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and this application, and should not be interpreted as having an idealized or excessively formal meaning unless expressly specified herein.

[0015] Some related techniques mainly use frame difference to detect passerby behavior. That is, the abnormal behavior area is roughly located by analyzing the change in frame difference of consecutive image frames in the video, and then behavior recognition is performed on the roughly located area to determine whether abnormal behavior exists or the type of abnormal behavior. The abnormal behavior detection and recognition technology of passerby based on frame difference is sensitive to the change in light, and not all frame difference changes of consecutive image frames are caused by abnormal behavior of passersby, and many non-abnormal behaviors may also cause large changes in image frames, and some abnormal behaviors do not cause large changes in image frames. In addition, since the frame difference method needs to use consecutive image frames to locate abnormal behavior, the behavior recognition of the roughly located area is also based on consecutive image frames. If the number of consecutive image frames used in behavior recognition is small, it will result in waste of time domain information, and if the number of consecutive image frames used in behavior recognition is large, it will result in increased time cost and resource consumption for abnormal behavior detection and recognition. Therefore, the frame difference method is more suitable for scenes with few passersby and a single background.

[0016] In some other related technologies, the presence of abnormal behavior is determined by analyzing the behavior of each passerby, for example, by detecting passersby or analyzing skeleton points, the spatial position of each passerby is determined and tracked, and the time-latitude trajectory of each passerby is obtained, and then the motion image sequence or skeleton point motion sequence of each passerby is compiled to perform behavior recognition and identify the type of abnormal behavior. When detecting passersby or analyzing skeleton points, there are strict requirements for the viewing angle of the monitoring device, and if the monitoring device is at a high angle, there is a problem in identifying the skeleton points of passersby, and if the monitoring device is at an eye angle, the spatial position of the passersby cannot be identified due to mutual obstruction between passersby, which may result in false detection and missed detection. In addition, both passerby detection and skeleton point detection require the consumption of large amounts of computing resources and have slow processing speeds, so they cannot meet the real-time analysis requirements for abnormal behavior detection and recognition.

[0017] Some related technologies use three-dimensional convolution or dual networks to learn the time series information of abnormal behavior and then realize abnormal behavior detection and recognition, but three-dimensional convolution and dual networks can only learn short-term time series information, for example, one calculation of 3×5×5 three-dimensional convolution can only associate three frames of images, and the optical flow calculation in the dual network is obtained by calculating adjacent frames, and both three-dimensional convolution and dual networks consume a large amount of resources during execution. Some related technologies use two-dimensional convolution to overlap features and then use three-dimensional convolution to fuse time series information to realize abnormal behavior detection and recognition, but because three-dimensional convolution is used, there is a limit to improving the execution speed. Some related technologies directly use a single frame of video image frame or overlap the features of multiple frames of video image frames to classify abnormal behavior, but they ignore the association relationship between frames and waste time series information, which results in a decrease in the accuracy of detection and recognition.

[0018] In view of the above, related abnormal behavior detection and recognition technologies have difficulty meeting the deployment needs in some real scenes.

[0019] In view of this, in a first aspect, as can be seen by referring to FIG. 1, the present embodiment provides a behavior detection method including steps S100 and S200.

[0020] In step S100, video image frame data for a plurality of frames is obtained from a video stream.

[0021] In step S200, the actions of passersby in the video stream are detected based on the video image frame data of multiple frames.

[0022] The above step S200 includes at least step S300.

[0023] In step S300, the video image frame data of the multiple frames are input into a two-dimensional convolutional neural network, and the behavior of passersby in the video stream is recognized based on the time series association relationship between the video image frame data of the multiple frames and the video image frame data of the multiple frames.

[0024] In some embodiments, the video stream in step S100 is obtained by a monitoring device. In the present embodiment, the video stream may be obtained in real time by a monitoring device, or may be obtained by a monitoring device and then stored in a computer-readable medium. This is not particularly limited in the present embodiment. It should be noted that in the present embodiment, each behavior detection corresponds to a video stream of a certain length, that is, a plurality of frames of video image frame data are obtained from a video stream of a certain length.

[0025] In some embodiments, the multiple frames of video image frame data in step S100 are data that can be processed by a two-dimensional convolutional neural network in the present embodiment. In the present embodiment, multiple frames of video images can be obtained after decoding the video stream, and each frame of video image frame data corresponds to one frame of video image frame.

[0026] The actions of passersby have temporal continuity, and accordingly, there are time-series related relationships between the multiple video image frames obtained by decoding the video stream, and there are time-series related relationships between the multiple video image frame data corresponding to the multiple video image frames. The two-dimensional convolutional neural network used in the embodiment of the present application can not only learn the features of the video image frame data of each frame, but also learn the time-series related relationships between the video image frame data of the multiple frames, so that the actions of passersby can be detected based on the time-series related relationships between the video image frame data of the multiple frames and the video image frame data of the multiple frames.

[0027] In some embodiments, the step of detecting the behavior of passersby in the video stream based on a plurality of video image frames may only recognize the behavior of passersby in the video stream, and the recognition of the behavior of passersby may include, but is not limited to, determining whether the behavior of passersby exists, determining the type of the behavior of passersby, etc. In some embodiments, the step of detecting the behavior of passersby in the video stream based on a plurality of video image frames may recognize the behavior of passersby in the video stream and then detect the spatial location of the behavior of passersby. In the embodiment of the present application, the type of the behavior of passersby is not particularly limited. In some embodiments, the behavior of passersby may include, for example, abnormal behavior such as falling, punching, etc., and may include normal behavior such as running, jumping, etc.

[0028] In the behavior detection method provided in the embodiment of the present application, the behavior of passersby is detected based on a two-dimensional convolutional neural network. The two-dimensional convolutional neural network used can not only learn the features of the video image frame data of each frame, but also learn the time-series relationship between the video image frame data of multiple frames, so that the behavior of passersby in the video stream can be recognized based on the time-series relationship between the video image frame data of multiple frames and the video image frame data of multiple frames. Compared with the use of three-dimensional convolutional or dual networks to perform behavior detection, the behavior detection of passersby using the two-dimensional convolutional neural network in the embodiment of the present application requires less calculation, faster execution speed, less resource consumption, and can meet the real-time requirements in actual deployment. Compared with the use of a single frame of video image frame directly or the use of the features of multiple frames of video image frame superimposed before behavior detection, the two-dimensional convolutional neural network in the embodiment of the present application learns the time-series features of the behavior of passersby, so that it can effectively avoid false detection and detection omission, and improve the detection accuracy. In addition, in the embodiments of the present application, the behavior type of passersby can be directly recognized, or the spatial location of the behavior of passersby can be determined after recognizing the behavior type of passersby, thereby avoiding the limitation of applicable scenes caused by roughly locating the area of ​​the behavior of passersby and then determining the behavior type, and greatly improving the scene adaptability of behavior detection.

[0029] In the present embodiment, the structure of the two-dimensional convolutional neural network is not particularly limited. In some embodiments, the two-dimensional convolutional neural network includes at least one convolutional layer and at least one fully connected layer. In some embodiments, a plurality of frames of video image frame data are input into the two-dimensional convolutional neural network in a batch processing manner, the convolutional layer in the two-dimensional convolutional neural network can perform feature extraction on the input data, and the fully connected layer identifies the behavior of the passerby in the video stream corresponding to the current passerby behavior detection according to the feature data obtained by the convolutional layer.

[0030] Relatedly, in some embodiments, the two-dimensional convolutional neural network includes at least one convolutional layer and at least one fully connected layer, and as can be seen with reference to FIG. 2, step S300 includes steps S310 and S320.

[0031] In step S310, the at least one convolutional layer performs feature extraction on the video image frame data of multiple frames to obtain feature data, which fuses time series information of the video image frame data of multiple frames, and the time series information characterizes a time series association relationship between the video image frame data of multiple frames.

[0032] In step S320, the at least one fully connected layer recognizes the behavior of passersby in the video stream based on the feature data.

[0033] In addition, in some embodiments, the feature data combines time series information of multiple frames of video image frame data, meaning that the feature data can characterize the features of the video image frame data of each frame, and can also characterize the time series relationship between the video image frame data of multiple frames.

[0034] In some embodiments, the two-dimensional convolutional neural network further comprises a pooling layer.

[0035] In some embodiments, the multiple convolution layers in the two-dimensional convolutional neural network are connected in series, and each convolution layer performs feature extraction on the input data to obtain a feature map corresponding to the input data. When multiple frames of video image frame data are input to the two-dimensional convolutional neural network in a batch processing manner, each convolution layer can perform feature extraction on the input data to obtain multiple feature maps. In the present embodiment, the multiple frames of video image frame data input to the two-dimensional convolutional neural network in a batch processing manner are arranged in chronological order, and the multiple feature maps obtained by each convolution layer are also arranged in chronological order. In some embodiments, after each convolutional layer, some feature channels of the multiple feature maps are exchanged to realize mutual fusion of different time series information, and finally realize fusion of time series data of the multiple frames of the video image frame data in the feature data.

[0036] Relatedly, in some embodiments, the two-dimensional convolutional neural network includes a plurality of the convolutional layers connected in series, and as can be seen with reference to FIG. 3, step S310 includes step S311 and step S312.

[0037] In step S311, for each of the convolutional layers, input data of the convolutional layer is input to the convolutional layer to perform feature extraction to obtain a plurality of feature maps, the plurality of feature maps corresponding one-to-one to a plurality of frames of the video image frame data, and each of the feature maps includes a plurality of feature channels.

[0038] In step S312, some feature channels of the plurality of feature maps are exchanged to obtain first data.

[0039] In some embodiments, when the convolutional layer is a first convolutional layer, input data of the convolutional layer is the video image frame data for a plurality of frames.

[0040] In some embodiments, when the convolutional layer is neither the last convolutional layer nor the first convolutional layer, the first data is used as input data for a next convolutional layer.

[0041] In some embodiments, when the convolutional layer is a last convolutional layer, the first data is the feature data.

[0042] In addition, in some embodiments, the exchange of some feature channels of multiple feature maps only involves moving data and no addition / multiplication operations, so the amount of calculations does not increase when exchanging time-series information, the speed of data movement is fast, and the execution efficiency of behavior detection is not affected.

[0043] Relatedly, in some embodiments, the plurality of frames of video image frame data comprises N frames of sequentially ordered video image frame data, and the plurality of feature maps comprises N sequentially ordered feature maps, and as can be seen with reference to FIG. 4, step S312 includes steps S3121 to S3123.

[0044] In step S3121, the feature channels in each of the feature maps are separated into N sets of ordered feature channels.

[0045] In step S3122, for an ith feature map in the N ordered feature maps, a jth feature map corresponding to the ith feature map is identified, where the ith feature map is any one of the N ordered feature maps, and the jth feature map is any one of the N ordered feature maps.

[0046] In step S3123, the i-th set of feature channels in the i-th feature map is exchanged with any one set of feature channels in the j-th feature map to obtain the first data, where N, i, and j are positive integers.

[0047] In the present embodiment, there is no particular limitation on how to identify the jth feature map corresponding to the ith feature map in step S3122. In some embodiments, the jth feature map corresponding to the ith feature map is identified based on an algebraic relationship between i and j, in some embodiments, the jth feature map corresponding to the ith feature map is identified based on an adjacent relationship between i and j, and in some embodiments, one feature map is randomly selected from the N feature maps arranged in the order and set as the jth feature map corresponding to the ith feature map.

[0048] In some embodiments, a classification feature vector is obtained by a fully connected layer of a two-dimensional convolutional neural network, and each element of the classification feature vector characterizes the classification probability of one behavior type. Based on the classification probabilities of various behavior types, the behavior of the passerby can be identified from the video stream corresponding to the current behavior detection of the passerby. Identifying the behavior of the passerby from the video stream corresponding to the current behavior detection of the passerby includes, but is not limited to, determining whether the target behavior to be detected exists in the video stream corresponding to the current behavior detection of the passerby, and determining the type of the target behavior that exists.

[0049] Relatedly, in some embodiments, step S320 includes steps S321 through S323, as can be seen with reference to FIG.

[0050] In step S321, the at least one fully connected layer obtains a classification feature vector based on the feature data, and each element of the classification feature vector corresponds to one type of behavior.

[0051] In step S322, the classification probability of each behavior type is determined based on the classification feature vector.

[0052] In step S323, the behavior of passersby in the video stream is recognized based on the classification probability of each behavior type.

[0053] In some embodiments, in step S322, the classification feature vector obtained in step S321 is input into a classifier to obtain classification probabilities for various behavior types.

[0054] Relatedly, in some embodiments, step S323 includes steps S3231 through S3234, as can be seen with reference to FIG.

[0055] In step S3231, it is determined whether the classification probability of each behavior type exceeds a filter threshold.

[0056] In step S3232, when the classification probability of at least one type of behavior exceeds the filter threshold, it is determined that the target behavior has been recognized.

[0057] In step S3233, a behavior type whose classification probability exceeds the filter threshold is identified as the target behavior type.

[0058] In step S3234, if none of the classification probabilities of the various behavior types exceeds the filter threshold, it is determined that the target behavior has not been recognized.

[0059] The convolution layer of the two-dimensional convolutional neural network has spatial invariance, that is, there is a spatial correspondence between the feature map acquired through the convolution layer and the initial image, and the convolution layer of the two-dimensional convolutional neural network acquired through training can increase the area feature value related to the classification in the feature map and decrease the area feature value unrelated to the classification. In some embodiments, when a target action is recognized, a contour analysis is performed on the feature map output by the convolution layer of the two-dimensional convolutional neural network, thereby identifying the edge contour of the area related to the target action and identifying the spatial location of the recognized target action. In some embodiments, one convolution layer is designated as a target convolution layer from at least one convolution layer of the two-dimensional convolutional neural network, and contour analysis is performed based on the feature map output by the target convolution layer and the classification feature vector acquired by the fully connected layer of the two-dimensional convolutional neural network.

[0060] In some embodiments, as can be seen with reference to FIG. 7, after step S300, step S200 further includes step S400.

[0061] In step S400, the spatial location of the behavior of passersby in the video stream is detected based on the output data of the two-dimensional convolutional neural network.

[0062] In some embodiments, step S400 may be performed in the context of a passerby's behavior recognized by step S300. In some embodiments, step S400 is performed after each execution of step S300, that is, step S400 is performed regardless of whether a passerby's behavior is recognized or not. In some embodiments, execution or skipping of step S400 may be selected for different application scenarios.

[0063] Relatedly, in some embodiments, the two-dimensional convolutional neural network includes at least one convolutional layer and at least one fully connected layer, and output data of the two-dimensional convolutional neural network includes a classification feature vector obtained by the at least one fully connected layer and a plurality of feature maps output by a target convolutional layer, the target convolutional layer being one of the at least one convolutional layer, and each element of the classification feature vector corresponds to a behavior type. As can be seen from FIG. 8, step S400 includes step S410.

[0064] In step S410, the spatial location of a target action is identified based on the multiple feature maps output by the target convolution layer and the classification feature vector.

[0065] Relatedly, in some embodiments, as can be seen with reference to FIG. 9, step S410 includes steps S411 and S412. In step S411, an edge contour of the target action is identified based on the multiple feature maps output by the target convolution layer and the classification feature vector.

[0066] In step S412, the spatial position of the target behavior is identified based on the edge contour of the target behavior.

[0067] Relatedly, in some embodiments, the above step S412 may include: calculating a derivative of the classification feature vector with respect to a plurality of feature maps output by the target convolutional layer to obtain a weight map; multiplying the weight map by the plurality of feature maps output by the target convolutional layer to obtain a first spatial prediction map corresponding to a plurality of behavior types; extracting a first spatial prediction map corresponding to a behavior type having the highest classification confidence based on the first spatial prediction map as a second spatial prediction map; generating a third spatial prediction map based on the second spatial prediction map, the size of the third spatial prediction map being the same as the size of the video image frame; and extracting edges of the third spatial prediction map to identify an edge contour of the target behavior.

[0068] Relatedly, in some embodiments, the video image frame data of multiple frames is collected from multiple video image frames of a predetermined time length, and the above step S412 may include: calculating a derivative of the classification feature vector with respect to a plurality of feature maps output by the target convolution layer, setting derivative values ​​less than zero to zero to obtain a weight map; multiplying the weight map by the plurality of feature maps output by the target convolution layer, setting products less than zero to zero to zero to obtain a first spatial prediction map corresponding to a plurality of behavior types; extracting a first spatial prediction map corresponding to a behavior type with the highest classification confidence based on the first spatial prediction map, as a second spatial prediction map; normalizing the second spatial prediction map; scaling the size of the second spatial prediction map after normalization to the size of the video image frame and performing a binarization process to obtain a third spatial prediction map; and performing edge extraction on the third spatial prediction map to identify an edge contour of the target behavior.

[0069] In some embodiments, step S412 includes delineating an edge profile of the target activity in a plurality of the video image frames.

[0070] In some embodiments, after recognizing the target behavior and determining the spatial location of the target behavior, a video image frame with an outline of the target behavior is written to a video generation cache area, and a video file is generated and stored in a file system.

[0071] Relatedly, in some embodiments, step S412 further includes, after drawing edge contours of the target behavior in the plurality of video image frames, populating the plurality of video image frames with the drawn edge contours of the target behavior into a video generation cache area.

[0072] In some embodiments, each passerby action detection corresponds to a certain length of video stream, and step S100 obtains a number of frames of video image frame data from the certain length of video stream. In the present embodiment, the certain length of time is not particularly limited. In some embodiments, the certain length of time is determined based on the length of time that the passerby action to be detected typically occurs. For example, a fight may take 2 seconds, a fall may take 1 second, etc.

[0073] In some embodiments, the duration of the passerby's behavior may be longer than the above-mentioned fixed duration. In some embodiments, when the target behavior is recognized in each detection, a video image frame with the outline of the target behavior is written into the video generation cache area, and continues until the target behavior is not recognized in a certain detection, indicating the target behavior is ended; in such a case, the video image frame in the video generation cache area is converted into a video segment, and a video segment in which the entire process of the target behavior from the start to the end can be obtained, and information such as the start time, end time, and duration of the target behavior can also be determined based on the video segment. In relation to this, as can be seen with reference to FIG. 10, in some embodiments, the video image frame data of multiple frames is input into a two-dimensional convolutional neural network, and based on the time-series association relationship between the video image frame data of the multiple frames and the video image frame data of the multiple frames, the result of recognizing the passerby's behavior in the video stream includes recognizing the target behavior or not recognizing the target behavior, and when the target behavior is not recognized, the behavior detection method further includes steps S501 to S503.

[0074] In step S501, it is determined whether the edge-contoured video image frame is stored in the video generation cache area.

[0075] In step S502, when the edge-contoured video image frame is stored in the video cache area, a video segment is generated based on the edge-contoured video image frame stored in the video cache area.

[0076] In step S503, the video segment is retrieved from the video generation cache area.

[0077] In some embodiments, the step of retrieving the video segments from the video generation cache area includes storing the video segments in the video generation cache area to a file system and emptying the video generation cache area.

[0078] In some embodiments, step S100 includes obtaining the video stream, decoding the video stream to obtain a plurality of consecutive video image frames, sampling the plurality of consecutive video image frames to obtain a plurality of target video image frames, and pre-processing the plurality of target video image frames to obtain a plurality of frames of the video image frame data.

[0079] In the present embodiment, there is no particular limitation on how the video stream is decoded. In some embodiments, the video stream is decoded using a graphics processor (GPU, Graphics Processing Unit).

[0080] In the present embodiment, there is no particular limitation on how the consecutive video image frames are sampled. In some embodiments, the consecutive video image frames are sampled randomly. In some embodiments, the consecutive video image frames are sampled at a predetermined interval. In some embodiments, the consecutive video image frames are sampled continuously.

[0081] It should be noted that sampling multiple video image frames at the predetermined intervals as described above can obtain more time series information than consecutively sampling multiple video image frames, thereby improving detection accuracy.

[0082] In the present embodiment, there is no particular limitation on how to pre-process the video image frames of the plurality of detection targets. In some embodiments, pre-processing the video image frames of the plurality of detection targets includes: adjusting the size of each of the video image frames of the detection targets to a predetermined size; and performing color space conversion, pixel value normalization, and subtracting the average value and dividing by the standard deviation on the video image frames of the detection targets adjusted to the predetermined size to obtain video image frame data of the plurality of frames.

[0083] In some embodiments, each passerby behavior detection corresponds to a fixed length of video stream, and a predetermined number of video image frame data are obtained in the fixed length of video image frames in a uniform sampling manner.

[0084] Relatedly, in some embodiments, the above step S100 includes: determining an area of ​​a foreground image area in a current video image frame in the video stream; determining a momentum of two adjacent video image frames when the area of ​​the foreground image area exceeds an area threshold; determining the current video image frame as a sampling starting point when the momentum of the two adjacent video image frames exceeds a momentum threshold; and uniformly sampling and pre-processing a predetermined number of the video image frames from a plurality of consecutive video image frames of a predetermined length of time to obtain the video image frame data of the plurality of frames.

[0085] In the present embodiment, there is no particular limitation on how to determine the area of ​​the foreground image region in the current video image frame. In some embodiments, a frame difference method is used to obtain the foreground image of the current video image frame.

[0086] In the present embodiment, there is no particular limitation on how to determine the momentum of two adjacent video image frames. In some embodiments, sparse optical flow is used to calculate the momentum of two adjacent video image frames.

[0087] In some embodiments, before performing activity detection by steps S100 to S200, the method further includes the steps of training a two-dimensional convolutional neural network, acquiring a video stream, decoding the video stream to generate video image frames, performing data cleansing to acquire sample video segments, marking types of passerby activity in the sample video segments, marking the sample video segments without passerby activity requiring detection as background, training the two-dimensional convolutional neural network using the marked sample video segments, and performing a quantification operation and format conversion on the trained two-dimensional convolutional neural network.

[0088] In a second embodiment, as can be seen with reference to FIG. one or more processors 101; A memory 102 in which one or more computer programs are stored, the one or more computer programs being executed by the one or more processors 101 to cause the one or more processors 101 to realize the behavior detection method provided by the embodiments of the present application in the first aspect; An electronic device is provided, the electronic device including: one or more I / O interfaces 103 connected between a processor 101 and a memory 102 and arranged to enable the processor 101 and the memory 102 to exchange information with each other.

[0089] The processor 101 is a device having a data processing function, and includes, but is not limited to, a central processor (CPU), etc.; the memory 102 is a device having a data storage function, and includes, but is not limited to, a random access memory (RAM, more specifically, SDRAM, DRR, etc.), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize the exchange of information between the processor 101 and the memory 102, and includes, but is not limited to, a data bus (Bus), etc.

[0090] In some embodiments, the processor 101, memory 102, and I / O interface 103 are connected to each other and to other components of a computing device by a bus 104.

[0091] In a third aspect, as can be seen with reference to FIG. 12, an embodiment of the present application provides a computer-readable storage medium having a computer program stored therein, the computer program realizing the behavior detection method provided by the embodiment of the present application in the first aspect when the computer program is executed by a processor.

[0092] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be described in detail below through specific examples.

[0093] Example 1 FIG. 13 is a schematic diagram of an example behavior detection device and system configuration in an embodiment of the present invention.

[0094] As shown in Figure 13, the behavior detection device includes a behavior recognition module, a behavior location detection module, and a video automatic storage module. In this embodiment 1, the behavior detection device is disposed in a server, and the server further includes a GPU, a CPU, a network interface, a video memory, and an internal memory, and are connected to each other by an internal bus.

[0095] Example 2 The flow of behavior detection in this example 2 is as follows.

[0096] The 2D convolutional neural network is loaded into video memory or internal memory and initialized, and parameters such as input image size limit, batch size, filter threshold, area threshold, typical duration of behavioral occurrence, frame rate, etc. are configured.

[0097] A video stream is acquired from the camera, and the acquired video stream is sent to the GPU of the system server for hardware decoding to generate a plurality of video image frames.

[0098] A set of video image frames corresponding to the typical duration of an action is extracted from the multi-frame video image frames, and N frames are uniformly sampled from it, where N is determined during training of a two-dimensional convolutional neural network.

[0099] The N frames of video image frames obtained by uniform sampling are preprocessed, their long sides are fixed to the size specified by the two-dimensional convolutional neural network, equiproportional scaling is performed, the short side fill pixel values ​​are set to 0, and then converted to RGB color space. Finally, the pixel values ​​are normalized between 0 and 1, the average value is subtracted, and then divided by the standard deviation to obtain the preprocessed N frames of video image frame data.

[0100] The N video image frames obtained after preprocessing are batched into N two-dimensional datasets and sent to a two-dimensional convolutional neural network. Each convolutional layer of the two-dimensional convolutional neural network is characterized as having a size of NCHW. The size of the N frames is characterized as a feature map of CHW, where C represents the number of feature channels, and H and W represent the width and height of the feature map, respectively.

[0101] The feature channels of the feature maps of N frames output by each convolutional layer are divided into N sets. For the i-th feature map, one feature map is randomly extracted from the other feature maps other than the i-th feature map and denoted as the j-th feature map. One set of feature channels is randomly selected from the N sets of feature channels of the j-th feature map and exchanged with the i set of feature channels of the i-th feature map, thereby exchanging time-series information without additionally increasing the amount of calculation.

[0102] The feature map after the feature channel exchange is sent to the next convolutional layer for calculation, and the feature data is obtained after the last convolutional layer, and a classification feature vector is generated by a fully connected layer. The classification feature vector is sent to a classifier to obtain the classification probability of various behavior types.

[0103] When the classification probability of at least one type of behavior exceeds the filter threshold, it is determined that the target behavior has been recognized.

[0104] A behavior type whose classification probability exceeds a filter threshold is identified as a target behavior type.

[0105] When the classification probability of each behavior type does not exceed the filter threshold, it is determined that the target behavior has not been recognized.

[0106] The step of identifying a spatial location of the target behavior upon recognition of the target behavior includes: extracting a feature map of a target convolutional layer in a two-dimensional convolutional neural network and a classification feature vector output by a fully connected layer, calculating a differential coefficient of the classification feature vector with respect to the feature map of the target convolutional layer, setting the value of a differential coefficient less than 0 to 0, and obtaining a weight map whose spatial size is consistent with that of the target convolutional feature map; multiplying the weight map and the feature map of the target convolutional layer to obtain a first spatial prediction map with values ​​in the product less than zero set to zero; Convert the first spatial prediction map into a storage format of N*Class*H*W, where Class represents the number of behavioral types; then perform a softmax operation on the first spatial prediction map at the type latitude to extract the latitude corresponding to the type with the highest reliability of the classification result, to obtain an N*H*W spatial prediction map; calculate the maximum value of all elements in the feature map at the location latitude of the batch processing number N, to obtain an H*W second spatial prediction map; normalizing the second spatial prediction map between 0 and 1 by subtracting the minimum value of all elements from every element in the second spatial prediction map and dividing the result by the maximum value of all elements; Scaling the normalized second spatial prediction map to the size of the video image frame and performing a binarization process (if the element value is greater than 0.5, it is set to 1, otherwise it is set to 0) to obtain a third spatial prediction map; An edge contour is extracted from the third spatial prediction map, and the obtained edge contour is a spatial position where the target behavior exists; Depicting the contour boundary in the video image frame as the detection result, fetching the video image frame into the video generation cache area, and initiating the next detection. The step of storing the video segments includes: determining whether there is a video image frame in the video generation cache area when the target action is not recognized (i.e., determining whether a previous detection recognized the target action); generating a video segment based on the video image frames in the video cache area when the video image frames are in the video cache area (i.e., the previous detection recognized the target behavior); storing the video segments; Empty the video generation cache area to start the next discovery, or and initiating a next detection when there are no video image frames in the video cache area (i.e., the previous detection did not recognize the target behavior).

[0107] It is understood by those skilled in the art that all or some steps of the methods and functional modules / units in the apparatus disclosed herein may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware embodiments, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components, for example, one physical component may have multiple functions, or one function or step may be performed by multiple physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor (such as a central processing unit, digital signal processor, or microprocessor), or as hardware, or as an integrated circuit such as a dedicated integrated circuit. Such software may be located on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the technical term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk memory, magnetic boxes, magnetic tapes, magnetic disk memory or other magnetic storage devices, or any other medium for storing the desired information and accessible by a computer. In addition, those skilled in the art will appreciate that communication media typically include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0108] Although exemplary embodiments are disclosed herein and specific terms are used, they are used and should be interpreted as merely general illustrative and not limiting. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics and / or elements described in connection with a particular embodiment may be used alone or in combination with features, characteristics and / or elements described in connection with other embodiments. Thus, those skilled in the art will appreciate that changes in various forms and details may be made without departing from the scope of the present application as set forth in the appended claims.

Claims

1. obtaining a plurality of frames of video image frame data from a video stream; detecting passerby actions in the video stream based on a plurality of frames of the video image frame data; The step of detecting a passerby's behavior in the video stream based on a plurality of frames of the video image frame data includes: The method includes at least the step of inputting the video image frame data of the plurality of frames into a two-dimensional convolutional neural network, and recognizing the behavior of passersby in the video stream based on a time-series relation between the video image frame data of the plurality of frames and the video image frame data of the plurality of frames; The step of detecting a passerby's behavior in the video stream based on a plurality of frames of the video image frame data includes: The method further includes inputting the video image frame data of multiple frames into a two-dimensional convolutional neural network; recognizing the actions of passersby in the video stream based on the time-series association relationship between the video image frame data of multiple frames and the video image frame data of multiple frames; and then detecting the spatial location of the actions of passersby in the video stream based on the output data of the two-dimensional convolutional neural network; The two-dimensional convolutional neural network includes at least one convolutional layer; The step of inputting the video image frame data of a plurality of frames into a two-dimensional convolutional neural network and recognizing the behavior of passersby in the video stream based on a time series relation between the video image frame data of the plurality of frames and the video image frame data of the plurality of frames, includes: performing feature extraction on the video image frame data of a plurality of frames by the at least one convolutional layer to obtain a plurality of feature maps in one-to-one correspondence with the video image frame data of a plurality of frames, each feature map including a plurality of feature channels; and exchanging a portion of the feature channels of the plurality of feature maps to obtain feature data fusing time series information of the video image frame data of the plurality of frames, the time series information characterizing a time series association relationship between the video image frame data of the plurality of frames; and recognizing behavior of passers-by in the video stream based on the feature data. Behavioral detection methods.

2. The two-dimensional convolutional neural network further includes at least one fully connected layer; The step of recognizing a behavior of a passerby in the video stream based on the feature data includes: and recognizing, by the at least one fully connected layer, actions of passers-by in the video stream based on the feature data. The method of claim 1 .

3. The two-dimensional convolutional neural network includes a plurality of serially connected convolutional layers, The step of performing feature extraction on a plurality of frames of the video image frame data by the at least one convolutional layer includes: For each of the convolutional layers, inputting input data of the convolutional layer into the convolutional layer to perform feature extraction; The step of exchanging some of the feature channels of the plurality of feature maps to obtain feature data fusing time series information of the plurality of frames of the video image frame data includes: Acquiring first data by exchanging a portion of feature channels of the plurality of feature maps; When the convolution layer is a first convolution layer, input data of the convolution layer is the video image frame data of a plurality of frames; When the convolution layer is neither the last convolution layer nor the first convolution layer, the first data is used as input data for a next convolution layer; When the convolutional layer is the last convolutional layer, the first data is set as the feature data. The method of claim 1 .

4. The plurality of frames of video image frame data includes N frames of sequentially ordered video image frame data, and the plurality of feature maps includes N sequentially ordered feature maps; The step of exchanging some of the feature channels of the plurality of feature maps to obtain first data includes: Separating the plurality of feature channels in each of the feature maps into an ordered set of N feature channels; for an ith feature map in the N ordered feature maps, identifying a jth feature map corresponding to the ith feature map, the ith feature map being any one of the N ordered feature maps, and the jth feature map being any one of the N ordered feature maps; exchanging the i-th set of feature channels in the i-th feature map with any one set of feature channels in the j-th feature map to obtain the first data; N, i, j are positive integers. The behavior detection method according to claim 3 .

5. The step of recognizing, by the at least one fully connected layer, the behavior of passersby in the video stream based on the feature data, comprises: obtaining a classification feature vector based on the feature data by the at least one fully connected layer, each element of the classification feature vector corresponding to one behavior type; determining a classification probability of various behavior types based on the classification feature vector; recognizing passerby behaviors in the video stream based on the classification probabilities of various behavior types. The behavior detection method according to claim 2 .

6. The step of recognizing passerby behavior in the video stream based on classification probabilities of various behavior types includes: determining whether the classification probability of each behavior type exceeds a filter threshold; determining that a target behavior has been recognized when a classification probability of at least one behavior type exceeds the filter threshold; identifying a behavior type having a classification probability exceeding the filter threshold as the target behavior type; and determining that the target behavior is not recognized when none of the classification probabilities of the various behavior types exceeds the filter threshold. The behavior detection method according to claim 5 .

7. the two-dimensional convolutional neural network further includes at least one fully connected layer, and output data of the two-dimensional convolutional neural network includes a classification feature vector obtained by the at least one fully connected layer based on the feature data and the plurality of feature maps output by a target convolutional layer, the target convolutional layer being one of the at least one convolutional layer, and each element of the classification feature vector corresponds to one type of behavioral type; The step of detecting a spatial position of a passerby's action in the video stream based on the output data of the two-dimensional convolutional neural network includes: A step of identifying a spatial location of a target action based on a plurality of feature maps output by the target convolution layer and the classification feature vector; The method of claim 1 .

8. The step of identifying a spatial location of a target action based on a plurality of feature maps output by the target convolution layer and the classification feature vector includes: Identifying an edge contour of the target action based on a plurality of feature maps output by the target convolution layer and the classification feature vector; and determining a spatial location of the target activity based on an edge profile of the target activity. The behavior detection method according to claim 7.

9. The video image frame data of the plurality of frames is collected from a plurality of video image frames of a predetermined length of time; The step of identifying an edge contour of the target action based on a plurality of feature maps output by the target convolution layer and the classification feature vector includes: Calculating differential coefficients of the classification feature vector with respect to a plurality of feature maps output by the target convolutional layer to obtain a weight map; multiplying the weight map by a plurality of feature maps output by the target convolutional layer to obtain a first spatial prediction map corresponding to a plurality of types of behavioral patterns; extracting a first spatial prediction map corresponding to a behavior type having the highest classification reliability based on the first spatial prediction map, and setting the first spatial prediction map as a second spatial prediction map; generating a third spatial prediction map based on the second spatial prediction map, the third spatial prediction map having a size equal to the size of the video image frame; extracting edges of the third spatial predictive map to identify edge contours of the target activity. The behavior detection method according to claim 8.

10. The step of identifying a spatial location of the target activity based on an edge profile of the target activity includes: delineating an edge profile of the target action in the plurality of video image frames. The behavior detection method according to claim 9 .

11. The step of identifying a spatial location of the target activity based on an edge profile of the target activity includes: and after the step of delineating the edge profile of the target activity in the plurality of video image frames, further comprising: populating a video generation cache area with the plurality of video image frames having the delineated edge profile of the target activity. The behavior detection method according to claim 10.

12. The video image frame data of the plurality of frames is input to a two-dimensional convolutional neural network, and the result of recognizing the behavior of passersby in the video stream based on the time series association relationship between the video image frame data of the plurality of frames and the video image frame data of the plurality of frames includes recognizing a target behavior or not recognizing a target behavior, and when the target behavior is not recognized, determining whether the edge-contoured video image frame is stored in a video generation cache area; generating a video segment based on the edge-contoured video image frames stored in the video generation cache area when the edge-contoured video image frames are stored in the video generation cache area; retrieving the video segment from the video generation cache area. The method of claim 1 .

13. The step of obtaining a plurality of frames of video image frame data from the video stream comprises: determining an area of ​​a foreground image region in a current video image frame within the video stream; determining an amount of motion between two adjacent video image frames when an area of ​​the foreground image region exceeds an area threshold; When the momentum of two adjacent video image frames is greater than a momentum threshold, determining a current video image frame as a sampling start point; uniformly sampling and pre-processing a predetermined number of video image frames from a plurality of consecutive video image frames of a predetermined time length to obtain the plurality of frames of video image frame data; The method of claim 1 .

14. At least one processor; a memory for storing at least one computer program, the at least one computer program being configured to cause the at least one processor to implement the behavior detection method according to any one of claims 1 to 13 when executed by the at least one processor; At least one I / O interface connected between the processor and the memory and arranged to enable the processor and the memory to communicate with each other; electronic equipment.

15. A computer program is stored, the computer program implementing the behavior detection method according to any one of claims 1 to 13 when executed by a processor. A computer-readable storage medium.

Images