Method and device for fighting behavior recognition based on video stream data

Abstract

The invention discloses a fight behavior identification method and device based on video stream data, and the method comprises the steps: configuring the working time and storage setting of cameras, and creating a video stream capturing task based on the working time for each camera; using the video analysis network model to detect whether the personnel in the video stream have limb intersection, and if yes, endowing the limb intersection personnel with labels; outputting fighting alarm information according to the limb intersection personnel labels and the associated information; and packaging the fight alarm information and uploading the fight alarm information to an alarm database. According to the method, the fight behavior can be quickly identified by analyzing the video streaming data of the camera by using an image identification technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for identifying fighting behavior based on video stream data. Background Art

[0002] Fighting is a serious threat to social security and is extremely dangerous. With the development of technology, it is now possible to use image recognition to monitor and reduce the underreporting rate of manual monitoring.

[0003] Existing technologies mainly identify people through two directions. One is to use target detection to identify the target frame of people, and then determine whether there is a fight by judging whether there is an intersection between the frames and classifying the behaviors. This solution relies on the behavior classification model and has a high false negative rate. The other is to use the video stream data to input into the three-dimensional convolutional network, and determine whether there is a fight in the monitoring screen through spatiotemporal feature analysis. However, due to the large amount of calculation of the three-dimensional convolutional network, the equipment requirements are high and it does not support covering multiple cameras.

[0004] Most of the existing technical means have the disadvantages of poor recognition effect and high requirements on equipment. To solve the above problems, it is necessary to comprehensively consider the equipment performance and recognition effect. Summary of the invention

[0005] In view of the above technical problems, the present invention provides a method and device for identifying fighting behavior based on video stream data, which can automatically and efficiently identify whether there are fighting people in real-time video.

[0006] The first aspect of the present invention provides a method for identifying fighting behavior based on video stream data, including configuring the working time and storage settings of the camera, and creating a video stream capture task based on the working time for each camera. Use a video analysis network model to detect whether there is a limb intersection between people in the video stream, and if there is a limb intersection, assign a limb intersection person label. Output fighting alarm information based on the limb intersection person label and related information; encapsulate the fighting alarm information and upload it to the alarm database.

[0007] In an optional implementation manner, configuring the working time and storage settings of the camera includes:

[0008] Configure the camera ID, configure the camera rtsp stream address, configure the camera capture duration, configure the camera capture interval, and configure the camera working time information.

[0009] In an optional implementation, the using of a video analysis network model to detect whether there is a body intersection of persons in a video stream includes:

[0010] Use the video analysis network module to obtain the spatial features of each frame image in the video stream, and perform time offset processing on the spatial features of the current frame image to the spatial features of the next frame image to obtain the fused spatial features;

[0011] Based on the fused spatial features, it is detected whether there is limb intersection between persons in the video stream.

[0012] In an optional implementation, the video analysis network module is used to obtain the spatial features of each frame image in the video stream respectively, and the spatial features of the current frame image are subjected to time offset processing on the spatial features of the next frame image to obtain the fused spatial features, including:

[0013] The video analysis network module is used to obtain the spatial features of eight frames in the video stream respectively, and the feature information of 1 / 8 in the current frame is temporally offset with the spatial features of the corresponding position in the next frame to obtain the fused spatial features.

[0014] In an optional implementation, the using of a video analysis network model to detect whether there is a body intersection of persons in a video stream includes:

[0015] A video analysis network model is used to calculate the fighting confidence and non-fighting confidence of whether the persons in each frame have body intersections, and the total fighting confidence and total non-fighting confidence are calculated based on the fighting confidence and non-fighting confidence;

[0016] The total fighting confidence and the total non-fighting confidence are normalized, and a judgment category is output, and whether there is a limb intersection between persons in the video stream is determined according to the judgment category.

[0017] In an optional implementation manner, outputting fight warning information according to the body intersection person labels and associated information includes:

[0018] The personnel coordinate information, fighting behavior label, fighting time, and number of personnel are output as the fighting alarm information.

[0019] A second aspect of the present invention provides a fighting behavior recognition device based on video stream data, comprising:

[0020] Configuration module, used to configure the camera's working hours and storage settings, and create a video stream capture task based on working hours for each camera;

[0021] The recognition module is used to detect whether there is a body intersection between people in the video stream using a video analysis network model, and if there is a body intersection, a body intersection person label is assigned;

[0022] The alarm module is used to output fight alarm information according to the body intersection person labels and related information; and encapsulate the fight alarm information and upload it to the alarm database.

[0023] In an optional embodiment, the recognition module is also used to use the video analysis network module to obtain the spatial features of each frame image in the video stream respectively, and perform time-shift processing on the spatial features of the current frame image with the spatial features of the next frame image to obtain a fused spatial feature; based on the fused spatial feature, detect whether there is any limb intersection between people in the video stream.

[0024] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0025] At least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method described in the first aspect of the embodiment of the present invention.

[0026] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the method according to the first aspect of the embodiment of the present invention is executed.

[0027] The present invention utilizes image recognition technology to realize rapid recognition of fighting behavior by analyzing camera video stream data.

[0028] The present invention utilizes time offset processing to transform a two-dimensional convolutional network and intercepts video stream data for identification and analysis to realize fighting behavior identification, which can greatly reduce equipment performance requirements and effectively improve fighting identification effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flow chart of a fighting behavior recognition method based on video stream data in an embodiment of the present invention.

[0030] Figure 2 The module diagram of a fighting behavior recognition device based on video stream data in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0033] See also Figure 1 The present invention provides a fighting behavior recognition method based on video stream data, which includes the following steps.

[0034] S100: configuring the working hours and storage settings of the cameras, and creating a video stream capture task based on the working hours for each camera.

[0035] For example, configure the camera ID, configure the camera RTSP stream address, configure the camera capture duration, configure the camera capture interval, and configure the camera working time information.

[0036] The video stream data captured by the camera is stored in the database. The database can be configured, queried and modified through the video stream storage module. Create a video stream analysis task for each camera. The process of each analysis task is as follows.

[0037] Use the camera ID to create a multi-threaded task, and determine whether to perform the video stream analysis task based on the working time information. If it is not within the working hours, the task will be dormant. Otherwise, connect to the camera rtsp stream address within the working hours to capture the video according to the capture time and store it in the memory space. After the analysis is completed, the relevant memory resources will be released immediately to reduce the burden on system resources. When there is a camera information modification task, the camera information will be updated first, and the video stream will be captured after the update is completed.

[0038] For example, 8 pm to 8 am is working time, so the video stream of this time period is captured and stored in the memory space for image recognition.

[0039] S200: Use a video analysis network model to detect whether there is body intersection between people in the video stream, and if there is body intersection, assign a body intersection person label.

[0040] The spatiotemporal information of the video stream data is analyzed through the video analysis network model to obtain information such as the movement trajectory, action behavior, and movement pattern of the personnel. The efficient and high-performance video classification network is used to exchange and analyze the information between adjacent frames and build a time model to connect all video frame information from the time dimension. The video stream data is input into the video stream analysis module to determine whether there is a fight in the video stream data. If there is no fight, it will sleep according to the capture interval; if there is a fight, the image will be captured for target detection.

[0041] S300: Outputting fight alarm information according to the body intersection person labels and related information; packaging the fight alarm information and uploading it to an alarm database.

[0042] For example, the personnel coordinate information, fighting behavior label, fighting time, and number of personnel are output as the fighting alarm information.

[0043] For example, a fight event may be determined based on the video capture time, personnel coordinate information may be determined based on the position of the camera, and the number of personnel may be determined based on personnel identification.

[0044] In one embodiment, the image is uploaded to the database via the http protocol and the alarm message is uploaded via the mqtt service.

[0045] Furthermore, when using the video analysis network model to detect whether there is a limb intersection between people in the video stream, in some embodiments, the video analysis network module can be used to obtain the spatial features of each frame image in the video stream, and the spatial features of the current frame image are subjected to time offset processing on the spatial features of the next frame image to obtain a fused spatial feature. Then, based on the fused spatial feature, it is detected whether there is a limb intersection between people in the video stream.

[0046] For example, the video analysis network module can be used to obtain the spatial features of eight frames in the video stream respectively, and the feature information of 1 / 8 in the current frame and the spatial features of the corresponding position in the next frame are time-shifted in turn to obtain the fused spatial features.

[0047] There is an input feature X. After the input is processed by the time shift module and the convolution module, the matrix addition is performed with the original X to obtain the output feature Y. The simplified formula is: Y = F (X) + X. Sample 8 frames of images in the video stream data, process each frame separately through the convolution network, obtain the corresponding spatial features, use the time shift module to perform time shift from the initial frame to the future frame, and combine the 1 / 8 feature map of the current frame with the 7 / 8 feature map of the next frame to generate the next layer of features to fuse the spatial features between different frames.

[0048] The present invention utilizes time offset processing to transform a two-dimensional convolutional network and intercepts video stream data for identification and analysis to realize fighting behavior identification, which can greatly reduce equipment performance requirements and effectively improve fighting identification effects.

[0049] Specifically, the time offset module is inserted into the convolutional layer of the convolutional network to offset part of the past frame channel to the future frame along the time dimension to reduce the memory movement cost. In addition, since the residual structure has good feature retention ability, inserting the time offset module into the residual branch of the convolutional network can alleviate the problem of spatial feature learning degradation.

[0050] For example, the first frame image is first input into the convolutional network to obtain the spatial features of each channel of the current frame image, and then the spatial features of each channel of the second frame are calculated. The feature information of 1 / 8 in the first frame is offset with the spatial features of the corresponding position in the second frame image, and this step is repeated to fuse the features of the sampled frame images.

[0051] The specific fusion operation takes a one-dimensional convolution as an example. Assume that the convolution kernel size is 3, the convolution weight is W = (w1, w2, w3), and the input X is a one-dimensional vector of infinite length. The convolution operator Y = Conv (W, X) can be written as: Yi = w1 X (i-1) +w2X i +w3X (i+1) The offset operation is to offset the input X by -1, 0, +1 respectively, and multiply it by w1, w2, w3 respectively. The sum of these results is the fusion feature Y. i -1 =X (i-1) , X i 0 =X i , X i +1 =X i+1 , then Y = w1 X -1 +w1 X 0 +w1X 1 .

[0052] After this operation, the feature maps of 8 sampling frames are finally obtained, and matrix splicing operation is performed on them (for example, 8 1*8 matrices are spliced ​​in the "row dimension" to obtain an 8*8 matrix) to obtain the fusion features of the sampling frame.

[0053] Finally, the fused features of the sampled frames are pooled and input into the fully connected layer of the two set categories (fighting and non-fighting), and finally the confidence of the sampled frames in each category is obtained.

[0054] For example, the video analysis network model is used to calculate the fighting confidence and non-fighting confidence of whether there is limb intersection between people in each frame of the image, and the total fighting confidence and total non-fighting confidence are calculated based on the fighting confidence and non-fight confidence.

[0055] Finally, the total fighting confidence and the total non-fighting confidence are normalized, and a judgment category is output, and whether there is a limb intersection between the persons in the video stream is determined based on the judgment category.

[0056] For example, in video stream data of a given length, 8 frames of images are sampled in sequence for analysis. Finally, the classification label results of all sampled frames are counted, and the total confidence of the fighting and non-fighting categories is calculated. The confidence is normalized to obtain the confidence of each category and sorted. The category result ranked first is taken out to determine whether there is fighting behavior in the video stream data of the entire length.

[0057] In summary, the present invention utilizes time offset processing to transform a two-dimensional convolutional network and intercepts video stream data for identification and analysis to realize fighting behavior recognition, which can greatly reduce equipment performance requirements and effectively improve fighting recognition effects.

[0058] The present invention uses image recognition technology to analyze the video stream data of the camera to realize the rapid identification of fighting behavior. Figure 2 A fighting behavior recognition device based on video stream data is shown.

[0059] The fighting behavior recognition device based on video stream data includes:

[0060] The configuration module 21 is used to configure the working hours and storage settings of the camera, and to create a video stream capture task based on the working hours for each camera.

[0061] The recognition module 22 is used to use the video analysis network model to detect whether there is any body intersection between people in the video stream, and if there is any body intersection, assign a body intersection person label.

[0062] The alarm module 23 is used to output fight alarm information according to the body intersection person labels and related information; and encapsulate the fight alarm information and upload it to the alarm database.

[0063] Furthermore, the recognition module 22 is also used to use the video analysis network module to obtain the spatial features of each frame image in the video stream respectively, and perform time-shift processing on the spatial features of the current frame image with the spatial features of the next frame image to obtain fused spatial features; based on the fused spatial features, detect whether there is any limb intersection between people in the video stream.

[0064] The functions or methods performed by the above modules can be referred to the description of the fighting behavior recognition method based on video stream data, which will not be repeated here.

[0065] The present invention also provides an electronic device, comprising:

[0066] At least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned fighting behavior recognition method based on video stream data.

[0067] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned fighting behavior recognition method based on video stream data is implemented.

[0068] It can be understood that computer-readable storage media may include: any entity or device capable of carrying a computer program, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. A computer program includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form, etc. A computer-readable storage medium may include: any entity or device capable of carrying a computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0069] In certain embodiments of the present invention, the device may include a controller, which is a single-chip microcomputer chip that integrates a processor, a memory, a communication module, etc. The processor may refer to the processor included in the controller. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0070] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0071] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fighting behavior recognition method based on video stream data, characterized in that: include: Configure the camera's working hours and storage settings, and create a video stream capture task based on the working hours for each camera; Use the video analysis network model to detect whether there is body intersection between people in the video stream, and if there is body intersection, assign a body intersection person label; Output fight alarm information according to the body intersection person labels and related information; encapsulate the fight alarm information and upload it to the alarm database.

2. The method for identifying fighting behavior based on video stream data according to claim 1, characterized in that: The configuration of the camera's working time and storage settings includes: Configure the camera ID, configure the camera rtsp stream address, configure the camera capture duration, configure the camera capture interval, and configure the camera working time information.

3. The method for identifying fighting behavior based on video stream data according to claim 1, characterized in that: The method of using a video analysis network model to detect whether there is a body intersection between persons in a video stream includes: Use the video analysis network module to obtain the spatial features of each frame image in the video stream, and perform time offset processing on the spatial features of the current frame image to the spatial features of the next frame image to obtain the fused spatial features; Based on the fused spatial features, it is detected whether there is limb intersection between persons in the video stream.

4. The method for identifying fighting behavior based on video stream data according to claim 3, characterized in that: The video analysis network module is used to obtain the spatial features of each frame image in the video stream respectively, and the spatial features of the current frame image are subjected to time offset processing on the spatial features of the next frame image to obtain the fused spatial features, including: The video analysis network module is used to obtain the spatial features of eight frames in the video stream respectively, and the feature information of 1 / 8 in the current frame is temporally offset with the spatial features of the corresponding position in the next frame to obtain the fused spatial features.

5. The method for identifying fighting behavior based on video stream data according to claim 3, characterized in that: The method of using a video analysis network model to detect whether there is a body intersection between persons in a video stream includes: A video analysis network model is used to calculate the fighting confidence and non-fighting confidence of whether the persons in each frame have body intersections, and the total fighting confidence and total non-fighting confidence are calculated based on the fighting confidence and non-fighting confidence; The total fighting confidence and the total non-fighting confidence are normalized, and a judgment category is output, and whether there is a limb intersection between persons in the video stream is determined according to the judgment category.

6. The method for identifying fighting behavior based on video stream data according to claim 1, characterized in that: Outputting the fight warning information according to the body intersection person labels and associated information includes: The personnel coordinate information, fighting behavior label, fighting time, and number of personnel are output as the fighting alarm information.

7. A fighting behavior recognition device based on video stream data, characterized in that: include: Configuration module, used to configure the camera's working hours and storage settings, and create a video stream capture task based on working hours for each camera; The recognition module is used to detect whether there is a body intersection between people in the video stream using a video analysis network model, and if there is a body intersection, a body intersection person label is assigned; The alarm module is used to output fight alarm information according to the body intersection person labels and related information; and encapsulate the fight alarm information and upload it to the alarm database.

8. The fighting behavior recognition device based on video stream data according to claim 7 is characterized in that: The recognition module is also used to use the video analysis network module to obtain the spatial features of each frame image in the video stream, and perform time offset processing on the spatial features of the current frame image to the spatial features of the next frame image to obtain fused spatial features; Based on the fused spatial features, it is detected whether there is limb intersection between persons in the video stream.

9. An electronic device, characterized in that: include: at least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the fighting behavior recognition method based on video stream data as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is run by a computer, the fighting behavior recognition method based on video stream data as claimed in any one of claims 1 to 6 is executed.

Images