An abnormal behavior detection method, device and equipment based on video stream and medium

Abstract

This invention discloses a method, apparatus, device, and medium for detecting abnormal behavior based on video streams. The method involves acquiring the video stream to be detected in real time and performing image acquisition on the video stream according to a preset image acquisition method to obtain an image stream to be detected. If the image stream to be detected meets the input conditions of a classification model, it is input into a pre-trained image stream classification model to obtain the image stream category and the confidence score corresponding to that category. If, based on the image stream category and the confidence score, the type of the video stream to be detected is determined to be abnormal behavior, a real-time alarm is triggered. This solves the technical problem of inaccurately identifying and alarming on fighting behavior in large-scale video surveillance scenarios, improves the accuracy of abnormal behavior detection, and can better maintain social order.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an abnormal behavior detection method and device based on video stream, equipment and medium. BACKGROUND

[0002] In real life, fighting incidents occur from time to time. Fighting not only affects environmental order, but also causes personal injury. The traditional fighting detection method mainly relies on manual observation or video monitoring. This method is not only inefficient, but also prone to missed detection or false detection, and has a certain lag, so it is difficult for relevant personnel to alarm in the first time. With the continuous development of artificial intelligence, intelligent monitoring system has become an important part of modern city safety management. In a large-scale video monitoring scene, how to use artificial intelligence method to identify fighting and brawling abnormal behavior in real time, and give real-time warning according to the detected fighting and brawling abnormal behavior, and remind the security personnel to respond in time, is very important to maintain social order.

[0003] The inventor found that the prior art has the following defects in the process of implementing the present application: at present, static pictures collected are generally analyzed to determine whether there is a fighting behavior, but since fighting has a strong dependence on time sequence, using single frame to judge is prone to missed detection and false detection. In addition, the posture information of each person can also be analyzed through skeleton posture flow to determine whether there is a fighting behavior. However, in the monitoring scene, there are phenomena such as small target, wide field of view, and target overlap. Only extracting skeleton sequence to construct spatio-temporal convolution graph ignores other key spatial semantic features, and the robustness is very poor. Moreover, when multiple people overlap, the skeleton posture information detection effect is poor. SUMMARY

[0004] The present application provides an abnormal behavior detection method and device based on video stream, which can improve the accuracy of detecting abnormal behavior and better maintain social order.

[0005] According to one aspect of the present application, an abnormal behavior detection method based on video stream is provided, which comprises:

[0006] Real-time acquisition of the video stream to be detected, and image acquisition of the video stream to be detected according to a preset image acquisition method to obtain a picture stream to be detected;

[0007] The picture stream to be detected includes a first number of images.

[0008] If it is determined that the picture stream to be detected meets the classification model input condition, the picture stream to be detected is input into a pre-trained picture stream classification model to obtain a picture stream category and a confidence degree corresponding to the picture stream category.

[0009] If the picture stream category and the confidence level are determined, the type of the video stream to be detected is determined as abnormal behavior, and a real-time alarm processing operation is performed.

[0010] According to another aspect of the present application, there is provided an abnormal behavior detection device based on a video stream, comprising:

[0011] A picture stream to be detected determination module is configured to acquire a video stream to be detected in real time, and perform image acquisition on the video stream to be detected according to a preset image acquisition method, to obtain a picture stream to be detected.

[0012] The picture stream to be detected includes a first number of images.

[0013] A picture stream category and confidence level determination module is configured to input the picture stream to be detected into a pre-trained picture stream classification model if it is determined that the picture stream to be detected meets a classification model input condition, to obtain a picture stream category and a confidence level corresponding to the picture stream category.

[0014] A real-time alarm processing module is configured to determine the type of the video stream to be detected as abnormal behavior according to the picture stream category and the confidence level, and perform a real-time alarm processing operation.

[0015] According to another aspect of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the abnormal behavior detection method based on a video stream according to any one of the embodiments of the present application when executing the computer program.

[0016] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement the abnormal behavior detection method based on a video stream according to any one of the embodiments of the present application when executed.

[0017] The technical scheme of the embodiment of the present application acquires a video stream to be detected in real time, and acquires images of the video stream to be detected according to a preset image acquisition method, to obtain a picture stream to be detected; if it is determined that the picture stream to be detected meets a classification model input condition, the picture stream to be detected is input into a pre-trained picture stream classification model, to obtain a picture stream category and a confidence degree corresponding to the picture stream category; if the picture stream category and the confidence degree are determined, it is determined that the type of the video stream to be detected is an abnormal behavior, and a real-time alarm processing operation is performed. The technical scheme solves the technical problem that in a large-scale video monitoring scene, fighting and brawling behaviors cannot be accurately recognized and alarmed, improves the accuracy of detecting abnormal behaviors, and better maintains social order.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flow chart of an abnormal behavior detection method based on a video stream according to the first embodiment of the present application;

[0021] Figure 2 is a structural schematic diagram of an abnormal behavior detection device based on a video stream according to the second embodiment of the present application;

[0022] Figure 3 is a structural schematic diagram of an electronic device according to the third embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0024] It is to be understood that the terminology "target", "current" and the like used in the specification and the claims of the application and the above-described drawings is intended to describe differentiable objects, and is not necessarily intended to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0025] Embodiment one

[0026] Figure 1 A flowchart of a video stream-based abnormal behavior detection method is provided for the first embodiment of the application. The embodiment can be applied to the identification and alarm of fighting behavior in a large-scale video monitoring scene. The method can be executed by a video stream-based abnormal behavior detection device, which can be implemented in the form of hardware and / or software.

[0027] Correspondingly, as shown in Figure 1 the method comprises:

[0028] S110, real-time acquisition of a video stream to be detected, and image acquisition of the video stream to be detected according to a preset image acquisition method, to obtain a picture stream to be detected.

[0029] The picture stream to be detected includes a first number of images.

[0030] The video stream to be detected can be a video stream collected by a camera. The image acquisition method can be a method of image acquisition of the video stream according to a certain image acquisition period. The picture stream to be detected can be a picture stream composed of multiple pictures obtained by image acquisition of the video stream to be detected.

[0031] In this embodiment, the video stream to be detected needs to be acquired in real time. Generally, the time length of the collected video stream to be detected is 5-10 seconds. According to the preset image acquisition method, the image acquisition method can be set to collect 8 images according to a certain image acquisition period, which is not limited here. The 8 images constitute the picture stream to be detected. It can be understood that the picture stream to be detected includes 8 images.

[0032] In addition, the image acquisition method can first acquire the video time length of the to-be-detected video stream, match the video time length with a preset time period mapping table, determine an image acquisition period corresponding to the to-be-detected video stream, and then acquire images of the to-be-detected video stream according to the image acquisition period, so as to ensure that a certain number of images can be acquired. It can be understood that the image threshold of the to-be-detected picture stream can be preset here. Since multiple images are input, it is better to judge whether there is abnormal behavior, and the accuracy of system alarm can be further improved.

[0033] Optionally, before the real-time acquisition of the to-be-detected video stream and the image acquisition of the to-be-detected video stream according to the preset image acquisition method to obtain the to-be-detected picture stream, the method further comprises: acquiring a historical abnormal behavior video stream, and acquiring images of the historical abnormal behavior video stream by the preset image acquisition method to obtain a historical abnormal behavior picture stream; performing data preprocessing and data enhancement processing on the historical abnormal behavior picture stream to obtain a standard historical abnormal behavior picture stream; wherein the types of the historical abnormal behavior picture stream include: a first-grade non-compliance behavior picture stream type, a second-grade non-compliance behavior picture stream type, a third-grade non-compliance behavior picture stream type, a fourth-grade non-compliance behavior picture stream type, and a compliance behavior picture stream type; the first-grade non-compliance behavior picture stream type and the third-grade non-compliance behavior picture stream type correspond to a first-grade abnormal behavior category; the second-grade non-compliance behavior picture stream type and the fourth-grade non-compliance behavior picture stream type correspond to a second-grade abnormal behavior category; the compliance behavior picture stream type corresponds to a normal behavior category; the severity of the first-grade non-compliance behavior picture stream type is lower than that of the second-grade non-compliance behavior picture stream type; the severity of the second-grade non-compliance behavior picture stream type is lower than that of the third-grade non-compliance behavior picture stream type; the severity of the third-grade non-compliance behavior picture stream type is lower than that of the fourth-grade non-compliance behavior picture stream type; the standard historical abnormal behavior picture streams of various types are respectively input into an initial picture stream classification model for model training, and the model is optimized according to a cosine annealing strategy algorithm, so as to train the picture stream classification model; wherein the picture stream classification model is a Resnet50_vd model with a time sequence moving module.

[0034] The first level non-compliant behavior picture stream type can be a type in which two people exist on the picture and the two people have abnormal behaviors, specifically, the two people are in a mutual fighting state. The second level non-compliant behavior picture stream type can be a type in which two people exist on the picture, the two people have abnormal behaviors, and carry an instrument, specifically, the two people are in a mutual fighting state with an instrument. The third level non-compliant behavior picture stream type can be a type in which more than two people exist on the picture and have abnormal behaviors, specifically, the more than two people are in a mutual fighting state. The fourth level non-compliant behavior picture stream type can be a type in which more than two people exist on the picture, have abnormal behaviors, and carry an instrument, specifically, the more than two people are in a mutual fighting state with an instrument. The compliant behavior picture stream type can be a type in a normal state, for example, one person can exist, or two people and more than two people exist, but there is no abnormal behavior.

[0035] Specifically, the first level abnormal behavior category can be a category of two people or more than two people and having abnormal behaviors, and the first level abnormal behavior category is a behavior category determined by the picture stream classification model. The second level abnormal behavior category can be a category of two people or more than two people, having abnormal behaviors, and carrying an instrument, and the second level abnormal behavior category is a behavior category determined by the picture stream classification model. It can be understood that the severity of the second level abnormal behavior category is higher than that of the first level abnormal behavior category, and therefore the confidence threshold corresponding to the second level abnormal behavior category is also relatively high.

[0036] In this embodiment, a historical abnormal behavior video stream needs to be obtained, and image acquisition, data preprocessing, and data enhancement processing are performed on the historical abnormal behavior video stream to obtain a standard historical abnormal behavior picture stream. Specifically, the length and width corresponding to the collected image are adjusted. For example, the width of the image can be adjusted to 320 pixels, and then the length is adjusted according to the scaling scale of the width to obtain a data preprocessing image.

[0037] Further, data enhancement processing is performed on the data preprocessing image. Specifically, multi-scale cropping can be performed first, that is, a cropping scale is randomly selected from multiple scales, and the starting position and width and height of the specific cropping are calculated to crop a fixed region in the original image sequence. Then, the image is randomly flipped, and the images in the image sequence are flipped at a random angle.

[0038] In addition, the specific process of video enhancement processing on the historical abnormal behavior video stream is as follows: two videos with the same label are superimposed according to a certain weight to form a new input video, and image acquisition and other operations are performed on the new input video, which can effectively improve the anti-interference ability of the network in space and time.

[0039] Further, each type of standard historical abnormal behavior picture stream is input into an initial picture stream classification model for model training, and the model is optimized according to a cosine annealing strategy algorithm, and the picture stream classification model is trained.

[0040] Specifically, after the feature extractor corresponding to the initial picture stream classification model obtains the feature representation of the input picture stream, the feature representation is input into the classifier to obtain the probability value of each video action, and the output probability value and the real label of the sample are used to construct a loss function for training the initial picture stream classification model. Multiple rounds of training are performed to continuously adjust the parameters to achieve an ideal result.

[0041] Correspondingly, the classification process specifically includes averaging the multi-frame level features corresponding to the historical abnormal behavior picture stream to obtain picture stream features, and then classifying through the classifier. Further, the learning rate is adjusted using the cosine annealing strategy algorithm to continuously optimize the model. In addition, the Label smooth method can be used to convert the 0-1 hard label of the real class into a soft label to achieve the purpose of regularization, prevent overfitting, and improve the generalization ability of the model. The picture stream classification model can be trained.

[0042] S120, if it is determined that the to-be-detected picture stream meets the classification model input condition, inputting the to-be-detected picture stream into the pre-trained picture stream classification model to obtain a picture stream category and a confidence degree corresponding to the picture stream category.

[0043] The classification model input condition can be a pre-set condition for inputting the picture stream classification model. The picture stream category can include a first-level abnormal behavior category, a second-level abnormal behavior category, and a normal behavior category. The confidence degree can be a degree of confidence in belonging to a certain picture stream category.

[0044] For example, if the to-be-detected picture stream meets the classification model input condition, it is input into the pre-trained picture stream classification model to obtain a picture stream category and a confidence degree corresponding to the picture stream category. Assuming that the obtained picture stream category is a first-level abnormal behavior category, and the obtained confidence degree is 0.75.

[0045] Optionally, the determination that the to-be-detected picture stream meets the classification model input condition includes inputting the to-be-detected picture stream into a human body detection model for human body detection to determine a second number of human body images, comparing the first number and the second number, and determining whether the to-be-detected picture stream meets the classification model input condition according to the comparison result.

[0046] For example, it is assumed that the picture stream to be detected contains 8 frames of images, i.e., the first quantity is 8. The 8 frames of images forming the picture stream to be detected are input into the human body detection model for human body detection, and the second quantity of human body images is obtained, which is assumed to be 4 frames. The 8 frames and 4 frames are compared to obtain a comparison result, and the comparison result is used to determine whether the picture stream to be detected meets the classification model input condition.

[0047] Optionally, the comparison between the first quantity and the second quantity and the determination of whether the picture stream to be detected meets the classification model input condition according to the comparison result include: calculating a comparison result K according to a formula ; wherein M1 represents the first quantity, and M2 represents the second quantity; and comparing the comparison result with a preset classification model input condition threshold to determine whether the picture stream to be detected meets the classification model input condition.

[0048] For the foregoing example, it is assumed that the preset classification model input condition threshold is 0.25. According to the first quantity and the second quantity, the value of K can be calculated as 0.5, and because 0.5 is greater than 0.25, it can be determined whether the picture stream to be detected meets the classification model input condition.

[0049] It can be understood that when the first quantity is 8 frames, the second quantity needs to be greater than or equal to 2 frames to meet the comparison result greater than the preset classification model input condition threshold. That is, if the number of frames in which the human body is detected in the 8 frames of pictures is less than 2 frames, the model is not sent to the model for the fighting action determination. If the number of frames in which the human body is detected in the 8 frames of images is not less than 2 frames, the model is sent to the model for the fighting action determination, i.e., the picture stream to be detected meets the classification model input condition.

[0050] S130, if according to the picture stream category and the confidence, it is determined that the type of the video stream to be detected is an abnormal behavior, and a real-time alarm processing operation is performed.

[0051] In this embodiment, after obtaining the picture stream category and the confidence corresponding to the current video stream to be detected, the historical picture stream category and the confidence are also combined to determine whether the type of the video stream to be detected is an abnormal behavior. If it is an abnormal behavior, a real-time alarm processing operation is performed.

[0052] Optionally, if the picture stream category and the confidence level are determined, the type of the video stream to be detected is determined as abnormal behavior, and a real-time alarm processing operation is performed, comprising: if the picture stream category is a first-level abnormal behavior category, a first-level abnormal behavior threshold corresponding to the first-level abnormal behavior category is obtained; a previous picture stream category and a previous confidence level corresponding to the video stream to be detected are obtained; if the previous picture stream category and the picture stream category are the same, and the previous confidence level and the confidence level both satisfy the first-level abnormal behavior threshold, the type of the video stream to be detected is determined as abnormal behavior, and the real-time alarm processing operation is performed.

[0053] The first-level abnormal behavior threshold can be a pre-set threshold of the first-level abnormal behavior, and is used to describe the size of the threshold of the confidence level.

[0054] In the previous example, it is assumed that the current picture stream category is the first-level abnormal behavior category, and the confidence level is 0.75. Because the current picture stream category is the first-level abnormal behavior category, the obtained first-level abnormal behavior threshold is 0.8.

[0055] Further, the previous picture stream category and the previous confidence level corresponding to the video stream to be detected are obtained. Specifically, it is assumed that the previous picture stream category is the first-level abnormal behavior category, and the previous confidence level is 0.82.

[0056] Correspondingly, because the current picture stream category and the previous picture stream category are both the first-level abnormal behavior category, it is then necessary to determine whether the previous confidence level and the confidence level both satisfy the first-level abnormal behavior threshold. Because 0.75<0.8 and 0.82>0.8, it is determined that the first-level abnormal behavior threshold is not satisfied, and the type of the video stream to be detected is determined as normal behavior.

[0057] In addition, it is assumed that the next picture stream category is the first-level abnormal behavior category, and the previous confidence level is 0.85. Specifically, because the current picture stream category and the next picture stream category are both the first-level abnormal behavior category, it is then necessary to determine whether the next confidence level and the confidence level both satisfy the first-level abnormal behavior threshold. Because 0.82>0.8 and 0.85>0.8, it is determined that the first-level abnormal behavior threshold is satisfied, and the type of the video stream to be detected is determined as abnormal behavior, and the real-time alarm processing operation is performed.

[0058] Optionally, if the picture stream category and the confidence level are determined, the type of the video stream to be detected is determined as abnormal behavior, and a real-time alarm processing operation is performed, comprising: if the picture stream category is a second-level abnormal behavior category, a second-level abnormal behavior threshold corresponding to the second-level abnormal behavior category is obtained; wherein the severity of the second-level abnormal behavior category is higher than that of the first-level abnormal behavior category; the second-level abnormal behavior threshold is greater than the first-level abnormal behavior threshold; the previous picture stream category corresponding to the video stream to be detected, the previous confidence level, the previous two picture stream categories and the previous two confidence levels are obtained; if the previous picture stream category and the previous two picture stream categories are the same as the picture stream category, and the previous confidence level, the previous two confidence levels and the confidence level all satisfy the second-level abnormal behavior threshold, the type of the video stream to be detected is determined as abnormal behavior, and a real-time alarm processing operation is performed.

[0059] In this embodiment, since the severity of the second-level abnormal behavior category is higher than that of the first-level abnormal behavior category, it can be determined that the second-level abnormal behavior threshold is greater than the first-level abnormal behavior threshold. Therefore, for the second-level abnormal behavior category, the picture stream category and the confidence level are compared for three times in succession.

[0060] For example, it is assumed that the current picture stream category is a second-level abnormal behavior category, and the confidence level is 0.88. It is assumed that the previous picture stream category is a second-level abnormal behavior category, and the previous confidence level is 0.86. It is assumed that the previous two picture stream categories are second-level abnormal behavior categories, and the previous two confidence levels are 0.9.

[0061] Similarly, the current picture stream category, the previous picture stream category and the previous two picture stream categories need to be compared first. If they are all second-level abnormal behavior categories, the confidence levels are compared.

[0062] It is assumed that the second-level abnormal behavior threshold is 0.85. Since 0.88>0.85, 0.86>0.85, and 0.9>0.85, it can be determined that the previous confidence level, the previous two confidence levels and the confidence level all satisfy the second-level abnormal behavior threshold, and the type of the video stream to be detected is determined as abnormal behavior, and a real-time alarm processing operation is performed.

[0063] Optionally, after the picture stream to be detected is input into the pre-trained picture stream classification model to obtain the picture stream category and the confidence level corresponding to the picture stream category, if it is determined that the picture stream category is a normal behavior category, the type of the video stream to be detected is determined as normal behavior.

[0064] In the embodiment, the picture stream category includes the normal behavior category, the first-level abnormal behavior category, and the second-level abnormal behavior category. When the picture stream category is determined as the normal behavior category, it can be determined that the type of the video stream to be detected is normal behavior.

[0065] The technical solution of the embodiment of the application comprises the following steps: acquiring a video stream to be detected in real time, performing image acquisition on the video stream to be detected according to a preset image acquisition method, obtaining a picture stream to be detected, determining that the picture stream to be detected meets a classification model input condition, inputting the picture stream to be detected into a pre-trained picture stream classification model, obtaining a picture stream category and a confidence degree corresponding to the picture stream category, determining that the type of the video stream to be detected is abnormal behavior according to the picture stream category and the confidence degree, and performing real-time alarm processing operation. The technical problem that fighting and brawling behavior cannot be accurately recognized and alarmed in a large-scale video monitoring scene is solved, the accuracy of detecting abnormal behavior is improved, and social order can be better maintained.

[0066] Embodiment two

[0067] Figure 2 A structure schematic diagram of an abnormal behavior detection device based on a video stream provided by the embodiment two of the application. The abnormal behavior detection device based on a video stream provided by the embodiment can be realized by software and / or hardware, and can be configured in a terminal device or a server to realize the abnormal behavior detection method based on a video stream in the embodiment of the application. As shown in the figure, the device comprises a picture stream to be detected determination module 210, a picture stream category and confidence degree determination module 220, and a real-time alarm processing module 230. Figure 2

[0068] The picture stream to be detected determination module 210 is configured to acquire a video stream to be detected in real time, perform image acquisition on the video stream to be detected according to a preset image acquisition method, and obtain a picture stream to be detected.

[0069] The picture stream to be detected comprises a first number of images.

[0070] The picture stream category and confidence degree determination module 220 is configured to input the picture stream to be detected into a pre-trained picture stream classification model if it is determined that the picture stream to be detected meets a classification model input condition, obtain a picture stream category and a confidence degree corresponding to the picture stream category.

[0071] The real-time alarm processing module 230 is configured to determine that the type of the video stream to be detected is abnormal behavior according to the picture stream category and the confidence degree, and perform real-time alarm processing operation. ​

[0072] The technical scheme of the embodiment of the application acquires a video stream to be detected in real time, acquires images from the video stream to be detected according to a preset image acquisition method, obtains a picture stream to be detected, determines whether the picture stream to be detected meets a classification model input condition, inputs the picture stream to be detected into a pre-trained picture stream classification model if the picture stream to be detected meets the classification model input condition, obtains a picture stream category and a confidence degree corresponding to the picture stream category, and determines that the type of the video stream to be detected is abnormal behavior according to the picture stream category and the confidence degree, and performs real-time alarm processing. The technical scheme solves the technical problem that fighting behavior cannot be accurately recognized and alarmed in a large-scale video monitoring scene, improves the accuracy of detecting abnormal behavior, and better maintains social order.

[0073] Optionally, the picture stream category and confidence degree determination module 220 can be specifically configured to input the picture stream to be detected into a human body detection model to detect human bodies, determine a second number of human body images, compare the first number and the second number, and determine whether the picture stream to be detected meets the classification model input condition according to a comparison result.

[0074] Optionally, the picture stream category and confidence degree determination module 220 can be specifically configured to calculate a comparison result K according to a formula wherein M1 represents the first number, M2 represents the second number, compare the comparison result with a preset classification model input condition threshold, and determine whether the picture stream to be detected meets the classification model input condition.

[0075] Optionally, the real-time alarm processing module 230 can be specifically configured to acquire a first-level abnormal behavior threshold corresponding to a first-level abnormal behavior category if the picture stream category is the first-level abnormal behavior category, acquire a previous picture stream category and a previous confidence degree corresponding to the video stream to be detected, determine that the type of the video stream to be detected is abnormal behavior and perform real-time alarm processing if the previous picture stream category is the same as the picture stream category and the previous confidence degree and the confidence degree both meet the first-level abnormal behavior threshold.

[0076] Optionally, the real-time alarm processing module 230 can be specifically configured to: if the picture stream category is a second-level abnormal behavior category, obtain a second-level abnormal behavior threshold corresponding to the second-level abnormal behavior category; the severity of the second-level abnormal behavior category is higher than that of the first-level abnormal behavior category; the second-level abnormal behavior threshold is greater than the first-level abnormal behavior threshold; obtain a previous picture stream category, a previous confidence, a previous two picture stream categories and a previous two confidences corresponding to the video stream to be detected; if the previous picture stream category and the previous two picture stream categories are the same as the picture stream category, and the previous confidence, the previous two confidences and the confidence all satisfy the second-level abnormal behavior threshold, it is determined that the type of the video stream to be detected is abnormal behavior, and a real-time alarm processing operation is performed.

[0077] Optionally, the normal behavior category determination module can be specifically configured to: after the picture stream to be detected is input into the pre-trained picture stream classification model to obtain the picture stream category and the confidence corresponding to the picture stream category if the picture stream to be detected satisfies the classification model input condition, if it is determined that the picture stream category is a normal behavior category, it is determined that the type of the video stream to be detected is normal behavior.

[0078] Optionally, the picture stream classification model training module can be specifically configured to: acquire a historical abnormal behavior video stream before acquiring the video stream to be detected in real time, and acquire the historical abnormal behavior video stream through a preset image acquisition method to obtain a historical abnormal behavior picture stream; perform data preprocessing and data enhancement processing on the historical abnormal behavior picture stream to obtain a standard historical abnormal behavior picture stream; wherein the types of the historical abnormal behavior picture stream include: a first-grade non-compliance behavior picture stream type, a second-grade non-compliance behavior picture stream type, a third-grade non-compliance behavior picture stream type, a fourth-grade non-compliance behavior picture stream type, and a compliance behavior picture stream type; the first-grade non-compliance behavior picture stream type and the third-grade non-compliance behavior picture stream type correspond to a first-grade abnormal behavior category; the second-grade non-compliance behavior picture stream type and the fourth-grade non-compliance behavior picture stream type correspond to a second-grade abnormal behavior category; the compliance behavior picture stream type corresponds to a normal behavior category; the severity of the first-grade non-compliance behavior picture stream type is lower than that of the second-grade non-compliance behavior picture stream type; the severity of the second-grade non-compliance behavior picture stream type is lower than that of the third-grade non-compliance behavior picture stream type; the severity of the third-grade non-compliance behavior picture stream type is lower than that of the fourth-grade non-compliance behavior picture stream type; the standard historical abnormal behavior picture stream of each type is input into an initial picture stream classification model for model training, and the model is optimized according to a cosine annealing strategy algorithm to obtain the picture stream classification model; wherein the picture stream classification model is a Resnet50_vd model with a time sequence moving module.

[0079] The abnormal behavior detection device based on a video stream provided in the embodiments of the present application can execute the abnormal behavior detection method based on a video stream provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0080] Embodiment three

[0081] Figure 3 A structural schematic diagram of an electronic device 10 that can be used to implement embodiment three of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the applications described and / or claimed in this document.

[0082] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0083] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0084] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as anomaly behavior detection methods based on video streams.

[0085] In some embodiments, the video stream-based abnormal behavior detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video stream-based abnormal behavior detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video stream-based abnormal behavior detection method by any other suitable means (e.g., by means of firmware).

[0086] The method comprises: acquiring a to-be-detected video stream in real time, and performing image acquisition on the to-be-detected video stream according to a preset image acquisition method to obtain a to-be-detected picture stream; if it is determined that the to-be-detected picture stream meets a classification model input condition, inputting the to-be-detected picture stream into a pre-trained picture stream classification model to obtain a picture stream category and a confidence degree corresponding to the picture stream category; and if the picture stream category and the confidence degree are determined, determining that a type of the to-be-detected video stream is an abnormal behavior, and performing real-time alarm processing operation.

[0087] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0088] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a standalone software package, or entirely on a remote machine or server.

[0089] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0091] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0092] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0093] It should be understood that the various forms of flow shown above can be reordered, additional or deleted steps. For example, each step described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0094] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0095] Example four

[0096] The embodiment four of the present application also provides a computer readable storage medium comprising computer readable instructions for executing a video stream based abnormal behavior detection method when executed by a computer processor, the method comprising: acquiring a video stream to be detected in real time, and performing image acquisition on the video stream to be detected according to a preset image acquisition method to obtain a picture stream to be detected; if it is determined that the picture stream to be detected meets a classification model input condition, inputting the picture stream to be detected into a pre-trained picture stream classification model to obtain a picture stream category and a confidence degree corresponding to the picture stream category; and if the type of the video stream to be detected is determined to be abnormal behavior according to the picture stream category and the confidence degree, performing real-time alarm processing operation.

[0097] Of course, the computer executable instructions of the computer readable storage medium provided by the embodiment of the present application are not limited to the method operations as described above, but can also perform related operations in the video stream based abnormal behavior detection method provided by any embodiment of the present application.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a FLASH memory, a hard disk, or an optical disc, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0099] It is worth noting that in the above embodiments of the video stream-based abnormal behavior detection device, each unit and module included is only divided according to functional logic, but is not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy mutual differentiation, and do not limit the protection scope of the present application.

[0100] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting abnormal behavior based on video stream, characterized in that, The method comprises the following steps: real-time acquisition of a video stream to be detected, and image acquisition of the video stream to be detected according to a preset image acquisition method, to obtain a picture stream to be detected; wherein the picture stream to be detected comprises a first number of images; if it is determined that the picture stream to be detected meets the classification model input condition, then the picture stream to be detected is input into a pre-trained picture stream classification model, to obtain a picture stream category and a confidence degree corresponding to the picture stream category; if the picture stream category and the confidence degree are determined, then the type of the video stream to be detected is determined to be an abnormal behavior, and real-time alarm processing operation is performed; wherein, before the real-time acquisition of the video stream to be detected, and the image acquisition of the video stream to be detected according to the preset image acquisition method, to obtain the picture stream to be detected, the method further comprises the following steps: acquisition of a historical abnormal behavior video stream, and video acquisition of the historical abnormal behavior video stream through a preset image acquisition method, to obtain a historical abnormal behavior picture stream; data preprocessing and data enhancement processing of the historical abnormal behavior picture stream, to obtain a standard historical abnormal behavior picture stream; wherein the type of the historical abnormal behavior picture stream comprises: a first grade non-compliance behavior picture stream type, a second grade non-compliance behavior picture stream type, a third grade non-compliance behavior picture stream type, a fourth grade non-compliance behavior picture stream type, and a compliance behavior picture stream type; the first grade non-compliance behavior picture stream type and the third grade non-compliance behavior picture stream type correspond to a first grade abnormal behavior category; the second grade non-compliance behavior picture stream type and the fourth grade non-compliance behavior picture stream type correspond to a second grade abnormal behavior category; the compliance behavior picture stream type corresponds to a normal behavior category; the severity of the first grade non-compliance behavior picture stream type is lower than that of the second grade non-compliance behavior picture stream type; the severity of the second grade non-compliance behavior picture stream type is lower than that of the third grade non-compliance behavior picture stream type; the severity of the third grade non-compliance behavior picture stream type is lower than that of the fourth grade non-compliance behavior picture stream type; input of each type of standard historical abnormal behavior picture stream into an initial picture stream classification model for model training, and optimization of the model according to a cosine annealing strategy algorithm, to train the picture stream classification model; wherein the picture stream classification model is a Resnet50_vd model with a time sequence moving module; wherein the first grade non-compliance behavior picture stream type is a type in which two people exist on a picture and both of them have abnormal behaviors; the second grade non-compliance behavior picture stream type is a type in which two people exist on a picture, both of them have abnormal behaviors, and carry instruments; the third grade non-compliance behavior picture stream type is a type in which more than two people exist on a picture and have abnormal behaviors; the fourth grade non-compliance behavior picture stream type is a type in which more than two people exist on a picture, have abnormal behaviors, and carry instruments.

2. The method of claim 1, wherein, The determination that the picture stream to be detected meets the classification model input condition comprises: human body detection of the picture stream to be detected into a human body detection model, to determine a second number of human body images; The first quantity and the second quantity are compared, and whether the picture stream to be detected meets the classification model input condition is determined according to a comparison result.

3. The method of claim 2, wherein, The comparison of the first quantity and the second quantity and the determination of whether the picture stream to be detected meets the classification model input condition according to a comparison result include: The comparison result K is calculated according to the formula ; wherein represents the first number, represents the second number; The comparison result is compared with a preset classification model input condition threshold value to determine whether the picture stream to be detected meets the classification model input condition.

4. The method of claim 1, wherein, If the picture stream category is a first-level abnormal behavior category, a first-level abnormal behavior threshold value corresponding to the first-level abnormal behavior category is obtained. The previous picture stream category and the previous confidence corresponding to the video stream to be detected are obtained. If the previous picture stream category and the picture stream category are the same, and the previous confidence and the confidence both meet the first-level abnormal behavior threshold value, it is determined that the type of the video stream to be detected is abnormal behavior, and real-time alarm processing operation is performed. The comparison of the first quantity and the second quantity and the determination of whether the picture stream to be detected meets the classification model input condition according to a comparison result include:

5. The method of claim 4, wherein, If the picture stream category is a second-level abnormal behavior category, a second-level abnormal behavior threshold value corresponding to the second-level abnormal behavior category is obtained. The severity of the second-level abnormal behavior category is higher than that of the first-level abnormal behavior category, and the second-level abnormal behavior threshold value is greater than the first-level abnormal behavior threshold value. The previous picture stream category, the previous confidence, the previous two picture stream categories and the previous two confidences corresponding to the video stream to be detected are obtained. If the previous picture stream category and the previous two picture stream categories are the same as the picture stream category, and the previous confidence, the previous two confidences and the confidence all meet the second-level abnormal behavior threshold value, it is determined that the type of the video stream to be detected is abnormal behavior, and real-time alarm processing operation is performed. After the picture stream to be detected is input into the pre-trained picture stream classification model to obtain the picture stream category and the confidence corresponding to the picture stream category, the method further includes:

6. The method of claim 1, wherein, If it is determined that the picture stream category is a normal behavior category, it is determined that the type of the video stream to be detected is normal behavior. The method includes:

7. An abnormal behavior detection device based on video stream, characterized in that, A picture stream to be detected determination module is configured to acquire a video stream to be detected in real time, and perform image acquisition on the video stream to be detected according to a preset image acquisition method to obtain a picture stream to be detected. The picture stream to be detected includes a first quantity of images. A picture stream category and confidence determination module is configured to input the picture stream to be detected into a pre-trained picture stream classification model to obtain a picture stream category and a confidence corresponding to the picture stream category if it is determined that the picture stream to be detected meets a classification model input condition. ​ The real-time alarm processing module is configured to determine that the type of the video stream to be detected is abnormal behavior according to the picture stream category and the confidence, and perform real-time alarm processing operation. The picture stream classification model training module is configured to: before obtaining the picture stream to be detected by acquiring the video stream to be detected in real time and performing image acquisition on the video stream to be detected according to a preset image acquisition method, obtain a historical abnormal behavior video stream, and perform video acquisition on the historical abnormal behavior video stream by using the preset image acquisition method to obtain a historical abnormal behavior picture stream; perform data preprocessing and data enhancement processing on the historical abnormal behavior picture stream to obtain a standard historical abnormal behavior picture stream; wherein the type of the historical abnormal behavior picture stream includes: a first-level non-compliance behavior picture stream type, a second-level non-compliance behavior picture stream type, a third-level non-compliance behavior picture stream type, a fourth-level non-compliance behavior picture stream type, and a compliance behavior picture stream type; the first-level non-compliance behavior picture stream type and the third-level non-compliance behavior picture stream type correspond to a first-level abnormal behavior category; the second-level non-compliance behavior picture stream type and the fourth-level non-compliance behavior picture stream type correspond to a second-level abnormal behavior category; the compliance behavior picture stream type corresponds to a normal behavior category; the severity of the first-level non-compliance behavior picture stream type is lower than that of the second-level non-compliance behavior picture stream type; the severity of the second-level non-compliance behavior picture stream type is lower than that of the third-level non-compliance behavior picture stream type; the severity of the third-level non-compliance behavior picture stream type is lower than that of the fourth-level non-compliance behavior picture stream type; the standard historical abnormal behavior picture stream of each type is input into an initial picture stream classification model for model training, and the model is optimized according to a cosine annealing strategy algorithm to obtain the picture stream classification model; wherein the picture stream classification model is a Resnet50_vd model with a time sequence moving module. The first-level non-compliance behavior picture stream type is a type in which two people exist on a picture and the two people have abnormal behavior; the second-level non-compliance behavior picture stream type is a type in which two people exist on a picture, the two people have abnormal behavior, and carry equipment; the third-level non-compliance behavior picture stream type is a type in which more than two people exist on a picture and have abnormal behavior; and the fourth-level non-compliance behavior picture stream type is a type in which more than two people exist on a picture, have abnormal behavior, and carry equipment.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the video stream-based abnormal behavior detection method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the video stream-based abnormal behavior detection method of any one of claims 1-6 when executed.

Images