Behavior recognition method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a behavior recognition method, including: obtaining a detection frame of each object in each frame of the video sequence to be recognized; determining a first recognition result of the video sequence to be recognized based on the distribution of the detection frame in each frame; when the first recognition result satisfies a first condition, performing behavior recognition on a trajectory sequence of a group of objects to obtain a fourth recognition result; wherein the trajectory sequence of the group of objects is generated based on the video sequence to be recognized; the group of objects includes at least two target objects whose spatial distance is less than a distance threshold; determining a target recognition result of the video sequence to be recognized based on at least the first recognition result or the fourth recognition result. The embodiment of the present application also provides a behavior recognition device, equipment, and storage medium.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and is related to but not limited to behavior recognition methods and devices, equipment and storage media. Background Art

[0002] Behavior recognition in video sources is an important application in the field of computer vision. It is extremely difficult to identify abnormal events in video sources. Possible challenges include scarcity of labeled data due to low-probability events, large variance between / within classes, differences in the subjective definition of abnormal events, and low resolution of video acquisition devices.

[0003] The main problems that exist in practice include: the number of action executors of an event is uncertain (according to the number of action executors of an event, it can usually be divided into single-object behavior and group-object behavior); different types of behavioral event recognition require different spatial receptive fields (in the recognition scenario, there may be behavioral subjects with different positions and fields of view, and the location of the behavioral event that actually needs to be detected usually only occupies a small area of ​​the screen); there may be spatial overlap between the behavioral subjects that have not performed the preset behavior and the behavioral subjects that have performed the preset behavior, etc. The above problems will interfere with the behavior recognition results. Summary of the invention

[0004] The embodiments of the present application provide a behavior recognition method, apparatus, device and storage medium.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a behavior recognition method, the method comprising:

[0007] Obtaining a detection frame of each object in each frame of the video sequence to be identified;

[0008] Determining a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0009] When the first recognition result satisfies the first condition, behavior recognition is performed on the trajectory sequence of the group object to obtain a fourth recognition result; wherein the trajectory sequence of the group object is generated based on the video sequence to be recognized; and the group object includes at least two target objects whose spatial distance is less than a distance threshold;

[0010] A target recognition result of the to-be-recognized video sequence is determined based at least on the first recognition result or the fourth recognition result.

[0011] In some possible implementations, determining the target recognition result of the video sequence to be recognized at least based on the first recognition result or the fourth recognition result includes: when the first recognition result represents that no abnormal behavior has occurred in the video sequence to be recognized, determining the target recognition result of the video sequence to be recognized based on the first recognition result; or, when the first recognition result represents that abnormal behavior has occurred in the video sequence to be recognized, determining the target recognition result of the video sequence to be recognized based on the fourth recognition result.

[0012] In this way, when it is logically predicted that the first recognition result of the video sequence to be recognized represents that no abnormal behavior has occurred in the video sequence to be recognized, the first recognition result is directly used as the final target recognition result, and the video sequence to be recognized that obviously does not have abnormal behavior can be directly determined; at the same time, for the situation where abnormal behavior may exist in the video sequence to be recognized by logical judgment, the neural network is further combined for recognition, and the fourth recognition result is used as the target recognition result to improve the robustness of behavior recognition.

[0013] In some possible implementations, determining the first recognition result of the video sequence to be identified based on the distribution of the detection frames in each frame of image includes: determining a dense area in each frame of image based on the distribution of the detection frames in each frame of image; the dense area includes a central detection frame whose number of overlaps with other detection frames in each frame of image is greater than or equal to a first number threshold; and determining the first recognition result of the video sequence to be identified based on the number of objects included in the dense area in each frame of image.

[0014] In this way, by guiding the reasoning of local dense areas through the heuristic detection box density distribution and dense group position estimation method, the effective perception area of ​​the model can be increased and the retrieval range of irrelevant background can be reduced. At the same time, combined with logical prediction, the problem of insufficient robustness and accuracy of pure neural network recognition can be improved.

[0015] In some possible implementations, when the first recognition result represents that abnormal behavior has occurred in the video sequence to be recognized, behavior recognition is performed on the trajectory sequence of the group objects to obtain the fourth recognition result, including: when the first recognition result represents that abnormal behavior has occurred in the video sequence to be recognized, determining the area change value of the dense area between adjacent frames in the video sequence to be recognized; determining the second recognition result of the video sequence to be recognized based on the area change value and the change threshold of the dense area; when the second recognition result represents that no abnormal behavior has occurred in the video sequence to be recognized, behavior recognition is performed on the trajectory sequence of the group objects to obtain the fourth recognition result.

[0016] In this way, the first recognition result characterizes the situation where abnormal behavior occurs in the video sequence to be identified, and the second recognition result is further determined based on the area change value of the dense area between adjacent frames, thereby combining the two-stage heuristic logical method to improve the robustness and accuracy of the neural network recognition scheme.

[0017] In some possible implementations, when the second recognition result indicates that the video sequence to be recognized has not undergone abnormal behavior, behavior recognition is performed on the trajectory sequence of the group objects to obtain the fourth recognition result, including: when the second recognition result indicates that the video sequence to be recognized has not undergone abnormal behavior, based on the coverage of the dense area in each frame image in the video sequence to be recognized, determining the bounding box area in each frame image in the video sequence to be recognized; the bounding box areas in each frame image are located at the same position, have the same size and surround the dense area; based on the pixel changes of the bounding box area between adjacent frames in the video sequence to be recognized, determining the third recognition result of the video sequence to be recognized; when the third recognition result indicates that the video sequence to be recognized has not undergone abnormal behavior, behavior recognition is performed on the trajectory sequence of the group objects to obtain the fourth recognition result.

[0018] In this way, after making heuristic logical judgments on the video sequence to be identified through the area changes of dense areas and the pixel changes of the bounding box area after taking the union, the behavior recognition model is used for behavior recognition, which can improve the robustness and accuracy of the neural network recognition scheme.

[0019] In some possible implementations, the method further includes: determining that abnormal behavior occurs in the video sequence to be identified when the second recognition result represents that abnormal behavior occurs in the video sequence to be identified; or determining that abnormal behavior occurs in the video sequence to be identified when the third recognition result represents that abnormal behavior occurs in the video sequence to be identified.

[0020] In this way, when the second recognition result or the third recognition result judged by the logical rule represents abnormal behavior in the video sequence to be identified, the second recognition result or the third recognition result is directly used as the target recognition result of the video sequence to be identified, thereby shortening the recognition process and improving the efficiency of behavior recognition for the video sequence to be identified in which obvious abnormal behavior has occurred.

[0021] In some possible implementations, determining the third recognition result of the video sequence to be identified based on the pixel changes of the bounding box area between adjacent frames in the video sequence to be identified includes: when the pixel difference of the bounding box area between adjacent frames in the video sequence to be identified is greater than or equal to a pixel threshold, determining that the third recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the pixel difference of the bounding box area between adjacent frames in the video sequence to be identified is less than the pixel threshold, determining that the third recognition result of the video sequence to be identified is that no abnormal behavior has occurred.

[0022] In this way, by comparing the pixel change degree of the bounding box area between adjacent frames in the video sequence to be identified, it is determined whether the video sequence to be identified has abnormal behavior, and the third identification result is obtained. In this way, the video sequence to be identified that obviously has abnormal behavior is quickly determined through logical rules, shortening the identification process and enhancing the accuracy of the identification method.

[0023] In some possible implementations, determining the second recognition result of the video sequence to be identified based on the area change value and the change threshold of the dense area includes: when the area change values ​​of the dense area between the adjacent frames are greater than or equal to the change threshold, determining that the second recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the area change values ​​of the dense area between the adjacent frames are less than the change threshold, determining that the second recognition result of the video sequence to be identified is that no abnormal behavior has occurred.

[0024] In this way, by comparing the area change of the dense area between adjacent frames in the video sequence to be identified, it is determined whether the video sequence to be identified has abnormal behavior, and the second identification result is obtained. This makes it possible to quickly determine the video sequence to be identified that has obviously abnormal behavior in the case of intense movement, shorten the identification process, and enhance the accuracy of the identification method.

[0025] In some possible implementations, determining the first recognition result of the video sequence to be identified based on the number of objects included in the dense area in each frame image includes: when the number of objects included in the dense area in each frame image is greater than or equal to 2, determining that the first recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the number of objects included in the dense area in each frame image is less than 2, determining that the first recognition result of the video sequence to be identified does not have abnormal behavior.

[0026] In this way, by determining whether the number of objects included in the dense area of ​​each frame of the video sequence to be identified is greater than or equal to 2, it is determined whether the video sequence to be identified has abnormal behavior, and the first identification result is obtained. In this way, in the case where the number of objects does not meet the requirements, the video sequence to be identified that obviously will not have abnormal behavior of group objects is quickly determined, shortening the identification process and enhancing the accuracy of the identification method.

[0027] In some possible implementations, determining the dense area in each frame of image based on the distribution of the detection frames in each frame of image includes: generating an adjacency matrix corresponding to each frame of image according to the intersection-and-union ratio between every two detection frames in each frame of image; taking the detection frames in the adjacency matrix whose matching times are greater than or equal to a second number threshold as the central detection frames in each frame of image; wherein the second number threshold is greater than the first number threshold; and expanding the central detection frame in each frame of image outward at a specific ratio to obtain the dense area in each frame of image.

[0028] In this way, the center detection frame is first determined by using the distribution of detection frames in each frame, and then the center detection frame where group object behavior is predicted to occur is expanded outward, so that the expanded dense area includes all objects centered on the center detection frame and surrounding the group object behavior as much as possible. This reduces the behavior receptive field of the action execution subject and can solve the problem of small effective perception range when identifying the full-view video sequence to be identified.

[0029] In some possible implementations, the method further includes: generating a trajectory sequence of the group objects based on a dense area in each frame of the image.

[0030] In this way, the dense area in each frame of the image is processed to obtain the trajectory sequence of the group object, so that the behavior of the group object can be recognized through the behavior recognition model. At the same time, the trajectory sequence of the group object reduces the behavior receptive field of the action execution subject, which can solve the problem of small effective perception range when recognizing the video sequence to be recognized in the full field of view.

[0031] In some possible implementations, generating the trajectory sequence of the group objects based on the dense area in each frame of the image includes: determining the bounding box area in each frame of the image based on the coverage of the dense area in each frame of the image in the video sequence to be identified; the bounding box area in each frame of the image is located at the same position, has the same size and surrounds the dense area; generating the trajectory sequence of the group objects based on the timestamp of each frame of the image and the bounding box area in each frame of the image.

[0032] In this way, by determining the bounding box area in each frame and further generating the trajectory sequence of the group object, the receptive field of the group object behavior event can be reduced, thereby improving the accuracy and efficiency of behavior recognition. At the same time, the relative position loss of the group object behavior is avoided, and the performance of behavior detection with similar space but different movement rhythm is improved.

[0033] In a second aspect, an embodiment of the present application provides a behavior recognition device, including:

[0034] An acquisition module, used for acquiring a detection frame of each object in each frame of the video sequence to be identified;

[0035] A first determination module, configured to determine a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0036] an identification module, configured to perform behavior identification on a trajectory sequence of a group of objects to obtain a fourth identification result when the first identification result satisfies a first condition; wherein the trajectory sequence of the group of objects is generated based on the video sequence to be identified; and the group of objects includes at least two target objects whose spatial distance is less than a distance threshold;

[0037] The second determination module is used to determine a target recognition result of the to-be-recognized video sequence based at least on the first recognition result or the fourth recognition result.

[0038] In a third aspect, an embodiment of the present application provides a device including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the above-mentioned behavior recognition method are implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned behavior recognition method when executed by a processor.

[0040] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0041] In an embodiment of the present application, the detection frame of each object in each frame of the image is first obtained, and then the first recognition result of the video sequence to be recognized is determined based on the distribution of the detection frame in each frame of the image, and then, when the first recognition result satisfies the first condition, the trajectory sequence of the group object generated based on the video sequence to be recognized is subjected to behavior recognition to obtain a fourth recognition result; finally, based on at least the first recognition result or the fourth recognition result, the target recognition result of the video sequence to be recognized is determined. The embodiment of the present application proposes a behavior recognition method based on the fusion of heuristic logic and deep learning, which can support the needs of abnormal behavior detection of group objects and improve the robustness and accuracy of pure neural network recognition schemes. At the same time, different from the behavior recognition method based on the full-image image sequence, the embodiment of the present application converts the video classification problem into the recognition problem of the trajectory sequence of the group object, which effectively improves the model's perception of the target area in the video source and greatly reduces the retrieval scope and calculation amount. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0043] Figure 1A A schematic diagram of a network architecture of a behavior recognition method provided in an embodiment of the present application;

[0044] Figure 1B A flowchart of a behavior recognition method provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a behavior recognition method provided in an embodiment of the present application;

[0046] Figure 3 A flowchart of a behavior recognition method provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of a behavior recognition method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a process for generating a trajectory sequence of a group of objects provided in an embodiment of the present application;

[0049] Figure 6 A logical flow chart of a multi-person behavior recognition method provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of a trajectory sequence of multiple people provided in an embodiment of the present application;

[0051] Figure 8 A schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present application;

[0052] Fig. 9 A schematic diagram of a hardware entity of a behavior recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following embodiments are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0054] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0055] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0057] Behavior detection in videos is an important problem in the field of computer vision and has a wide range of applications in the field of smart cities, such as detecting illegal behaviors, traffic accidents, and some uncommon events. Most devices that collect video sources only record the dynamics of each moment, but do not have the ability to automatically identify (often requiring special personnel to be responsible for manual inspection). Due to the huge number of videos, it is obviously unrealistic to filter the content in the video by manpower alone. It is necessary to use computer vision and deep learning technology to automatically detect behavioral events occurring in the video.

[0058] Figure 1A A network architecture diagram of a behavior recognition method provided in an embodiment of the present application is shown as follows: Figure 1A As shown, the network architecture includes: camera 101, object detection module 102, logic recognition module 103 and model recognition module 104: wherein, object detection module 102, logic recognition module 103 and model recognition module 104 can be set in server 100. To realize supporting an exemplary application, camera 101 establishes communication connection with server 100 through network. Video in a specific scene is collected by camera 101, and then a video sequence 11 to be identified is sampled, i.e., a multi-frame image containing a group of objects, and the video sequence 11 to be identified is input into object detection module 102; object detection module 102 can make full use of relevant detection algorithms such as inter-frame difference method, background subtraction method, optical flow method, etc. to realize the positioning and analysis of each object in the video sequence 11 to be identified, and obtain a multi-frame image 12 with detection results (detection frame of each object is marked); then, the multi-frame image 12 with detection results is subjected to the method of density distribution of detection frame and heuristic dense group position estimation based on the logic recognition module 103. The reasoning of dense areas is guided to preliminarily detect the abnormal behavior of obvious group objects in the video sequence 11 to be identified, or some unlikely situations are filtered out from some geometric information at the surface level, so as to improve the recognition efficiency of the video sequence and the accuracy of behavior recognition; finally, for the video sequence 11 to be identified that does not conform to the logical detection rules, the trajectory sequence 13 of the group objects is generated based on the dense areas in each frame of the image; the trajectory sequence 13 is input into the model recognition module 104 together; the model recognition module 104 can make full use of the relevant video understanding model to perform behavior recognition on the trajectory sequence 13, and finally output the recognition result at the video level. Therefore, for the video sequence to be identified that does not conform to the logical detection rules, the neural network is used for further confirmation and judgment, which enhances the robustness of the entire recognition scheme. Based on this network architecture, a recognition method framework for group object behavior based on logical rules and deep learning can be designed.

[0059] The embodiment of the present application provides a behavior recognition method, which is applied to a server, a terminal device or other devices, wherein the terminal device includes but is not limited to a mobile phone, a laptop computer, a tablet computer, a handheld Internet device, a multimedia device, a streaming media device, a mobile Internet device, a wearable device or other types of devices.

[0060] Figure 1B A flow chart of a behavior recognition method provided in an embodiment of the present application is shown as follows: Figure 1B As shown, the method comprises at least the following steps:

[0061] Step S110, obtaining a detection frame of each object in each frame of the video sequence to be identified;

[0062] Here, the to-be-recognized video sequence can be obtained by sampling a specific video source. For example, a video shot at a fixed angle often covers a wider field of view and contains more information, such as pedestrians, vehicles, animals, buildings, and other complex background information.

[0063] It can be understood that the video sequence to be identified is a frame sequence composed of multiple frames, wherein each frame may contain at least one object or no object, and different frames may contain different objects, but all frames of the video sequence to be identified contain at least two objects in which group object behavior events may occur. The objects here can be pedestrians, moving vehicles, animals, etc., which are determined according to the actual scene in implementation, and are not limited in the embodiments of the present application.

[0064] The detection and location analysis of objects in video images can be achieved through relevant image or video processing technology. For example, the object detection algorithm pre-processes the video sequence to be identified to obtain multiple frames of images with detection frames, and then extracts the detection frame of the object in each frame. Among them, the detection algorithm, such as template matching, can be implemented for the detection of moving targets in the video through frame difference method, background subtraction method, optical flow method, etc., which is not limited in the embodiments of the present application.

[0065] For different objects appearing in the video sequence to be identified, after object detection, the detection frame of each object and the unique object identifier associated with the detection frame will be obtained. That is, the detection frames of different objects are distinguished by the corresponding object identifiers, so that the main object of the behavior can be automatically identified and processed in time.

[0066] Step S120, determining a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0067] Here, based on the distribution of detection frames in each frame image, each frame image can be divided into a sparse area and a dense area, wherein the sparse area includes spatially independent detection frames or no detection frames, and the dense area includes at least two overlapping detection frames.

[0068] It can be understood that the dense area in each frame of the image can be considered as a potential target area where group object behaviors are prone to occur, so that the dense area can be analyzed and processed, and a preliminary judgment can be made as to whether there are abnormal group object behaviors such as "fighting", "pushing", and "hugging" in the dense area, and the first recognition result of the video sequence to be identified can be determined.

[0069] In this way, by guiding the reasoning of local dense areas through the heuristic dense group position estimation method, the effective perception area of ​​the model can be increased and the retrieval range of irrelevant background can be reduced.

[0070] Step S130, when the first recognition result satisfies the first condition, performing behavior recognition on the trajectory sequence of the group object to obtain a fourth recognition result;

[0071] Here, the trajectory sequence of the group objects is generated based on the video sequence to be identified, and the trajectory sequence is the sequence of activity areas of the group objects in each frame image in the video sequence to be identified; wherein the group objects include at least two target objects whose spatial distance is less than a distance threshold.

[0072] In the implementation, the video sequence to be identified, which is determined by step S120 as a first identification result as a possible abnormal behavior, is first preprocessed to generate a trajectory sequence of group objects; then the trajectory sequence of the group objects is input into the trained behavior recognition model for further identification and judgment to obtain a fourth identification result: if the discrimination score output by the behavior recognition model is higher than a given threshold, it is determined that the trajectory sequence has abnormal behavior and the specific group object behavior type is output.

[0073] For example, taking pedestrians as the detection objects, the abnormal behavior types of group objects may include "trampling", "two-person fight", "group fight", "pushing", "armed" and "hugging"; if the discrimination score output by the model does not meet the given threshold, the fourth recognition result is determined to be that no abnormal behavior of the group objects has occurred in the video sequence to be identified.

[0074] Step S140: determining a target recognition result of the to-be-recognized video sequence based at least on the first recognition result or the fourth recognition result.

[0075] Here, the first recognition result is the result of judging whether abnormal behavior occurs in the video sequence to be recognized based on logic rules, and the third recognition result is the result of further recognition by a neural network when the video sequence to be recognized is judged by logic rules to be likely to have abnormal behavior. The first recognition result obtained by the heuristic logic judgment method and the fourth recognition result obtained by the deep learning method are combined to determine the target recognition result of the video sequence to be recognized, thereby improving the accuracy and efficiency of abnormal behavior recognition.

[0076] In some embodiments, when the first recognition result indicates that no abnormal behavior has occurred in the video sequence to be recognized, a target recognition result of the video sequence to be recognized is determined based on the first recognition result. In this way, when it is logically predicted that the first recognition result of the video sequence to be recognized indicates that no abnormal behavior has occurred in the video sequence to be recognized, the first recognition result is directly used as the final target recognition result, and the video sequence to be recognized that obviously does not have abnormal behavior can be directly determined, thereby shortening the recognition process.

[0077] In other embodiments, when the first recognition result indicates that an abnormal behavior occurs in the video sequence to be recognized, a target recognition result of the video sequence to be recognized is determined based on the fourth recognition result. In this way, for the case where abnormal behavior may exist in the video sequence to be recognized by logical judgment, the neural network is further combined for recognition, and the fourth recognition result is used as the target recognition result to improve the robustness of behavior recognition.

[0078] It should be noted that in step S110, relevant detection algorithms can be flexibly used to accurately locate and analyze the subject of the behavior; in step S120, the distribution of the detection box in each frame image is used to preliminarily determine whether the video sequence to be identified has obvious abnormal behavior, or to filter out some unlikely situations from some surface-level geometric information, thereby improving the recognition efficiency of the video sequence and the accuracy of behavior recognition; in the behavior recognition process in step S130, the existing video understanding model can be fully utilized, thereby constructing a behavior recognition algorithm framework based on logical rules and deep learning, and improving the accuracy and robustness of behavior recognition. By converting the entire video sequence to be identified into a trajectory sequence of group objects, that is, the accurate behavior occurrence area, the image range of the input model is reduced, so that the model only recognizes the area where the group objects are located in each frame image, which can improve the efficiency and accuracy of behavior recognition. Therefore, the embodiment of the present application can be adapted to more video information with a large coverage field of view and more information, such as video sources obtained in outdoor urban street scenes and indoor rail transit scenes.

[0079] In some possible implementations, after obtaining the target recognition result of the video sequence to be identified through two-stage analysis of logical prejudgment and model recognition, an alarm notification can also be issued to the relevant departments or platforms to achieve timely handling of behaviors that endanger the safety of oneself and others. A possible implementation is as follows: when there is abnormal behavior of a group of objects in the video sequence to be identified, determine the spatial location and behavior category of the abnormal behavior; determine the alarm content based on the spatial location and behavior category; and send an alarm notification to the terminal device corresponding to the spatial location based on the alarm content, so that the administrator holding the terminal device can handle the abnormal behavior.

[0080] It is understandable that different location areas have corresponding managers holding terminal devices. By receiving notifications from the alarm system through terminal devices, multiple object behaviors occurring in the location area can be quickly located and processed. When behaviors that endanger the safety of oneself and others occur in outdoor urban street scenes and indoor rail transit scenes, the system will automatically identify the subject of the behavior and issue an alarm, providing personnel with relevant needs with an efficient and convenient detection capability.

[0081] In an embodiment of the present application, the detection frame of each object in each frame of the image is first obtained, and then the first recognition result of the video sequence to be identified is determined based on the distribution of the detection frame in each frame of the image, and then, when the first recognition result satisfies the first condition, the trajectory sequence of the group object generated based on the video sequence to be identified is subjected to behavior recognition to obtain a fourth recognition result. The embodiment of the present application proposes a behavior recognition method based on the fusion of heuristic logic and deep learning, which can support the needs of abnormal behavior detection of group objects and improve the robustness and accuracy of pure neural network recognition schemes. At the same time, different from the behavior recognition method based on the whole image sequence, the embodiment of the present application converts the video classification problem into the recognition problem of the trajectory sequence of the group object, which effectively improves the model's perception of the target area in the video source and greatly reduces the retrieval scope and calculation amount.

[0082] Figure 2 A flow chart of a behavior recognition method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method comprises at least the following steps:

[0083] Step S210, obtaining a detection frame of each object in each frame of the video sequence to be identified;

[0084] Step S220, determining a dense area in each frame of image based on the distribution of the detection frame in each frame of image;

[0085] Here, the dense area (patch) includes a central detection frame whose number of overlaps with other detection frames in each frame of the image is greater than or equal to a first number threshold. That is, for each frame of the video sequence to be identified, the embodiment of the present application expects to find a detection frame that overlaps more with other detection frames as the central detection frame to generate a dense area including group objects.

[0086] The dense area includes at least two objects, one of which is an object associated with the central detection frame; and the object identifier associated with the central detection frame is used as the object identifier associated with the dense area.

[0087] Step S230, determining a first recognition result of the to-be-recognized video sequence based on the number of objects included in the dense area in each frame of the image;

[0088] Here, by judging whether the number of objects in the dense area of ​​each frame image meets the requirements, it is preliminarily judged whether the video sequence to be identified may have abnormal behavior, and the first recognition result is obtained, that is, the first recognition result includes two situations: abnormal behavior occurs in the video sequence to be identified and no abnormal behavior occurs in the video sequence to be identified. It is generally believed that the number of objects involved in group object behavior is greater than or equal to two. In this way, the inference of local dense areas can be guided by the heuristic detection box density distribution and dense group position estimation method, which can increase the effective perception area of ​​the model and reduce the retrieval range of irrelevant background.

[0089] In some embodiments, when the number of objects included in the dense area in each frame image is greater than or equal to 2, the first recognition result of the video sequence to be identified is determined to be that abnormal behavior has occurred; in other embodiments, when the number of objects included in the dense area in each frame image is less than 2, the first recognition result of the video sequence to be identified is determined to be that no abnormal behavior has occurred.

[0090] In this way, by determining whether the number of objects included in the dense area of ​​each frame of the video sequence to be identified is greater than or equal to 2, it is determined whether the video sequence to be identified has abnormal behavior, and the first identification result is obtained. In this way, in the case where the number of objects does not meet the requirements, the video sequence to be identified that obviously will not have abnormal behavior of group objects is quickly determined, shortening the identification process and enhancing the accuracy of the identification method.

[0091] Step S240, when the first recognition result indicates that an abnormal behavior occurs in the video sequence to be recognized, behavior recognition is performed on the trajectory sequence of the group object to obtain the fourth recognition result;

[0092] Here, for the case where the number of objects included in the dense area in each frame of the image determines that the video sequence to be identified has abnormal behavior, it is necessary to further use the neural network to identify the specific behavior type, so as to obtain the fourth recognition result of the video sequence to be identified. In this way, the problem of insufficient robustness and accuracy of pure neural network recognition is improved by combining logical prediction.

[0093] Step S250: determining a target recognition result of the to-be-recognized video sequence based at least on the first recognition result or the fourth recognition result.

[0094] In some embodiments, when the first recognition result indicates that no abnormal behavior has occurred in the video sequence to be identified, a target recognition result of the video sequence to be identified is determined based on the first recognition result; in other embodiments, when the first recognition result indicates that abnormal behavior has occurred in the video sequence to be identified, a target recognition result of the video sequence to be identified is determined based on the fourth recognition result.

[0095] In the embodiment of the present application, based on the number of objects included in the dense area in each frame of the image, a trajectory sequence that obviously does not have abnormal behavior is preliminarily detected, thereby avoiding further detection in the input behavior recognition model and improving the recognition efficiency of the video sequence to be recognized. In the case of determining that abnormal behavior occurs in the video sequence to be recognized based on the number of objects included in the dense area, it is necessary to further use a neural network to identify the specific behavior type, thereby enhancing the accuracy and robustness of the overall recognition process.

[0096] Figure 3 A flow chart of a behavior recognition method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method comprises at least the following steps:

[0097] Step S310, obtaining a detection frame of each object in each frame of the video sequence to be identified;

[0098] Step S320, determining a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0099] Step S330, when the first recognition result indicates that abnormal behavior occurs in the video sequence to be recognized, determining an area change value of the dense region between adjacent frames in the video sequence to be recognized;

[0100] Here, the first recognition result represents the situation where abnormal behavior occurs in the video sequence to be recognized, and the area change values ​​of the dense regions between adjacent frames are further compared.

[0101] For example, the area of ​​the dense region in the n-1th frame is 0.6 square centimeters, and the area of ​​the same dense region in the nth frame is 0.8 square centimeters. The area change value of the dense region between the n-1th frame and the nth frame is 0.2 square centimeters, indicating that the position range of the group objects has increased and the movement amplitude has changed dramatically.

[0102] Step S340, determining a second recognition result of the to-be-recognized video sequence based on the area change value and the change threshold of the dense region;

[0103] Here, by comparing whether the area change value of the dense area between adjacent frames is greater than or equal to the change threshold, it is determined whether the video sequence to be identified has abnormal behavior, and a second identification result is obtained, that is, the second identification result includes two situations: the video sequence to be identified has abnormal behavior and the video sequence to be identified has no abnormal behavior. It is generally believed that when the area of ​​a dense area changes significantly, the movements of multiple objects included in the dense area are violent, and abnormal behavior of group objects is likely to occur.

[0104] If the area change values ​​of the dense areas between consecutive adjacent frames in the video sequence to be identified are all less than the change threshold, then the second recognition result of the video sequence to be identified is judged as no abnormal behavior has occurred; if there is any area change value of the dense areas between adjacent frames that is greater than the change threshold, then the second recognition result of the video sequence to be identified is judged as abnormal behavior has occurred.

[0105] The change threshold is a preset empirical value. For example, it can be preset according to the object type and behavior type. When the detected object is a pedestrian and the detected behavior is a fighting type, the change threshold of the dense area can be set to a specific percentage of the pedestrian detection frame. The embodiment of the present application does not limit this.

[0106] In some embodiments, determining the second recognition result of the video sequence to be identified based on the area change value and the change threshold of the dense area includes: when the area change values ​​of the dense area between the adjacent frames are greater than or equal to the change threshold, determining that the second recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the area change values ​​of the dense area between the adjacent frames are less than the change threshold, determining that the second recognition result of the video sequence to be identified is that no abnormal behavior has occurred.

[0107] In this way, by comparing the area change of the dense area between adjacent frames in the video sequence to be identified, it is determined whether the video sequence to be identified has abnormal behavior, and the second identification result is obtained. This makes it possible to quickly determine the video sequence to be identified that has obviously abnormal behavior in the case of intense movement, shorten the identification process, and enhance the accuracy of the identification method.

[0108] Step S350, when the second recognition result indicates that the to-be-recognized video sequence has no abnormal behavior, behavior recognition is performed on the trajectory sequence of the group object to obtain the fourth recognition result;

[0109] Here, for the case where the video sequence to be identified does not have abnormal behavior based on the area of ​​the dense area in each frame of the image, it is necessary to further use the neural network to identify the specific behavior type, so as to obtain the fourth recognition result of the video sequence to be identified. In this way, the problem of insufficient robustness and accuracy of pure neural network recognition is improved by combining logical prediction.

[0110] Step S360: Determine a target recognition result of the video sequence to be recognized based at least on the first recognition result or the fourth recognition result.

[0111] Here, based on the first recognition result, the second recognition result or the fourth recognition result, a target recognition result of the video sequence to be recognized is determined.

[0112] In some embodiments, when the first recognition result represents that no abnormal behavior has occurred in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the first recognition result; in some embodiments, when the second recognition result represents that abnormal behavior has occurred in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the second recognition result; in some embodiments, when the first recognition result represents that abnormal behavior has occurred in the video sequence to be recognized and the second recognition result represents that no abnormal behavior has occurred in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the fourth recognition result.

[0113] In an embodiment of the present application, the first recognition result characterizes the situation where abnormal behavior occurs in the video sequence to be identified, and the second recognition result is further determined based on the area change value of the dense region between adjacent frames, thereby combining the two-stage heuristic logic method to improve the robustness and accuracy of the neural network recognition scheme.

[0114] Figure 4 A flow chart of a behavior recognition method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the method comprises at least the following steps:

[0115] Step S410, obtaining a detection frame of each object in each frame of the video sequence to be identified;

[0116] Step S420, determining a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0117] Step S430, when the first recognition result indicates that the abnormal behavior occurs in the video sequence to be recognized, determining an area change value of the dense region between adjacent frames in the video sequence to be recognized;

[0118] Step S440, determining a second recognition result of the to-be-recognized video sequence based on the area change value and the change threshold of the dense region;

[0119] The implementation process of the above steps S410 to S440 is similar to the above steps S310 to S340. For technical details not disclosed in the embodiments of the present application, please refer to the description of the previous embodiment for understanding.

[0120] Step S450, when the second recognition result indicates that the video sequence to be recognized has no abnormal behavior, determining a bounding box area in each frame of the video sequence to be recognized based on the coverage of the dense area in each frame of the video sequence to be recognized;

[0121] Here, in view of the situation that the second recognition result indicates that the video sequence to be recognized does not send abnormal behavior, a bounding box area with a larger range than the dense area is further determined.

[0122] The ranges of dense regions in different frame images in the video sequence to be identified can be merged in spatial position to obtain a larger bounding box, and each frame image can be intercepted to obtain the bounding box area in the corresponding frame image.

[0123] Step S460, determining a third recognition result of the video sequence to be recognized based on pixel changes in the bounding box area between adjacent frames in the video sequence to be recognized;

[0124] Here, when it is preliminarily determined that abnormal behavior exists in the video sequence to be identified by the change in area of ​​the dense region, a further comparison is made to see whether the pixels in the larger bounding box area change between adjacent frames, so as to determine whether abnormal behavior occurs in the video sequence to be identified through multi-stage logic rules and obtain a third identification result.

[0125] In implementation, the pixel changes between adjacent frames in the video sequence to be identified can be determined by an image processing algorithm in the related art. Usually, the size and dimensions of adjacent frame images are set to be the same. For grayscale images, since there is only a single channel, the pixel subtraction at the corresponding position can be performed directly. For color images, the corresponding color components should be subtracted separately.

[0126] In some possible implementations, determining the third recognition result of the video sequence to be identified based on the pixel changes of the bounding box area between adjacent frames in the video sequence to be identified includes: when the pixel difference of the bounding box area between adjacent frames in the video sequence to be identified is greater than or equal to a pixel threshold, determining that the third recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the pixel difference of the bounding box area between adjacent frames in the video sequence to be identified is less than the pixel threshold, determining that the third recognition result of the video sequence to be identified is that no abnormal behavior has occurred.

[0127] In some possible implementations, determining the third recognition result of the video sequence to be identified based on the pixel changes of the bounding box area between adjacent frames in the video sequence to be identified includes: when the cumulative sum of pixel difference values ​​of the bounding box area between adjacent frames in the video sequence to be identified is greater than or equal to a pixel threshold, determining that the third recognition result of the video sequence to be identified is that abnormal behavior has occurred; when the cumulative sum of pixel difference values ​​of the bounding box area between adjacent frames in the video sequence to be identified is less than the pixel threshold, determining that the third recognition result of the video sequence to be identified is that no abnormal behavior has occurred.

[0128] Here, for each two adjacent frames of the video sequence to be identified, it is determined whether the pixel difference or the cumulative sum of the pixel difference satisfies the pixel threshold, and whether abnormal behavior occurs in the corresponding video sequence to be identified. Thus, the video sequence to be identified that obviously has abnormal behavior can be quickly identified through logical rules, shortening the identification process and enhancing the accuracy of the identification method.

[0129] Step S470, when the third recognition result indicates that the video sequence to be recognized has no abnormal behavior, behavior recognition is performed on the trajectory sequence of the group object to obtain the fourth recognition result;

[0130] Here, for the case where no abnormal behavior occurs in the video sequence to be identified based on the pixel changes in the bounding box area in each frame image, it is necessary to further use the neural network to identify the specific behavior type, thereby obtaining the fourth identification result of the video sequence to be identified. In this way, the problem of insufficient robustness and accuracy of pure neural network recognition is improved by combining logical prediction.

[0131] Step S480: Determine a target recognition result of the video sequence to be recognized based at least on the first recognition result or the fourth recognition result.

[0132] Here, based on the first recognition result, the second recognition result, the third recognition result or the fourth recognition result, a target recognition result of the video sequence to be recognized is determined.

[0133] In some embodiments, when the first recognition result represents that no abnormal behavior occurs in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the first recognition result; in some embodiments, when the second recognition result represents that abnormal behavior occurs in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the second recognition result; in some embodiments, when the third recognition result represents that abnormal behavior occurs in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the third recognition result; in some embodiments, when the first recognition result represents that abnormal behavior occurs in the video sequence to be recognized, and the second recognition result and the third recognition result both represent that no abnormal behavior occurs in the video sequence to be recognized, the target recognition result of the video sequence to be recognized is determined based on the fourth recognition result.

[0134] In the embodiment of the present application, for the case where the second recognition result indicates that the video sequence to be recognized has not sent abnormal behavior, the third recognition result of the video sequence to be recognized is further determined based on the pixel changes in the bounding box area between adjacent frame images. When the third recognition result indicates that the video sequence to be recognized has not sent abnormal behavior, the behavior recognition model is used to recognize the trajectory sequence of the group object. Thus, after performing heuristic logical judgment on the video sequence to be recognized through the area changes of the dense area and the pixel changes of the bounding box area after taking the union, the behavior recognition model is used to perform behavior recognition, which can improve the robustness and accuracy of the neural network recognition scheme.

[0135] Figure 5 A schematic diagram of a process for generating a trajectory sequence of a group of objects provided in an embodiment of the present application, such as Figure 5 As shown, the method comprises at least the following steps:

[0136] Step S510, determining a dense area in each frame of image based on the distribution of the detection frame in each frame of image;

[0137] Here, the dense area includes a central detection frame whose number of overlaps with other detection frames in each frame image is greater than or equal to a first number threshold. The central detection frames in different frame images may be of the same object or different objects, and the central detection frame of an object may appear in each frame image in the video sequence to be identified or only in a part of the frames.

[0138] In implementation, the center detection frame with the most matches with other detection frames in each frame can be determined by constructing an adjacency matrix. It is generally believed that the spatial location of the center detection frame is prone to group object behavior. Therefore, the dense area of ​​each frame can be obtained by expanding a certain range outward from the center detection frame.

[0139] In some embodiments, the dense area in each frame of the image is determined by the following process: based on the distribution of the detection boxes in each frame of the image, a central detection box in each frame of the image is determined; and the central detection box in each frame of the image is expanded outward at a specific ratio to obtain the dense area in each frame of the image.

[0140] The specific ratio is an empirical value, which is generally set to 2 times. The specific ratio may also be determined based on the actual application scene images taken, so that the expanded dense area can surround all the execution subjects of the group object behavior as much as possible.

[0141] In this way, the center detection frame is first determined by using the distribution of the detection frames in each frame, and then the center detection frame is expanded outward so that the expanded dense area includes all objects centered on the center detection frame and surrounding the group object behavior as much as possible. This reduces the behavior receptive field of the action execution subject and can solve the problem of small effective perception range when identifying the video sequence to be identified in the full field of view.

[0142] In some embodiments, the number of detection frames included in each frame of image is greater than or equal to 2, and determining the central detection frame in each frame of image based on the distribution of the detection frames in each frame of image includes: generating an adjacency matrix corresponding to each frame of image according to the intersection-and-union ratio between every two detection frames in each frame of image; taking the detection frames in the adjacency matrix whose matching times are greater than or equal to a second number threshold as the central detection frame in each frame of image; wherein the second number threshold is greater than the first number threshold.

[0143] Here, the intersection over union (IoU) between each two detection frames is the result of dividing the overlapping part of the two regions by the collective part of the two regions. The value of (i, j) in the adjacency matrix represents the intersection over union of detection frames i and j in the frame image. The number of values ​​greater than 0 in the i-th row (except itself) is taken as the number of matches of detection frame i. If there are many repetitions between a detection frame j and multiple other detection frames, and there are multiple object detection frames around the location of the detection frame j, group object behavior is likely to occur, so the detection frame j can be used as the center detection frame of the frame image.

[0144] In this way, for the case where each frame image includes multiple detection frames, by calculating the intersection-over-union ratio between any two detection frames and counting the number of matches of each detection frame, the central detection frame that overlaps with other detection frames the most times can be accurately screened out, so as to facilitate the subsequent generation of trajectory sequences of dense areas and group objects, and effectively extract useful information in the video to be identified.

[0145] In some other embodiments, when each frame of the image includes at least two detection frames, first, the detection frame of each object is expanded outward by a fixed ratio to obtain an expanded detection frame; at least two first detection frames are selected from the expanded detection frame; wherein the area of ​​the first detection frame is larger than that of other detection frames in the expanded detection frame; and then, the intersection-and-union ratio between the at least two first detection frames is determined. For example, when the fixed ratio is 1.5 times, the length and width of each detection frame of the object are expanded by 1.5 times. This can increase the image resolution, thereby better calculating the intersection-and-union ratio between overlapping detection frames.

[0146] Step S520: generating a trajectory sequence of the group of objects based on the dense area in each frame of the image.

[0147] Here, firstly, based on the dense area in each frame, the smallest bounding box that includes the same group of group objects in each frame is determined, and the bounding box area in each frame is extracted based on the smallest bounding box to form the trajectory sequence of the group of group objects. This facilitates the subsequent behavior recognition of group objects through the behavior recognition model.

[0148] The trajectory sequence of the group object can be generated by the following process:

[0149] Step S5201, determining a bounding box area in each frame of the image based on the coverage of the dense area in each frame of the image in the to-be-identified video sequence;

[0150] Here, we can first take the union of the coverage ranges of each dense area in the video sequence to be identified, and combine the maximum coordinates of the boundary points of each dense area to determine the minimum bounding box that can include all detection boxes that overlap with the central detection box. Then, we can use image segmentation technology or other image processing technology in related technologies to capture the bounding box area corresponding to the spatial position of the minimum bounding box in each frame of the image. The embodiments of the present application are not limited to this.

[0151] It should be noted that after the area corresponding to the central detection frame is adaptively expanded outward according to a specific ratio, the obtained dense area is still a rectangular area. In order to facilitate subsequent processing, the size of the bounding box area after taking the union of the dense areas of each frame can be normalized. For example, the long side size of the bounding box area is adjusted to 224 pixels, the short side of the bounding box area is scaled proportionally with the long side, and the short side that is less than 224 pixels after scaling is filled with black edges at the top and bottom.

[0152] Step S5202: Generate a trajectory sequence of the group objects based on the timestamp of each frame of image and the bounding box area in each frame of image.

[0153] Here, each frame image in the video sequence to be identified carries its own timestamp, which can be a timestamp set when the camera or other acquisition device is acquiring the image, or a timestamp set in the subsequent sampling process of the original image sequence, or a timestamp set in other achievable ways, and the embodiments of the present application are not limited to this.

[0154] Replacing the full-image video sequence with a trajectory sequence generated in the bounding box area in timestamp order can reduce the receptive field of group object behavior events, thereby improving the accuracy and efficiency of behavior recognition. At the same time, it avoids the loss of the relative position of group object behaviors, and has a good performance improvement for the detection of behaviors with similar space but different motion rhythms.

[0155] In the embodiment of the present application, the distribution of the detection frame in the acquired frame image is first used to determine the dense area, and then the dense area in each frame image is preprocessed to obtain the trajectory sequence of the group object, thereby solving the problem of identifying the group object behavior. At the same time, the trajectory sequence of the group object reduces the behavior receptive field of the action execution subject, which can solve the problem of small effective perception range when identifying the full-view video sequence to be identified.

[0156] The above behavior recognition method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0157] The embodiment of the present application is explained by taking the group object as a group of pedestrians and the abnormal behavior fighting of the group object as an example. Traditional behavior recognition methods usually perform full-image data enhancement or other preprocessing on the input video sequence and then send it to the classification model for prediction. However, this method is only applicable to human-centric video behavior recognition, and this type of data is common in public video academic data sets. For online video streams, it often contains more information and covers a larger field of view. At the same time, the event location and human body scale of the target are also random. Therefore, it is obviously unreasonable to simply use the full image as the model input. In addition, there is uncertainty and inaccuracy in relying solely on neural networks for recognition.

[0158] For the identification of abnormal fighting behaviors in online video streams, it is necessary to locate the approximate area of ​​the fight in the entire image (especially the high-angle view) in the video sequence, increase the effective perception range of the machine for the input video stream, and accurately identify fighting events including "two-person fight", "multi-person fight", "pushing", "armed" and "hugging". At the same time, it supports common indoor and outdoor scenes such as urban streets and rail transit, so that the automatic analysis of behavioral events in video content provides convenience and evidence collection capabilities for relevant departments.

[0159] The embodiment of the present application provides a method for identifying abnormal fighting behaviors based on the fusion of heuristics and deep learning. First, at the data processing level, different from the traditional video behavior recognition method based on the whole image sequence, the embodiment of the present application proposes a method based on the density distribution of the detection box and the heuristic dense group position estimation to guide the reasoning of the local dense area, which increases the effective perception area of ​​the model and reduces the retrieval range of irrelevant background.

[0160] Secondly, at the level of behavior recognition algorithms: first determine the minimum bounding box of the group objects in the video sequence through the preprocessing step, and determine the bounding box area in each frame image. For video sequences that meet the number of pedestrians required, calculate the degree of area change of the bounding box area between adjacent frames. If the area change is large, it is determined that a fight has occurred in the corresponding video sequence; otherwise, further calculate the pixel difference between adjacent frames. For cases where the pixel change amplitude is large, that is, the object moves violently, directly output that the corresponding video sequence has a fight; and for cases where the pixel change amplitude is small, that is, the movement intensity does not meet the threshold, it needs to be further sent to the behavior recognition model for final judgment.

[0161] Figure 6 The logic flow chart of the multi-person behavior recognition method provided in the embodiment of the present application is as follows: Figure 6 As shown, the process includes the following steps:

[0162] Step S601, extracting the detection frame of each person in the video sequence to be identified;

[0163] Here, the full-image video sequence captured by the image acquisition device is obtained, and the detection frame of the pedestrian in the video image is obtained by calling the upstream detection component.

[0164] Step S602, expanding the detection frame of each pedestrian at a fixed ratio and sorting them by area;

[0165] Here, all detection results of the current frame are expanded by a fixed ratio (the default is 1.5 times) and then sorted in descending order by area.

[0166] Step S603, constructing an adjacency matrix based on the intersection-and-union ratios between the detection frames, and determining the central detection frame with the most matching times;

[0167] Here, after sorting, several detection frames with larger areas are selected to determine the intersection-union ratio, and an adjacency matrix is ​​constructed; the detection frame with the most matching times is calculated based on the adjacency matrix as the central detection frame.

[0168] Step S604, restoring the resolution of the central detection frame and calculating the dense area containing the pedestrian;

[0169] Here, the central detection frame obtained above is reduced to the original resolution, and a larger expansion ratio is defined. It is adaptively expanded outward according to the expansion ratio to obtain a larger rectangular frame as the dense area of ​​the current frame.

[0170] It is generally believed that multi-person behaviors are more likely to occur in dense areas, and the central detection box is the center of the multi-person behavior. Therefore, the rectangular box obtained by expanding the central detection box outward, i.e., the dense area, should surround the group where the multi-person behavior occurs as much as possible.

[0171] Step S605, determining whether there are more than or equal to 2 people in the dense area;

[0172] Here, if the number of pedestrians in the dense area is greater than or equal to 2, step S607 is executed; otherwise, step S606 is executed and the judgment ends.

[0173] Step S606, determining that no fight occurs;

[0174] Step S607, determining whether the area of ​​the dense region between adjacent frames changes dramatically;

[0175] Here, if the area of ​​the dense region between adjacent frames changes dramatically, step S608 is executed to end the judgment; otherwise, step S609 is continued to be executed.

[0176] Step S608, determining that a fight has occurred;

[0177] Step S609, taking a union of the dense regions of all frame sequences to obtain a larger bounding box region, and calculating the pixel difference of the bounding box regions between adjacent frames;

[0178] The process of intercepting a larger bounding box area in each frame image according to the dense area is as follows: first, the bounding box area in each frame image is intercepted according to the size of the minimum bounding box, and then the long side of the bounding box area is resized to 224 pixels, and then the short side of the bounding box area is scaled proportionally, and the upper and lower black edges are filled if it is less than 224 pixels.

[0179] Step S610, determining whether the accumulated sum of pixel differences is greater than a pixel threshold;

[0180] Here, if the accumulated sum of the pixel differences of each frame is greater than the pixel threshold, step S611 is executed to end the judgment; otherwise, step S612 is continued to be executed.

[0181] Step S611, determining that a fight has occurred;

[0182] Step S612, input the trained behavior recognition network for further recognition.

[0183] Here, the specified bounding box areas in each frame are deducted to form the motion trajectory sequence of the group of pedestrians, and sent to the neural network for fighting event recognition. The trajectory sequence obtained through the above process is as follows Figure 7 As shown, the trajectory sequence of the group of pedestrians includes 5 frames of bounding box area images, and each bounding box area image includes a dense area composed of abnormal behaviors occurring by two behavior subjects.

[0184] The embodiment of the present application uses a heuristic and scalable estimation scheme for the positions of dense groups of people, which effectively improves the model's perception of dense areas in video sequences and greatly reduces the retrieval scope and calculation amount; at the same time, it combines a two-stage heuristic logical method to improve the robustness and accuracy of the pure neural network recognition scheme.

[0185] The embodiments of the present application can be applied to outdoor urban street scenes and indoor rail transit scenes. When abnormal behavior events occur among a group of objects in these scenes, the device collecting the video source will automatically identify the area and type of the event, and send out an alarm to personnel with related needs, providing an efficient and convenient detection capability.

[0186] Based on the foregoing embodiments, the embodiments of the present application further provide a behavior recognition device, which includes the modules included, the sub-modules included in each module, and the units included in each sub-module, which can be implemented by a processor in the device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0187] Figure 8 A schematic diagram of the structure of a behavior recognition device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the identification device 800 includes an acquisition module 810, a first determination module 820, an identification module 830 and a second determination module 840, wherein:

[0188] The acquisition module 810 is used to acquire the detection frame of each object in each frame of the video sequence to be identified;

[0189] The first determination module 820 is used to determine a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image;

[0190] The recognition module 830 is used to perform behavior recognition on the trajectory sequence of the group object to obtain a fourth recognition result when the first recognition result satisfies the first condition; wherein the trajectory sequence of the group object is generated based on the video sequence to be recognized; and the group object includes at least two target objects whose spatial distance is less than a distance threshold;

[0191] The second determination module 840 is used to determine a target recognition result of the to-be-recognized video sequence based at least on the first recognition result or the fourth recognition result.

[0192] In some possible embodiments, the second determination module 840 includes: a fifth determination submodule, which is used to determine the target recognition result of the video sequence to be identified based on the first recognition result when the first recognition result indicates that no abnormal behavior has occurred in the video sequence to be identified; or, when the first recognition result indicates that abnormal behavior has occurred in the video sequence to be identified, determine the target recognition result of the video sequence to be identified based on the fourth recognition result.

[0193] In some possible implementations, the first determination module 820 includes: a first determination submodule, used to determine a dense area in each frame of the image based on the distribution of the detection frames in each frame of the image; the dense area includes a central detection frame whose number of overlaps with other detection frames in each frame of the image is greater than or equal to a first number threshold; and a second determination submodule, used to determine a first recognition result of the video sequence to be recognized based on the number of objects included in the dense area in each frame of the image.

[0194] In some possible implementations, the recognition module 830 includes: a third determination submodule, used to determine the area change value of the dense area between adjacent frames in the video sequence to be recognized when the first recognition result indicates that abnormal behavior has occurred in the video sequence to be recognized; a fourth determination submodule, used to determine the second recognition result of the video sequence to be recognized based on the area change value and the change threshold of the dense area; and a recognition submodule, used to perform behavior recognition on the trajectory sequence of the group objects to obtain the fourth recognition result when the second recognition result indicates that no abnormal behavior has occurred in the video sequence to be recognized.

[0195] In some possible embodiments, the recognition submodule includes: a first determination unit, which, when the second recognition result indicates that the video sequence to be recognized has not undergone abnormal behavior, determines the bounding box area in each frame image in the video sequence to be recognized based on the coverage of the dense area in each frame image in the video sequence to be recognized; the bounding box area in each frame image is located at the same position, has the same size and surrounds the dense area; a second determination unit, which is used to determine the third recognition result of the video sequence to be recognized based on the pixel change of the bounding box area between adjacent frames in the video sequence to be recognized; and a recognition unit, which is used to perform behavior recognition on the trajectory sequence of the group object to obtain the fourth recognition result when the third recognition result indicates that the video sequence to be recognized has not undergone abnormal behavior.

[0196] In some possible embodiments, the identification submodule also includes a fifth determination unit, which is used to determine that the target recognition result of the video sequence to be identified is the occurrence of abnormal behavior when the second recognition result represents that the video sequence to be identified has abnormal behavior; or, when the third recognition result represents that the video sequence to be identified has abnormal behavior, determine that the target recognition result of the video sequence to be identified is the occurrence of abnormal behavior.

[0197] In some possible embodiments, the second determination unit is further used to determine that the third recognition result of the video sequence to be identified is that abnormal behavior has occurred when the pixel difference value of the bounding box area between adjacent frames in the video sequence to be identified is greater than or equal to a pixel threshold; and to determine that the third recognition result of the video sequence to be identified is that no abnormal behavior has occurred when the pixel difference value of the bounding box area between adjacent frames in the video sequence to be identified is less than the pixel threshold.

[0198] In some possible implementations, the fourth determination submodule is further used to determine that the second recognition result of the video sequence to be identified is that abnormal behavior has occurred when the area change value of the dense area between the adjacent frames is greater than or equal to the change threshold; and to determine that the second recognition result of the video sequence to be identified is that no abnormal behavior has occurred when the area change value of the dense area between the adjacent frames is less than the change threshold.

[0199] In some possible implementations, the second determination submodule is further used to determine that the first recognition result of the video sequence to be identified is that abnormal behavior has occurred when the number of objects included in the dense area in each frame image is greater than or equal to 2; and to determine that the first recognition result of the video sequence to be identified is that no abnormal behavior has occurred when the number of objects included in the dense area in each frame image is less than 2.

[0200] In some possible implementations, the first determination submodule includes: a first generation unit, used to generate an adjacency matrix corresponding to each frame of the image according to the intersection-and-union ratio between every two detection frames in each frame of the image; a third determination unit, used to use the detection frame in the adjacency matrix whose matching times is greater than or equal to a second number threshold as the central detection frame in each frame of the image; wherein the second number threshold is greater than the first number threshold; and an expansion unit, used to expand the central detection frame in each frame of the image outward at a specific ratio to obtain a dense area in each frame of the image.

[0201] In some possible implementations, the first determining module 820 further includes a generating submodule, configured to generate a trajectory sequence of the group objects based on the dense area in each frame of the image.

[0202] In some possible implementations, the generation submodule includes: a fourth determination unit, used to determine the bounding box area in each frame image based on the coverage of the dense area in each frame image in the video sequence to be identified; the bounding box area in each frame image is located at the same position, has the same size and surrounds the dense area; a second generation unit, based on the timestamp of each frame image and the bounding box area in each frame image, generates a trajectory sequence of the group object.

[0203] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0204] It should be noted that in the embodiment of the present application, if the above-mentioned behavior recognition method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable the device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0205] Correspondingly, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the behavior recognition methods described in the above embodiments are implemented. Correspondingly, in an embodiment of the present application, a chip is also provided, and the chip includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the steps in any of the behavior recognition methods described in the above embodiments. Correspondingly, in an embodiment of the present application, a computer program product is also provided, and when the computer program product is executed by a processor of a device, it is used to implement the steps in any of the behavior recognition methods described in the above embodiments.

[0206] Based on the same technical concept, an embodiment of the present application provides a behavior recognition device for implementing the behavior recognition method recorded in the above method embodiment. Fig. 9 A hardware entity diagram of a behavior recognition device provided in an embodiment of the present application, such as Fig. 9 As shown, the identification device 900 includes a memory 910 and a processor 920, the memory 910 stores a computer program that can be run on the processor 920, and the processor 920 implements the steps in any behavior identification method described in the embodiments of the present application when executing the program.

[0207] The memory 910 is configured to store instructions and applications executable by the processor 920, and can also buffer data to be processed or processed by the processor 920 and various modules in the device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0208] When the processor 920 executes the program, the steps of any of the above-mentioned behavior recognition methods are implemented. The processor 920 generally controls the overall operation of the recognition device 900.

[0209] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0210] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and the like; it can also be various devices including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0211] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0212] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0213] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0215] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0216] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0217] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0218] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0219] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0220] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A behavior recognition method, characterized in that: The method comprises: Obtain the detection frame of each object in each frame of the video sequence to be identified; Determining a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image; In a case where the first recognition result indicates that abnormal behavior occurs in the video sequence to be recognized, determining an area change value of a dense region between adjacent frames in the video sequence to be recognized; Determining a second recognition result of the to-be-recognized video sequence based on the area change value and the change threshold of the dense region; In the case where the second recognition result indicates that the video sequence to be recognized has no abnormal behavior, determining a bounding box area in each frame of the video sequence to be recognized based on the coverage of the dense area in each frame of the video sequence to be recognized; the bounding box area in each frame of the video sequence is located at the same position, has the same size, and surrounds the dense area; Determining a third recognition result of the video sequence to be recognized based on pixel changes in the bounding box area between adjacent frames in the video sequence to be recognized; When the third recognition result indicates that the video sequence to be recognized has no abnormal behavior, behavior recognition is performed on the trajectory sequence of the group objects to obtain a fourth recognition result; wherein the trajectory sequence of the group objects is generated based on the video sequence to be recognized; and the group objects include at least two target objects whose spatial distance is less than a distance threshold; A target recognition result of the to-be-recognized video sequence is determined based at least on the first recognition result or the fourth recognition result.

2. The method according to claim 1, characterized in that The determining, based at least on the first recognition result or the fourth recognition result, a target recognition result of the to-be-recognized video sequence includes: In a case where the first recognition result indicates that no abnormal behavior occurs in the video sequence to be recognized, determining a target recognition result of the video sequence to be recognized based on the first recognition result; or In a case where the first recognition result indicates that abnormal behavior occurs in the video sequence to be recognized, a target recognition result of the video sequence to be recognized is determined based on the fourth recognition result.

3. The method according to claim 1 or 2, characterized in that The determining, based on the distribution of the detection frames in each frame of the image, a first recognition result of the to-be-recognized video sequence comprises: Based on the distribution of the detection frames in each frame of image, a dense area in each frame of image is determined; the dense area includes a central detection frame whose number of overlaps with other detection frames in each frame of image is greater than or equal to a first number threshold; Based on the number of objects included in the dense area in each frame of the image, a first recognition result of the to-be-recognized video sequence is determined.

4. The method according to claim 1, characterized in that The method further comprises: In the case where the second recognition result indicates that the video sequence to be recognized has abnormal behavior, determining that the target recognition result of the video sequence to be recognized is abnormal behavior; or In a case where the third recognition result indicates that an abnormal behavior occurs in the video sequence to be recognized, it is determined that a target recognition result of the video sequence to be recognized is the occurrence of an abnormal behavior.

5. The method according to claim 1, characterized in that The determining, based on pixel changes in the bounding box area between adjacent frames in the video sequence to be identified, a third identification result of the video sequence to be identified, comprises: When the pixel difference of the bounding box area between adjacent frames in the to-be-identified video sequence is greater than or equal to the pixel threshold, determining that the third identification result of the to-be-identified video sequence is that abnormal behavior occurs; or, When the pixel difference value of the bounding box area between adjacent frames in the to-be-identified video sequence is less than the pixel threshold, it is determined that the third recognition result of the to-be-identified video sequence is that no abnormal behavior occurs.

6. The method according to claim 1, characterized in that The determining, based on the area change value and the change threshold of the dense area, a second recognition result of the to-be-recognized video sequence comprises: In the case where the area change values ​​of the dense regions between the adjacent frames are all greater than or equal to the change threshold, determining that the second recognition result of the video sequence to be recognized is that abnormal behavior occurs; or, When the area change value of the dense region between the adjacent frames is less than the change threshold, it is determined that the second recognition result of the to-be-recognized video sequence is that no abnormal behavior occurs.

7. The method according to claim 3, characterized in that The determining, based on the number of objects included in the dense area in each frame of the image, a first recognition result of the to-be-recognized video sequence comprises: When the number of objects included in the dense area in each frame of the image is greater than or equal to 2, determining that a first recognition result of the to-be-recognized video sequence is that an abnormal behavior occurs; When the number of objects included in the dense area in each frame of the image is less than 2, it is determined that no abnormal behavior occurs in the first recognition result of the video sequence to be recognized.

8. The method according to claim 3, characterized in that The determining of the dense area in each frame of image based on the distribution of the detection frame in each frame of image includes: Generate an adjacency matrix corresponding to each frame of image according to the intersection-and-union ratio between every two detection frames in each frame of image; Using the detection frame whose matching times in the adjacency matrix is ​​greater than or equal to the second number threshold as the central detection frame in each frame of the image; wherein the second number threshold is greater than the first number threshold; The central detection frame in each frame of image is expanded outward in proportion to obtain a dense area in each frame of image.

9. The method according to any one of claims 4 to 8, characterized in that The method further comprises: Based on the dense area in each frame of the image, a trajectory sequence of the group of objects is generated.

10. The method according to claim 9, characterized in that The step of generating a trajectory sequence of the group objects based on the dense area in each frame of the image comprises: Based on the coverage of the dense area in each frame of the image in the to-be-identified video sequence, determining the bounding box area in each frame of the image; the bounding box area in each frame of the image is located at the same position, has the same size, and surrounds the dense area; A trajectory sequence of the group objects is generated based on the timestamp of each frame of image and the bounding box area in each frame of image.

11. A behavior recognition device, characterized in that: The device comprises an acquisition module, a first determination module, an identification module and a second determination module, wherein: The acquisition module is used to acquire the detection frame of each object in each frame of the video sequence to be identified; The first determination module is used to determine a first recognition result of the to-be-recognized video sequence based on the distribution of the detection frames in each frame of the image; The recognition module is used to determine the area change value of the dense area between adjacent frames in the video sequence to be recognized when the first recognition result indicates that abnormal behavior occurs in the video sequence to be recognized; determine the second recognition result of the video sequence to be recognized based on the area change value and the change threshold of the dense area; when the second recognition result indicates that the video sequence to be recognized does not have abnormal behavior, determine the bounding box area in each frame of the video sequence to be recognized based on the coverage of the dense area in each frame of the video sequence to be recognized; the bounding box area in each frame of the video sequence is located at the same position, has the same size and surrounds the dense area; determine the third recognition result of the video sequence to be recognized based on the pixel change of the bounding box area between adjacent frames in the video sequence to be recognized; when the third recognition result indicates that the video sequence to be recognized does not have abnormal behavior, perform behavior recognition on the trajectory sequence of the group object to obtain a fourth recognition result; wherein the trajectory sequence of the group object is generated based on the video sequence to be recognized; the group object includes at least two target objects whose spatial distance is less than the distance threshold; The second determination module is used to determine a target recognition result of the to-be-recognized video sequence based at least on the first recognition result or the fourth recognition result.

12. A behavior recognition device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images