A method, device, system and storage medium for early warning of abnormal behavior

Abstract

The present application discloses an abnormal behavior early warning method, device, system and storage medium, the method comprising: processing monitoring video data to obtain a first monitoring image; performing abnormal behavior detection on the first monitoring image to obtain a first danger hotspot map, the first danger hotspot map including the abnormal behavior type and the corresponding danger probability value of each pixel position in the first monitoring image; processing the first danger hotspot map to obtain an abnormal area where abnormal behavior exists; generating early warning information based on the abnormal area. Through the above methods, the present application can improve the effectiveness of early warning.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to an abnormal behavior early warning method, device, system and storage medium. Background Art

[0002] With the continuous increase in the number of vehicles, the incidence of traffic accidents is also increasing. Many traffic accidents occur due to factors such as roads or weather, which cause motor vehicle drivers to not understand the location and movement of pedestrians / non-motor vehicles and fail to respond in time, resulting in collisions. Therefore, how to promptly and effectively remind drivers of abnormal behavior in the current scene has become an urgent problem that needs to be solved. Summary of the invention

[0003] The present application provides an abnormal behavior early warning method, device, system and storage medium, which can improve the effectiveness of early warning.

[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide an abnormal behavior early warning method, the method comprising: processing monitoring video data to obtain a first monitoring image; performing abnormal behavior detection on the first monitoring image to obtain a first danger hotspot map, the first danger hotspot map including the abnormal behavior type and the corresponding danger probability value at each pixel position in the first monitoring image; processing the first danger hotspot map to obtain an abnormal area where abnormal behavior exists; and generating early warning information based on the abnormal area.

[0005] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide an early warning device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the early warning method of abnormal behavior in the above-mentioned technical solution.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in the present application is: to provide a monitoring and early warning system, which includes an early warning device, a speaker and a display device, the early warning device is connected to the speaker and the display device, and is used to send notification information to the speaker and the display device, wherein the early warning device is the early warning device in the above-mentioned technical solution.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the early warning method of abnormal behavior in the above technical solution.

[0008] Through the above scheme, the beneficial effects of the present application are: first, the acquired monitoring video data is processed to obtain at least one first monitoring image; then, abnormal behavior detection is performed on the first monitoring image to obtain a first danger hotspot map, which includes the abnormal behavior type of each pixel position in the first monitoring image and the danger probability value corresponding to the abnormal behavior type; then, the first danger hotspot map is processed to obtain abnormal areas where abnormal behavior exists, and early warning information related to the abnormal area is generated, so as to monitor whether abnormal behavior occurs at any point on the specified road section, and comprehensively evaluate potential abnormal behavior; and abnormal behavior detection based on the first danger hotspot map can more effectively detect potential abnormal behavior, improve the accuracy of abnormal behavior warning, reduce the probability of traffic accidents, and help improve road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flowchart of an embodiment of an early warning method for abnormal behavior provided by the present application;

[0011] Figure 2 It is a schematic diagram of the installation of the camera, display device and speaker provided in this application;

[0012] Figure 3 It is a flowchart of another embodiment of the abnormal behavior early warning method provided by the present application;

[0013] Figure 4 It is a structural schematic diagram of an embodiment of an early warning device provided by the present application;

[0014] Figure 5 It is a structural diagram of an embodiment of a monitoring and early warning system provided by the present application;

[0015] Figure 6 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] "Ghost popping out" often occurs during the driving of motor vehicles. "Ghost popping out" is a traffic accident caused by pedestrians or non-motor vehicles suddenly appearing from the driver's blind spot and the driver failing to react in time. It accounts for a large proportion of all types of accidents. Therefore, there is an urgent need for an early warning solution to inform the possibility of a collision in advance and remind the driver to slow down or stop early, so as to reduce or even avoid this type of accident.

[0018] In order to reduce the occurrence of "ghosting", some solutions in the related technology determine whether a collision occurs based on the distance between the vehicle and the pedestrian. However, there are the following disadvantages: (1) It is necessary to install warning devices on the pedestrian's mobile phone and the vehicle respectively, which is cumbersome and not conducive to popularization. It also involves the maintenance of remote servers, increasing the burden of traffic management; (2) For pedestrians or people driving non-motor vehicles, when they "ghost", the mobile phone text message reminder may completely lose its warning function due to the coverage of surrounding noise or the mobile phone itself is set to silent mode; (3) For motor vehicle drivers, when the warning system is not working, it is difficult to detect the collision. By the time the system determines the distance, the distance between pedestrians and vehicles is already shrinking. By the time the system completes the determination and sends the warning information to the driver, it is difficult for the driver to respond quickly by slowing down, which may result in the driver hitting a pedestrian or non-motor vehicle. This delay actually greatly reduces the effectiveness of the warning. (4) Using only the distance parameter as the basis for issuing a warning cannot comprehensively evaluate potential abnormal behaviors, is not suitable for multi-target environments, and has a high false alarm rate. (5) The application scenario is limited to zebra crossings, which is inconsistent with the random location of "ghosting" and cannot effectively prevent the "ghosting" phenomenon.

[0019] In view of the above problems existing in the relevant technologies, the present application proposes a scheme for early warning of abnormal behaviors of pedestrians and non-motor vehicles, which can timely and effectively detect potential "ghost peeking" behaviors in the current monitoring scenario, and can monitor all types of "ghost peeking" in the entire designated road section, and can use effective methods to promptly remind pedestrians and drivers, thereby significantly reducing traffic accidents caused by "ghost peeking".

[0020] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an abnormal behavior early warning method provided by the present application, the method comprising:

[0021] Step 11: Process the surveillance video data to obtain a first surveillance image.

[0022] By installing a surveillance camera above a specific road section or all road sections, the real-time video stream of the current surveillance section can be obtained; considering the occasional nature of the "ghost peeking" phenomenon, real-time video stream collection can be performed on the current surveillance section, that is, surveillance video data can be sampled from the real-time video stream; then the sampled surveillance video data is preprocessed (for example: scaling, denoising, cropping or geometric transformation, etc.) to obtain the first surveillance image.

[0023] Step 12: Perform abnormal behavior detection on the first monitoring image to obtain a first danger hotspot map.

[0024] After acquiring the first surveillance image, the first surveillance image in the first surveillance image can be input into a pre-trained abnormal behavior detection network for identification, so that the abnormal behavior detection network processes the first surveillance image to generate a first danger hotspot map. The first danger hotspot map includes the abnormal behavior type at each pixel position in the first surveillance image and the danger probability value corresponding to the abnormal behavior type, and the danger probability value is the probability of abnormal behavior occurring.

[0025] Furthermore, the abnormal behavior detection network can be a network with target detection function such as a YOLO (You Only Look Once) network, a Long Short-Term Memory (LSTM) network or a Convolutional Neural Network (CNN). The size of the first danger hotspot map is the same as the size of the monitoring image. The abnormal behavior can be "ghost peeking", and "ghost peeking" includes 5 types of abnormal behaviors: (1) pedestrians or non-motor vehicles suddenly run out from the front of the motor vehicle; (2) pedestrians or non-motor vehicles run red lights; (3) pedestrians or non-motor vehicles suddenly rush out from the intersection; (4) climb over the guardrail; (5) children suddenly rush to the road. For example, taking the size of the first monitoring image as M1×N1 as an example, the size of the first danger hotspot map is M1×N1, and the value of each position in the first danger hotspot map is the type of abnormal behavior occurring at the corresponding position in the first monitoring image and the corresponding danger probability value of the type. For example, the abnormal behavior type corresponding to the position [a, b] is the first abnormal behavior type, and the danger probability value is P1, 1≤a≤M1, 1≤b≤N1.

[0026] Step 13: Process the first danger hot spot map to obtain abnormal areas where abnormal behaviors exist.

[0027] The output of the abnormal behavior detection network is a heat map of risk factors in the current monitoring scenario. The first risk heat map can be further post-processed to extract the area (i.e., abnormal area) in the current monitoring scenario where abnormal behavior is most likely to occur. Specifically, the first risk heat map can be clustered, and the obtained cluster center is the high-risk abnormal area. The coordinates of the abnormal area are calculated based on the calculated cluster center, and the image corresponding to the abnormal area can be captured. Alternatively, the position with the largest risk probability value in the first risk heat map can be directly obtained, and the area centered on this position is used as the abnormal area. The size of the abnormal area can be pre-set, and the shape of the abnormal area can be a regular shape such as a rectangle or a circle, or the shape of the abnormal area can be set to an irregular shape according to the specific application scenario.

[0028] Step 14: Generate warning information based on abnormal areas.

[0029] After the abnormal area is acquired, an early warning message may be generated, which includes warning information, abnormal behavior type or video data of the abnormal area. A first notification message may be sent to a speaker to cause the speaker to play a preset warning message, which may be a whistle, a buzzer or a preset human voice warning sound, but is not limited thereto. The content or volume of the warning message may be adjusted according to specific application needs; or a second notification message may be sent to a display device to cause the display device to display the abnormal behavior type, a warning message or play a video of the abnormal area, thereby reminding the driver of the motor vehicle that an abnormal behavior is currently occurring and that he needs to slow down or stop, while also reminding pedestrians or drivers of non-motor vehicles.

[0030] In a specific embodiment, Figure 2 As shown, an equipment placement pole 21 is installed at intervals of N meters (for example, 100 meters) on both sides of the road, and a camera 22 is installed on a cross arm (not shown in the figure) of the equipment placement pole 21. One or more cameras 22 can be installed according to needs and conditions; a display device 23 is installed on the same equipment placement pole 21, and a speaker 24 can be installed or not installed on the same equipment placement pole 21 according to conditions.

[0031] In order to ensure full-scene monitoring, the installation height of the camera 22 and the interval between the two equipment placement poles 21 are controlled so that the monitoring field of the camera 22 can cover the road conditions within N meters on one side of the road; and in order to avoid the impact of vehicle occlusion on monitoring, the equipment placement pole 21 can be set on the other side of the road to expand the field of view. The point can be directly opposite or offset with the point on the opposite side (such as Figure 2 As shown). Since the abnormal behavior of pedestrians or non-motor vehicles can only be monitored on the road sections within the monitoring field of view of the camera 22, if the "ghost peeking" behavior is to be monitored on the entire road section, additional points can be set every N meters on all road sections.

[0032] When abnormal behavior is detected in the current monitoring scene, the speaker 24 is used to broadcast warning information to warn pedestrians to turn back and remind passing vehicles to avoid the pedestrians. In addition, the display device 23 can output warning words and play the video stream of the road section where the pedestrians are "looking out" in real time. Figure 2 As shown, a pedestrian 25 appears in front of a motor vehicle 26a. At this time, the loudspeaker 24 can play voice messages such as "pedestrians, please retreat" or "vehicles give way" to remind the pedestrian 25 and the motor vehicles 26a-26b to prevent traffic accidents. The warning message displayed by the display device 23 includes the type of "ghostly poking its head out", which is the first type of abnormal behavior, that is, "a pedestrian or non-motor vehicle suddenly runs out from the front of a motor vehicle."

[0033] The present embodiment provides a method for issuing early warnings for abnormal behaviors of pedestrians / non-motor vehicles, and can cover early warnings for "ghost peeking" on the entire road section, that is, it can monitor the "ghost peeking" behavior at any point on the designated road section, and can comprehensively evaluate potential abnormal behaviors to reduce the probability of danger; and abnormal behaviors can be detected based on a dangerous heat map, so that potential abnormal behaviors can be detected more effectively, and the accuracy of abnormal behavior early warnings can be improved, which helps to improve the effectiveness of early warnings; in addition, voice reminders and screen reminders are combined to use a more direct and effective prompting method to warn pedestrians and drivers, thereby improving the effectiveness of early warnings, reducing the probability that drivers or pedestrians cannot receive early warning information, and further improving road safety.

[0034] See also Figure 3 , Figure 3 : is a flow chart of another embodiment of the abnormal behavior early warning method provided by the present application, the method comprising:

[0035] Step 31: Process the surveillance video data to obtain a first surveillance image.

[0036] Determine the second monitoring image in the monitoring video data. The second monitoring image can be obtained by sampling the monitoring video data at intervals, or the second monitoring image is each frame of the monitoring video data, or the monitoring video data is first detected, and the image with motor vehicles and pedestrians / non-motor vehicles is used as the second monitoring image; first scale the second monitoring image to obtain a scaled image; then crop the scaled image to obtain a sampled image; then remove the mean and normalize the sampled image to obtain the first monitoring image. It can be understood that the existing means are used to achieve the removal of the mean and normalization, which will not be repeated here.

[0037] Furthermore, it can be determined whether the size of the scaled image is larger than a preset size; if the size of the scaled image is larger than the preset size, the scaled image is cropped along the width and / or length direction of the scaled image so that the size of the sampled image is equal to the preset size.

[0038] In a specific embodiment, the sampled second monitoring image is scaled proportionally to obtain a scaled image with an image size of m*k, where m is a preset positive integer, m≤k, and m and k are both the number of pixels; the scaled image is cropped to obtain m*m pixels in the central area of ​​the scaled image as the sampling image; the sampling image is de-meaned and normalized to obtain a final first monitoring image for input into an abnormal behavior detection network.

[0039] In other embodiments, the second monitoring image may be scaled non-proportionally to generate a sampling image; and then the sampling image may be de-meaned and normalized to obtain the first monitoring image.

[0040] Step 32: Obtain a data set, use the training set to train the abnormal behavior detection network, and use the verification set to verify it until the preset training termination condition is met, thereby obtaining a trained abnormal behavior detection network.

[0041] The data set includes a training set and a validation set. The training set and the validation set include multiple training test images and label heat maps corresponding to the training test images. The label heat map is a map generated by setting the danger probability value corresponding to the abnormal behavior type for each location based on Gaussian distribution.

[0042] Furthermore, a dataset with abnormal behaviors is prepared first. The dataset contains training and test images of various abnormal behavior types and dangerous scenes. The abnormal area in each training and test image can be manually selected, and the center point of the abnormal area is used as the center point of the Gaussian distribution. According to the Gaussian distribution, the corresponding danger probability value is set for each of the training and test images to obtain a label hotspot map. Then, the training set is used to train the abnormal behavior detection network, and finally the network parameters with the best performance on the validation set are selected as the network parameters of the abnormal behavior detection network.

[0043] It can be understood that a commonly used loss function can be used to calculate the loss value (recorded as the current loss value) between the training test image and the label heat map corresponding to the training test image; the preset training stop conditions may include: the loss value converges, that is, the difference between the previous loss value and the current loss value is less than the set value; determining whether the current loss value is less than the preset loss value, the preset loss value is a preset loss threshold; the number of training times reaches the set value (for example: training 10,000 times); or the accuracy obtained when testing using the verification set is greater than the set accuracy, etc.

[0044] Step 33: Perform abnormal behavior detection on the first monitoring image to obtain a first danger hotspot map.

[0045] The first monitoring image is input into the abnormal behavior detection network, and the abnormal behavior detection network outputs a danger hotspot map (i.e., the first danger hotspot map) of the same size as the input image through calculation. The value of each position in the first danger hotspot map represents the danger probability value corresponding to the abnormal behavior type at the position.

[0046] Step 34: cluster the first danger hot spot map to obtain a clustering result.

[0047] The first danger hotspot map is clustered using the K-means clustering method to obtain a clustering result, which includes at least one cluster, and the cluster center of the cluster includes a central position and a danger probability value corresponding to the central position; specifically, the cluster includes at least one sample, and the data corresponding to each sample includes a position (i.e., a horizontal coordinate and a vertical coordinate) and a corresponding danger probability value, the central position includes a central horizontal coordinate and a central vertical coordinate, the central horizontal coordinate is a value obtained by averaging the horizontal coordinates of all samples in the cluster, the central vertical coordinate is a value obtained by averaging the vertical coordinates of all samples in the cluster, and the danger probability value corresponding to the central position is a value obtained by averaging the danger probability values ​​of all samples in the cluster.

[0048] In a specific embodiment, it is assumed that the width of the first danger hotspot map is ow, the height is oh, and the upper left corner is the origin of the coordinates. i ,n i ) is s i ,s i Representing the danger probability value output by the abnormal behavior detection network, the first danger heat map can be represented as a set of 3D data of size n (n=ow×oh), as shown below:

[0049]

[0050] in, is the data corresponding to the i-th sample in the first danger hotspot map.

[0051] For the first danger hotspot map defined above, the K-means clustering method is used to segment it. Specifically, for a given first danger hotspot map, the sample set is divided into K clusters according to the distance between each sample in the first danger hotspot map. Assume that the clusters are divided into C = (C1, C2, ..., C K ), C=(m,n,s) T , obtain these K clusters by minimizing the square error:

[0052]

[0053] Where E is the square error, μ i is the cluster C i The mean vector, μ i =[μ im ,μ in ,μ is ],μ i The calculation formula is as follows:

[0054]

[0055] It is understandable that in addition to the K-means clustering method, other clustering methods may also be used, such as hierarchical clustering or density-based clustering methods.

[0056] Step 35: Screening the clustering results to obtain screening results.

[0057] Determine whether the danger probability value of the cluster center of the cluster cluster is greater than the preset danger threshold; if the danger probability value of the cluster center is greater than the preset danger threshold, the cluster cluster is placed in the screening result; if the danger probability value of the cluster center is less than / equal to the preset danger threshold, no processing is performed.

[0058] In a specific embodiment, after obtaining the optimal K cluster centers C = (C1, C2, ..., C K ), substitute the output K cluster centers into the following formula:

[0059]

[0060] in, is the preset danger threshold, s is the danger probability value of the cluster center, δ(C i ) is the judgment result of whether the danger probability value of the cluster center of the i-th cluster is greater than the preset danger threshold.

[0061] Through the above processing, we can finally get the cluster centers C=(C1, C2, …, C Q ).

[0062] Step 36: Based on the screening results, determine the abnormal area.

[0063] The abnormal area includes multiple sub-areas. Based on the central position, a sub-area (i.e., the area where the abnormal behavior occurs) is constructed. The sub-area is an area centered on the central position. Specifically, the sub-area is a rectangular area, and an area with the central position as the center and a set size as the sub-area. For example, assuming that the sub-area is recorded as D = (D1, D2, ..., D Q ), the width of each sub-region is w d , the height of each sub-region is hd , then the definition of the sub-region is as follows:

[0064]

[0065] Among them, D j is the jth sub-region, (m j ,n j ) is the center position of the jth sub-region.

[0066] Step 37: Send the first notification information to the speaker to control the speaker to play the warning information; send the second notification information to the display device to control the display device to play the video data and display the abnormal behavior type.

[0067] The early warning information includes warning information, abnormal behavior type and video data of the abnormal area, and can generate a first notification information and a second notification information. The first notification information is sent to the speaker so that the speaker plays the warning information; at the same time, the second notification information is sent to the display device so that the display device plays the video data, displays the abnormal behavior type or displays a warning message.

[0068] The early warning solution provided in this embodiment has the following advantages:

[0069] (1) The entire road section has a “ghost peeking” warning coverage, which can monitor “ghost peeking” behavior at any point on the designated road section.

[0070] The "ghost peeking" warning system in the related art is limited to specific traffic areas such as "zebra crossings", which is inconsistent with the random location of "ghost peeking" behavior, and most of them require the installation of warning devices on pedestrians' mobile phones and vehicles, which is cumbersome and not conducive to popularization. It also involves the maintenance of remote servers, increasing the burden of traffic management. In the solution provided in this embodiment, the camera, speaker and display device are combined into a monitoring and warning system, and installed on both sides of the road at intervals, which can realize the monitoring and warning of the entire section of the designated road, and only need to be installed on the designated section, which is more conducive to popularization and maintenance.

[0071] (2) Use more direct and effective prompting methods to warn pedestrians and drivers.

[0072] Most of the related technologies send warning information to pedestrians or drivers through multimedia terminals such as mobile phones. The delay and terminal reminder method of this method make it impossible for pedestrians or drivers to obtain the warning information in the first time, which greatly reduces the effectiveness of receiving the warning information. In the solution provided by this embodiment, a loudspeaker is used to broadcast voice warnings, and a display device is used to display warning information and play videos of abnormal areas in real time, so that vehicles and pedestrians on the entire road section can obtain warning information in the first time, which greatly improves the effectiveness of warnings and reduces the possibility of accidents.

[0073] (3) Detecting abnormal behavior based on full-image analysis can more effectively detect potential abnormal areas.

[0074] Related technologies usually rely only on the distance and speed calculated by the radar for early warning, which is not suitable for complex environments with multiple targets and has a high false alarm rate. For motor vehicle drivers, when the early warning system determines the distance, the distance between pedestrians and vehicles is already shrinking. When the judgment is completed and the early warning information is sent to the driver, it is difficult for the driver to quickly respond such as slowing down in a short period of time, which makes it easy to hit pedestrians or non-motor vehicles. This delay actually greatly reduces the effectiveness of the early warning. In the solution provided in this embodiment, by inputting real-time video frame images into the abnormal behavior detection network, the full map information is integrated to predict the danger probability value of the potential dangerous situation in the current monitoring scene. By clustering the output danger hot spot map, the potential abnormal dangerous phenomena in multiple areas can be warned and displayed, improving the safety of the road, and having a higher efficiency in the early warning of the "ghost head" phenomenon.

[0075] See also Figure 4 , Figure 4 It is a structural diagram of an embodiment of the early warning device provided in the present application. The early warning device 40 includes a memory 41 and a processor 42 connected to each other. The memory 41 is used to store a computer program. When the computer program is executed by the processor 42, it is used to implement the early warning method of abnormal behavior in the above-mentioned embodiment.

[0076] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of a monitoring and early warning system provided in the present application. The monitoring and early warning system 50 includes an early warning device 51, a speaker 52 and a display device 53. The early warning device 51 is connected to the speaker 52 and the display device 53, and is used to send notification information to the speaker 52 and the display device 53, wherein the early warning device 51 is the early warning device in the above embodiment.

[0077] The solution provided in this embodiment can be applied to the field of video surveillance in artificial intelligence, especially the field of vehicle warning for "ghost peeking out"; the camera and the display device are combined into a monitoring and warning system, and points are installed at intervals on both sides of the road, so that the entire section of the designated road can be monitored and warned; moreover, a loudspeaker is used to broadcast voice warnings, and a display device is used to display warning information and real-time video of abnormal areas, so that vehicles and pedestrians on the entire road can obtain warning information in the first time, which greatly improves the effectiveness of the warning and reduces the possibility of accidents; in addition, the real-time video frame image is input into the abnormal behavior detection network, and the full map information is integrated to predict the potential danger probability value in the current monitoring scene. By clustering the output danger hot spot map, the potential abnormal danger phenomenon in multiple areas can be warned and displayed, so as to achieve the effect of nipping the danger in the bud, and the warning of the "ghost peeking out" phenomenon is more efficient.

[0078] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 60 is used to store a computer program 61. When the computer program 61 is executed by the processor, it is used to implement the early warning method of abnormal behavior in the above embodiment.

[0079] The computer-readable storage medium 60 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which may store program codes.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0083] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for early warning of abnormal behavior, characterized in that: include: Processing the monitoring video data to obtain a first monitoring image, wherein the monitoring video data is sampled from a real-time video stream, and the first monitoring image is at least two frames of images in the monitoring video data; Performing abnormal behavior detection on the first monitoring image to obtain a first danger hotspot map, wherein the first danger hotspot map includes an abnormal behavior type and a corresponding danger probability value at each pixel position in the first monitoring image; Performing clustering processing on the first danger hot spot map to obtain a clustering result; wherein the clustering result includes at least one cluster, and the cluster center of the cluster includes a center position and a danger probability value corresponding to the center position; Determining whether the danger probability value of the cluster center of the cluster cluster is greater than a preset danger threshold; If yes, the cluster is put into the screening results; Based on the screening results, determining abnormal areas; Generate warning information based on the abnormal area.

2. The abnormal behavior early warning method according to claim 1, characterized in that: The size of the first danger hotspot map is the same as the size of the monitoring image, and the step of performing clustering processing on the first danger hotspot map to obtain a clustering result includes: The first danger hot spot map is clustered using a K-means clustering method to obtain the clustering result.

3. The abnormal behavior early warning method according to claim 2, characterized in that: The abnormal area includes a plurality of sub-areas, and the step of determining the abnormal area based on the screening result includes: Based on the central position, construct the sub-area; The sub-region is a region centered on the central position.

4. The abnormal behavior early warning method according to claim 1, characterized in that: The early warning information includes warning information, the abnormal behavior type and video data of the abnormal area, and the step of generating the early warning information based on the abnormal area includes: Sending a first notification message to a speaker, and controlling the speaker to play a warning message; Sending a second notification message to the display device to control the display device to play the video data and display the abnormal behavior type.

5. The abnormal behavior early warning method according to claim 1, characterized in that: Before the step of performing abnormal behavior detection on the first monitoring image, the method further includes: Obtaining a data set, wherein the data set includes a training set and a validation set; The abnormal behavior detection network is trained using the training set, and verified using the verification set until a preset training termination condition is met, thereby obtaining a trained abnormal behavior detection network; The training set and the validation set include a plurality of training test images and label heat maps corresponding to the training test images, and the label heat map is a map generated by setting a danger probability value corresponding to an abnormal behavior type for each position based on Gaussian distribution.

6. The abnormal behavior early warning method according to claim 1, characterized in that: The step of processing the monitoring video data to obtain the first monitoring image includes: Determining a second surveillance image in the surveillance video data; Scaling the second monitoring image to obtain a scaled image; Cutting the scaled image to obtain a sampled image; The sampled image is de-averaged and normalized to obtain the first monitoring image.

7. The abnormal behavior early warning method according to claim 6, characterized in that: The step of clipping the scaled image to obtain a sampled image comprises: Determining whether the size of the scaled image is greater than a preset size; If so, the scaled image is cropped along the width direction and / or the length direction of the scaled image so that the size of the sampled image is equal to the preset size.

8. An early warning device, characterized in that: It comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the early warning method for abnormal behavior described in any one of claims 1 to 7.

9. A monitoring and early warning system, characterized in that: It comprises an early warning device, a speaker and a display device, wherein the early warning device is connected to the speaker and the display device and is used to send notification information to the speaker and the display device, wherein the early warning device is the early warning device according to claim 8.

10. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the abnormal behavior early warning method according to any one of claims 1 to 7.

Images