Abnormal behavior detection method and device based on patrol robot, medium and equipment

Abstract

The application discloses an abnormal behavior detection method based on a patrol robot, and comprises the following steps: acquiring video stream information of a specified scene, wherein the video stream information is obtained by a patrol robot during patrol in the specified scene; traversing each frame of image information in the video stream information, performing human body detection on the image information in a deep learning manner to obtain behavior feature information corresponding to the human body; performing specified object detection on the image information in a deep learning manner to obtain specified object feature information; and fusing the behavior feature information and the specified object feature information to perform abnormal behavior analysis and acquire abnormal behavior information in the video stream information. The application solves the problems of low intelligentization and automation, low accuracy and high labor cost in existing abnormal behavior detection of a specified scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to an abnormal behavior detection method and device based on a patrol robot, a medium and equipment. BACKGROUND

[0002] In some specific scenarios, the behavior of personnel in the scene must be monitored for abnormalities to avoid safety problems. For example, target personnel in a designated place may exhibit abnormal behaviors such as climbing a fence, hanging themselves, and fighting in a room.

[0003] The prior art mainly uses a fixed camera to take pictures of a room for image analysis to detect abnormal behavior. This scheme uses a fixed camera to take pictures of the corresponding designated room to obtain corresponding video stream information, and then analyzes the video stream using a series of deep learning schemes to determine whether the target personnel have exhibited abnormal behavior. However, the coverage of a single camera is very limited, and if all the rooms in the place are to be covered, the overall cost of the cameras is very high, and the abnormal detection capability needs to be improved.

[0004] The prior art also uses a simple patrol or work robot to manage the place to detect abnormal behavior. Unlike the first scheme, this scheme uses a robot to patrol each room to complete daily guard duty and some simple delivery tasks. The disadvantage is that the robot is too single-functioned, and the degree of intelligence and automation is not high, and the management personnel need to assist in visual analysis to determine whether an abnormality has occurred, which consumes the energy and time of the management staff to some extent. SUMMARY

[0005] The embodiments of the present application provide an abnormal behavior detection method and device based on a patrol robot, a medium and equipment to solve the problems of low intelligence and automation, low accuracy, and high labor cost in the existing abnormal behavior detection of a designated scene.

[0006] An abnormal behavior detection method based on a patrol robot, the method comprising:

[0007] Obtaining video stream information of a designated scene, the video stream information being obtained by a patrol robot patrolling and collecting the designated scene;

[0008] Traversing each frame of image information in the video stream information, performing human body detection on the image information using a deep learning method to obtain behavior feature information corresponding to the human body;

[0009] Performing designated object detection on the image information using a deep learning method to obtain designated object feature information;

[0010] Fusion of the behavior feature information and the designated item feature information is performed for abnormal behavior analysis to obtain abnormal behavior information in the video stream information.

[0011] Optionally, the deep learning-based manner for human body detection on the image information includes:

[0012] The deep learning model YOLOV5 and the top-down method based on a heat map are used for human body detection and human body key node detection on the image information respectively to obtain a bounding box of each human body and human body key nodes and their position information.

[0013] The image information and the human body key nodes and their position information are input into a preset first feature encoding network to obtain behavior feature information corresponding to the human body.

[0014] The behavior feature information is action or behavior information reflected according to a connection relationship of the human body key nodes.

[0015] Optionally, the deep learning-based manner for designated item detection on the image information includes:

[0016] The deep learning model YOLOv5 is used for designated item detection on the image information to obtain designated items and their position information.

[0017] The image information and the position information of the designated items are input into a preset second feature encoding network to obtain designated item feature information.

[0018] Optionally, the fusion of the behavior feature information and the designated item feature information for abnormal behavior analysis to obtain abnormal behavior information in the video stream information includes:

[0019] Based on a preset 3DCNN time series classification network, feature abnormality scores are obtained according to the behavior feature information and the designated item feature information.

[0020] Based on a preset BP neural network, abnormal dangerous item and its logical position abnormality scores are obtained according to the behavior feature information and the designated item feature information.

[0021] The feature abnormality scores and the logical position abnormality scores are used to determine abnormal behavior information in the video stream information.

[0022] Optionally, the 3DCNN time series classification network based on a preset network is used to obtain feature abnormality scores according to the behavior feature information and the designated item feature information.

[0023] For each frame of image information, the behavior feature information and the designated article feature information are fused to obtain fused features of the image information;

[0024] The fused features of the image information of every M frames are input as input parameters into a preset 3DCNN time sequence classification network;

[0025] Abnormal behavior information and a feature abnormality score thereof output by the 3DCNN time sequence classification network are obtained.

[0026] Optionally, the obtaining of the dangerous abnormal article and a logical position abnormality score thereof based on the preset BP neural network according to the behavior feature information and the designated article feature information comprises:

[0027] The left hand, the right hand and / or the neck are selected as the target position;

[0028] For each frame of image information, a pixel distance between the target position and the designated article is obtained according to the behavior feature information and the designated article feature information;

[0029] The pixel distances of all frames of image information in every N seconds are input as input parameters into a preset BP neural network;

[0030] The dangerous abnormal article and the logical position abnormality score thereof output by the BP neural network are obtained.

[0031] Optionally, the determining of the abnormal behavior information in the video stream information according to the feature abnormality score and the logical position abnormality score comprises:

[0032] Respective weight information corresponding to the feature abnormality score and the logical position abnormality score is obtained;

[0033] A comprehensive score is calculated according to the feature abnormality score and the weight information corresponding thereto, and the logical position abnormality score and the weight information corresponding thereto;

[0034] When the comprehensive score is greater than or equal to a preset abnormality threshold, it is determined that the video stream information contains abnormal behavior information, and the abnormal behavior information is the abnormal behavior information output by the 3DCNN time sequence classification network.

[0035] An abnormal behavior detection device based on a patrol robot, the device comprising:

[0036] An acquisition module, configured to acquire video stream information of a designated scene, the video stream information being acquired by a patrol robot during patrol in the designated scene;

[0037] A human body detection module is configured to traverse each frame of image information in the video stream information, and detect the human body in the image information by using deep learning to obtain behavior feature information corresponding to the human body.

[0038] An article detection module is configured to detect a specified article in the image information by using deep learning to obtain specified article feature information.

[0039] An anomaly analysis module is configured to analyze abnormal behavior by fusing the behavior feature information and the specified article feature information to obtain abnormal behavior information in the video stream information.

[0040] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the abnormal behavior detection method based on a patrol robot.

[0041] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the abnormal behavior detection method based on a patrol robot when executing the computer program.

[0042] The embodiments of the present application use a patrol robot to cover all monitoring ranges, effectively reducing the cost of equipment; video stream information of a specified scene is obtained by the patrol robot patrolling and collecting the specified scene; each frame of image information in the video stream information is traversed, and human body detection is performed on the image information by using deep learning to obtain behavior feature information corresponding to the human body; specified article detection is performed on the image information by using deep learning to obtain specified article feature information; abnormal behavior analysis is performed by fusing the behavior feature information and the specified article feature information to obtain abnormal behavior information in the video stream information; thereby realizing abnormal behavior detection in the specified scene, improving the accuracy and robustness of abnormal behavior detection through multi-modal fusion schemes such as human body detection and specified article detection, improving the intelligent and automated degree, and effectively saving the labor cost without personnel assistance in the whole process. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0044] Figure 1 is a patrol robot inspection schematic diagram provided by an embodiment of the present application;

[0045] Figure 2 The abnormal behavior detection method based on a patrol robot is provided by an embodiment of the present application.

[0046] Figure 3 The schematic diagram of the abnormal behavior detection device based on a patrol robot is provided by an embodiment of the present application.

[0047] Figure 4 The schematic diagram of the computer device is provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0049] The embodiments of the present application are based on the fact that a patrol robot constantly patrols a designated place back and forth, and captures images through a camera located at the head of the robot to collect video stream information, so as to analyze whether a phenomenon of abnormal behavior of a person occurs. Figure 1 The patrol robot provided by the embodiments of the present application is provided with a patrol schematic diagram. The black diagonal rectangular frame represents a room, the black dot represents the position of the patrol robot, the short arrow on the black dot represents the direction of the camera on the patrol robot, and the long arrow straight line represents the patrol track of the patrol robot.

[0050] The embodiments of the present application obtain video stream information of a designated scene, the video stream information being obtained by a patrol robot in a designated scene; each frame of image information in the video stream information is traversed, human body detection is performed on the image information in a deep learning manner to obtain behavior feature information corresponding to the human body; specified object detection is performed on the image information in a deep learning manner to obtain specified object feature information; the behavior feature information and the specified object feature information are fused for abnormal behavior analysis to obtain abnormal behavior information in the video stream information; and thus the abnormal behavior detection in the designated scene is realized, the accuracy and robustness of the abnormal behavior detection are improved through the multi-modal fusion scheme such as human body detection and specified object detection, and the intelligent and automatic degree is improved, the whole process does not need personnel assistance, and the labor cost is effectively saved.

[0051] The abnormal behavior detection method based on a patrol robot provided by the embodiments of the present application will be described in detail. Figure 2 The abnormal behavior detection method based on a patrol robot is provided by the embodiments of the present application. As shown in Figure 2 the abnormal behavior detection method based on a patrol robot includes:

[0052] In step S101, video stream information of a specified scene is obtained, which is collected by a patrol robot patrolling the specified scene.

[0053] like Figure 1 As shown, the patrol robot's patrol route is fixed, and the camera on the patrol robot is angled directly towards the room, so that the video stream information captured is basically the scene in front of the room, which greatly facilitates subsequent detection and analysis.

[0054] For video stream information captured by a camera, this embodiment of the invention uses deep learning to analyze and process each frame of the video stream information to determine whether abnormal human behavior has occurred in the room. The analysis process can be divided into the following three stages: human body key point localization, designated object localization, and fusion analysis. Each stage is described in detail below.

[0055] In step S102, each frame of image information in the video stream is traversed, and human body detection is performed on the image information using deep learning to obtain the behavioral feature information corresponding to the human body.

[0056] Here, in this embodiment of the invention, human detection is performed on image information to obtain key node localization, and then behavioral feature information is obtained based on the key node localization. Optionally, step S102 further includes:

[0057] In step S1021, the deep learning model YOLOv5 and a heatmap-based top-down method are used to perform human body detection and human key node detection on the image information to obtain the bounding box of each human body and the key nodes of the human body and their location information.

[0058] Here, the key human body nodes refer to nodes corresponding to joints in the human body, including but not limited to elbows, shoulders, knees, waists, cervical spine, left hand, right hand, left foot, and right foot. This embodiment of the invention uses a heatmap method to perform human body detection and corresponding key node localization on each frame of image information in a video.

[0059] In step S1022, the image information and key human body nodes and their location information are input into a preset first feature encoding network to obtain the behavioral feature information corresponding to the human body.

[0060] Here, the behavior feature information is action or behavior information reflected according to the connection relationship of the human body key nodes. The embodiment of the application obtains the human body key point position corresponding to each frame of image information and the connection relationship of the key points through a preset neural network. The image information and the human body key nodes and their position information are input into a preset first feature encoding network AutoEncoder, a series of convolution calculations are performed, features are extracted, and a 512-dimensional vector representation, that is, the behavior feature information, is obtained for subsequent fusion processing. Among them, different human postures and behaviors correspond to different behavior feature information.

[0061] In step S103, the image information is detected for specified items in a deep learning manner to obtain specified item feature information.

[0062] Here, the specified item detection refers to potential dangerous item detection, including but not limited to cloth objects such as clothes and towels, box-shaped objects that can be used for foot padding or fighting such as boxes and boxes, and sticks such as toothbrushes and pens. Alternatively, the step S103 includes:

[0063] In step S1031, the image information is detected for specified items by using a deep learning model YOLOv5 to obtain the specified items and their position information.

[0064] In step S1032, the image information and the position information of the specified items are input into a preset second feature encoding network to obtain the specified item feature information.

[0065] Here, in a similar manner to the behavior feature information corresponding to the human body, the embodiment of the application inputs the image information and the position information of the specified items into the second feature encoding network AutoEncoder to extract features and obtain a 512-dimensional vector representation, that is, the specified item feature information, for subsequent fusion processing. The specified item feature information refers to the feature information of the potential dangerous item.

[0066] It should be understood that the first feature encoding network is used for human feature extraction, and the second feature encoding network is used for specified item feature extraction, and the network parameters of the two are different, and the extracted feature information is different.

[0067] In step S104, the behavior feature information and the specified item feature information are fused for abnormal behavior analysis to obtain abnormal behavior information in the video stream information.

[0068] After obtaining the behavior feature information through step S102 and obtaining the designated item feature information through step S103, the embodiment of the present application collects and analyzes the behavior feature information and the designated item feature information in the above two steps, and judges whether there is abnormal behavior information in the video stream information. Optionally, the step S104 further includes:

[0069] In step S1041, based on the preset 3DCNN time series classification network, the feature abnormal score is obtained according to the behavior feature information and the designated item feature information.

[0070] Here, the embodiment of the present application first performs feature abnormality analysis based on the 3DCNN time series classification network to obtain a feature abnormal score. The feature abnormal score is a score for judging whether the video frame information has abnormal behavior. The feature abnormal score is a value between 0 and 1, and the closer to 1, the more likely the video frame information has abnormal behavior, and vice versa.

[0071] Optionally, as a preferred example of the present application, the step S1041 further includes:

[0072] In step S411, for each frame of image information, the behavior feature information and the designated item feature information are fused to obtain the fusion feature of the image information.

[0073] Optionally, as described above, the behavior feature information and the designated item feature information are 512-dimensional feature vectors, and the embodiment of the present application connects the behavior feature information and the designated item feature information in front and back to obtain a 1024-dimensional fusion feature.

[0074] In step S412, the fusion feature of every M frames of image information is input as an input parameter to the preset 3DCNN time series classification network.

[0075] Here, the embodiment of the present application takes every 3 seconds as a collection window, and takes the 1024-dimensional fusion feature of 32 frames of image information collected in a uniform skip frame manner as an input sample of the 3DCNN time series classification network.

[0076] Since the input parameters in the embodiment of the present application are feature information instead of original images, the 3DCNN time series classification network in the embodiment of the present application is different from the existing 3DCNN, only using the subsequent full connection layer of the existing 3DCNN, and being composed of the subsequent full connection layer of the 3DCNN and the LSTM network structure. The 3DCNN time series classification network is trained by using the pre-collected positive sample data and negative sample data, and using the sample data with labeled labels as supervision information, and using a common cross-entropy loss function to perform supervision training of the model, so as to obtain a time series analysis model capable of judging whether the video frame information has abnormal behavior.

[0077] In step S413, the abnormal behavior information and the feature abnormal score output by the 3DCNN time series classification network are obtained.

[0078] The 3DCNN time series analysis network outputs a score for judging whether the video frame information has abnormal behavior, that is, the feature abnormal score.

[0079] In order to improve the stability and accuracy of abnormal behavior detection, the embodiment of the present application also provides a logical position abnormality analysis of potential dangerous articles. When a user has an abnormal behavior, for some potential dangerous articles such as towels and boxes, the positions where they are placed are usually deviated from the conventional use positions. The embodiment of the present application assists in judging whether an abnormal behavior occurs by detecting the abnormal positions of the potential dangerous articles.

[0080] In step S1042, based on the preset BP neural network, the behavior feature information and the specified article feature information are used to obtain an abnormal dangerous article and a logical position abnormal score of the abnormal dangerous article.

[0081] Here, the embodiment of the present application performs logical position abnormality analysis based on the BP neural network to obtain a logical position abnormal score. The logical position abnormal score is a score for judging whether the video frame information has an abnormal dangerous article. Here, the embodiment of the present application defaults the potential dangerous article whose placement position is greatly deviated from the conventional position as an abnormal dangerous article. The logical position abnormal score is a value between 0 and 1, and the closer to 1, the more likely that the video frame information has an abnormal dangerous article, and vice versa.

[0082] Optionally, as a preferred example of the present application, the step S1042 further includes:

[0083] In step S421, the left hand, the right hand and / or the neck are selected as the target positions.

[0084] Here, the embodiment of the present application selects the hand and / or the neck as the target position according to common sense experience, and the hand includes the left hand and the right hand.

[0085] In step S422, for each frame of image information, a pixel distance between a target position and a specified article is obtained according to the behavior feature information and the specified article feature information.

[0086] Similarly, in each 3-second acquisition window, the embodiment of the application obtains the behavior feature information and the specified article feature information of all frames in the acquisition window, obtains the hand and the neck according to the behavior feature information, and respectively calculates the pixel distances between the specified article and the left hand, the specified article and the right hand, and the specified article and the neck.

[0087] In step S423, the pixel distances of all frames of image information in each N seconds are taken as input parameters and input into a preset BP neural network.

[0088] Optionally, as a preferred example of the application, the N seconds are preferably 3 seconds to correspond to the input parameters of the 3DCNN time sequence classification network. Assuming that there are 180 frames of image information in 3 seconds, 180*3 pixel distances can be obtained through step S423, and the 180*3 pixel distances are combined as input parameters of the preset BP neural network.

[0089] In step S424, an abnormal dangerous article and a logical position abnormality score thereof output by the BP neural network are obtained.

[0090] Here, the structure of the BP neural network is the same as that of the existing BP neural network, which will not be described here. Similarly, the BP neural network is trained using pre-collected positive sample data and negative sample data, and sample data with labeled labels are used as supervision information, and a general cross-entropy loss function is used for supervised training of the model, so as to obtain a classification model that can determine whether the video frame information contains an abnormal dangerous article and its position.

[0091] After the BP neural network, a score for determining whether the video frame information contains an abnormal article position, i.e., the logical position abnormality score, is output.

[0092] The embodiment of the application further combines the feature abnormality score and the logical position abnormality score for comprehensive abnormality analysis, so as to further improve the robustness and accuracy of abnormal behavior detection.

[0093] In step S1043, the abnormal behavior information in the video stream information is determined according to the feature abnormality score and the logical position abnormality score.

[0094] Optionally, when both the human body and the specified object exist in the image information, the feature anomaly score and the logical position anomaly score are obtained through the above steps, and whether an abnormal behavior appears in the video frame information is determined according to the weighted scores of the two by the embodiment of the application. The step S1043 further comprises:

[0095] In step S431, the weight information corresponding to the feature anomaly score and the logical position anomaly score is obtained respectively.

[0096] In step S432, the comprehensive score is calculated according to the feature anomaly score and the weight information corresponding thereto, the logical position anomaly score and the weight information corresponding thereto.

[0097] In step S433, when the comprehensive score is greater than or equal to a preset anomaly threshold, it is determined that the abnormal behavior information exists in the video stream information, and the abnormal behavior information is the abnormal behavior information output by the 3DCNN time sequence classification network.

[0098] Exemplarily, assuming that the preset anomaly threshold is set to 0.5, the feature anomaly score and the logical position anomaly score are denoted as s1 and s2 respectively, and the weight information is 0.5, then the comprehensive score s = 0.5 * s1 + 0.5 * s2 is calculated. If s is greater than or equal to 0.5, it is considered that an abnormal behavior appears in the video frame information, and the background control system needs to be alarmed, otherwise it is considered that the video frame information is normal.

[0099] Optionally, when only the human body exists in the image information, the feature anomaly score is obtained through the above steps, and the feature anomaly score is taken as the comprehensive score by the embodiment of the application. At this time, the preset anomaly threshold is set to 0.6. If the feature anomaly score is greater than or equal to 0.6, it is considered that an abnormal behavior appears in the video frame information, otherwise it is considered that the video frame information is normal.

[0100] In summary, compared with the fixed camera mode, the abnormal behavior detection method based on the patrol robot provided by the application not only reduces the camera cost, but also expands the detection coverage. Further, the human body feature information and the specified object feature information are detected by using the deep learning method, and the multi-modal fusion analysis is performed, which greatly improves the accuracy and robustness of the abnormal behavior detection, and the intelligent and automatic degree is further improved. The whole process does not need personnel assistance, which effectively saves the labor cost.

[0101] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0102] In an embodiment, the present application also provides a patrol robot-based abnormal behavior detection device corresponding to the patrol robot-based abnormal behavior detection method in the above embodiment. As shown in FIG. 17, the patrol robot-based abnormal behavior detection device includes an acquisition module 31, a human body detection module 32, an article detection module 33, and an anomaly analysis module 34. The functions of each module are described in detail as follows. Figure 3

[0103] The acquisition module 31 is configured to acquire video stream information of a specified scene, which is obtained by a patrol robot during patrol in the specified scene.

[0104] The human body detection module 32 is configured to traverse each frame of image information in the video stream information, and perform human body detection on the image information in a deep learning manner to obtain behavior feature information corresponding to the human body.

[0105] The article detection module 33 is configured to perform specified article detection on the image information in a deep learning manner to obtain specified article feature information.

[0106] The anomaly analysis module 34 is configured to perform abnormal behavior analysis by fusing the behavior feature information and the specified article feature information to obtain abnormal behavior information in the video stream information.

[0107] Optionally, the human body detection module 32 includes:

[0108] A human body detection unit is configured to perform human body detection and human body key node detection on the image information in a top-down manner based on a heat map and by using a deep learning model YOLOV5 to obtain a bounding box of each human body and position information of the human body key node.

[0109] A human body feature extraction unit is configured to input the image information and the position information of the human body key node into a preset first feature encoding network to obtain the behavior feature information corresponding to the human body.

[0110] The behavior feature information is action or behavior information reflected according to a connection relationship of the human body key node.

[0111] Optionally, the article detection module 33 includes:

[0112] An article detection unit is configured to perform specified article detection on the image information by using a deep learning model YOLOv5 to obtain the specified article and position information thereof.

[0113] An article feature extraction unit is configured to input the image information and the position information of the specified article into a preset second feature encoding network to obtain the specified article feature information. ​

[0114] Optionally, the anomaly analysis module 34 comprises:

[0115] a first obtaining unit, configured to obtain a feature anomaly score based on a preset 3DCNN time series classification network according to the behavior feature information and the designated item feature information;

[0116] a second obtaining unit, configured to obtain an abnormal dangerous item and a logic position anomaly score thereof based on a preset BP neural network according to the behavior feature information and the designated item feature information;

[0117] an anomaly analysis unit, configured to determine abnormal behavior information in the video stream information according to the feature anomaly score and the logic position anomaly score.

[0118] Optionally, the first obtaining unit comprises:

[0119] a fusion sub-unit, configured to fuse the behavior feature information and the designated item feature information to obtain a fusion feature of the image information for each frame of image information;

[0120] a first input sub-unit, configured to input the fusion feature of the image information of every M frames as an input parameter to the preset 3DCNN time series classification network;

[0121] a first obtaining sub-unit, configured to obtain abnormal behavior information and a feature anomaly score thereof output by the 3DCNN time series classification network.

[0122] Optionally, the second obtaining unit comprises:

[0123] a selection sub-unit, configured to select a left hand, a right hand and / or a neck as a target position;

[0124] a distance obtaining sub-unit, configured to obtain a pixel distance between the target position and the designated item according to the behavior feature information and the designated item feature information for each frame of image information;

[0125] a second input sub-unit, configured to input the pixel distance of all frames of image information within every N seconds as an input parameter to the preset BP neural network;

[0126] a second obtaining sub-unit, configured to obtain an abnormal dangerous item and a logic position anomaly score thereof output by the BP neural network.

[0127] Optionally, the anomaly analysis unit comprises:

[0128] a weight obtaining sub-unit, configured to obtain weight information corresponding to the feature anomaly score and the logic position anomaly score, respectively;

[0129] a computing subunit, configured to calculate a comprehensive score according to the feature anomaly score and the corresponding weight information, and the logical position anomaly score and the corresponding weight information;

[0130] an abnormal behavior determination subunit, configured to determine that there is abnormal behavior information in the video stream information when the comprehensive score is greater than or equal to a preset abnormal threshold, and the abnormal behavior information is the abnormal behavior information output by the 3DCNN time sequence classification network.

[0131] The specific limitations of the abnormal behavior detection device based on the patrol robot can be referred to the limitations of the abnormal behavior detection method based on the patrol robot, which will not be repeated here. Each module in the above abnormal behavior detection device based on the patrol robot can be realized by software, hardware and their combinations. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the form of software so as to be called and executed by the processor to perform the operations corresponding to the above modules.

[0132] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 4 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an abnormal behavior detection method based on a patrol robot.

[0133] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0134] obtaining video stream information of a specified scene, the video stream information being obtained by a patrol robot during patrol in the specified scene;

[0135] traversing each frame of image information in the video stream information, and performing human body detection on the image information in a deep learning manner to obtain behavior feature information corresponding to the human body;

[0136] performing specified object detection on the image information in a deep learning manner to obtain specified object feature information;

[0137] Fuse the behavior feature information and the designated item feature information to perform abnormal behavior analysis, and obtain abnormal behavior information in the video stream information.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0140] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for detecting abnormal behavior based on a patrol robot, characterized by, The method comprises: acquiring video stream information of a specified scene, the video stream information being obtained by a patrol robot patrolling and collecting the specified scene; traversing each frame of image information in the video stream information, and performing human body detection on the image information in a deep learning manner to obtain behavior feature information corresponding to the human body; specifically, performing human body detection and human body key node detection on the image information respectively by using a deep learning model YOLOV5 and a top-down method based on a heat map to obtain a bounding box of each human body and the human body key node and position information thereof; inputting the image information and the human body key node and position information thereof into a preset first feature encoding network to obtain the behavior feature information corresponding to the human body; the behavior feature information is action or behavior information reflected according to a connection relationship of the human body key node; performing specified article detection on the image information in a deep learning manner to obtain specified article feature information; specifically, performing specified article detection on the image information by using a deep learning model YOLOv5 to obtain the specified article and position information thereof; inputting the image information and the position information of the specified article into a preset second feature encoding network to obtain the specified article feature information; fusing the behavior feature information and the specified article feature information to perform abnormal behavior analysis, and acquiring abnormal behavior information in the video stream information; specifically, based on a preset 3DCNN time series classification network, acquiring a feature abnormal score according to the behavior feature information and the specified article feature information; based on a preset BP neural network, acquiring an abnormal dangerous article and a logical position abnormal score according to the behavior feature information and the specified article feature information; the logical position abnormal score is a value between 0 and 1, which is used to determine whether there is an abnormal dangerous article whose placement position deviates from a regular use position in the video frame information, and the closer the value is to 1, the more likely it is that the abnormal dangerous article appears in the video frame information; and determining the abnormal behavior information in the video stream information according to the feature abnormal score and the logical position abnormal score.

2. The abnormal behavior detection method based on a patrol robot according to claim 1, wherein, The method comprises: for each frame of image information, fusing the behavior feature information and the specified article feature information to obtain fusion features of the image information; inputting the fusion features of every M frames of image information as input parameters into the preset 3DCNN time series classification network; acquiring abnormal behavior information and a feature abnormal score output by the 3DCNN time series classification network.

3. The abnormal behavior detection method based on a patrol robot according to claim 1, wherein The method comprises: selecting a left hand, a right hand and / or a neck as a target position; for each frame of image information, acquiring a pixel distance between the target position and the specified article according to the behavior feature information and the specified article feature information; inputting the pixel distances of all frames of image information in every N seconds as input parameters into the preset BP neural network; acquire the dangerous abnormality and its logic position abnormality score output by the BP neural network.

4. The abnormal behavior detection method based on a patrol robot according to claim 1, wherein, The determination of the abnormal behavior information in the video stream information according to the feature abnormality score and the logic position abnormality score comprises: respectively acquiring weight information corresponding to the feature abnormality score and the logic position abnormality score; calculating a comprehensive score according to the feature abnormality score and its corresponding weight information, the logic position abnormality score and its corresponding weight information; when the comprehensive score is greater than or equal to a preset abnormality threshold, determining that there is abnormal behavior information in the video stream information, and the abnormal behavior information is the abnormal behavior information output by the 3DCNN time series classification network.

5. An abnormal behavior detection device based on a patrol robot, characterized in that, The device comprises: an acquisition module, configured to acquire video stream information of a specified scene, the video stream information being acquired by a patrol robot in a specified scene; a human body detection module, configured to traverse each frame of image information in the video stream information, and perform human body detection on the image information in a deep learning manner to obtain behavior feature information corresponding to a human body; the human body detection module is configured to perform human body detection and human body key node detection on the image information respectively by using a deep learning model YOLOV5 and a top-down method based on a heat map to obtain a bounding box of each human body and position information of human body key nodes; input the image information and the position information of the human body key nodes into a preset first feature encoding network to obtain the behavior feature information corresponding to the human body; the behavior feature information is action or behavior information reflected according to a connection relationship of the human body key nodes; an article detection module, configured to perform specified article detection on the image information in a deep learning manner to obtain specified article feature information; the article detection module is specifically configured to perform specified article detection on the image information by using a deep learning model YOLOv5 to obtain specified articles and position information of the specified articles; and input the image information and the position information of the specified articles into a preset second feature encoding network to obtain the specified article feature information; an abnormality analysis module, configured to perform abnormal behavior analysis by fusing the behavior feature information and the specified article feature information to acquire abnormal behavior information in the video stream information; the abnormality analysis module is configured to acquire a feature abnormality score according to the behavior feature information and the specified article feature information based on a preset 3DCNN time series classification network; acquire dangerous abnormality and its logic position abnormality score according to the behavior feature information and the specified article feature information based on a preset BP neural network; the logic position abnormality score is a value between 0 and 1, used to determine whether there is dangerous abnormality deviating from a regular use position in the video frame information, and the closer the value is to 1, the more likely it is that there is dangerous abnormality in the video frame information; and determine the abnormal behavior information in the video stream information according to the feature abnormality score and the logic position abnormality score.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by a processor to implement the abnormal behavior detection method based on a patrol robot according to any one of claims 1 to 4. The computer program is executed by a processor to implement the abnormal behavior detection method based on a patrol robot according to any one of claims 1 to 4.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the abnormal behavior detection method based on the patrol robot according to any one of claims 1 to 4 when executing the computer program.

Images