Intelligent inspection behavior detection method, device and equipment based on space-time motion detection
Through space-time action detection technology, the feature fusion and behavior analysis of the video stream in the smart inspection scenario is solved, and the problem of difficulty in taking into account detection accuracy and efficiency in the existing technology is achieved, and efficient abnormal behavior detection in complex personnel scenarios is achieved.
Patent Information
- Application Number
- CN202411098820.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
It is difficult for existing behavioral detection technologies to take into account both detection accuracy and efficiency in smart inspection scenarios, especially in complex scenarios, there are problems of action ambiguity and high computing power requirements.
Using a method based on spatiotemporal action detection, the video stream frame sequence is characterized by fusion analysis and behavior analysis through the feature analysis model group, and abnormal behavior detection is performed in combination with the prediction model group, and spatial-temporal feature fusion results and frame sequence analysis results are generated to realize abnormal behavior detection. .
It effectively improves the accuracy and efficiency of behavior detection algorithms, can be applied to complex scenarios of personnel, solves the problem that the existing technology cannot take into account both accuracy and efficiency, and enhances the algorithm's recognition ability.
Smart Images

Figure CN119091501A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of behavioral activity detection, and in particular to a method, device and equipment for intelligent patrol behavior detection based on spatiotemporal motion detection. Background Art
[0002] At present, behavior detection technology is increasingly being used, such as in the smart inspection system. Specifically, smart inspection mainly uses detection and recognition algorithms to detect and recognize the behaviors of examinees and invigilators in real time. Therefore, the accuracy and efficiency of the detection and recognition algorithm are key indicators for evaluating its performance in practical applications.
[0003] There are two main types of existing behavior detection technologies used in smart inspections. One is the method of detection and identification through algorithms. Usually, the target detection algorithm is used to directly complete detection and identification. Although the algorithm is efficient, since this method detects a single frame, there will be action ambiguity in scenarios with multiple frame rates. The other is to first perform target detection screening and then connect the video time series classification model in series. This method will achieve a certain accuracy rate due to the addition of time series information, but for the inspection scenario, a standard examination room contains at least 32 people, and both model parallel reasoning and serial reasoning will require high computing power support. Other solutions such as multi-camera collaborative analysis and 2D base video classification models have this problem. It can be seen that the existing behavior detection technology / algorithms cannot take into account both accuracy and efficiency at the same time, and there are problems such as limited usage scenarios. Summary of the invention
[0004] The present application provides a method, device and equipment for intelligent patrol behavior detection based on spatiotemporal motion detection. By performing feature fusion analysis and behavior analysis on the frame sequence of the video stream, feature fusion results and frame sequence analysis results are obtained. Abnormal behavior detection is then performed based on the feature fusion results and frame sequence analysis results to obtain abnormal behavior detection results. The application effectively takes into account the requirements of abnormal behavior detection algorithms for accuracy and efficiency, enhances the algorithm recognition capability, and can be applied to scenes with complex personnel to solve the problems existing in the prior art.
[0005] In a first aspect, the present application provides a method for detecting intelligent inspection behavior based on spatiotemporal motion detection, comprising:
[0006] Acquire a target video stream containing a target object, and extract a frame sequence from the target video stream;
[0007] Performing feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result;
[0008] Based on the spatiotemporal feature fusion result and the frame sequence, a behavior analysis is performed through a preset prediction model group to obtain a frame sequence analysis result;
[0009] Abnormal behavior detection is performed based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain an abnormal behavior detection result of the target object.
[0010] Optionally, the feature analysis model group includes a first feature analysis model and a second feature analysis model, and the feature analysis fusion of the frame sequence is performed by using a preset feature analysis model group to obtain a spatiotemporal feature fusion result, including:
[0011] Extracting features from key frames in the frame sequence using the first feature analysis model to obtain spatial feature information, and extracting features from the frame sequence using the second feature analysis model to obtain spatiotemporal feature information;
[0012] Perform feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information;
[0013] Performing behavior prediction on the spatiotemporal feature fusion information to obtain a predicted category, a predicted target box, and a first confidence level corresponding to the target object;
[0014] Based on the prediction category, the prediction target box, the first confidence level, and the spatiotemporal feature fusion information, a spatiotemporal feature fusion result is generated.
[0015] Optionally, extracting features from the key frame using the first feature analysis model to obtain spatial feature information includes:
[0016] Performing image scaling on the key frame to obtain a target key image frame;
[0017] For the target key image frame, extract features using the first feature analysis model to obtain a multi-scale feature map, and use the multi-scale feature map as the spatial feature information;
[0018] The multi-scale feature map includes at least three two-dimensional branch feature maps.
[0019] Optionally, extracting features from the frame sequence using the second feature analysis model to obtain spatiotemporal feature information includes:
[0020] Performing image scaling on each image frame in the frame sequence to obtain a scaled frame sequence, wherein the scaled frame sequence includes at least one scaled image frame;
[0021] For the zoomed image frame, feature extraction is performed using the second feature analysis model to obtain a three-dimensional branch feature map, and the three-dimensional branch feature map is used as the spatiotemporal feature information.
[0022] Optionally, the performing feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information includes:
[0023] Perform image sampling based on the three-dimensional branch feature map to obtain sampling information;
[0024] Feature fusion is performed according to the multi-scale feature map, the three-dimensional branch features and the sampling information to obtain a multi-scale fusion feature map, and the multi-scale fusion feature map is used as the spatiotemporal feature fusion information.
[0025] Optionally, the prediction model group includes a key point prediction model and a classification model, and the behavior analysis is performed through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence to obtain a frame sequence analysis result, including:
[0026] Extracting the target frame coordinates of the target object from the spatiotemporal feature fusion result;
[0027] Performing image cropping on the image frames in the frame sequence based on the target frame coordinates to obtain a cropped image of the target object;
[0028] Performing skeleton construction on the cropped image by using the key point prediction model to obtain skeleton motion information of the target object, wherein the skeleton motion information includes a time-series pedestrian skeleton motion map, and the time-series pedestrian skeleton motion map includes abnormal behavior data;
[0029] For the skeleton motion information, abnormal classification is performed using the classification model to obtain a second confidence level, and the second confidence level is used as the frame sequence analysis result.
[0030] Optionally, performing abnormal behavior detection according to the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object includes:
[0031] Extracting a first confidence level and a predicted category from the spatiotemporal feature fusion result, and extracting a second confidence level and abnormal behavior analysis data from the frame sequence analysis result;
[0032] Performing weighted processing based on the first confidence level and the second confidence level to obtain a final behavior confidence level;
[0033] According to the abnormal behavior data, the final behavior confidence, the prediction category and the preset action behavior set are combined to perform discrimination processing to obtain the abnormal behavior detection result.
[0034] Optionally, after obtaining the abnormal behavior detection result of the target object, the method further includes:
[0035] Based on the abnormal behavior detection result, performing behavior frame analysis on the frame sequence to obtain the target abnormal behavior of the target object and the starting frame where the target abnormal behavior occurs;
[0036] For the target object, data is constructed using the target abnormal behavior and the starting frame to obtain an initial structure of the target object;
[0037] For the target abnormal behavior, continuously perform behavior deduplication detection on the image frames after the start frame in the frame sequence to obtain a continuous frame set of the target abnormal behavior, wherein the continuous frame set is used to characterize the duration period of the target abnormal behavior from the start to the end;
[0038] Update the initial structure based on the continuous frame set to obtain a target structure;
[0039] A behavior warning is issued within the time period corresponding to the target structure.
[0040] In the second aspect, the present application provides a smart inspection behavior detection device based on spatiotemporal motion detection, comprising:
[0041] An acquisition module, used for acquiring a target video stream containing a target object and extracting a frame sequence from the target video stream;
[0042] A feature analysis and fusion module is used to perform feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result;
[0043] A behavior analysis module, used to perform behavior analysis through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence to obtain a frame sequence analysis result;
[0044] The abnormal behavior detection module is used to perform abnormal behavior detection according to the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object.
[0045] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0046] Memory, used to store computer programs;
[0047] The processor is used to implement the steps of the intelligent patrol behavior detection method based on spatiotemporal motion detection as described in any embodiment of the first aspect when executing the program stored in the memory.
[0048] In summary, the embodiments of the present application obtain a target video stream containing a target object, extract a frame sequence from the target video stream, perform feature analysis and fusion on the frame sequence through a preset feature analysis model group, and obtain a spatiotemporal feature fusion result, perform behavior analysis based on the spatiotemporal feature fusion result and the frame sequence through a preset prediction model group, and obtain a frame sequence analysis result, perform abnormal behavior detection based on the feature fusion result and the frame sequence analysis result, and obtain an abnormal behavior detection result of the target object, thereby effectively taking into account the requirements of the abnormal behavior detection algorithm for accuracy and efficiency, enhancing the algorithm recognition capability, and being applicable to scenes with complex personnel, solving the problem that the existing behavior detection technology cannot take into account both accuracy and efficiency and is limited in usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0051] Figure 1 A flowchart of a smart inspection behavior detection method based on spatiotemporal motion detection provided in an embodiment of the present application;
[0052] Figure 2 It is a schematic diagram of the steps of a smart inspection behavior detection method based on spatiotemporal action detection provided by an optional embodiment of the present application;
[0053] Figure 3 is a flow chart of behavior detection technology provided by an example of this application;
[0054] Figure 4 This is a framework diagram of a temporal action recognition model of a behavior detection method provided by an example of this application;
[0055] Figure 5 This is a feature fusion module diagram of a behavior detection method provided by an example of this application;
[0056] Figure 6This is a flow chart of a behavior prediction and deduplication algorithm of a behavior detection method provided by an example of this application;
[0057] Figure 7 A structural block diagram of a smart inspection behavior detection device based on spatiotemporal action detection provided in an embodiment of the present application;
[0058] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0060] In related technologies, with the widespread application of behavior detection technology, smart inspection is also known as one of the application scenarios of behavior detection technology. Smart inspection is a test supervision method that combines advanced technologies such as artificial intelligence, the Internet of Things, and big data analysis. It aims to improve the fairness, security, and efficiency of the test, and overcome the limitations of traditional manual invigilation through automation and intelligent means. The smart inspection system captures video and audio information in the test room in real time by deploying high-definition surveillance cameras, sensors and other equipment in the test room. Using artificial intelligence algorithms, the system can automatically identify suspicious behaviors of candidates, such as cheating and whispering. At the same time, through big data analysis, the system can predict potential violations, thereby achieving comprehensive monitoring of the test process. In addition, smart inspection also reduces dependence on manual invigilation by optimizing human resource allocation. It not only improves the response speed and processing efficiency of invigilation, but also reduces supervision costs, providing an innovative supervision solution for education and examination departments. With the continuous advancement of technology, smart inspection is expected to play a more important role in future education examinations.
[0061] To facilitate the understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.
[0062] In a specific implementation, the behavior detection method provided in the embodiment of the present application can be applied to the behavior analysis of students and invigilators in the examination room, such as smart inspection, non-perception identity verification in the examination room, classroom student behavior analysis and other scenarios.
[0063] Figure 1The flowchart of a method for detecting intelligent patrol behavior based on spatiotemporal motion detection provided by the embodiment of the present application is as follows. Figure 1 As shown, the smart inspection behavior detection method based on spatiotemporal action detection provided in the embodiment of the present application may specifically include the following steps:
[0064] Step 110: Acquire a target video stream containing a target object, and extract a frame sequence from the target video stream.
[0065] In this embodiment, the target video stream may include multiple captured target objects, that is, multiple target objects may be captured in one video; the frame sequence may include one or more video image frames (or video frame images, image frames, frame images, etc.), and the frame sequence may be obtained by extracting frames from the target video stream, and the frame extraction method may be sequential frame extraction or random frame extraction. For example, for sequential frame extraction, starting from a certain frame of the target video stream, a certain number of continuous frames after or before the frame may be extracted to form a continuous frame sequence; for random frame extraction, a certain number of frames may be randomly extracted from the target video stream to obtain a frame sequence, and this example does not impose any restrictions on this.
[0066] Exemplarily, taking the application scenario of smart examination inspection as an example, the target video stream can be obtained by shooting video through the surveillance camera installed in the examination room. At this time, the target video stream may capture one or more candidates and invigilators, and the captured candidates and / or invigilators can be used as target objects. Then a certain number of video image frames are extracted from the target video stream. For example, taking the image frames captured by the current target video stream as an example, taking the current frame as a reference, a certain number of image frames are obtained from the target video stream according to a preset number to form a frame sequence. As a result, the present embodiment obtains a frame sequence containing multiple single frames, and the frame sequence can be subsequently subjected to action / behavior detection and recognition, which can avoid problems such as action ambiguity caused by the existing algorithm that only detects single frames.
[0067] Step 120, performing feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result.
[0068] Specifically, the feature analysis model group may include at least two feature analysis models, and the types of these feature analysis models may be different, so as to extract feature information from the frame sequence from multiple angles and dimensions and obtain spatiotemporal feature fusion results. It should be noted that this embodiment does not limit the number and types of feature analysis models included in the feature analysis model group.
[0069] In a specific implementation, the feature analysis model group can extract features from the frame sequence through multiple feature analysis models. Specifically, by inputting the frame sequence into each feature analysis model, the feature analysis and feature extraction of the frame sequence can be performed through the feature analysis model, and at least feature information such as spatial features and spatiotemporal features can be extracted from the frame sequence. Subsequently, the spatial features and spatiotemporal features can be used to perform category prediction, position recognition, and confidence prediction on the target objects involved in the frame sequence, and generate spatiotemporal feature fusion results.
[0070] Specifically, for category prediction, the present embodiment can use the extracted feature information to classify the target object, such as identifying which specific type of person the target object is (taking smart examination inspection as an example, it can predict whether the target object is a candidate or an invigilator, etc.), and obtain a category prediction result; for position recognition, the present embodiment uses feature information to perform position recognition on the target object in the image frame, that is, to identify the position coordinates of the target object in the image frame, and frame the target object in the form of a target frame, so that subsequent behavior analysis can be performed on the target object; for confidence prediction, when the present embodiment performs category prediction on the target object, the credibility of the category prediction result can be obtained as the confidence.
[0071] Step 130, based on the spatiotemporal feature fusion result and the frame sequence, a behavior analysis is performed through a preset prediction model group to obtain a frame sequence analysis result.
[0072] Specifically, the frame sequence analysis results may include the behavior analysis results of the target object (including the identified abnormal behavior of the target object) and the confidence level corresponding to the behavior analysis results (such as the confidence level of the abnormal behavior). This embodiment does not impose any restrictions on this. The prediction model group may include multiple prediction models, and the types of these prediction models may be different, so that different types of prediction models can be used to predict the abnormal behavior of the target object and information such as the confidence level of the abnormal behavior.
[0073] Specifically, this embodiment inputs the confidence and frame sequence in the spatiotemporal feature fusion result into the prediction model group, and uses the prediction model in the prediction model group to perform continuous behavior prediction on the target object in the target frame using the target frame carried by the spatiotemporal feature fusion result, and obtains the predicted abnormal behavior corresponding to the target object, thereby realizing the behavior analysis of the target object, and the prediction model can also perform confidence prediction for the predicted abnormal behavior, and obtain the confidence corresponding to the abnormal behavior. Therefore, the embodiment of the present application realizes high-precision detection of action behavior.
[0074] In actual processing, in order to take into account both high-precision detection and efficiency of action behaviors, this embodiment can extract key frames from the frame sequence when predicting the abnormal behavior of the target object, and perform behavioral analysis on the key frames through the prediction model group, thereby eliminating the need to perform behavioral analysis on all frame sequences, thereby effectively improving the efficiency of action behavior detection.
[0075] Step 140: Perform abnormal behavior detection based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain an abnormal behavior detection result of the target object.
[0076] In the specific implementation, both the spatiotemporal feature fusion results and the frame sequence analysis results include confidence levels. This embodiment can combine the confidence levels of the above two results to obtain a final confidence level to judge the predicted category of the target object. For example, a confidence threshold can be set. When the final confidence level is greater than the confidence threshold, it indicates that the predicted category of the target object in the spatiotemporal feature fusion result is the correct category. The target object can then be judged by combining the predicted category and the abnormal behavior in the frame sequence analysis results. For example, for the intelligent inspection scenario, when the predicted category of the target object is a candidate, if the abnormal behavior of the target object indicates that the target object has suspicious behaviors such as cheating and whispering, then it can be determined that the target object is a candidate and has abnormal behavior. At this time, the abnormal behavior detection result is an abnormal behavior result, which contains the abnormal behavior of the target object. Subsequently, an alarm can be issued for the abnormal behavior result to remind the invigilator that the candidate is suspected of cheating. If the abnormal behavior in the frame sequence analysis result is an empty set, or other related results that indicate that the target object has no abnormal behavior, it can be determined that the target object has no suspicious behaviors such as cheating. At this time, a non-abnormal behavior result can be generated as an abnormal behavior detection result. When the predicted category of the target object is an invigilator, a non-abnormal behavior result can be directly generated.
[0077] In addition, when the final confidence is not greater than the confidence threshold, that is, the predicted category of the target object in the spatiotemporal feature fusion result may not be the correct category, then the frame sequence can be re-detected and recognized.
[0078] It can be seen that the embodiment of the present application obtains a target video stream containing a target object, extracts a frame sequence from the target video stream, performs feature analysis and fusion on the frame sequence through a preset feature analysis model group, and obtains a spatiotemporal feature fusion result, performs behavior analysis through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence, and obtains a frame sequence analysis result, performs abnormal behavior detection based on the spatiotemporal feature fusion result and the frame sequence analysis result, and obtains an abnormal behavior detection result of the target object, thereby effectively taking into account the requirements of the abnormal behavior detection algorithm for accuracy and efficiency, enhancing the algorithm recognition capability, and being applicable to scenes with complex personnel, solving the problem that the existing behavior detection technology cannot take into account both accuracy and efficiency and is limited in usage scenarios.
[0079] Reference Figure 2 , shows a schematic flow chart of the steps of a smart patrol behavior detection method based on spatiotemporal action detection provided by an optional embodiment of the present application. The method may specifically include the following steps:
[0080] Step 210: Acquire a target video stream containing a target object, and extract a frame sequence from the target video stream.
[0081] For example, refer to Figure 3 Taking the smart inspection as an example, the examination room is filmed by using the surveillance cameras and other shooting equipment installed in the examination room, thereby obtaining a surveillance video stream of multiple target objects including candidates and / or invigilators as the target video stream. Then, based on the current image frame, the 16 image frames before the current image frame can be extracted from the target video stream as a frame sequence, and the frame sequence can be stored.
[0082] Step 220: extracting features from key frames in the frame sequence using the first feature analysis model to obtain spatial feature information, and extracting features from the frame sequence using the second feature analysis model to obtain spatiotemporal feature information.
[0083] The feature analysis model group includes a first feature analysis model and a second feature analysis model.
[0084] In this embodiment, the first feature analysis model can be a 2D convolution model, and the second feature analysis model can be a 3D convolution model, which is not limited in this embodiment. A key frame is a frame at a fixed position in a frame sequence. Taking a video stream of 25 frames per second (s for short) as an example, a key frame can be the first frame per second in the video stream. Of course, the key frame in the frame sequence can also be set according to actual needs, which is not limited in this embodiment.
[0085] In a specific implementation, the embodiment of the present application can extract key frames from a frame sequence and input the key frames into a 2D convolution model to extract spatial features from the key frames through the 2D convolution model. At the same time, the frame sequence can be input into a 3D convolution model to extract spatiotemporal features through the 3D convolution model.
[0086] In actual processing, the yolov8 backbone network can be selected as the 2D convolution model, and the I3D model can be selected as the 3D convolution model. In order to improve recognition efficiency and reduce model resource consumption, this embodiment can also scale the image frame according to actual needs before inputting the image frame into the convolution model. For example, refer to Figure 4 For key frames extracted from a frame sequence, the key frames can be scaled to a target resolution, which can be 640*640, that is, the key frames can be scaled to a resolution of 640*640 and input into the yolov8 backbone network to extract spatial features; for a frame sequence, the resolution of all image frames in the frame sequence can be scaled before the frame sequence is input into the I3D model to obtain a 320*320 image frame sequence, which can be input into I3D to extract spatiotemporal features.
[0087] In an optional embodiment, the embodiment of the present application performs feature extraction on the key frame through the first feature analysis model to obtain spatial feature information, which may specifically include: performing image scaling on the key frame to obtain a target key image frame; for the target key image frame, using the first feature analysis model to perform feature extraction to obtain a multi-scale feature map, and using the multi-scale feature map as the spatial feature information; wherein the multi-scale feature map includes at least three two-dimensional branch feature maps.
[0088] In a specific implementation, the present embodiment can characterize spatial features through a multi-scale feature map. Specifically, the scaled key image frame is input into the first feature analysis model, and the first feature analysis model can perform feature analysis and feature extraction on the scaled key image frame, and extract multiple 2D branch feature maps from the key image frame as two-dimensional branch feature maps.
[0089] Optionally, an embodiment of the present application performs feature extraction on the frame sequence through the second feature analysis model to obtain spatiotemporal feature information, which may specifically include: performing image scaling on each image frame in the frame sequence to obtain a scaled frame sequence, wherein the scaled frame sequence includes at least one scaled image frame; for the scaled image frame, performing feature extraction through the second feature analysis model to obtain a three-dimensional branch feature map, and using the three-dimensional branch feature map as the spatiotemporal feature information.
[0090] In the specific implementation, considering the requirement of algorithm efficiency and to reduce the amount of calculation, this embodiment can scale the image frames in the frame sequence to obtain scaled image frames, and then input the scaled image frames into the second feature analysis model for feature extraction to obtain a 3D branch feature map as a three-dimensional branch feature map.
[0091] Step 230: Perform feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information.
[0092] In this embodiment, the spatial feature information includes multiple two-dimensional branch feature maps, and the spatiotemporal feature information includes a three-dimensional branch feature map. The two-dimensional branch feature map and the three-dimensional branch feature map can be fused using feature fusion to obtain a multi-scale fused feature map that can simultaneously characterize spatial features and spatiotemporal features as spatiotemporal feature fusion information.
[0093] In an optional embodiment, the embodiment of the present application performs feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information, which may specifically include: performing image sampling based on the three-dimensional branch feature map to obtain sampling information; performing feature fusion based on the multi-scale feature map, the three-dimensional branch features and the sampling information to obtain a multi-scale fusion feature map, and using the multi-scale fusion feature map as the spatiotemporal feature fusion information.
[0094] For example, refer to Figure 4 and Figure 5 , S1, S2, and S3 can be used to refer to the three extracted two-dimensional branch feature maps, and ST can be used to refer to the extracted three-dimensional branch feature map. The multi-scale feature maps S1, S2, and S3 are extracted from the scaled key frames through the yolov8 backbone network, and the 2D branch feature map is represented by {S1, S2, S3}. For the extraction of three-dimensional branch feature maps, this example can use the I3D model as the spatiotemporal feature extraction model. The basic model is Resnet50 pre-trained with the ImageNet dataset, and then the pre-trained 2D convolution kernel weights are self-replicated to increase the temporal dimension, and the N×N filter is repeatedly replicated N times to become N×N×N as the initialization of the I3D model. In addition, considering the efficiency of the algorithm, for some branches of the 3D convolution, the image frames contained in the frame sequence are scaled to 320*320 resolution to reduce the amount of calculation. Then the scaled frame sequence is input into the 3D convolution branch output feature map ST as the 3D branch feature map, thereby obtaining the 2D branch feature map {S1, S2, S3} and the 3D branch feature map ST. Figure 5As shown in the feature fusion module diagram, after obtaining the 2D branch feature map {S1, S2, S3} and the 3D branch feature map ST, upsampling can be performed on the ST feature map, and feature fusion can be performed with the 2D branch feature map {S1, S2, S3} to obtain a new multi-scale fusion feature map RS, RS = {S1+(ST*4upsample), S2+(ST*2upsample), S3+(ST*1upsample)}, and the multi-scale fusion feature map is used as spatiotemporal feature fusion information for subsequent behavior prediction processing.
[0095] Step 240 , performing behavior prediction on the spatiotemporal feature fusion information to obtain a predicted category, a predicted target box, and a first confidence level corresponding to the target object.
[0096] Step 250, generating a spatiotemporal feature fusion result based on the prediction category, the prediction target box, the first confidence and the spatiotemporal feature fusion information.
[0097] A unified description of steps 240 to 250 is given as follows:
[0098] In actual implementation, the multi-scale fusion feature map can be input into the first feature analysis model. Through the first feature analysis model, behavior prediction can be performed to preliminarily predict whether the target object has abnormal behavior. If the target object has abnormal behavior, the first feature model can predict the category of the target object and obtain the predicted category. The first feature analysis model can also calculate the confidence level based on the predicted category, etc., as the first confidence level. In addition, the first feature analysis model can identify / predict the coordinate position of the target object on the image frame of the frame sequence based on the multi-scale fusion feature map, and frame the target object in the form of a target frame. Therefore, by inputting the spatiotemporal feature fusion information into the first feature analysis model, the predicted category, predicted target frame and first confidence level of the target object output by the first feature analysis model are obtained, and the spatiotemporal feature fusion information can be combined to generate the spatiotemporal feature fusion result.
[0099] For example, refer to Figure 4Taking the intelligent inspection scene as an example, the multi-scale fusion feature map RS is input into the decode_box layer and non-maximum suppression layer (Non-MaximumSuppression) of the original yolov8. Yolov8 is used to predict the action behavior of the multi-scale fusion feature map to determine whether the target object has abnormal behavior. For the predicted abnormal behavior, yolov8 is used to predict the category of the target object involved in the frame sequence. It can be predicted which type of person the target object is, such as whether the target object is a candidate or an invigilator, thereby obtaining the predicted category corresponding to the target object, and the confidence 1 can be calculated for the predicted category, as well as the position of the target object in the predicted frame sequence to obtain the predicted target box.
[0100] Step 260: extracting the target frame coordinates of the target object from the spatiotemporal feature fusion result.
[0101] Step 270 , cropping the image frames in the frame sequence based on the target frame coordinates to obtain a cropped image of the target object.
[0102] Step 280: construct a skeleton of the cropped image using the key point prediction model to obtain skeleton motion information of the target object.
[0103] The skeleton motion information includes a time-series pedestrian skeleton motion graph, and the time-series pedestrian skeleton motion graph includes abnormal behavior data.
[0104] Wherein, the prediction model group includes a key point prediction model and a classification model.
[0105] Step 290, for the skeleton motion information, perform abnormal classification through the classification model to obtain a second confidence level, and use the second confidence level as the frame sequence analysis result.
[0106] A unified description of steps 260 to 290 is given as follows:
[0107] In a specific implementation, the prediction model group may include multiple prediction models such as key point prediction models and classification models, wherein the key point prediction models include but are not limited to openpose, HRnet, etc., and the classification model may also be called a skeleton behavior classification model, including but not limited to the STGCN model, which is not limited in this embodiment.
[0108] Specifically, this embodiment can extract the target frame coordinates of the target object from the spatiotemporal feature fusion result, and then use the target frame coordinates to crop the image frames of the original frame sequence, thereby obtaining a cropped image containing the target object, and input the cropped image into the key point prediction model. The key point prediction model can construct the motion skeleton of the target object from the cropped image to obtain a time-series pedestrian skeleton motion map, which contains the abnormal behavior of the target object and is used to characterize whether the target object has abnormal behavior. Subsequently, the time-series pedestrian skeleton motion map can be input into the classification model, and the confidence of the abnormal behavior of the target object is calculated by the classification model to obtain the abnormal behavior confidence as the second confidence. Therefore, this embodiment adds the I3D model to provide multi-frame timing information on the basis of the single-frame behavior detection algorithm based on yolov8, and cascades the key point timing classification algorithm, effectively combining the image space features, the inter-frame timing features and the human body posture skeleton motion information to enhance the recognition ability of the algorithm, and realizes a smart patrol action detection algorithm that takes both accuracy and efficiency into consideration.
[0109] For example, refer to Figure 4 For the predicted abnormal behavior, the corresponding target frame coordinates and the original image are extracted and expanded and cropped before being input into the human key point prediction model to obtain the temporal pedestrian skeleton motion graph K output by the key point prediction model. The skeleton motion information K is then input into the skeleton behavior classification model to obtain the abnormal behavior confidence 2 output by the skeleton behavior classification model.
[0110] Reference Figure 3 In actual implementation, a temporal action recognition model can be preset. The temporal action recognition model can aggregate the first feature analysis model and the second feature analysis model in the feature analysis model group and the key point prediction model and the classification model in the prediction model group. A 16-frame frame sequence is input into the temporal action recognition model to detect the abnormal behavior and predict the category of the target object. That is, after detecting the abnormal behavior of the target object, the object identity of the target object with abnormal behavior is detected. Subsequently, the object with abnormal behavior can be deduplicated and warned.
[0111] In the model training stage, for the model training of each model in the temporal action recognition model, a real examination room video can be used as the model training set. The examination room video is a video stream of multiple segments (such as N segments) of indefinite length. The video stream is decoded at a frame rate of 25 frames / second, thereby obtaining a frame set F = {f1, f2, f3...
[0112] f25,f26…fn}; where f1 to fn represent all video frames parsed from multiple video streams. Then, the first frame per second in the frame set F is annotated. Since it is a 25-frame / s video stream, its annotations can be f26,f51,f76, etc. The annotation categories include behaviors such as the examinee standing, the examinee walking, the examinee raising his hand, and the invigilator showing the test paper bag. Then, openpose is used to extract the pedestrian key points FK of the frame set F. A data contains the annotation category and frame sequence such as {fk26,fk27,...,fk50}. Then, the annotated training set is used to train each model until the model training is completed.
[0113] Step 300, performing abnormal behavior detection based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain an abnormal behavior detection result of the target object.
[0114] In actual implementation, this embodiment can combine the first confidence in the spatiotemporal feature fusion result and the second confidence in the frame sequence analysis result to obtain the final confidence. Then, according to the final confidence, it is determined whether the predicted category of the target object is correct. If it is correct, the preset action module corresponding to the predicted category is obtained according to the predicted category, and the action module is matched with the abnormal behavior for determination. If the action module matches the abnormal behavior, it indicates that the target object does have abnormal behavior, and an early warning can be issued for the target object and the abnormal behavior. For example, the current predicted category of the target object is an examinee. If the final confidence determines that the predicted category of the target object is correct, the preset action module corresponding to the examinee is matched with the abnormal behavior of the target object. If the action module matches the abnormal behavior of the target object, it indicates that the examinee may have suspicious behaviors such as cheating and whispering, and an early warning can be issued. Therefore, this embodiment effectively takes into account the requirements of the abnormal behavior detection algorithm for accuracy and efficiency, and can be applied to complex scenes such as smart inspection, non-perceptual identity authentication in the examination room, and classroom student behavior analysis.
[0115] In an optional embodiment, the embodiment of the present application performs abnormal behavior detection based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object, which may specifically include: extracting a first confidence and a prediction category from the spatiotemporal feature fusion result, and extracting a second confidence and abnormal behavior analysis data from the frame sequence analysis result; performing weighted processing based on the first confidence and the second confidence to obtain a final behavior confidence; performing discrimination processing based on the abnormal behavior data, in combination with the final behavior confidence, the prediction category, and a preset action behavior set, to obtain the abnormal behavior detection result.
[0116] For example, refer to Figure 4, for confidence 1 and confidence 2, a weighted method is used to weight confidence 1 and confidence 2 to obtain the final behavior confidence. Then the final behavior confidence can be compared with the preset confidence threshold. If the final behavior confidence is higher than the confidence threshold, the target object is judged as a student / invigilator based on the target frame and the predicted category, and the corresponding action module is further screened (the action module can be an action behavior set, including multiple preset action behaviors), and the action module is used to combine the abnormal behavior of the target object for discrimination processing to obtain the abnormal behavior detection result.
[0117] In actual processing, after the embodiment of the present application realizes the behavior detection and identification of the target object, it can also issue an early warning for the detected abnormal behavior, such as outputting an early warning sound through an alarm, a buzzer, etc. Since the abnormal behavior of the target object may be continuous, such as it may last for N seconds, that is, in the same or multiple consecutive frame sequences, there may be multiple image frames involving the abnormal behavior. In order to avoid multiple early warnings for the same abnormal behavior, the embodiment of the present application also proposes a behavior prediction deduplication algorithm. Through the behavior prediction deduplication algorithm, the early warnings involving the same abnormal behavior in the frame sequence are deduplicated, and only one reminder is required, avoiding waste of resources, and solving the problem of multiple early warnings for the same abnormal behavior in the existing behavior detection technology during early warning.
[0118] Optionally, after obtaining the abnormal behavior detection result of the target object, the embodiment of the present application may further specifically include: based on the abnormal behavior detection result, performing behavior frame analysis on the frame sequence to obtain the target abnormal behavior of the target object and the starting frame where the target abnormal behavior occurs; for the target object, using the target abnormal behavior and the starting frame to construct data to obtain the initial structure of the target object; for the target abnormal behavior, continuously performing behavior deduplication detection on the image frames after the starting frame in the frame sequence to obtain a continuous frame set of the target abnormal behavior, and the continuous frame set is used to characterize the duration period of the target abnormal behavior from the start to the end; based on the continuous frame set, updating the initial structure to obtain the target structure; and performing a behavior warning within the time period corresponding to the target structure.
[0119] The start frame refers to the first image frame in which the algorithm detects that the target object has abnormal behavior, the continuous frame set refers to the number of continuous frames of abnormal behavior, the continuous frame set includes multiple image frames including the start frame until the end frame, and the end frame refers to the image frame in which the target object stops abnormal behavior. The initial structure includes but is not limited to the current abnormal behavior of the target object and the number of continuous frames of abnormal behavior. The number of continuous frames of abnormal behavior of the initial structure can be 0 or 1; the target structure includes but is not limited to the abnormal behavior of the target object and the continuous frame set, which is not limited in the embodiments of the present application.
[0120] For example, refer to Figure 6 , assuming that examinee A performs an abnormal behavior that lasts for N seconds, the existing algorithm will prompt the behavior in N*25 frames, but in actual use, only one reminder is often needed. For this reason, the embodiment of the present application proposes a behavior prediction deduplication algorithm, so as to achieve only one alarm for the same abnormal behavior. Specifically, refer to Figure 6 For candidate A, a structure can be constructed, including {current behavior act, number of frames actF for the behavior}, and then the structure can be initialized: act=None, actF=0, to obtain the initial structure. Assuming that the algorithm detects abnormal behavior in frame f26, then actF+1, and f27, f28...f50 continue to detect the same behavior, then actF+1 will be continuously triggered. When the behavior ends, subsequent image frames, such as f51, trigger actF-1. When actF=0, it indicates that the abnormal behavior has ended. In addition, the behavior prompt threshold T can also be set to start an early warning at the beginning of an image frame where abnormal behavior is detected. For example, when actF=T, that is, starting from the Tth frame after the initial frame where the abnormal behavior is detected, the system prompts that abnormal behavior is detected, and the structure act=the abnormal behavior until actF=0 or act=None, ending the alarm for a behavior.
[0121] In summary, the embodiment of the present application obtains a target video stream containing a target object, extracts a frame sequence from the target video stream, then extracts features of key frames in the frame sequence through a first feature analysis model included in a feature analysis model group to obtain spatial feature information, and extracts features of the frame sequence through a second feature analysis model included in the feature analysis model group to obtain spatiotemporal feature information, performs feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information, then performs behavior prediction on the spatiotemporal feature fusion information to obtain a predicted category, a predicted target box, and a first confidence level corresponding to the target object, generates a spatiotemporal feature fusion result based on the predicted category, the predicted target box, the first confidence level, and the spatiotemporal feature fusion information, extracts the target box coordinates of the target object from the spatiotemporal feature fusion result, and generates a spatiotemporal feature fusion result based on the predicted category, the predicted target box, the first confidence level, and the spatiotemporal feature fusion information. The target box coordinates are used to crop the image frames in the frame sequence to obtain a cropped image of the target object, and then the skeleton of the cropped image is constructed by analyzing the key point prediction model included in the model group to obtain the skeleton motion information of the target object including the time series pedestrian skeleton motion map, and the skeleton motion information is analyzed by using the classification model included in the model group to perform abnormal classification to obtain a second confidence level, and the second confidence level is used as the frame sequence analysis result, and abnormal behavior detection is performed based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object. Therefore, the embodiment of the present application effectively takes into account the requirements of the abnormal behavior detection algorithm for accuracy and efficiency, enhances the algorithm recognition capability, can be applied to scenes with complex personnel, and solves the problem that the existing behavior detection technology cannot take into account both accuracy and efficiency and is limited in usage scenarios.
[0122] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.
[0123] like Figure 7 As shown, the embodiment of the present application also provides a smart inspection behavior detection device 700 based on spatiotemporal action detection, including:
[0124] An acquisition module 710 is used to acquire a target video stream containing a target object and extract a frame sequence from the target video stream;
[0125] The feature analysis and fusion module 720 is used to perform feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result;
[0126] A behavior analysis module 730 is used to perform behavior analysis based on the spatiotemporal feature fusion result and the frame sequence through a preset prediction model group to obtain a frame sequence analysis result;
[0127] The abnormal behavior detection module 740 is used to perform abnormal behavior detection according to the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object.
[0128] Optionally, the feature analysis model group includes a first feature analysis model and a second feature analysis model, and the feature analysis fusion module 720 includes:
[0129] a feature extraction submodule, configured to extract features of key frames in the frame sequence using the first feature analysis model to obtain spatial feature information, and to extract features of the frame sequence using the second feature analysis model to obtain spatiotemporal feature information;
[0130] A feature fusion submodule, used to perform feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information;
[0131] A behavior prediction submodule, used to perform behavior prediction based on the spatiotemporal feature fusion information, and obtain a predicted category, a predicted target box, and a first confidence level corresponding to the target object;
[0132] The spatiotemporal feature fusion result generation submodule is used to generate a spatiotemporal feature fusion result based on the prediction category, the prediction target box, the first confidence level and the spatiotemporal feature fusion information.
[0133] Optionally, the feature extraction submodule includes:
[0134] A first image scaling unit, configured to perform image scaling on the key frame to obtain a target key image frame;
[0135] A first feature extraction unit is used to extract features for the target key image frame using the first feature analysis model to obtain a multi-scale feature map, and use the multi-scale feature map as the spatial feature information, wherein the multi-scale feature map includes at least three two-dimensional branch feature maps;
[0136] A second image scaling unit, configured to perform image scaling on each image frame in the frame sequence to obtain a scaled frame sequence, wherein the scaled frame sequence includes at least one scaled image frame;
[0137] The second feature extraction unit is used to perform feature extraction on the zoomed image frame through the second feature analysis model to obtain a three-dimensional branch feature map, and use the three-dimensional branch feature map as the spatiotemporal feature information.
[0138] Optionally, the feature fusion submodule includes:
[0139] An image sampling unit, used to perform image sampling based on the three-dimensional branch feature map to obtain sampling information;
[0140] A feature fusion unit is used to perform feature fusion according to the multi-scale feature map, the three-dimensional branch features and the sampling information to obtain a multi-scale fusion feature map, and use the multi-scale fusion feature map as the spatiotemporal feature fusion information.
[0141] Optionally, the prediction model group includes a key point prediction model and a classification model, and the behavior analysis module 730 includes:
[0142] A first extraction submodule, used to extract the target frame coordinates of the target object from the spatiotemporal feature fusion result;
[0143] An image cropping submodule, used for cropping the image frames in the frame sequence based on the target frame coordinates to obtain a cropped image of the target object;
[0144] A skeleton construction submodule, configured to construct a skeleton of the cropped image by using the key point prediction model to obtain skeleton motion information of the target object, wherein the skeleton motion information includes a time-series pedestrian skeleton motion map, and the time-series pedestrian skeleton motion map includes abnormal behavior data;
[0145] The abnormal classification submodule is used to perform abnormal classification on the skeleton motion information through the classification model to obtain a second confidence level, and use the second confidence level as the frame sequence analysis result.
[0146] Optionally, the abnormal behavior detection module 740 includes:
[0147] A second extraction submodule, used to extract a first confidence level and a prediction category from the spatiotemporal feature fusion result, and to extract a second confidence level and abnormal behavior analysis data from the frame sequence analysis result;
[0148] A weighting submodule, configured to perform weighted processing based on the first confidence and the second confidence to obtain a final behavior confidence;
[0149] The discrimination processing submodule is used to perform discrimination processing according to the abnormal behavior data in combination with the final behavior confidence, the prediction category and the preset action behavior set to obtain the abnormal behavior detection result.
[0150] Optionally, the behavior detection device 700 further includes:
[0151] A behavior frame analysis module, configured to perform behavior frame analysis on the frame sequence based on the abnormal behavior detection result, to obtain a target abnormal behavior of the target object and a starting frame where the target abnormal behavior occurs;
[0152] A data construction module, used for constructing data for the target object by using the target abnormal behavior and the starting frame to obtain an initial structure of the target object;
[0153] a deduplication detection module, configured to continuously perform behavior deduplication detection on image frames after the start frame in the frame sequence for the target abnormal behavior, so as to obtain a continuous frame set of the target abnormal behavior, wherein the continuous frame set is used to characterize a duration period of the target abnormal behavior from the start to the end;
[0154] An updating module, used for updating the initial structure based on the continuous frame set to obtain a target structure;
[0155] The early warning module is used to issue a behavior early warning within a time period corresponding to the target structure.
[0156] It should be noted that the intelligent patrol behavior detection device based on spatiotemporal motion detection provided in the embodiments of the present application can execute the intelligent patrol behavior detection method based on spatiotemporal motion detection provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.
[0157] In a specific implementation, the above-mentioned intelligent inspection behavior detection device based on spatiotemporal motion detection can be integrated into a device, so that the device can extract fusion features from the frame sequence of the video stream, and determine the abnormal behavior detection result of the target object based on the fusion features. As an electronic device, it can realize the requirements of the abnormal behavior detection algorithm for accuracy and efficiency, enhance the algorithm recognition ability, and can be applied to complex scenes with personnel. The electronic device can be composed of two or more physical entities, or it can be composed of one physical entity. For example, the electronic device can be a personal computer (PC), a computer, a server, etc., and the embodiments of the present application do not impose specific restrictions on this.
[0158] like Figure 8As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to execute the program stored in the memory 113, and implement the steps of the intelligent patrol behavior detection method based on spatiotemporal motion detection provided by any of the above-mentioned method embodiments. Exemplarily, the steps of the intelligent patrol behavior detection method based on spatiotemporal motion detection may include the following steps: obtaining a target video stream containing a target object, and extracting a frame sequence from the target video stream; performing feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result; performing behavior analysis through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence to obtain a frame sequence analysis result; performing abnormal behavior detection based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain an abnormal behavior detection result of the target object.
[0159] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the intelligent patrol behavior detection method based on spatiotemporal motion detection as provided in any of the aforementioned method embodiments are implemented.
[0160] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0161] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A smart inspection behavior detection method based on spatiotemporal action detection, characterized in that: include: Acquire a target video stream containing a target object, and extract a frame sequence from the target video stream; Performing feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result; Based on the spatiotemporal feature fusion result and the frame sequence, a behavior analysis is performed through a preset prediction model group to obtain a frame sequence analysis result; Abnormal behavior detection is performed based on the spatiotemporal feature fusion result and the frame sequence analysis result to obtain an abnormal behavior detection result of the target object.
2. The method according to claim 1, characterized in that The feature analysis model group includes a first feature analysis model and a second feature analysis model, and the feature analysis fusion of the frame sequence is performed by the preset feature analysis model group to obtain a spatiotemporal feature fusion result, including: Extracting features from key frames in the frame sequence using the first feature analysis model to obtain spatial feature information, and extracting features from the frame sequence using the second feature analysis model to obtain spatiotemporal feature information; Perform feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information; Performing behavior prediction on the spatiotemporal feature fusion information to obtain a predicted category, a predicted target box, and a first confidence level corresponding to the target object; Based on the prediction category, the prediction target box, the first confidence level, and the spatiotemporal feature fusion information, a spatiotemporal feature fusion result is generated.
3. The method according to claim 2, characterized in that The step of extracting features from the key frame using the first feature analysis model to obtain spatial feature information includes: Performing image scaling on the key frame to obtain a target key image frame; For the target key image frame, extract features using the first feature analysis model to obtain a multi-scale feature map, and use the multi-scale feature map as the spatial feature information; The multi-scale feature map includes at least three two-dimensional branch feature maps.
4. The method according to claim 3, characterized in that The extracting features of the frame sequence by using the second feature analysis model to obtain spatiotemporal feature information includes: Performing image scaling on each image frame in the frame sequence to obtain a scaled frame sequence, wherein the scaled frame sequence includes at least one scaled image frame; For the zoomed image frame, feature extraction is performed using the second feature analysis model to obtain a three-dimensional branch feature map, and the three-dimensional branch feature map is used as the spatiotemporal feature information.
5. The method according to claim 4, characterized in that The performing feature fusion based on the spatial feature information and the spatiotemporal feature information to obtain spatiotemporal feature fusion information includes: Perform image sampling based on the three-dimensional branch feature map to obtain sampling information; Feature fusion is performed according to the multi-scale feature map, the three-dimensional branch features and the sampling information to obtain a multi-scale fusion feature map, and the multi-scale fusion feature map is used as the spatiotemporal feature fusion information.
6. The method according to claim 1, characterized in that The prediction model group includes a key point prediction model and a classification model. The behavior analysis is performed through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence to obtain a frame sequence analysis result, including: Extracting the target frame coordinates of the target object from the spatiotemporal feature fusion result; Performing image cropping on the image frames in the frame sequence based on the target frame coordinates to obtain a cropped image of the target object; Performing skeleton construction on the cropped image by using the key point prediction model to obtain skeleton motion information of the target object, wherein the skeleton motion information includes a time-series pedestrian skeleton motion map, and the time-series pedestrian skeleton motion map includes abnormal behavior data; For the skeleton motion information, abnormal classification is performed using the classification model to obtain a second confidence level, and the second confidence level is used as the frame sequence analysis result.
7. The method according to claim 1, characterized in that The abnormal behavior detection is performed according to the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object, including: Extracting a first confidence level and a predicted category from the spatiotemporal feature fusion result, and extracting a second confidence level and abnormal behavior analysis data from the frame sequence analysis result; Performing weighted processing based on the first confidence level and the second confidence level to obtain a final behavior confidence level; According to the abnormal behavior data, the final behavior confidence, the prediction category and the preset action behavior set are combined to perform discrimination processing to obtain the abnormal behavior detection result.
8. The method according to any one of claims 1 to 7, characterized in that: After obtaining the abnormal behavior detection result of the target object, the method further includes: Based on the abnormal behavior detection result, performing behavior frame analysis on the frame sequence to obtain the target abnormal behavior of the target object and the starting frame where the target abnormal behavior occurs; For the target object, data is constructed using the target abnormal behavior and the starting frame to obtain an initial structure of the target object; For the target abnormal behavior, continuously perform behavior deduplication detection on the image frames after the start frame in the frame sequence to obtain a continuous frame set of the target abnormal behavior, wherein the continuous frame set is used to characterize the duration period of the target abnormal behavior from the start to the end; Update the initial structure based on the continuous frame set to obtain a target structure; A behavior warning is issued within the time period corresponding to the target structure.
9. A smart inspection behavior detection device based on spatiotemporal action detection, characterized in that: include: An acquisition module, used for acquiring a target video stream containing a target object and extracting a frame sequence from the target video stream; A feature analysis and fusion module is used to perform feature analysis and fusion on the frame sequence through a preset feature analysis model group to obtain a spatiotemporal feature fusion result; A behavior analysis module, used to perform behavior analysis through a preset prediction model group based on the spatiotemporal feature fusion result and the frame sequence to obtain a frame sequence analysis result; The abnormal behavior detection module is used to perform abnormal behavior detection according to the spatiotemporal feature fusion result and the frame sequence analysis result to obtain the abnormal behavior detection result of the target object.
10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the intelligent patrol behavior detection method based on spatiotemporal motion detection as described in any one of claims 1 to 8 when executing the program stored in the memory.
Citation Information
Patent Citations
A method and a system for analyzing human behavior of an educational examination monitoring video
CN109271886A
Human body action recognition method based on a TP-STG framework
CN109492581A
time sequence behavior detection method based on 3D human body key points
CN109919122A
Method and equipment for identifying call making
CN110110631A
Abnormal behavior identification method and system based on skeleton extraction
CN113688797A
