A method, apparatus, device, and storage medium for detecting behavioral actions.

Abstract

This invention discloses a method, apparatus, device, and storage medium for behavior and action detection. The method includes: extracting a sequence of image frames from a video stream; extracting spatiotemporal features from the image frames in the sequence to obtain the spatiotemporal features of the image frame sequence; performing person bounding box detection and tracking on the image frames in the sequence to obtain person bounding box information for each person contained in the image frame sequence; and determining the behavior and action information of each person in the image frame sequence based on the spatiotemporal features and the person bounding box information. Using this method, a behavior recognition algorithm is run once for different targets in the video to extract the spatiotemporal features. Then, the person bounding box information obtained from target detection and tracking is used to determine the behavior and action information of each person in the image frame sequence. This eliminates the need to run the behavior recognition algorithm multiple times, improving the efficiency of behavior and action detection. Furthermore, by obtaining the spatiotemporal features and then extracting the behavior and action information of each target, it better considers the interaction between humans and the environment.

Description

Technical Field

[0001] This invention relates to the field of equipment testing technology, and in particular to a method, apparatus, device, and storage medium for detecting behavioral actions. Background Technology

[0002] Behavior recognition has become a research hotspot in the field of computer vision due to its wide application in video surveillance, virtual reality, human-computer intelligent interaction and other fields.

[0003] Currently, most mainstream behavior recognition methods are based on single-target video data, with very few based on multi-target video data. However, in real life, it's extremely rare for video data to contain only a single target; most situations involve behavior recognition of multiple targets simultaneously. Furthermore, recognizing a person's behavior often requires considering their interaction with their surroundings, and cannot focus solely on localized information. Existing technologies, when detecting and tracking multiple targets in a video, require running a behavior recognition algorithm for each target, resulting in low computational efficiency. Summary of the Invention

[0004] This invention provides a behavior action detection method, apparatus, device, and storage medium to solve the problem of low computational efficiency when multiple targets are present in a video and need to be detected and tracked, requiring the behavior recognition algorithm to be run for each target.

[0005] In a first aspect, embodiments of the present invention provide a behavior action detection method, the method comprising:

[0006] Extract a sequence of image frames from a video stream;

[0007] Spatiotemporal features are extracted from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence;

[0008] The image frames in the image frame sequence are subjected to person bounding box detection and tracking to obtain the person bounding box information of each person contained in the image frame sequence;

[0009] Based on the spatiotemporal features and the frame information of each person, the behavioral action information of each person in the image frame sequence is determined.

[0010] Secondly, embodiments of the present invention provide a behavior action detection device, comprising:

[0011] The frame sequence extraction module is used to extract a sequence of image frames from a video stream;

[0012] The spatiotemporal feature extraction module is used to extract spatiotemporal features from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence.

[0013] The detection and tracking module is used to detect and track person bounding boxes in the image frames of the image frame sequence, and obtain the person bounding box information of each person contained in the image frame sequence.

[0014] The behavior and action determination module is used to determine the behavior and action information of each person in the image frame sequence based on the spatiotemporal features and the information of each person's frame.

[0015] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the behavior detection method according to any embodiment of the present invention.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute the behavior detection method described in any embodiment of the present invention.

[0020] This invention provides a method, apparatus, device, and storage medium for behavior and action detection. The method involves extracting a sequence of image frames from a video stream; extracting spatiotemporal features from the image frames in the sequence to obtain their spatiotemporal characteristics; performing person bounding box detection and tracking on the image frames to obtain bounding box information for each person contained in the sequence; and determining the behavior and action information of each person in the sequence based on the spatiotemporal features and the bounding box information. This technical solution runs a behavior recognition algorithm once on different targets in the video to extract spatiotemporal features, and then uses the bounding box information obtained from target detection and tracking to determine the behavior and action information of each person in the image frame sequence. This eliminates the need for multiple runs of the behavior recognition algorithm, improving the efficiency of behavior and action detection. Furthermore, by obtaining the spatiotemporal features and then extracting the behavior and action information of each target, it better considers the interaction between humans and the environment.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a behavior action detection method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is an application example diagram of a behavior action detection method provided in Embodiment 1 of the present invention;

[0025] Figure 3 This is a flowchart illustrating a behavior action detection method according to Embodiment 1 of the present invention;

[0026] Figure 4 This is a flowchart of a behavior action detection method provided in Embodiment 1 of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a behavior action detection device according to Embodiment 2 of the present invention;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Example 1

[0032] Figure 1 The flowchart of a behavior action detection method provided in Embodiment 1 of the present invention is applicable to the behavior recognition of multi-target video data. The method can be executed by a behavior action detection device, which can be implemented in hardware and / or software. Optionally, it can be implemented by an electronic device as an execution terminal, such as a mobile terminal, a PC, or a server.

[0033] like Figure 1 As shown in the embodiments of this disclosure, a behavior action detection method may specifically include the following steps:

[0034] S110. Extract a sequence of image frames from the video stream.

[0035] In this embodiment, a video stream can be viewed as a data structure composed of a set of image frames arranged in chronological order, adding a time dimension compared to images. It's important to understand that a typical video often contains twenty to thirty frames per second. Identifying human behavior in a video stream requires not only analyzing the content of each individual frame but also combining the analysis of adjacent frames. An image frame sequence can be a collection of image frames sampled at a predetermined number of frames before and after a central image frame, with the sampling results arranged in chronological order.

[0036] Specifically, to identify a person's behavior at a specific moment in a video, a sequence of image frames needs to be formed by extracting a certain number of frames before and after the image frame corresponding to that moment from the video stream, which consists of image frames. The length of the sequence can be adjusted according to actual needs. Optionally, image frames can be sampled at intervals. The specific moment can be a period of time, such as 2 seconds or 4 seconds.

[0037] For example, input a long video, take a certain frame as the center, and sample one image every other frame forward and backward, for a total of 32 frames, to form an image frame sequence.

[0038] S120. Extract spatiotemporal features from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence.

[0039] In this embodiment, spatiotemporal features can be understood as the information features of the people contained in the image in the temporal and spatial domains.

[0040] Specifically, for common videos, one second often contains twenty to thirty frames, with minimal differences between adjacent frames, thus eliminating the need for dense sampling. A reference frame can be identified as the center within the aforementioned image frame sequence, and frames in the sequence can be sampled forward and backward at set intervals. This reference frame and the sampled frames are used as input data to a trained spatiotemporal feature extraction model for spatiotemporal feature extraction. The model's output is then used as the spatiotemporal features of the image frame sequence. This technical solution uses the SlowFast model for spatiotemporal feature extraction. Different datasets are input into the model for training, resulting in different trained spatiotemporal feature extraction models. These models can identify different types of behaviors and actions.

[0041] As described above, the SlowFast model in this solution is a dual-channel model, consisting of a slow channel and a fast channel, which extract spatial and temporal information respectively. The biggest difference between SlowFast and the traditional dual-stream video processing model is that the dual-stream structure does not explore different temporal speeds, and for both streams, the dual-stream structure uses the exact same backbone network; however, the Fast path is faster and more lightweight.

[0042] Following the description above, the SlowFast model is a very lightweight model. This is because this path is designed to process spatial information using fewer channels / kernels and with weaker capabilities. This information can be provided by the first path in a less redundant manner. Based on this advantage, intermediate layers on this path do not require temporal pooling to reduce temporal resolution. These two paths are named the slow and fast paths based on their difference in processing speed. After the information from the two paths is pooled into feature vectors, they are fused through lateral connections.

[0043] S130. Perform person bounding box detection and tracking on the image frames in the image frame sequence to obtain the person bounding box information of each person contained in the image frame sequence.

[0044] In this embodiment, the person bounding box information may include the position and number of the person bounding box. It's important to understand that for a long input video, target detection can be performed on specific frames at predicted intervals. However, subsequent target tracking requires the detection results of adjacent frames as input; if the interval is too large, the accuracy will be very low. Therefore, this technical solution chooses to use a target detection algorithm to detect each frame of the video, and the detection results are provided to the tracking algorithm.

[0045] Specifically, the image frames in the image frame sequence are first input into the trained object detection network model, which outputs image frames with bounding boxes containing character information. Then, the trained object tracking network model assigns numerical numbers to successfully matched targets based on the bounding boxes in the preceding and following frames; it assigns new numbers to newly appearing targets and retains the numbers of unmatched targets (those that have disappeared or are occluded) for a period of time. The image frame sequence, after passing through the object detection and tracking network models, yields numbered bounding boxes, which serve as the final tracking result. The object detection network model can use the YOLOv5 model, and the object tracking network model can use the ByteTrack model.

[0046] S140. Based on the spatiotemporal characteristics and the information of each person's frame, determine the behavioral and action information of each person in the image frame sequence.

[0047] Specifically, the aforementioned character bounding box position information and character bounding box numbers are summarized to form character bounding box information for each character appearing in the image frame sequence. Based on the character bounding box position information, the image frame size is projected to the same size as the aforementioned spatiotemporal features; that is, it is scaled to the same size as the spatiotemporal features and then projected onto the corresponding position. At this point, a region of interest (ROI) feature of the same size is determined based on the acquired spatiotemporal features of the target character bounding box. This ROI feature is then input into the trained behavior classification network model, undergoing standard operations for classification tasks such as global average pooling, fully connected layers, and softmax normalization to obtain the final behavior category. The behavior classification network model can be the aforementioned SlowFast model.

[0048] This disclosure provides a behavior and action detection method. This method runs a behavior recognition algorithm once on different targets in a video to extract spatiotemporal features. Then, it uses the bounding box information obtained from target detection and tracking to determine the behavior and action information of each person in the image frame sequence. This eliminates the need to run the behavior recognition algorithm multiple times, improving the efficiency of behavior and action detection. Furthermore, by obtaining spatiotemporal features and then extracting the behavior and action information of each target, it better considers the interaction between humans and the environment.

[0049] Based on the above solution, the embodiments disclosed herein can be further optimized as follows:

[0050] Each action information is used as the behavior detection result of the corresponding person at the current moment and displayed on the target image frame, where the target image frame is selected from the image frame sequence.

[0051] Specifically, Figure 2 This is an application example diagram of a behavior action detection method provided in Embodiment 1 of the present invention. Figure 2 As shown, the target image frame can be a selected frame from a sequence of image frames. The target image frame displays the behavior detection results at the current moment. People in the image are enclosed in bounding boxes, with the leftmost box number indicating the box's identifier, the middle letters (e.g., walk, stand, watch) representing the identified behavior, and the rightmost decimal representing the probability of the identified behavior. Correspondingly, each person recognition result at the current moment is displayed on the corresponding target image frame.

[0052] The types of human behaviors that this technical solution can identify are related to the dataset used to train the input model. The algorithm in this solution can identify any behavior contained in the dataset. For the dataset used to train the SlowFast model, this technical solution uses an algorithm trained on a publicly available dataset. The types of behaviors identified can include more than a dozen categories, such as walking, standing, talking, holding objects, and listening to others.

[0053] Figure 3 This is a flowchart illustrating a behavior action detection method according to Embodiment 1 of the present invention; as shown... Figure 3 As shown, an image frame sequence is extracted from the video stream. The image frame sequence is sampled forward and backward at set intervals and input into the trained SlowFast model to extract spatiotemporal features. Target detection and tracking are performed on each frame in the image frame sequence to obtain the bounding box information and bounding box number of each person contained in the image frame sequence. Based on the above spatiotemporal features and bounding box information, the region of interest features are determined. The region of interest features are input into the trained SlowFast model. After a series of standard operations of classification tasks, including global average pooling, fully connected layers, and normalization, the final behavior category is obtained. Finally, the behavior detection results and numbers are displayed on the target image frame.

[0054] As the first preferred embodiment of this example, Figure 4 This is a flowchart of a behavior action detection method provided according to Embodiment 1 of the present invention. Figure 4 As shown, the specific steps may include the following:

[0055] S210. Extract a sequence of image frames from the video stream.

[0056] S220. Determine a reference image frame from the image frame sequence, and obtain at least one first image frame from the image frame sequence based on the reference image frame and the filtering rules.

[0057] In this embodiment, the reference image frame is an image frame that can serve as a baseline, with sampling proceeding before and after it as the center. For typical videos, one second often contains twenty to thirty frames, and the difference between adjacent frames is very small, so dense sampling is not necessary. The filtering rules can be set according to the length of the image frame sequence and the sampling interval.

[0058] Specifically, a reference image frame can be determined as the center in the above image frame sequence, and image frames in the image frame sequence can be sampled forward and backward at a set interval.

[0059] Based on the above optimizations, as one implementation of S220, this embodiment of the present disclosure can determine a reference image frame from the image frame sequence, and obtain at least one first image frame from the image frame sequence based on the reference image frame and filtering rules. Specifically, the optimization is as follows:

[0060] a1) Use the image frame with the set sequence number in the image frame sequence as the reference image frame.

[0061] a2) Using the reference image frame as the center, sample image frames in the image frame sequence forward and backward at a set interval.

[0062] a3) The reference image frame and each sampled image frame are recorded as the first image frame.

[0063] In this embodiment, the sequence number can be set in advance. The sequence number of the frame used for behavior recognition is set. The image frame with the set sequence number in the image frame sequence is used as the reference image frame. With this as the center, the image frames in the image frame sequence are sampled forward and backward at set intervals. The combination of the reference image frame and each sampled image frame is recorded as the first image frame.

[0064] For example, the default sampling interval for the SlowFast algorithm used in this technical solution is 2 frames, meaning one frame is sampled every other frame. Similarly, we do not need to make a judgment on every single frame. In the algorithm, we will provide an arbitrarily adjustable prediction interval, i.e., how many frames between actions are performed. The default setting is 30 frames. Given a long video, with a certain reference image frame as the center, one image is sampled every other frame forward and backward, for a total of 32 frames. The 32 sampled image frames are recorded as the first image frame.

[0065] S230. Each first image frame is used as input data and input into the trained spatiotemporal feature extraction model.

[0066] Specifically, the spatiotemporal feature extraction model can be a SlowFast model. This involves training the model with a dataset containing over a dozen behavior categories to obtain the trained spatiotemporal feature extraction model. For example, behaviors might include walking, standing, talking, holding an object, and listening to others. This step involves inputting the aforementioned baseline image frame and pre- and post-sampled image frames as input data into the spatiotemporal feature extraction model.

[0067] S240. Obtain the feature vector information output by the spatiotemporal feature extraction model as the spatiotemporal features of the image frame sequence.

[0068] Specifically, the input data is fed into the trained spatiotemporal feature extraction model. The data is processed through a slow channel and a fast channel to extract spatial and temporal information respectively. Then, the temporal and spatial semantic information from different channels is fused through lateral connections and pooled into feature vectors. The output feature vector information is used as the spatiotemporal features of the image frame sequence.

[0069] S250. For at least one second image frame sampled from the image frame sequence, each second image frame is used as input data and input into the trained target detection network model.

[0070] In this embodiment, for a long input video, target detection can be performed on specific frames according to the predicted interval. However, subsequent target tracking requires the detection results of adjacent frames as input, and the accuracy will be very low if the interval is too large. No interval sampling is needed in the image frame sequence; each frame of the acquired video is used as a second image frame, and this second image frame is used as input data, which is then fed into the trained target detection network model.

[0071] The object detection network model chosen is YOLOv5, currently among the fastest object detection algorithms. The YOLOv5 model employs Mosaic data augmentation at its input, using random scaling, cropping, and arrangement for data concatenation. Adaptive anchor box calculation is used in the YOLO algorithm, where initial anchor boxes with predetermined widths and heights are set for different datasets. During network training, the network outputs predicted boxes based on these initial anchor boxes, compares them with the ground truth boxes, calculates the difference, and then updates the network parameters iteratively. When training on different datasets, the initial anchor box values ​​are calculated, and then, during each training iteration, the optimal anchor box values ​​are adaptively calculated for each training set.

[0072] S260. Obtain the position information of the person bounding box of the person contained in the second image frame output by the object detection network model.

[0073] Specifically, after the image frame passes through the trained object detection network model, all parts of the second image frame containing the location of the person are enclosed in bounding boxes, thus obtaining the bounding box location information containing the person.

[0074] In this technical solution, the target detection network model can use the ByteTrack algorithm model. This algorithm utilizes the similarity between the detection box and the tracking trajectory to remove the background from the low-scoring detection results while retaining the high-scoring detection results, thereby mining out the real objects (difficult samples such as occlusion and blurring), thus reducing missed detections and improving the coherence of the trajectory.

[0075] S270. Based on the position information of each character frame and the position information of the previous character frame corresponding to the previous second image frame, determine the character frame number of the characters contained in the second image frame.

[0076] Specifically, the position information of the person bounding boxes in the second image frame is input into the trained target tracking network model. Based on the person bounding box information in the second image frame, a numerical number is assigned to the same target corresponding to the successfully matched previous frame. Newly appearing targets are assigned a new number, while the numbers of unmatched targets (disappeared or occluded) are retained for a period of time. The image frame sequence is processed by the target detection network model and the target tracking network model to obtain numbered person bounding boxes, where the numbers serve as the final tracking result, i.e., determining the person bounding box number of the person contained in the second image frame.

[0077] Based on the above optimizations, as one implementation of S270, this embodiment of the present disclosure can determine the character frame number of the character contained in the second image frame according to the position information of each character frame and the position information of the previous character frame corresponding to the previous second image frame. The specific optimization is as follows:

[0078] b1) Obtain the position information of the previous person frame corresponding to the previous second image frame.

[0079] b2) Input the position information of each preceding character frame and the position information of each character frame as input data into the trained target tracking network model.

[0080] b3) Using a target tracking network model, the position information of each character frame is matched with the position information of the previous character frame.

[0081] b4) Determine the person frame number of the person contained in the second image frame based on the matching results.

[0082] Specifically, firstly, the position information of the previous person bounding box corresponding to the previous second image frame is obtained. Then, the position information of the person bounding box in the second image frame and the position information of the previous person bounding box are input into the trained target tracking network model. Based on the person bounding box information of the second image frame, a numerical number is assigned to the same target corresponding to the previous frame that is successfully matched. Newly appearing targets are assigned a new number, and the numbers of targets that are not successfully matched (disappeared or occluded) are retained for a period of time. The image frame sequence is processed by the target detection network model and the target tracking network model to obtain numbered person bounding boxes, where the number can be used as the final tracking result, that is, to determine the person bounding box number of the person contained in the second image frame.

[0083] S280. The position information of the character frame and the character frame number are used as the character frame information of the corresponding character, and the information is summarized to form the character frame information of each character contained in the image frame sequence.

[0084] Specifically, this step summarizes the above-mentioned character frame position information and character frame number to form the character frame information of each character contained in the image frame sequence.

[0085] S290. Select a target image frame from the image frames that are involved in determining the person frame information in the image frame sequence, and obtain the target person frame information of the target person contained in the target image frame.

[0086] Specifically, the target image frame is determined based on the image frame containing the position information of the person frame in the image frame sequence, and the target person frame information of the target person is obtained from the target image frame.

[0087] S2100. Based on the target person bounding box position information in the target person bounding box information, project the target person bounding box region of each target person onto the spatiotemporal features to obtain the spatiotemporal features of the target person bounding box of each target person.

[0088] Specifically, the size of the target person's bounding box position information in the target person bounding box information is projected onto the same spatiotemporal features as the aforementioned features; that is, it is scaled to the same size as the spatiotemporal features and then projected onto the corresponding position. At this point, the spatiotemporal features of the target person's bounding box for each target person are obtained.

[0089] S2110. Based on the spatiotemporal characteristics of each target person's frame, determine the region of interest features of each target person in the target image frame.

[0090] This step can use the spatiotemporal features of the target person frame as the region of interest feature, since both are the same size and do not need to be enlarged or scaled.

[0091] S2120. Based on the features of each region of interest, determine the behavioral action information of each target person in the target image frame, and use it as the behavioral action information of each person in the image frame sequence.

[0092] Specifically, the features of the aforementioned region of interest are input into the trained behavior classification network model. After undergoing standard classification operations such as global average pooling, fully connected layers, and normalization, the final behavior category is obtained. The behavior classification network model can be the SlowFast model described above.

[0093] Based on the above optimizations, as one implementation of S2120, this embodiment of the present disclosure can determine the behavioral action information of each target person in the target image frame based on the features of each region of interest. The specific optimization for determining the behavioral action information of each person in the image frame sequence is as follows:

[0094] c1) The features of each region of interest are used as input data and fed into a pre-trained behavior classification network model.

[0095] c2) Obtain the behavior category information of the behavior classification network model relative to the features output of each region of interest.

[0096] c3) Use the behavior category information as the behavior action information of the corresponding target person in the target image frame.

[0097] Specifically, the behavior classification network model can be a SlowFast model. The behavior classification network model and the spatiotemporal feature extraction model are different parts of the same model, used for spatiotemporal feature extraction and behavior recognition, respectively. The features of each region of interest are used as input data and fed into the pre-trained SlowFast model. After a series of standard operations for classification tasks, such as global average pooling, fully connected layers, and normalization, the final behavior category is obtained. The behavior category information is then used as the behavior action information of the corresponding target person in the target image frame.

[0098] In the above technical solution, after running the behavior recognition algorithm once to extract spatiotemporal features from different targets in the video, the bounding boxes obtained from target detection and tracking are used directly for behavior classification, eliminating the need to run the behavior recognition algorithm multiple times and improving its efficiency. Simultaneously, by obtaining the spatiotemporal features, the behavioral action information of each target can be extracted, better taking into account the interaction between humans and the environment.

[0099] Example 2

[0100] Figure 5 This is a schematic diagram of an enhanced behavior and action detection device provided in Embodiment 2 of the present invention. Figure 5As shown, the device includes: a frame sequence extraction module 310, a spatiotemporal feature extraction module 320, a detection and tracking module 330, and a behavior and action determination module 340.

[0101] The frame sequence extraction module 310 is used to extract a sequence of image frames from a video stream;

[0102] The spatiotemporal feature extraction module 320 is used to extract spatiotemporal features from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence.

[0103] The detection and tracking module 330 is used to detect and track person bounding boxes in the image frames of the image frame sequence, and obtain the person bounding box information of each person contained in the image frame sequence.

[0104] The behavior and action determination module 340 is used to determine the behavior and action information of each person in the image frame sequence based on the spatiotemporal features and the information of each person's frame.

[0105] The technical solution provided in this disclosure improves the efficiency of the behavior recognition algorithm by extracting spatiotemporal features from different targets in a video through a single behavior recognition algorithm run, and then directly classifying the behavior using the bounding boxes obtained from target detection and tracking. This eliminates the need to run the behavior recognition algorithm multiple times. Furthermore, by obtaining the spatiotemporal features, the behavioral action information of each target can be extracted, better taking into account the interaction between humans and the environment.

[0106] Furthermore, the device may also include a results display module.

[0107] Result display module: used to display the behavior and action information of each person as the behavior detection result of the corresponding person at the current moment on the target image frame, wherein the target image frame is selected from the image frame sequence.

[0108] Furthermore, the spatiotemporal feature extraction module 320 may include:

[0109] A reference image frame determination unit is used to determine a reference image frame from the image frame sequence, and to obtain at least one first image frame from the image frame sequence based on the reference image frame and a filtering rule;

[0110] The model training unit is used to input each of the first image frames as input data into the trained spatiotemporal feature extraction model.

[0111] The feature vector extraction unit is used to obtain the feature vector information output by the spatiotemporal feature extraction model as the spatiotemporal features of the image frame sequence.

[0112] Furthermore, the reference image frame determination unit can specifically be used for:

[0113] The image frame with the set sequence number in the image frame sequence is used as the reference image frame;

[0114] Centered on the reference image frame, image frames in the image frame sequence are sampled forward and backward at a set interval number of frames.

[0115] The reference image frame and each sampled image frame are denoted as the first image frame.

[0116] Furthermore, the detection and tracking module 330 may specifically include:

[0117] The data input unit is used to input the second image frame as input data to the trained target detection network model for each second image frame sampled from the image frame sequence.

[0118] The character bounding box acquisition unit is used to obtain the character bounding box position information of the characters contained in the second image frame output by the target detection network model;

[0119] The numbering unit is used to determine the character frame number of the character contained in the second image frame based on the position information of each character frame and the position information of the previous character frame corresponding to the previous second image frame.

[0120] The information aggregation unit is used to take the character frame position information and character frame number as the character frame information of the corresponding character, and aggregate them to form the character frame information of each character contained in the image frame sequence.

[0121] Furthermore, the numbering unit can specifically be used for:

[0122] Obtain the position information of the previous person frame corresponding to the previous second image frame;

[0123] The position information of each preceding character frame and the position information of each character frame are used as input data and input into the trained target tracking network model.

[0124] The target tracking network model is used to match the position information of each character frame with the position information of the previous character frame.

[0125] The person frame number of the person contained in the second image frame is determined based on the matching result.

[0126] Furthermore, the behavior and action determination module 340 may specifically include:

[0127] The information acquisition unit is used to select a target image frame from the image frames that participate in the determination of the person frame information in the image frame sequence, and to acquire the target person frame information of the target person contained in the target image frame.

[0128] The spatiotemporal feature acquisition unit is used to project the target person frame region of each target person onto the spatiotemporal feature based on the target person frame position information in the target person frame information, so as to obtain the spatiotemporal feature of the target person frame of each target person.

[0129] The region of interest determination unit is used to determine the region of interest features of each target person in the target image frame based on the spatiotemporal features of each target person bounding box.

[0130] The behavior and action determination unit is used to determine the behavior and action information of each target person in the target image frame based on the features of each region of interest, and to determine the behavior and action information of each person in the image frame sequence.

[0131] Furthermore, the behavior and action determination unit can be specifically used for:

[0132] The features of each region of interest are used as input data and fed into a pre-trained behavior classification network model.

[0133] Obtain the behavior category information output by the behavior classification network model relative to the features of each region of interest;

[0134] The behavioral category information is used as the behavioral action information of the corresponding target person in the target image frame.

[0135] The behavior and action detection device structure provided in the embodiments of this disclosure can execute the behavior and action detection method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0137] Example 3

[0138] Figure 6A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as action detection methods.

[0142] In some embodiments, the behavior detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the behavior detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the behavior detection method by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0148] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0149] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting behavioral actions, characterized in that, include: Extract a sequence of image frames from a video stream; Spatiotemporal features are extracted from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence; The image frames in the image frame sequence are subjected to person bounding box detection and tracking to obtain the person bounding box information of each person contained in the image frame sequence; Based on the spatiotemporal features and the character frame information, determine the behavioral action information of each character in the image frame sequence; The step of determining the behavioral action information of each person in the image frame sequence based on the spatiotemporal features and the person frame information includes: Select a target image frame from the image frames corresponding to the person frame information contained in the image frame sequence, and obtain the target person frame information of the target person contained in the target image frame; The target person frame position information in the target person frame information is scaled to the same size as the spatiotemporal feature, and the target person frame area of ​​each target person is projected onto the spatiotemporal feature to obtain the spatiotemporal feature of the target person frame of each target person. Based on the spatiotemporal features of each target person bounding box, the region of interest features of each target person in the target image frame are determined; Based on the features of each region of interest, determine the behavioral action information of each target person in the target image frame, and use it as the behavioral action information of each person in the image frame sequence; The step of performing person bounding box detection and tracking on the image frames in the image frame sequence to obtain person bounding box information for each person contained in the image frame sequence includes: At least one second image frame is sampled from the image frame sequence, and each second image frame is used as input data and input into the trained target detection network model. Obtain the bounding box position information of the person contained in the second image frame output by the target detection network model; Obtain the position information of the previous person frame corresponding to the previous second image frame; The position information of the previous character frame and the position information of each character frame are used as input data and input into the trained target tracking network model. The target tracking network model is used to match the position information of each character frame with the position information of the previous character frame. The person frame number of the person contained in the second image frame is determined according to the matching result; wherein, if the matching is successful, the person frame number of the person contained in the previous second image frame is used as the person frame number of the person contained in the second image frame. The position information and number of the character frame are used as the character frame information of the corresponding character, and are summarized to form the character frame information of each character contained in the image frame sequence.

2. The method according to claim 1, characterized in that, Also includes: The behavioral action information is used as the behavior detection result of the corresponding person and displayed on the target image frame, wherein the target image frame is selected from the image frame sequence.

3. The method according to claim 1, characterized in that, The step of extracting spatiotemporal features from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence includes: A reference image frame is determined from the image frame sequence, and at least one first image frame is obtained from the image frame sequence based on the reference image frame and the filtering rules. Each of the first image frames is used as input data and input into the trained spatiotemporal feature extraction model. The feature vector information output by the spatiotemporal feature extraction model is obtained as the spatiotemporal features of the image frame sequence.

4. The method according to claim 3, characterized in that, The step of determining a reference image frame from the image frame sequence and obtaining at least one first image frame from the image frame sequence based on the reference image frame and filtering rules includes: The image frame with the set sequence number in the image frame sequence is used as the reference image frame; Centered on the reference image frame, image frames in the image frame sequence are sampled forward and backward at a set interval number of frames. The reference image frame and each sampled image frame are denoted as the first image frame.

5. The method according to claim 1, characterized in that, The step of determining the behavioral action information of each target person in the target image frame based on the features of each region of interest includes: The features of each region of interest are used as input data and fed into a pre-trained behavior classification network model. Obtain the behavior category information output by the behavior classification network model relative to the features of each region of interest; The behavioral category information is used as the behavioral action information of the corresponding target person in the target image frame.

6. A behavior and action detection device, characterized in that, include: The frame sequence extraction module is used to extract a sequence of image frames from a video stream; The spatiotemporal feature extraction module is used to extract spatiotemporal features from the image frames in the image frame sequence to obtain the spatiotemporal features of the image frame sequence. The detection and tracking module is used to detect and track person bounding boxes in the image frames of the image frame sequence, and obtain the person bounding box information of each person contained in the image frame sequence. The behavior and action determination module is used to determine the behavior and action information of each person in the image frame sequence based on the spatiotemporal features and the information of each person frame; The behavior action determination module includes: The information acquisition unit is used to select a target image frame from the image frames corresponding to the person frame information contained in the image frame sequence, and acquire the target person frame information of the target person contained in the target image frame; The spatiotemporal feature acquisition unit is used to scale the target person frame position information in the target person frame information to the same size as the spatiotemporal feature, and project the target person frame area of ​​each target person onto the spatiotemporal feature to obtain the spatiotemporal feature of the target person frame of each target person. The region of interest determination unit is used to determine the region of interest features of each target person in the target image frame based on the spatiotemporal features of each target person bounding box; The behavior and action determination unit is used to determine the behavior and action information of each target person in the target image frame based on the features of each region of interest, and to use it as the behavior and action information of each person in the image frame sequence. The detection and tracking module includes: The data input unit is used to sample at least one second image frame from the image frame sequence, and input each second image frame as input data into the trained target detection network model. The character bounding box acquisition unit is used to obtain the character bounding box position information of the characters contained in the second image frame output by the target detection network model; The numbering unit is used to determine the character frame number of the character contained in the second image frame based on the position information of each character frame and the position information of the previous character frame corresponding to the previous second image frame. The information aggregation unit is used to take the character frame position information and character frame number as the character frame information of the corresponding character, and aggregate them to form the character frame information of each character contained in the image frame sequence. Specifically, the numbering unit is used for: Obtain the position information of the previous person frame corresponding to the previous second image frame; The position information of the previous character frame and the position information of each character frame are used as input data and input into the trained target tracking network model. The target tracking network model is used to match the position information of each character frame with the position information of the previous character frame. The person frame number of the person contained in the second image frame is determined according to the matching result; wherein, if the matching is successful, the person frame number of the person contained in the previous second image frame is used as the person frame number of the person contained in the second image frame.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the behavior detection method as described in any one of claims 1-5.

8. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the behavior detection method as described in any one of claims 1-5.

Images