Method, device, equipment, medium and product for constructing spatiotemporal action detection dataset

Abstract

This application discloses a method, apparatus, device, medium, and product for constructing a spatiotemporal action detection dataset, relating to the field of dataset construction. The method includes cropping surveillance video into short video clips and extracting images to obtain dense image files, sparse image files, and detection image files; detecting all targets in the dense image files and sparse image files using a target detector; traversing the sparse image detection results, annotating target behaviors, and obtaining a target behavior detection dataset; tracking targets in the dense image detection results and sparse image detection results using a target tracker to obtain a target identity dataset; and matching the target behavior detection dataset with the target identity dataset to obtain a spatiotemporal action detection dataset. This application can improve the efficiency and quality of dataset production.

Description

Technical Field

[0001] The present application relates to the field of dataset construction, and in particular to a method, apparatus, device, medium, and product for constructing a spatiotemporal action detection dataset. Background Art

[0002] With the development of deep learning and computer vision technologies, motion detection in videos is gradually evolving from single-frame image analysis to comprehensive video analysis that integrates temporal and spatial information. However, datasets based on object detection techniques and image annotation are no longer sufficient for researchers to monitor diverse dynamic motions. Therefore, video-based datasets that incorporate temporal and spatial information are needed to meet the demands of more sophisticated and complex motion detection tasks.

[0003] To meet the requirements of dynamic action detection, a spatiotemporal action detection dataset should include the bounding box, behavior information, and identity ID of each target in the video. Existing methods for creating spatiotemporal action detection datasets primarily involve the following steps: first, collecting a large amount of raw data; second, annotating the targets, their actions, and IDs in each frame of the video; and finally, setting the start and end time points of the actions in the video. However, existing spatiotemporal action detection datasets, based on the aforementioned foundation, focus on identity labeling and target detectors, taking into account factors such as high target volume and rapid motion speed. These methods suffer from the following shortcomings.

[0004] 1. Inefficiency: Manual data processing and labeling is time-consuming and labor-intensive, especially when labeling video data, which requires frame-by-frame labeling and results in a huge amount of data.

[0005] 2. Labeling the identity information of multiple targets is difficult: Manual labeling can solve this problem, but it is time-consuming and labor-intensive. Tracking algorithm labeling will lose the target's previous identity information when the detector fails to detect it, resulting in target matching errors.

[0006] 3. Single data format: The labeled dataset cannot simultaneously meet the frame-level and segment-level task requirements of target detection, behavior recognition, and spatiotemporal action detection. Often, a single labeling can only meet the requirements of one task.

[0007] Therefore, a method is needed to improve the production efficiency and quality of dataset construction. Summary of the Invention

[0008] The purpose of this application is to provide a method, device, equipment, medium and product for constructing a spatiotemporal action detection dataset, which can improve the efficiency and quality of dataset production.

[0009] To achieve the above objectives, this application provides the following solutions.

[0010] In the first aspect, the present application provides a method for constructing a spatiotemporal action detection dataset, including: obtaining a surveillance video of a target scene and cropping the surveillance video into short video clips; performing image extraction on the short video clips to obtain a dense image file, a sparse image file, and a detection image file; using a target detector to detect all targets in the dense image file and the sparse image file to obtain a dense image detection result and a sparse image detection result; the target detector is trained through the detection image file; traversing the sparse image detection results, annotating the target behavior, and obtaining a target behavior detection dataset; using a target tracker to track targets in dense images and sparse images based on the dense image detection results and the sparse image detection results to obtain a target identity dataset; the spatiotemporal action detection dataset is used for behavior recognition.

[0011] In the second aspect, the present application provides a device for constructing a spatiotemporal action detection dataset, including: an acquisition and cropping module, used to acquire a surveillance video of a target scene and crop the surveillance video into short video clips; an image extraction module, used to perform image extraction on the short video clips to obtain a dense image file, a sparse image file and a detection image file; a target detection module, used to use a target detector to detect all targets in the dense image file and the sparse image file to obtain a dense image detection result and a sparse image detection result; the target detector is trained by the detection image file; a traversal module, used to traverse the sparse image detection results, annotate the target behavior, and obtain a target behavior detection dataset; a target tracking module, used to use a target tracker to track targets in dense images and sparse images based on the dense image detection results and the sparse image detection results to obtain a target identity dataset; a matching module, used to match the target behavior detection dataset and the target identity dataset to obtain a spatiotemporal action detection dataset.

[0012] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a spatiotemporal action detection dataset.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a spatiotemporal action detection dataset.

[0014] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for constructing a spatiotemporal action detection dataset.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0016] The present application provides a method, apparatus, device, medium and product for constructing a spatiotemporal action detection dataset, which extracts images from short video clips, determines three types of image files, and then uses target detection to detect targets in dense image files and sparse image files. When annotating target behavior, only sparse image detection results are annotated, thereby improving annotation efficiency. Manual annotation is simplified by tracking using target detectors and target trackers. A target tracker is used to track targets in dense images and sparse images based on dense image detection results and sparse image detection results to obtain a target identity dataset; the target behavior detection dataset and the target identity dataset are matched to obtain a spatiotemporal action detection dataset, which combines the time, space, target identity information and target action information in the target scene, thereby improving the quality of the dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a diagram of the application environment of a method for constructing a spatiotemporal action detection dataset in one embodiment of the present application.

[0019] Figure 2 A flowchart of a method for constructing a spatiotemporal action detection dataset provided in one embodiment of the present application.

[0020] Figure 3 Schematic diagram of the method for building a spatiotemporal action detection dataset.

[0021] Figure 4 It is a multi-target image sample of a closed scene.

[0022] Figure 5 These are pictures of static and dynamic behaviors in closed scenes.

[0023] Figure 6 A densely cropped data graph.

[0024] Figure 7 This is a sparsely cropped data graph.

[0025] Figure 8 Schematic diagram of training data annotation for multi-object detectors in closed scenes.

[0026] Figure 9 Schematic diagram of behavior annotation on consecutive frames of multiple targets in a closed scene.

[0027] Figure 10 Flowchart of the improved tracking algorithm for identity information matching.

[0028] Figure 11 A schematic diagram of the functional modules of a device for constructing a spatiotemporal action detection dataset provided in another embodiment of the present application.

[0029] Figure 12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0032] The method for constructing a spatiotemporal action detection dataset provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the video to be processed to the server 104. After the server 104 receives the video to be processed, for the video to be processed, the server 104 cuts the surveillance video into short video clips; performs image extraction on the short video clips to obtain dense image files, sparse image files and detection image files; uses the target detector to detect all targets in the dense image files and the sparse image files to obtain dense image detection results and sparse image detection results; the target detector is obtained by training the detection image files; traverses the sparse image detection results, annotates the target behavior, and obtains a target behavior detection dataset; uses the target tracker to track the targets in the dense image and the sparse image based on the dense image detection results and the sparse image detection results to obtain a target identity dataset; matches the target behavior detection dataset with the target identity dataset to obtain a spatiotemporal action detection dataset. The server 104 can feed back the final spatiotemporal action detection dataset to the terminal 102. In addition, in some embodiments, the spatiotemporal action detection dataset construction method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly construct the spatiotemporal action detection dataset for the video to be processed, or the server 104 can obtain the video to be processed from the data storage system and construct the spatiotemporal action detection dataset for the video to be processed.

[0033] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0034] In an exemplary embodiment, Figure 2 As shown, a method for constructing a spatiotemporal action detection dataset is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used for explanation, including the following steps 201 to 206.

[0035] Step 201: Acquire a surveillance video of a target scene and cut the surveillance video into short video clips.

[0036] Step 202: performing image extraction on the short video clip to obtain a dense image file, a sparse image file, and a detection image file.

[0037] Step 203: Detect all objects in the dense image file and the sparse image file using an object detector to obtain a dense image detection result and a sparse image detection result; the object detector is trained by the detection image file.

[0038] Step 204: traverse the sparse image detection results, mark the target behavior, and obtain a target behavior detection dataset.

[0039] Step 205: Using a target tracker to track targets in the dense image and the sparse image based on the dense image detection results and the sparse image detection results, to obtain a target identity dataset.

[0040] Step 206: Match the target behavior detection dataset and the target identity dataset to obtain a spatiotemporal action detection dataset; the spatiotemporal action detection dataset is used for behavior recognition.

[0041] Implement the above-mentioned steps 201 to 206, extract images from short video clips, determine three image files, and then use target detection to detect targets in dense image files and sparse image files. When annotating target behavior, only the sparse image detection results are annotated, thereby improving the annotation efficiency. Tracking is performed using a target detector and a target tracker, simplifying manual annotation. Using a target tracker to track targets in dense images and sparse images based on dense image detection results and sparse image detection results, a target identity dataset is obtained; the target behavior detection dataset and the target identity dataset are matched to obtain a spatiotemporal action detection dataset, which combines the time, space, target identity information and target action information in the target scene, thereby improving the quality of the dataset.

[0042] In an exemplary embodiment, a surveillance video of a target scene is obtained and the surveillance video is cropped into short video clips, specifically comprising: capturing the surveillance video of the target scene from a bird's-eye view; the surveillance video includes a closed boundary of the target scene; defining a target behavior within the target scene based on the surveillance video; filtering the surveillance video according to the target behavior to obtain a filtered surveillance video; and reconstructing the filtered surveillance video at a fixed frame rate and cropping it into short video clips.

[0043] In an exemplary embodiment, image extraction is performed on the short video clip to obtain a dense image file, a sparse image file and a detection image file, specifically including: extracting the short video clip frame by frame to obtain a dense image file; extracting the short video clip second by second to obtain a sparse image file; and randomly extracting the short video clip to obtain a detection image file.

[0044] In an exemplary embodiment, the target detector uses a YOLOv8 model.

[0045] In an exemplary embodiment, a target tracker is used to track targets in dense images and sparse images based on dense image detection results and sparse image detection results to obtain a target identity dataset, specifically comprising: extracting a set number of images from the dense image detection results and the sparse image detection results as the starting point of the target tracker, and using the improved cascade matching DeepSort algorithm to track all targets in the dense image detection results and the sparse image detection results to obtain a target identity dataset.

[0046] In an exemplary embodiment, the method for constructing a spatiotemporal action detection dataset further includes: determining a final spatiotemporal action detection dataset based on the spatiotemporal action detection dataset using upsampling, format conversion, and random sampling. The method for determining a final spatiotemporal action detection dataset based on the spatiotemporal action detection dataset using upsampling, format conversion, and random sampling specifically includes: performing matching and filling based on upsampling according to the spatiotemporal action detection dataset and the dense image detection results to obtain a dense image spatiotemporal action detection dataset; and performing format conversion and random sampling on the dense image spatiotemporal action detection dataset to obtain the final spatiotemporal action detection dataset.

[0047] In an exemplary embodiment, Figure 3 As shown, this application provides a specific process of a spatiotemporal action detection dataset construction method in practical application, including the following steps.

[0048] Step 1: Use a camera to collect surveillance video of the target scene and define the target behavior.

[0049] A research scenario was selected that met the following characteristics: multiple targets, multiple behavior categories, and a small number of targets flowing in and out of the scenario. A camera was installed at a 45-degree bird's-eye view to collect spatiotemporal motion detection data videos of captive Hu sheep in closed scenarios. All closed scenarios were filmed from a bird's-eye view, and the videos contained clear closed boundaries within the scenarios. The number and names of target behavior types in the scenarios were determined. Based on actual needs, the seven behaviors that needed to be labeled were determined: standing, lying down, eating, drinking, licking, walking, and rumination. The first four are static and interactive actions that can be judged from a single image, while the last three are dynamic actions that cannot be distinguished from the first four actions from a single image.

[0050] After collecting video data, we manually screened the initial videos based on whether the defined target behaviors appeared in the videos. The cameras saved and uploaded the videos every hour during the video collection. During the initial video screening, we used manual observation to ensure that the seven behaviors mentioned above appeared in at least two original videos. A total of eight one-hour original videos were screened. Figure 4 This is an example of multi-target spatiotemporal action detection data image in a closed scene. Figure 5 The following are examples of static and dynamic behaviors. Figure 5 (a) is the starting image of the video. Figure 5 (b) is an example of a static behavior that can be judged based on an image. Figure 5 (c) is an example of walking behavior that can only be judged based on continuous video frames.

[0051] Step 2: Reconstruct the obtained video at a fixed frame rate and crop it into short video clips.

[0052] When cropping short video clips, the initial video needs to be reconstructed at f frames per second, and then cropped for s seconds every m minutes.

[0053] Specifically, the video streams of the eight videos were reconstructed at 30 frames per second, and then 10 seconds were cut every 6 minutes to obtain 10 short video clips of 10 seconds in length, for a total of 80 short video clips.

[0054] Step 3: Extract the short video clips frame by frame, second by second, and randomly. Perform image extraction on 80 short video clips to form three types of files: dense image, sparse image, and detection image.

[0055] There are three image extraction methods for video clips: the first is frame-by-frame extraction, where each frame of each short video clip is saved as an image and stored in a dense image folder; the second is second-by-second extraction, where the first frame of each short video clip is extracted every second, including the first and last frames, and stored in a sparse image folder; the third is random extraction, where two images are randomly extracted from each short video clip as detection images and stored in a detection image folder. In all three image extraction methods, images are named directly using the frame number.

[0056] Specifically, the first method is frame-by-frame extraction: a dense image folder is named, where a folder is named for each short video clip; for each short video clip, one image is extracted from each frame in chronological order, named with the current frame number of the image, including the first and last frames, and a total of 301 images are obtained from frame 0 to 300, which are used to extract images at different time intervals during data loading. Figure 6 For dense cropping and extraction of sample images, the first and second frames are saved with the naming number "video number_frame number". Figure 6 (a) is the first frame, Figure 6 (b) is the second frame, Figure 6 (c) is the third frame.

[0057] The second method is second-by-second extraction: a sparse image folder is named, where a folder is named for each short video clip; for each short video clip, an image is extracted every second in chronological order, named with the current frame number of the image, including the first and last frames, and 11 images are extracted at frames 0, 30, 60, ..., 300 to complete the sparse annotation. Figure 7 This is a sample image extracted by sparse cropping, indicating that the first and thirty-first frames are saved with the naming number "video number_frame number". Figure 7 (a) is the first frame, Figure 7 (b) is the thirty-first frame, Figure 7 (c) is the sixty-first frame.

[0058] The third method is random extraction: name a detection image folder, randomly extract 2 images from each short video clip, name them sequentially, and a total of 160 images are used to train the target detector.

[0059] Step 4: According to the requirements of the target detection task, mark all the targets in each image in the detection image, and then divide it into training set and validation set in proportion to train the target detector.

[0060] In step 4, an object detector is trained to accurately identify the corresponding object in the scene. The image data used is the detection image file portion obtained in step 3.

[0061] For the 160 images in the detection image file in step 3, use the X-AnyLabeling annotation tool and Segment Anything (Base) to achieve automatic auxiliary annotation. Label all the targets in the 160 images one by one to form a label file corresponding to the YOLO format. This is used as the label data required for target detector training, and then randomly divided into the training set and validation set of the target detector in an 8:2 ratio. Figure 8 Label example images for target detector training data, only labeling targets for training target detectors.

[0062] The model framework used is YOLO v8, and the pre-trained weights are YOLO v8-n. The hyperparameters in the training include batch size, optimizer, training rounds, and stopping training criteria. Among them, the batch size is 16; the optimizer is SGD; the training rounds are 300, and the stopping criterion is to stop the object detector training if the validation set loss value does not decrease in 50 training rounds.

[0063] After training is completed, the object detector model configuration and parameter information are exported to form a model file for subsequent video image detection.

[0064] The object detector is designed to localize objects and assist in tracking. The model training dataset is the same as the detection image file (240 annotated images: 200 for training and 40 for validation, with original images of 1920×1080 pixels). The model input is a batch-by-3×640×640 format, obtained by scaling the original images to 640×640 pixels and performing data augmentation such as translation and centrosymmetry. The model output, after post-processing, is the center point coordinates and bounding box width and height (x, y, w, h) of each object in the image (which are converted to the coordinates of the top left corner and bottom right corner (x1, y1, x2, y2) during plotting), an object label (e.g., 0: sheep), and the probability of being classified as that label. The output for each object in each image is (x, y, w, h, cls, conf), where cls is the class label number and conf is the confidence score (also known as the probability of being classified as that class). The loss calculation is the same as YOLOv8, which is DFL+CIOU+BCE.

[0065] The output of the above model is (x, y, w, h), but it will be converted to (x1, y1, x2, y2) in post-processing. Therefore, when making the spatiotemporal action detection dataset, a coordinate conversion is actually performed and (x1, y1, x2, y2) is used because it is convenient to directly use the dataset to draw in the image.

[0066] Step 5: Use the object detector to detect all objects in the dense image and the sparse image respectively.

[0067] Use the target detector obtained in step 4 to detect the images in the dense image file and the sparse image file obtained in step 3. Generate a detection file for each image. Each line in the file represents an object, and the value is the horizontal and vertical coordinates of the upper left and lower right corners of the object. Get the dense image detection results and the sparse image detection results.

[0068] Among them, for the generated detection files, a table file is generated for sparse and dense image files respectively, where the header is (video name, frame number, upper left corner point horizontal coordinate, upper left corner point vertical coordinate, upper right corner point horizontal coordinate, upper right corner point vertical coordinate, target category); among them, if the target type in the scene is 1, the target category label is removed, and the frame number is the name of the corresponding image.

[0069] Step 6: Traverse the sparse images of each short video clip, annotate the target behavior, and form a target behavior detection dataset.

[0070] For each target in each image in the sparse image detection results, an action label is assigned to it.

[0071] The first two frames of the sparse image detection results corresponding to each video are extracted to form a file containing 7 images; then, according to the 7 image files corresponding to each video, a json file containing the bounding box information and the seven category information of the sparse frames below is generated, and then the behavior annotation is performed using VIA's via_image_annotator annotation tool. Figure 9 This is an example diagram of behavior annotation. The behavior annotation interface allows single or multiple selections. A single-target behavior can be composed of two basic behaviors, or only one combined behavior can be annotated. When annotating, the annotator follows the following three steps.

[0072] The first step is to import 7 consecutive images and their JSON files corresponding to a video at a time.

[0073] In the second step, the labeler first traverses 7 images, determines the static and dynamic behaviors of the target, and then assigns action labels to the target boxes.

[0074] The third step is to export the JSON file after completing the action assignment of all target boxes in the 7 images in a video.

[0075] After completing all video annotations, the JSON file information is comprehensively extracted to form a target behavior detection dataset containing action labels, and finally merged into a table file with the file header (video name, frame number, upper left corner point horizontal coordinate, upper left corner point vertical coordinate, upper right corner point horizontal coordinate, upper right corner point vertical coordinate, target action category). It is possible to have single target multiple behavior annotations, that is, the two rows of information only differ in the target action category.

[0076] Step 7: Extract the first two frames of the dense image of each short video clip and input them into the target tracker to accurately track each target and form a target identity dataset.

[0077] For both sparse and dense image detection results, the first two images are extracted as the starting point of the target tracker. The DeepSort algorithm with improved cascade matching is used to track the targets in the sparse and dense images and assign a unique ID number. Because the sparse and dense image files are named in the same folder, they all come from the same short video, where the dense folder is collected for each frame (the dense image files can be integrated into a video). The sparse image files are sampled one frame per second. The detection results are used for behavior annotation, but the target box position of the tracking result is not completely consistent with the box coordinates of the detection result. By matching the detection results with the tracking results, an ID number can be added to each target of the behavior annotation. Therefore, in the tracking process, a trained detector is used for real-time detection rather than directly using the detection results.

[0078] Among them, the DeepSort algorithm for improved cascade matching is different in two aspects: first, the maximum survival time of the hyperparameter max_age in the algorithm is set to 10800, which indicates the number of frames that a target is still considered valid after it is not detected. This means that the target track will be deleted only when it fails to match the detection box for 10800 consecutive frames; second, the cascade matching algorithm starts by sorting all tracks from small to large based on the size of max_age, and gives priority to matching tracks with fewer mismatches during the matching process, such as Figure 10 Flowchart for the improved tracking algorithm.

[0079] (1) Create the corresponding Tracks for the first frame's detection results. Initialize the Kalman filter's motion variables and use the Kalman filter to predict the corresponding frames. At this point, the Tracks must be unconfirmed (the unconfirmed state in the figure).

[0080] (2) Perform IOU matching on the detected object bounding box in the current frame and the predicted bounding box in the previous frame using Tracks. The cost matrix (calculated as 1-IOU) is then calculated based on the IOU matching results. This is essentially the cost matrix that determines which objects in two consecutive frames are extremely close together with minimal displacement, and are therefore considered the same object.

[0081] (3) All the cost matrices obtained in (2) are used as the input of the Hungarian algorithm to obtain the linear matching results. There are three types of results at this time. The first is Tracks mismatch (Unmatched Tracks), which directly deletes the mismatched Tracks (because this Tracks is in an uncertain state. If it is in a certain state, it must reach a certain number of times (this is max_age) in a row (the default is 30 times, because in the scene we hope that the target can still match the previous ID after being occluded for a long time, so we set a large value of 10800)); the second is detection frame mismatch (Unmatched Detections), which initializes such a detection frame as a new Tracks (new Tracks); the third is that the detection frame and the predicted frame are successfully paired, which means that the previous frame and the next frame are tracked successfully, and the corresponding detection frame is updated with its corresponding Tracks variable through Kalman filtering.

[0082] (4) Repeat steps (2)-(3) until confirmed Tracks appear or the video frame ends.

[0083] (5) Use Kalman filtering to predict the corresponding frames of the confirmed Tracks and the unconfirmed Tracks. Perform cascade matching on the confirmed Tracks frame and the detection frame (previously, each time the Tracks match, the appearance features and motion information of the detection frame will be saved (the appearance features here refer to the (1, 128)-dimensional feature vector obtained by the REID model in the article. This model is a classification network, and the training data set is the image classification data set. After removing the last softmax classification head, only the 128-dimensional vector obtained by the second-to-last layer is taken out as the appearance feature of the ID target. The motion information is the speed and acceleration of the coordinates). By default, the first 100 frames are saved, and the appearance features and motion information are used to perform cascade matching on each detection frame in the new frame. This is done because the confirmed Tracks and the detection frame are more likely to match).

[0084] (6) There are three possible outcomes after cascade matching. The first is track matching. Such tracks are updated with their corresponding track variables through Kalman filtering. The second and third are mismatches between the detection boxes and the tracks. In this case, the previously unconfirmed tracks and the mismatched tracks are combined with the unmatched detections for IOU matching. The cost matrix (calculated as 1-IOU) is then calculated based on the IOU matching results.

[0085] (7) All the cost matrices obtained in (6) are used as the input of the Hungarian algorithm to obtain the linear matching results. There are three types of results. The first is Tracks mismatch (Unmatched Tracks), which deletes the mismatched Tracks (because the Tracks are in an uncertain state. If they are in a certain state, they must be deleted after a certain number of times (max_age, the default is 30 times)). The second is detection frame mismatch (Unmatched Detections), which initializes such detections as a new Tracks (new Tracks). The third is that the detection frame and the predicted frame are successfully paired, which means that the previous frame and the next frame are tracked successfully, and the corresponding detections are updated through the Kalman filter to update the corresponding Tracks variables.

[0086] The detection box mismatch refers to the fact that none of the detection boxes in the new frame can be matched with a certain ID.

[0087] (8) Repeat steps (5) to (7) until the video frame ends.

[0088] Improved cascade matching refers to:

[0089] In the DeepSort algorithm, if a target is lost during tracking, its track is not deleted until it has lost max_age frames. The algorithm traverses (0, max_age), meaning the loop must be repeated max_age times. Therefore, with the default value of 30, there are many targets, and even with more Tracks, only 30 traversals are required. Therefore, for each max_age, the distribution of target loss times is not sparse. Tracks with fewer lost frames are matched first, while those with more lost frames are matched later. This ensures that when an occluded target reappears, the unoccluded target has already matched most of the detection frames. When an occluded target appears, only a few frames remain to match, and different occluded targets have different lost frames, frame distances, and feature vectors, ensuring accurate matching.

[0090] In target scenarios, such as cattle and sheep in a farm, dense crowds often cause occlusion for extended periods, necessitating a large max_age. This is because after being obscured, they may not reappear for tens of seconds, and a successful match with a previously identified ID is desirable, as the target that appears is the same target from tens of seconds ago. Even with a value of 10800, there are often only 30-50 targets. Consequently, there will inevitably be many missing frame values ​​between (0 and 10800) where no target exists. However, the algorithm still requires 10800 iterations. Therefore, the improvement sorts all tracks by the value of their max_age attribute. Similarly, matching targets with fewer missing frames (max_age) is performed first, followed by those with greater missing frames. In this case, only as many iterations as there are targets are required. This reduces the computational time of the tracking algorithm in this scenario.

[0091] After accurately tracking all targets, a dataset and table file containing target identity information are generated for both dense and sparse image files. The header of the table is (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target identity ID).

[0092] Because the annotation work involves identity annotation and behavior annotation, these two annotations are combined into a single file to form a spatiotemporal action annotation. The identity annotations can be used as a target tracking dataset, and the behavior annotations can be used as an image-based action detection dataset. After merging, a spatiotemporal action detection dataset containing spatiotemporal information can be formed.

[0093] Both parts of the annotation are based on the target detection results of dense images and sparse images. The table only contains (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target category). The dense results are used for tracking and identity labeling, and the sparse results are used for behavior labeling.

[0094] The table formed after target tracking uses the target detection information of each image for matching and tracking. At this time, each target will be assigned an identity ID. Therefore, the tracking information (video and frame, target frame position, identity ID) obtained by the target tracker will be made into a table here with the header (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target identity ID).

[0095] Step 8: Match the box information in the target behavior detection dataset and the target identity information dataset one by one to form a closed scene multi-target spatiotemporal action detection dataset.

[0096] According to the target action data table and the target identity data table, all targets with the same video name and number are matched, and the target identity information is filled into the target action table to form a closed scene multi-target spatiotemporal action detection dataset of sparse images of (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action, target identity ID). If the IOU of the target bounding box is greater than 0.999, it is considered that the two are the same target. After filling the identity information into the action information, a closed scene multi-target spatiotemporal action detection dataset of sparse images of (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action, target identity ID) is formed. Figure 9 This is an example of the generated annotation file.

[0097] The target action table is (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category). The target identity data table is (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target identity ID). The target detection result table value is (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target category). There are three tables in total, which are constructed for sparsely captured and densely captured image files respectively.

[0098] Step 9: Based on the dataset labeled on sparse images, upsample to form a dataset labeled on dense images, and then perform format conversion and random sampling to form the final spatiotemporal action detection dataset. The purpose of Step 9 is to extract a dataset that can be directly used in the object detection model.

[0099] For the construction of dense image dataset, the sparse image upsampling method is adopted to form a closed scene multi-target spatiotemporal action detection dataset on dense images, and then random sampling is performed on the dense image dataset to form the final spatiotemporal action detection dataset based on target detection.

[0100] There are two steps to upsampling to form dense image annotations: the first step is to use the identity information of the sparse annotation dataset to match the target in the dense image identity information file. Specifically, the multi-target bounding box and identity information with the same video name and frame number in the sparse annotation dataset and the dense annotation dataset are used to determine the matching target, and the sparsely annotated behavior is filled in the last column of the corresponding frame number row to match the target in the dense image identity information file; the second step is to fill the action information that matches the identity information in the first frame per second in the sparse annotation dataset into the 30 frames of the dense image target identity information of this second to form the dataset annotation file on the dense image.

[0101] An object action recognition dataset based on object detection is formed by randomly sampling a dense image dataset.

[0102] There are two steps to randomly sample the target detection dataset: the first step is to copy the image with the corresponding video frame number from the dense image to the target detection image; the second step is to extract the bounding box coordinate information and target action information of the corresponding image from the dense image annotation file to form a TXT file in the YOLO annotation format, thus completing the target action recognition dataset based on target detection.

[0103] At this point, dense and sparse datasets that integrate time, space, target identity information, and target action information, as well as spatiotemporal action detection datasets for target detection, are completed. In subsequent work, different data loaders can be used according to different task categories.

[0104] Because there are too many dense images, only the target detection boxes in the sparse image data are manually labeled with behavior categories. That is, the table (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category) is only on the sparse image. The table on the dense image will also use the table on the sparse image, and the action categories of the sparse image will be used to fill in the dense image. The upsampling method is to use the behavior labeled per second in the sparse image. The dense image has 30 frames in this second. Then, the behavior labeled in this second in the sparse image is directly filled in after the corresponding target ID on the 30 frames of the dense image.

[0105] The final spatiotemporal action detection dataset is shown in Table 1.

[0106] Table 1 Spatiotemporal action detection dataset

[0107]

[0108] So, in general.

[0109] The first step is to perform target detection on sparse and dense images to obtain (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target category).

[0110] In the second step, behavior annotation is performed on the sparse image to obtain (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category) on the sparse image.

[0111] The third step is to track dense and sparse images and generate (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target identity ID).

[0112] In the fourth step, for sparse images, the results obtained in the second and third steps are matched according to the target frame coordinate information, and the identity information is integrated with the behavior information to form a spatiotemporal action detection dataset (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category, target identity ID).

[0113] In the fifth step, for the dense image, the target identity ID on the corresponding timestamp on the sparse image is first matched, and then the behavior category (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category) is filled in to obtain the (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category, target identity ID) on the dense image.

[0114] Step 6: In the sparse image spatiotemporal action detection dataset (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category, target identity ID), extract images according to the first two information, save the coordinates and action category information as labels, and form an action detection dataset based on static images.

[0115] In this step, four tables are included for each of the sparse and dense image formats, for a total of eight: 1. Target detection table (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target category). 2. Action annotation table (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category). 3. Tracking identity ID table (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category, target identity ID). 4. Spatiotemporal action detection table (also known as spatiotemporal action detection annotation file) (video name, frame number, upper left corner horizontal coordinate, upper left corner vertical coordinate, upper right corner horizontal coordinate, upper right corner vertical coordinate, target action category, target identity ID).

[0116] Finally, we randomly extract images and annotation information to create action detection annotations and image folders for static images. Each image corresponds to a txt file, with each line representing an object in the format (target action category, center point horizontal coordinate, center point vertical coordinate, width, height, target action category) (this is the YOLO annotation file format).

[0117] Step 10: Create a dataset loader to form the final dataset.

[0118] The purpose of the data loader is to load input images and label information for model training. A single sample input image is a collection of images sampled at a fixed sampling rate and number of images. The label contains the bounding boxes of all objects in the current frame, the behavior category, the start and end times of the action in the continuous segment, and the target ID. The data loader generates a single sample at a fixed sampling rate and number of images to analyze the identity and behavior of each object in the current frame. In dense images and annotations, the data loading formula is expressed as follows:

[0119]

[0120] in, x is the current frame, s is the sampling rate, and T is the number of sampled images, that is, how much time information is used to judge the target behavior. C is the label set of all action categories, For the t Frame No. i The detection box of the target, is the behavior category to which the detection box belongs. t b Target behavior c The start frame, t e Target behavior c The end frame of For the t b Frame to t e Frame i The set of target behaviors in each frame, is the corresponding detection box. F refers to the model or function or calculation method, that is, using F(X) through model calculation, we can finally get the result on the left. The content in F is the input information. Refers to the current frame image, x is the frame number, Refers to the image of this frame. t is t b Frame to t e The value between .

[0121] In sparse images and annotations t b = t e =x The left side of the formula is the label, and the right side is the image required to obtain these labels. In other words, the left side is the result of the model output, and the right side is the image required for the model input. b =t e =x indicates that the model input is the current frame plus the previous frames, and the output is the bounding box and action category for the current frame. This is a restriction for model training and differs from the Dataset loading method. In dense images, the model input is the current frame plus the previous frames (or a video), and the output is the bounding box and action category for multiple frames (or the bounding box and action category for each frame in the video).

[0122] Through data loaders with different parameter selections, the dataset obtained in this application can be used to train YOLO, DETR, Frame-level STAD, Clip-level STAD and other target detection and spatiotemporal action detection datasets.

[0123] This application makes full use of intelligent tools and proposes a method for constructing a multi-target spatiotemporal action detection dataset for closed scenes, targeting scenes with a small number of target inflows and outflows in the video, multiple targets, multiple behavior categories, easy occlusion between targets, and no rapid motion. It can accurately match the identity information of targets that have not appeared for a long time, and finally obtain dataset annotation files that can be used for four tasks at the frame and fragment levels: target detection, behavior recognition, and spatiotemporal action detection, which greatly improves the efficiency and quality of dataset production. It includes the following steps: collecting multi-target closed scene video data; cropping videos based on fixed intervals and frame rates; annotating targets to train target detectors; detecting sampled videos; annotating all target actions in the sampled frames based on detection results and video sequences; assigning identity information to targets for target tracking based on detection results and target features; merging action annotation and tracking annotation files to form an annotation file; based on the annotation file, forming a dataset format and data loader suitable for various computer vision tasks; creating the final multi-target spatiotemporal action detection dataset in closed scenes. This application achieves accurate tracking of multiple targets in closed scenes, and provides a dataset production paradigm for studying the complex behaviors of targets in spatiotemporal action detection.

[0124] This application has the following technical effects.

[0125] First, in the dataset production, this application combines the time, space, target identity information and target action information in the scene to obtain rich annotation files.

[0126] Second, in the preparation of the dataset, this application greatly simplifies the time cost of manual labeling by training target detectors and target tracking algorithms improved for the scenarios.

[0127] Third, in the process of dataset production, this application obtained dataset formats that meet the requirements of various tasks, further improving the efficiency of dataset production.

[0128] This application also provides an application scenario that utilizes the aforementioned method for constructing a spatiotemporal motion detection dataset. Specifically, the method provided in this embodiment can be applied to animal behavior recognition in farm animals, such as cattle, sheep, rabbits, and other farmed animals. Because many animal behaviors are continuous and coherent, a single image alone cannot accurately determine animal behavior. Therefore, a spatiotemporal motion detection dataset is required.

[0129] Based on the same inventive concept, embodiments of the present application also provide a spatiotemporal action detection dataset construction device for implementing the aforementioned method for constructing a spatiotemporal action detection dataset. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more of the following embodiments of the spatiotemporal action detection dataset construction device can be found in the aforementioned definition of the method for constructing a spatiotemporal action detection dataset, and will not be further elaborated here.

[0130] In an exemplary embodiment, Figure 11 As shown, a device for constructing a spatiotemporal action detection dataset is provided.

[0131] The acquisition and cropping module 1201 is used to acquire a surveillance video of a target scene and crop the surveillance video into short video clips.

[0132] The image extraction module 1202 is configured to perform image extraction on the short video clip to obtain a dense image file, a sparse image file, and a detection image file.

[0133] The target detection module 1203 is used to detect all targets in the dense image file and the sparse image file using a target detector to obtain dense image detection results and sparse image detection results; the target detector is trained by the detection image file.

[0134] The traversal module 1204 is configured to traverse the sparse image detection results, annotate the target behavior, and obtain a target behavior detection dataset.

[0135] The target tracking module 1205 is configured to track targets in the dense image and the sparse image using a target tracker based on the dense image detection result and the sparse image detection result to obtain a target identity dataset.

[0136] The matching module 1206 is configured to match the target behavior detection dataset with the target identity dataset to obtain a spatiotemporal action detection dataset.

[0137] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data for constructing a spatiotemporal action detection dataset. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a spatiotemporal action detection dataset is implemented.

[0138] Those skilled in the art will understand that Figure 12 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned method embodiments when executing the computer program.

[0139] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the above-mentioned method embodiments when executed by a processor.

[0140] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the above method embodiments are implemented.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0142] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0143] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0144] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0145] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing a spatiotemporal action detection dataset, characterized in that: The method for constructing a spatiotemporal action detection dataset includes: Acquire a surveillance video of a target scene and crop the surveillance video into short video clips; Performing image extraction on the short video clip to obtain a dense image file, a sparse image file, and a detection image file; performing image extraction on the short video clip to obtain a dense image file, a sparse image file, and a detection image file, specifically comprising: extracting the short video clip frame by frame to obtain a dense image file; extracting the short video clip second by second to obtain a sparse image file; and randomly extracting the short video clip to obtain a detection image file; Detecting all objects in the dense image file and the sparse image file using an object detector to obtain a dense image detection result and a sparse image detection result; wherein the object detector is trained by the detection image file; Traversing the sparse image detection results, annotating the target behavior, and obtaining a target behavior detection dataset; Using the target tracker to track the targets in the dense image and sparse image based on the dense image detection results and the sparse image detection results, a target identity dataset is obtained; The target behavior detection dataset and the target identity dataset are matched to obtain a spatiotemporal action detection dataset; the spatiotemporal action detection dataset is used for behavior recognition.

2. The method for constructing a spatiotemporal action detection dataset according to claim 1, wherein: Obtaining a surveillance video of a target scene and cropping the surveillance video into short video clips specifically includes: Capturing a surveillance video of a target scene from a bird's-eye view; wherein the surveillance video includes a closed boundary of the target scene; defining a target behavior within a target scene based on the surveillance video; Filtering the surveillance video according to the target behavior to obtain a filtered surveillance video; The filtered surveillance video is reconstructed according to a fixed frame rate and cut into short video clips.

3. The method for constructing a spatiotemporal action detection dataset according to claim 1, wherein: The model used by the target detector is YOLOv8.

4. The method for constructing a spatiotemporal action detection dataset according to claim 1, wherein: The target tracker is used to track the targets in dense and sparse images based on the dense image detection results and the sparse image detection results to obtain the target identity dataset, which specifically includes: A set number of images from the dense image detection results and the sparse image detection results are respectively extracted as the starting points of the target tracker, and the DeepSort algorithm with improved cascade matching is used to track all targets in the dense image detection results and the sparse image detection results to obtain a target identity dataset.

5. The method for constructing a spatiotemporal action detection dataset according to claim 1, wherein: The method for constructing a spatiotemporal action detection dataset further includes: A final spatiotemporal action detection dataset is determined based on the spatiotemporal action detection dataset by utilizing upsampling, format conversion, and random sampling.

6. A device for constructing a spatiotemporal action detection dataset, characterized in that: include: An acquisition and cropping module, configured to acquire a surveillance video of a target scene and crop the surveillance video into short video clips; An image extraction module is used to extract images from the short video clips to obtain dense image files, sparse image files and detection image files; Extracting images from the short video clip to obtain a dense image file, a sparse image file, and a detection image file, specifically comprising: extracting the short video clip frame by frame to obtain a dense image file; extracting the short video clip second by second to obtain a sparse image file; and randomly extracting the short video clip to obtain a detection image file; An object detection module is configured to detect all objects in the dense image file and the sparse image file using an object detector to obtain a dense image detection result and a sparse image detection result; the object detector is trained by the detection image file; A traversal module, configured to traverse the sparse image detection results, annotate the target behavior, and obtain a target behavior detection dataset; The target tracking module is used to track the targets in the dense image and the sparse image based on the dense image detection results and the sparse image detection results using the target tracker to obtain the target identity dataset; The matching module is used to match the target behavior detection dataset with the target identity dataset to obtain a spatiotemporal action detection dataset.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a spatiotemporal action detection dataset according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a spatiotemporal action detection dataset according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing a spatiotemporal action detection dataset according to any one of claims 1 to 5 is implemented.

Images