Video dense target detection data labeling method, device and equipment and storage medium

Abstract

This invention relates to the field of image recognition technology, and discloses a method, apparatus, device, and storage medium for annotating dense target detection data in video. The method includes: preprocessing a video file to obtain a sequence of sub-frame images and frame target information; setting a preset small target region and using the preset small target region as the Region of Interest (ROI) of the sub-frame image sequence; obtaining ROI region information corresponding to the ROI region based on the frame target information; and annotating the ROI region and ROI region information of the sub-frame image sequence to obtain ROI annotation results. In this embodiment, ROI regions are mainly used for annotating small or very dense targets. By selecting and enlarging ROI regions, automatic tracking of small, dense, and special target types is supported to accelerate the annotation speed and achieve accurate and efficient annotation on small target images.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, device, and storage medium for annotating dense target detection data in video. Background Technology

[0002] In the field of image object detection, deep learning technology has achieved a dominant position, leading the pack in various image recognition competitions. However, in practical applications, training a high-accuracy deep learning model requires a large amount of manually labeled image data. This process consumes enormous human, material, and financial resources. For example, the ImageNet dataset, considered a milestone in computer vision, contains tens of millions of labeled images, most of which were completed by 50,000 users on the crowdsourcing platform Mechanical Turk over two years. However, in real-world industry applications, such as "smart cities," the requirements are complex and varied, making it impossible to label a large-scale dataset for every specific AI task. The difficulty, high cost, and scarcity of labeled data have become bottlenecks that urgently need to be addressed in the development and application of artificial intelligence technology.

[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for annotating dense target detection data in video, aiming to solve the technical problems of existing image target detection datasets, such as the difficulty in annotating small or dense targets that are difficult for the human eye to distinguish, the high cost of annotation, and the scarcity of annotated data.

[0005] To achieve the above objectives, the present invention provides a method for annotating video dense target detection data, comprising:

[0006] The video file is preprocessed to obtain a sequence of subframe images and frame target information;

[0007] A preset small target area is defined, and the preset small target area is used as the ROI region of the subframe image sequence;

[0008] Based on the frame target information, obtain the ROI region information corresponding to the ROI region;

[0009] The ROI regions and ROI region information of the subframe image sequence are labeled to obtain ROI labeling results.

[0010] In some embodiments, setting a preset small target region and using the preset small target region as the ROI region of the subframe image sequence includes:

[0011] Determine the first frame image in the subframe image sequence;

[0012] Obtain the frame target information corresponding to the first frame image, and determine the coordinates of the target rectangle based on the frame target information;

[0013] Based on the coordinates of the target rectangle, at least one preset small target region is set, and the preset small target region is used as the ROI region of the subframe image sequence.

[0014] In some embodiments, the step of annotating the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results includes:

[0015] ROI configuration files are generated based on the ROI regions of the subframe image sequence, and the ROI region information is stored in the ROI subfolder;

[0016] Call the ROI subfolder and determine if an ROI configuration file exists;

[0017] When the ROI configuration file is detected, the current annotation mode is set to ROI mode;

[0018] In the ROI mode, the ROI regions and ROI region information of the subframe image sequence are labeled to obtain ROI labeling results.

[0019] In some embodiments, generating an ROI configuration file based on the ROI regions of the subframe image sequence and storing the ROI region information in an ROI subfolder includes:

[0020] Obtain the coordinate information and rectangle shape information corresponding to the ROI region of the subframe image sequence;

[0021] Generate an ROI configuration file based on the coordinate information and the rectangle shape information;

[0022] Based on the ROI region information, determine the target type and rectangle coordinate information within the ROI region, and store the target type and rectangle coordinate information within the ROI region into the ROI subfolder.

[0023] In some embodiments, the step of annotating the ROI regions and ROI region information of the subframe image sequence in the ROI mode to obtain ROI annotation results includes:

[0024] In the ROI mode, the ROI area is enlarged and displayed on the screen display area based on the ROI configuration file;

[0025] The ROI region and its information are then annotated to obtain the ROI annotation results.

[0026] In some embodiments, the step of labeling based on the magnified ROI region and ROI region information to obtain ROI labeling results includes:

[0027] Based on the magnified ROI region and ROI region information, annotations are performed to obtain the magnified annotation results of the image.

[0028] The ROI region coordinates of the specific target are obtained based on the annotation results after the image is magnified;

[0029] The coordinate restoration calculation is performed on the displayed coordinates of the ROI region to obtain the restored coordinate information of the specific target in the screen display area;

[0030] The coordinate information in the magnified image annotation result is updated based on the restored coordinate information to obtain the ROI annotation result.

[0031] In some embodiments, the method further includes:

[0032] Based on a preset unsupervised tracking algorithm, the subframe image sequence and frame target information are labeled to obtain the main region labeling result;

[0033] The main region annotation results and ROI annotation results are merged based on a preset merging algorithm to obtain the target annotation results.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes a video dense target detection data annotation device, comprising:

[0035] The preprocessing module is used to preprocess the video file to obtain a sequence of subframe images and frame target information;

[0036] The setting module is used to set a preset small target area and use the preset small target area as the ROI area of ​​the subframe image sequence;

[0037] The acquisition module is used to acquire ROI region information corresponding to the ROI region based on the frame target information;

[0038] The ROI annotation module is used to annotate the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

[0039] Furthermore, to achieve the above objectives, the present invention also proposes a video dense target detection data annotation device, which includes: a memory, a processor, and a video dense target detection data annotation program stored in the memory and executable on the processor. The video dense target detection data annotation program is configured to implement the video dense target detection data annotation method as described in the above embodiments.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a video dense target detection data annotation program, which is used to enable the processor to implement the video dense target detection data annotation method as described in the above embodiments.

[0041] This invention provides a method for annotating dense target detection data in video, comprising: preprocessing a video file to obtain a sequence of sub-frame images and frame target information; setting a preset small target region and using the preset small target region as the Region of Interest (ROI) of the sub-frame image sequence; obtaining ROI region information corresponding to the ROI region based on the frame target information; and annotating the ROI region and ROI region information of the sub-frame image sequence to obtain ROI annotation results. In this embodiment, the ROI region is mainly used for annotating small or very dense targets. By selecting and enlarging the ROI region, automatic tracking of small, dense, and special target types is supported to accelerate the annotation speed and achieve accurate and efficient annotation on small target images. This solves the technical problems of existing image target detection datasets, such as the difficulty in annotating small or dense targets that are difficult for the human eye to distinguish, the high annotation cost, and the scarcity of annotated data. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of the video dense target detection data annotation device in the hardware operating environment involved in the embodiments of the present invention;

[0043] Figure 2 This is a flowchart illustrating the first embodiment of the video dense target detection data annotation method of the present invention;

[0044] Figure 3 This is a schematic diagram of a video slice in the video dense target detection data annotation method of the present invention;

[0045] Figure 4 This is a first schematic diagram of ROI annotation in the video dense target detection data annotation method of the present invention;

[0046] Figure 5 This is a second schematic diagram of ROI annotation in the video dense target detection data annotation method of the present invention;

[0047] Figure 6 This is a flowchart illustrating the second embodiment of the video dense target detection data annotation method of the present invention;

[0048] Figure 7 This is a schematic diagram of the main region annotation in the video dense target detection data annotation method of the present invention;

[0049] Figure 8 This is a structural block diagram of the first embodiment of the video dense target detection data annotation device of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0053] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a video-dense target detection data annotation device in the hardware operating environment involved in the embodiments of the present invention.

[0055] like Figure 1As shown, the video dense target detection data annotation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0056] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on video dense target detection data annotation equipment, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a video dense target detection data annotation program.

[0058] exist Figure 1 In the video dense target detection data annotation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the video dense target detection data annotation device of the present invention can be set in the video dense target detection data annotation device, and the video dense target detection data annotation device calls the video dense target detection data annotation program stored in the memory 1005 through the processor 1001 and executes the video dense target detection data annotation method provided in the embodiment of the present invention.

[0059] Among common object detection annotation methods or tools, there are several typical types. First, Labelme / LabemImg, open-source annotation tools. Both have similar functions and can annotate rectangular targets in images. They are simple and easy to use, but they cannot process videos and do not have intelligent annotation functions. They require a fully manual annotation method, which is time-consuming and labor-intensive. When the targets are particularly dense or small, annotation becomes very difficult.

[0060] Secondly, DarkLabel, a closed-source multi-object annotation tool, can annotate images / videos and has some automatic target tracking capabilities, including interpolation. For example, it can annotate targets on two frames separated by a certain distance, and then use interpolation to achieve rectangular tracking of targets on unannotated frames in between. However, the results are not very accurate. It also supports robust / accurate tracking modes. The robust mode is not very accurate and can only track about 3 to 5 frames before the error becomes very large. The accurate mode works well for rigid bodies / targets with small deformations, but not very well for targets with large deformations (such as pedestrians walking). In addition, DarkLabel has poor video support performance. It cannot process large video files and does not easily achieve accurate frame localization. It also cannot import preset deep learning models.

[0061] Third, CVAT, an open-source, web-based interactive video / image annotation tool in a B / S mode, supports keyframe bounding box interpolation and automatic annotation using deep learning models with TensorFlow ODAPI and Intel OpenVINO IR format. However, its limitation to Chrome means that the tool will inevitably have performance issues when annotating complex multi-object objects, and the requirements for the training model framework will further limit its use.

[0062] Fourth, various methods utilize pre-set deep learning models to achieve automated annotation. For example, a small batch of models is first annotated to obtain the model, and then applied to the annotation process to accelerate the annotation process. Annotation based on pre-set deep learning models works well in the field of object detection under normal conditions (e.g., the target size is relatively large and the environmental background is relatively clear), but it is difficult to solve the recognition problem in two situations: a) Recognition of small targets. Taking a typical resolution of 1080p video in urban surveillance as an example, when the height of pedestrian targets is reduced to about 64 to 128 pixels, most deep learning pedestrian detection models will fail and cannot be recognized. However, in some smart city or other industry scenarios, the recognition of small targets is quite important and cannot be easily ignored or missed. At this time, pure manual annotation becomes the only option, but the workload of fully manual annotation per frame is unbearable in video business; b) Some special target types that need to be trained from scratch mean that there is still a pure manual annotation process before using the model, which will also slow down the annotation process in video business.

[0063] In view of the above problems, embodiments of the present invention provide a method for annotating video dense target detection data, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of a video dense target detection data annotation method according to the present invention.

[0064] like Figure 2 As shown, the video dense target detection data annotation method includes:

[0065] S10: Preprocess the video file to obtain several sub-frame image sequences and frame target information;

[0066] S20: Set a preset small target area and use the preset small target area as the ROI area of ​​the subframe image sequence;

[0067] S30: Obtain the ROI region information corresponding to the ROI region based on the frame target information;

[0068] S40: Annotate the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

[0069] It should be noted that the execution entity in this embodiment can be a video dense target detection data annotation device. This video dense target detection data annotation device can be a computer device with data processing capabilities, or other devices that can achieve the same or similar functions. This embodiment does not limit this; in this embodiment, a computer device is used as an example for explanation. It is understood that image recognition involves a wide range, including image segmentation, image detection, image tracking, image matching, etc. This embodiment provides a video dense target detection data annotation method, mainly involving annotation methods in the field of target detection and tracking, such as providing the bounding box position and category of the target, and continuously tracking the target in a video sequence.

[0070] For example, step S10 includes: preprocessing the video file to obtain several frame images and frame target information corresponding to each frame image; slicing the frame images based on a preset length to obtain several sub-frame image sequences.

[0071] In one example, the process of preprocessing a video file to obtain several frames and the corresponding frame target information for each frame may include: determining whether a preset video target detection model exists; if the preset video target detection model exists, invoking the preset video target detection model; preprocessing the video file based on the preset video target detection model to obtain several frames and the target type, target ID, and target bounding box coordinates corresponding to each frame; and storing the target type, target ID, and target bounding box coordinates as the frame target information corresponding to the frame image.

[0072] Specifically, a pre-defined video object detection model can be used to preprocess the video file, exporting each frame as an image and its corresponding frame object information. The frame object information includes the object detection type and detection bounding box coordinates exported frame by frame, and is saved to a text file. The pre-defined video object detection model includes, but is not limited to, open-source or self-trained deep learning models.

[0073] For example, if a pre-trained object detection model (e.g., an open-source model) exists to identify the target type to be labeled, it can be used as a preset video object detection model. The preset video object detection model prediction service is deployed to predict video files frame by frame, exporting each frame as an image and its corresponding target information. The output of the target information for each frame includes the target type, target ID, and target bounding box coordinates (x, y, w, h). In one example, each video frame is exported as an image, and the output of each frame (frame target information) is written to a text file with the same name. The directory hierarchy is: dir / image path; dir / in / text file path. It is understood that if the preset video object detection model does not exist, the output can be empty.

[0074] In one example, the process of slicing the frame image based on a preset length to obtain a sequence of sub-frame images may include: setting a preset slice time length; and slicing the frame image based on the preset slice time length to divide it into multiple sub-frame image sequences.

[0075] For example, the exported frame image sequence is sliced ​​into multiple sub-frame image sequences according to a certain length unit, and the corresponding target type, coordinate data, etc., are exported to a sub-text file. The process of slicing the frame image into multiple sub-frame image sequences based on the preset slice time length may include: determining the number of slices based on the video duration of the video file and the preset slice time length; determining the number of frames in each slice based on the video frame rate of the video file and the video duration; generating a slice sequence starting from the first frame at intervals of the specified number of frames to obtain several sub-frame image sequences; and storing the sub-frame image sequences and the corresponding frame target information.

[0076] Specifically, such as Figure 3 As shown, video slicing divides the preprocessed image sequence into slices according to a certain time unit. Each slice can serve as a basic unit for annotation tasks. The preset slice duration is set to sliceT. If the video duration is T, the number of slices is N = T / (sliceT), and the number of frames in each slice is frameNums ≤ (sliceT) * fps, where fps is the video frame rate. For each video result export directory dir, starting from the first frame, a slice sequence (subframe image sequence) is generated every frameNums frames. The frameNums images are copied to a new subdirectory, and the corresponding text files of the frame output results are also copied to the same subdirectory. The directory hierarchy is: newDir / image path, newDir / in / text file path. Each new subdirectory newDir contains frameNums images and the same number of text files with the same name. The number of images and text files in the last slice (subframe image sequence) may be less than the number of frames frameNums. The number of target categories K in the entire video export result dir is counted, and the maximum value of the target ID in each category is maxID(i), where i = 0 to K-1.

[0077] In one example, setting a preset small target region and using the preset small target region as the ROI region of the subframe image sequence includes: determining a first frame image in the subframe image sequence; obtaining frame target information corresponding to the first frame image, and determining the target rectangle coordinates based on the frame target information; setting at least one preset small target region based on the target rectangle coordinates, and using the preset small target region as the ROI region of the subframe image sequence.

[0078] In one example, the step of annotating the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results includes: generating an ROI configuration file based on the ROI regions of the subframe image sequence and storing the ROI region information in an ROI subfolder; calling the ROI subfolder and determining whether an ROI configuration file exists; when the ROI configuration file is detected, setting the current annotation mode to ROI mode; and in ROI mode, annotating the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

[0079] In one example, the step of generating an ROI configuration file based on the ROI region of the subframe image sequence and storing the ROI region information in an ROI subfolder includes: obtaining the coordinate information and rectangle shape information corresponding to the ROI region of the subframe image sequence; generating an ROI configuration file based on the coordinate information and rectangle shape information; determining the target type and rectangle coordinate information within the ROI region based on the ROI region information, and storing the target type and rectangle coordinate information within the ROI region in an ROI subfolder.

[0080] In one example, the step of annotating the ROI region and ROI region information of the subframe image sequence in the ROI mode to obtain ROI annotation results includes: in the ROI mode, enlarging the ROI region in the screen display area based on the ROI configuration file; and annotating the enlarged ROI region and ROI region information to obtain ROI annotation results.

[0081] In one example, the step of labeling based on the magnified ROI region and ROI region information to obtain ROI labeling results includes: labeling based on the magnified ROI region and ROI region information to obtain magnified image labeling results; obtaining the ROI region display coordinates of a specific target based on the magnified image labeling results; performing coordinate restoration calculation on the ROI region display coordinates to obtain the restored coordinate information of the specific target in the screen display area; and updating the coordinate information in the magnified image labeling results based on the restored coordinate information to obtain ROI labeling results.

[0082] In one example, areas that are difficult for the human eye to distinguish and label, such as small or dense targets, are designated as Regions of Interest (ROIs). Specifically, the process of labeling the ROIs and frame target information of the subframe image sequence to obtain ROI labeling results is exemplified as follows: A preset small target area is set, and the preset small target area is used as the ROI of the subframe image sequence; ROI information corresponding to the ROI is obtained based on the frame target information; and ROIs and ROI information of the subframe image sequence are labeled to obtain ROI labeling results.

[0083] Specifically, when a Region of Interest (ROI) is selected, it is designated as ROI annotation mode. In ROI annotation mode, the segmentation results are annotated. ROI regions are primarily used for annotating small or very dense targets. It should be noted that in ROI annotation mode, the ROI region needs to be enlarged before annotation. The specific ROI region division can be set according to actual needs, based on what is difficult for the human eye to observe and annotate; this embodiment does not impose any limitations on this.

[0084] For example, for a particularly small target area, a suitable ROI region, such as a rectangular ROI region, is selected on the image. The coordinate information, length, and width of this ROI region are exported to the ROI configuration file. Subfiles containing the target type within the ROI region and the processed rectangular bounding box coordinates are exported to the ROI subfolder. Multiple ROI regions can be set. The annotation tool imports the ROI subfolder and determines whether to use ROI annotation mode or normal annotation mode (e.g., main area annotation mode) depending on whether an ROI configuration file exists. For ROI annotation mode, the image is enlarged to a suitable size, and data annotation is performed after the enlargement. The annotation data is then transformed according to the appropriate coordinates and exported to the annotation file.

[0085] In one example, ROI annotation includes: defining a small / dense target area of ​​a certain size as the ROI region, exporting the ROI region information, reloading the exported information, enlarging the image, repeating the target annotation process, and setting the saved annotation information as the ROI annotation result. Specifically, areas that are difficult for the human eye to distinguish and annotate, such as small / dense targets, are designated as ROI regions.

[0086] Specifically, a rectangle is selected from the first frame of the segmented subframe image sequence. This rectangle is sufficient to enclose the small target. Depending on the number and distribution of the target, multiple ROI regions can be selected. These multiple ROI regions are denoted as Region(i)(x1,y1,x2,y2), where i = 0 to n-1, and n is the number of selected ROI regions.

[0087] Specifically, export ROI information for the j-th ROI region. For example, export the coordinates of Region(j): x1, y1, x2, y2 to the image folder / ROI subdirectory, denoted as roi.config (ROI configuration file). Based on Region(j)(x1, y1, x2, y2) and the imported coordinate information from video preprocessing, if a recorded target Object(Cls, id, x, y, w, h) is completely within the Region(j) region: x >= x1, y >= y1, x+w-1 <= x2, y+h-1 <= y2, then continue exporting the Object information to the image folder / ROI(j) / in subfolder.

[0088] Specifically, ROI region loading: When all information of the j-th ROI is exported according to the above process of exporting ROI information of the j-th ROI region, the annotation tool reloads the image folder / ROI(j) subdirectory. When roi.config (ROI configuration file) is detected, the current annotation mode is recorded as ROI mode, which is different from the normal annotation mode when no ROI region is defined.

[0089] Specifically, ROI region magnification and display: After reading roi.config (ROI configuration file) (x1,y1,x2,y2), the corresponding ROI region of the original image is magnified to the image display area. The magnification process includes: based on the ROI region coordinates of the original image: x1, y1, x2, y2, w, h, the original image information: W, H, the screen display area: W, H, and the type and coordinate data of a certain exported target in the / ROI / in subfolder, Object(Cls,ID,xx1,yy1,xx2,yy2), the coordinates of the Object displayed on the screen after the ROI region is loaded can be obtained as follows: X1=x1*W / w, Y1=y1*H / h, X2=x2*W / w, Y2=y2*H / h.

[0090] Specifically, ROI annotation is performed after the ROI area is zoomed in and displayed. It should be noted that the ROI annotation process is similar to the main area annotation process, with the difference being the restoration and saving of the ROI annotation results and the calculation of the size of the target matching area and the search area during target matching.

[0091] For example, for a target annotation result: screen coordinates Q(X1,Y1,X2,Y2), when saving the annotation result, what is actually saved is the target's annotation coordinates q restored from screen coordinates Q to the original image. Therefore, it is necessary to perform a restoration calculation on screen coordinates Q. It can be understood that the specific restoration calculation of screen coordinates Q includes: the corresponding coordinates q in the original image are: src_x1=X1 / W*w+x1, src_y1=Y1 / H*h+y1, Src_x2=X2 / W*w+x1, Src+y2=Y2 / H*h+y2. For the ROI region image matching process, the image to be matched is: the screen-annotated rectangle Q(X1,Y1,X2,Y2); the image to be searched is: the image T of the ROI region in the original image magnified to the screen area (W,H); after matching the result Q1, Q1 needs to be restored, referring to the coordinate restoration process from src_x1 to src_y2 described above. J = 0 to n-1. For each ROI region, repeat the above ROI labeling process, which will not be elaborated here.

[0092] This embodiment uses pedestrian data annotation in a smart city as a specific scenario for illustration. The minimum size of the pedestrian target to be annotated can be 32*64 pixels, and the video duration of the video file to be annotated is half an hour (1800 seconds, 25 frames per second). Pedestrians include ordinary pedestrians (category 0) and cyclists (bicycles, tricycles, motor vehicles, etc., category 1). The annotation results should provide the category C1s, pedestrian ID, and the rectangular coordinates (x1, y1, x2, y2) of all pedestrians. For cyclists, the rectangle should completely enclose both the person and the vehicle.

[0093] Specific scenario: Pedestrian targets at normal density (e.g., 5 to 15 people per frame), including small targets and cyclists. In this scenario, small pedestrian targets are difficult to observe even with the human eye due to their tiny pixel values, resulting in significant errors when drawing bounding boxes. General deep learning models also struggle to detect them, reducing annotation efficiency. Therefore, this embodiment introduces Region of Interest (ROI) annotation. PPHuman's detection of pedestrians cannot include bicycles; therefore, the imported pre-annotated data for pedestrian-cyclist types needs to be discarded and manually annotated again.

[0094] The specific process is as follows: Based on Baidu's open-source PPHuman, a pedestrian detection service is modified and deployed. The input is a video file sample.mp4, the export directory is Dir, and the output is video frame images and the pedestrian target data (id, x1, y1, x2, y2) detected in each frame. The total number of video frames is 1800 * 25 = 45000. The output format is: Dir / 1.jpg, Dir / 2.jpg, Dir / 3.jpg...Dir / 45000.jpg and Dir / in / 1.txt, Dir / in / 2.txt...Dir / in / 45000.txt. The txt format is: Cls, ID, (x1, x2, y1, y2), where x1, x2, y1, y2 are multiple lines. The type value is uniformly set to 0, as PPHuman does not have the ability to finely classify types; this will be modified later using an annotation tool. The maximum value of ID in all txt files is calculated, assumed to be 360. The exported image and text results are segmented into 10-second segments. The total number of segments is 1800 / 10 = 180, and each segment contains 10 * 25 = 250 frames. For example... Figure 3 As shown, the initial value of the new ID for each shard is used to generate a new ID. For example, the initial value of shard 1 is newID = 500, the initial value of shard 2 is newId = 1000, and so on, increasing by 500 every time, to ensure that the newly generated IDs for each shard are not duplicated.

[0095] For example, the process of labeling a person riding a bicycle is as follows: The labeling tool imports the first segment (frames 1-250 and 1-250 txt files), and draws and displays rectangles on the corresponding coordinates of the screen display area based on the pre-stored labeling information in the txt files. Due to the pre-labeling import, most targets do not need to be labeled again; only a small number of targets without rectangles need to be found. For targets without rectangles, there are two cases: one is that the target was not detected at all, in which case a new ID needs to be generated, ID = newID, and newID = newID + 1 (incrementing by 1); the other is that the target was detected in some frames, in which case a new ID does not need to be generated, and the already detected IDs can be reused.

[0096] Specifically, ROI region labeling is performed: Assuming the small target is concentrated in two regions of the image (the division criteria are based on achieving a magnification effect, with an optimal near-square effect), such as... Figure 4 and Figure 5 As shown, the detailed process is as follows: Select the ROI rectangle within the main annotation image area (original image size); export the ROI region image, ROI configuration file, and pre-annotated data within the ROI region to the dir / roi subfolder, in the following format: Dir / roi1 / roi.json

[0097] {

[0098] "xmin": 20,

[0099] “ymin”: 30

[0100] "width": 400,

[0101] "height": 300

[0102] }

[0103] Dir / roi1 / 1.jpg (1.jpg is a small image stored in the roi1 region, and the same applies to subsequent images) ...

[0105] Dir / roi1 / 250.jpg

[0106] Dir / roi1 / in / 1.txt (1.txt is the complete pre-annotated object within the roi1 region, and the same applies to subsequent files) ...

[0108] Dir / roi1 / in / 250.txt.

[0109] Specifically, import ROI region data: The annotation tool imports the roi1 folder and checks if the roi.json configuration file exists to determine if it's currently in ROI mode. If it is in ROI mode, a zoom-in operation is required, and the screen display area size needs to be set to the original image size. For example... Figure 4 As shown, the specific ROI annotation process can be referenced from the main area annotation mode above. The difference is that the coordinates after the screen is enlarged need to be calculated when loading, the enlarged image is also used when matching, and the enlarged screen data needs to be restored to the small image coordinates when saving.

[0110] This embodiment preprocesses the video file to obtain a sequence of sub-frame images and frame target information; it sets a preset small target region and uses this preset small target region as the Region of Interest (ROI) of the sub-frame image sequence; it obtains the ROI region information corresponding to the ROI region based on the frame target information; and it performs annotation based on the ROI region and ROI region information of the sub-frame image sequence to obtain ROI annotation results. In this embodiment, ROI regions are mainly used for annotating small or very dense targets. By selecting and enlarging ROI regions, automatic tracking of small, dense, and special target types is supported to accelerate the annotation speed and achieve accurate and efficient annotation on small target images.

[0111] In one embodiment, such as Figure 6As shown, a second embodiment of the video dense target detection data annotation method of the present invention is proposed based on the first embodiment. The method further includes...

[0112] S50: Based on a preset unsupervised tracking algorithm, the sub-frame image sequence and frame target information are labeled to obtain the main region labeling result;

[0113] S60: The main region annotation results and ROI annotation results are merged based on a preset merging algorithm to obtain the target annotation results.

[0114] In one example, the process of annotating the subframe image sequence and frame target information based on a preset unsupervised tracking algorithm to obtain the main region annotation result may include: annotating the target type and target rectangle in the screen display area according to the subframe image sequence and frame target information to obtain a first annotation result; determining whether the target rectangle exists in the subframe image sequence frame by frame; when the target rectangle does not exist in the subframe image sequence, annotating the subframe image sequence based on the preset unsupervised tracking algorithm to obtain a second annotation result; and using the first annotation result and the second annotation result as the main region annotation result.

[0115] Specifically, when no ROI region is selected, it is referred to as the main region annotation mode. In the main region annotation mode, the main region is annotated on the segmented sub-frame image sequence and frame target information. In the main region annotation mode, a preset unsupervised tracking algorithm, such as an unsupervised target matching algorithm, can be used to achieve semi-automatic annotation and target tracking. It can be understood that the main region annotation is mainly used for the annotation of relatively clear, large targets. The target size is required to be greater than 150*200. The specific target size requirement can be set according to actual needs, based on what is easy for the human eye to observe and annotate. This embodiment does not impose any restrictions on this.

[0116] For example, the annotation tool imports a sequence of subframe images and a sub-text file (including frame target information), displays the corresponding target types and target bounding boxes in the GUI window, and checks for and fills in missing target types across the subframe image sequence. For missing target types, manual annotation can be performed. For instance, for target types that require manual annotation, an appropriate number of automatic detection frames can be set based on an unsupervised target detection algorithm for automatic annotation, thereby speeding up the annotation process.

[0117] In this embodiment, under the main region annotation mode, the segmented sub-frame image sequence and frame target information are annotated with the main region. In the main region annotation mode, a preset unsupervised tracking algorithm, such as an unsupervised target matching algorithm, can be used to achieve semi-automatic annotation and target tracking.

[0118] For example, a single segmented image sequence, i.e., a subframe image sequence, and the corresponding preprocessed frame target information (target category, target ID, target bounding box (x, y, w, h), where the number of target categories is K, and the maximum value of the target ID in each category is maxID(i), i = 0 to K-1) are imported. The target category, target OD, and target bounding box of each target are displayed on a single image in the annotation interface. A specific target is selected, and its trajectory is continuously tracked frame by frame: if the target's coordinate information already exists, the target type and target bounding box are annotated in the screen display area according to the subframe image sequence and frame target information to obtain the first annotation result, which can then be used to fill in any gaps. If the target bounding box does not exist, the subframe image sequence is annotated based on a preset unsupervised tracking algorithm, i.e., missing frames without imported information are annotated to obtain the second annotation result.

[0119] In one example, the specific annotation supplementation process is as follows: Starting from the frame with missing annotation information, denoted as frame i, the target rectangle of this frame is manually drawn, and the annotation area of ​​the target rectangle is denoted as R(x1,y1,x2,y2). An appropriate automatic tracking distance (number of frames) D is set, and a search is performed in frame i+1 using R as the object to be matched, based on a preset unsupervised tracking algorithm (e.g., an unsupervised image matching algorithm). It should be noted that, to improve search accuracy and efficiency, the specific search area can be set as S(x1-deltaX,y1-deltaY,x2+deltaX,y2+deltaY), where deltaX and deltaY are the search distances enlarged based on the original rectangle, which can be set according to the actual situation. This embodiment does not impose any restrictions on this. When the target area is matched in frame i+1, the result is denoted as R1, and R1 is set as the object to be matched. The above process is repeated until frame i+D. It is understood that, depending on different scenarios and algorithm adaptability, different image matching algorithms can be selected as the preset unsupervised tracking algorithm, and this embodiment does not impose any restrictions on this.

[0120] It should be noted that the specific fine-tuning process is as follows: the results automatically generated between i+1 and i+D are manually fine-tuned as needed. The fine-tuned result is denoted as M(x1,y1,x2,y2), where i = 1 to D, and M is the final labeled rectangular area result of the target. The target category and target ID are the same as the target category and target of the i-th frame, respectively.

[0121] In one example, for target types with no imported coordinates, denoted as completely new target categories, the specific target ID generation rules may include: The maximum ID value of the video target in the video file preprocessing step is maxID(i), where i is the target category. The initial value of the newly generated ID for each video slice is newID = maxID(i) + a(j), where j = 0 to N-1, N is the number of video slices, and a(j) is a positive integer. It is required that a(j+1) and a(j) maintain a sufficiently large distance to ensure that the newly generated IDs between slices do not conflict. Subsequent new ID generation = newID + 1. It should be noted that the coordinate annotation process for completely new target categories refers to the specific annotation supplementation and fine-tuning process described above, and will not be repeated here.

[0122] Specifically, this embodiment uses pedestrian data annotation in a smart city as an example. The minimum size of the pedestrian target to be annotated can be 32*64 pixels, and the video duration of the video file to be annotated is half an hour (1800 seconds, 25 frames per second). Pedestrians include ordinary pedestrians (category 0) and cyclists (bicycles, tricycles, motor vehicles, etc., category 1). The annotation results should provide the category C1s, pedestrian ID, and the rectangular coordinates (x1, y1, x2, y2) of all pedestrians. For cyclists, the rectangle should completely enclose both the person and the vehicle.

[0123] Specific scenario: Sparse pedestrian target scene (less than 5 people per frame), no small targets, no bicycles or pedestrians. The specific processing steps are as follows: Modify and deploy the pedestrian detection service based on Baidu's open-source PPHuman. The input is the video file sample.mp4, the export directory is Dir, and the output is video frame images and the pedestrian target data detected in each frame (id, x1, y1, x2, y2). The total number of video frames is 1800 * 25 = 45000. The output format is: Dir / 1.jpg, Dir / 2.jpg, Dir / 3.jpg...Dir / 45000.jpg and Dir / in / 1.txt, Dir / in / 2.txt...Dir / in / 45000.txt. The txt format is: Cls, ID, (x1, x2, y1, y2), where x1, x2, y1, y2 are multiple lines. The type value is uniformly set to 0, as PPHuman does not have the ability to subdivide types; this will be modified later using annotation tools. Find the maximum value of the ID in all txt files, let's say it's 360. Divide the exported image and text results into segments of 10 seconds each. The total number of segments is 1800 / 10 = 180, and each segment contains 10 * 25 = 250 frames. For example... Figure 3As shown, the initial value of the new ID for each shard is used to generate a new ID. For example, the initial value of shard 1 is newID = 500, the initial value of shard 2 is newId = 1000, and so on, increasing by 500 every time, to ensure that the newly generated IDs for each shard are not duplicated.

[0124] For example, the annotation process begins: The annotation tool imports the first segment (frames 1-250 and 1-250 txt files), and draws and displays rectangles on the corresponding coordinates of the screen display area based on the pre-stored annotation information in the txt files. Due to the pre-annotation import, most targets do not need to be annotated again; only a small number of targets without rectangles need to be identified. For targets without rectangles, there are two scenarios: one is that the target was not detected at all, in which case a new ID needs to be generated, ID = newID, and newID = newID + 1 (incrementing by 1); the other is that the target was detected in some frames, in which case a new ID does not need to be generated, and the already detected IDs can be reused.

[0125] Specifically, for the annotation of the target bounding box, it is assumed that there is no bounding box between frames 10 and 30. Unsupervised target matching algorithms such as template matching or SIFT can be used to track the target. The specific target tracking process is as follows: Figure 7 As shown: On frame 10, draw a bounding box for the target, denoted as m, and the rectangular region as b. On frame 11, perform a matching search on m. To speed up the search, the search area can be expanded outward by about 200 pixels from b, resulting in a bounding box r. The bounding box of the annotation result in frame 11 is denoted as r, with the ID and Cls type values ​​the same as in frame 10. Update the template to r, and repeat the matching process for frames 12 to 30. It should be noted that a tracking frame number d can be set according to the tracking effect. When the tracking frame number exceeds d, restart drawing the bounding box and repeating the target tracking process. Export the main region annotation results: Export the final annotation results of all frames to dir / out / in the following format: Dir / out / 1.txt, Dir / out / 2.txt...Dir / out / 250.txt. For video files with sparse scenes, the main region annotation process is complete after exporting the main region annotation results. The entire main region annotation process requires only a small amount of manual intervention. For the steps that require manual intervention, since the tracking distance d is generally greater than or equal to 5, and can reach up to about 20, the main region annotation method proposed in this embodiment can improve the data annotation efficiency by at least 5 times.

[0126] It is understandable that the main area annotation is mainly used for the annotation of relatively clear, large targets. The target size is required to be greater than 150*200. The specific target size requirement can be set according to actual needs, based on what is easy for the human eye to observe and annotate. This embodiment does not impose any restrictions on this.

[0127] It should be noted that annotation data merging involves merging the annotation file data in the main region annotation mode and the annotation data in the ROI mode according to the target merging algorithm. The merged annotation data file is the final annotation file required, i.e., the target annotation result. For example, the main region annotation result dir / out and the ROI annotation result dir / roi / out are merged. Items with the same Cls and ID are considered as the same target. When the same target appears in both the main region and the ROI region, the coordinate data (x1, y1, x2, y2) of the target annotation result is based on the ROI annotation result.

[0128] In one example, the process of merging the main region annotation results and ROI annotation results based on a preset merging algorithm to obtain target annotation results may include: merging the main region annotation results and ROI annotation results based on a preset merging algorithm to obtain initial target annotation results; extracting the target type, target ID, and target bounding box coordinates from the main region annotation results and ROI annotation results respectively; setting targets with the same target type and target ID as the same target; setting the target bounding box coordinates of the ROI annotation results as the target bounding box coordinates of the same target to obtain target annotation results for the same target; and updating the initial target annotation results based on the target annotation results for the same target to obtain target annotation results.

[0129] For example, the main region annotation results and ROI annotation results are merged. If the same target appears in both the main region annotation results and the ROI annotation results, the ROI annotation result shall prevail.

[0130] Specifically, for a single segmented sequence (subframe image sequence), with M frames, let the main region annotation results be saved as file R(j), where j = 0 to M-1; and the ROI annotation results as file S(k), where k = 0 to n-1, and n is the number of ROI regions. The annotation results in R(j) and S(k) are saved in the following format: Cls (target category), ID (target ID), coordinates (x1, y1, x2, y2). All R(j) and S(k) files are named the same as the frame name.

[0131] It should be noted that the segmented sequence (subframe image sequence) is traversed starting from 0 to M. For the j-th frame, the main region annotation result R(j) and the ROI annotation result S(k) are recorded, k = 0 to n-1. The result of S(k) is merged into S. The results of R(j) and S are compared. If the Cls and ID in the target annotation record object(Cls,ID,x1,y1,x2,y2) in the two files are the same, they are considered to be the same target. If both S and R(j) have the target, the result in S is taken as the standard, and the record result in S is retained in the final merged result. If the target is in R(j) but not in S, or the target is not in R(j) but is in S, the target record with the corresponding target is added to the final merged result. After all frames have been traversed, the final merged result is the sequence annotation result U, which is the target annotation result.

[0132] Understandably, for N slice sequences of the same video file, if the annotation duration of a single slice is insufficient, the entire video file must be annotated. This necessitates ID clustering and rearrangement of the sequence annotation results U(j) of the N slices. Due to the nature of video slices, different slices of the same video may contain the same target but are scattered across different adjacent slices, and each slice sequence may have a different ID. For this situation, an image clustering algorithm can be used. Target clustering is performed between every certain number of connected slices, and for several slices containing the same target, only one target ID needs to be used.

[0133] In this embodiment, for core vision tasks, video files are pre-processed and pre-annotated using intelligent annotation algorithms and annotation tools. Combined with a preset unsupervised tracking algorithm and ROI annotation, the merged target annotation results are obtained, realizing a semi-automatic or fully automated annotation process. This achieves or even surpasses the performance of a large number of manually annotated images, which can greatly save human and material resources, reduce the investment threshold for AI application development, and promote industry development in fields such as "smart cities".

[0134] In this embodiment, video files are preprocessed to achieve pre-annotation. Combined with a preset unsupervised tracking algorithm and ROI region annotation, the merged target annotation result is obtained. It supports automatic tracking of small targets, dense targets, and special target types. Various semi-automatic and automatic methods are used to accelerate the annotation process. It can process a large number of video files at the same time, reduce manual intervention in the entire chain, and accelerate the annotation efficiency. This solves the technical problems of low data annotation efficiency, high annotation cost, and lack of annotation data in the existing image target detection dataset.

[0135] Furthermore, this embodiment of the invention also proposes a storage medium storing a video dense target detection data annotation program, which, when executed by a processor, implements the steps of the video dense target detection data annotation method described above.

[0136] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the video dense target detection data annotation device of the present invention.

[0137] like Figure 8 As shown, the video dense target detection data annotation device includes:

[0138] Preprocessing module 10 is used to preprocess the video file to obtain several subframe image sequences and frame target information;

[0139] The setting module 20 is used to set a preset small target area and use the preset small target area as the ROI area of ​​the subframe image sequence;

[0140] The acquisition module 30 is used to acquire ROI region information corresponding to the ROI region based on the frame target information;

[0141] The ROI annotation module 40 is used to annotate the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

[0142] It is understandable that image recognition involves a wide range of areas, including image segmentation, image detection, image tracking, and image matching. This embodiment provides a video dense target detection data annotation device, which mainly involves annotation in the field of target detection and tracking, such as giving the rectangular box position of the target and its category, and continuously tracking the target in a video sequence.

[0143] For example, the preprocessing module 10 is specifically used to: preprocess the video file to obtain several frame images and frame target information corresponding to each frame image; and slice the frame images based on a preset length to obtain several sub-frame image sequences.

[0144] In one example, the process of preprocessing a video file to obtain several frames and the corresponding frame target information for each frame may include: determining whether a preset video target detection model exists; if the preset video target detection model exists, invoking the preset video target detection model; preprocessing the video file based on the preset video target detection model to obtain several frames and the target type, target ID, and target bounding box coordinates corresponding to each frame; and storing the target type, target ID, and target bounding box coordinates as the frame target information corresponding to the frame image.

[0145] Specifically, a pre-defined video object detection model can be used to preprocess the video file, exporting each frame as an image and its corresponding frame object information. The frame object information includes the object detection type and detection bounding box coordinates exported frame by frame, and is saved to a text file. The pre-defined video object detection model includes, but is not limited to, open-source or self-trained deep learning models.

[0146] For example, if a pre-trained object detection model (e.g., an open-source model) exists to identify the target type to be labeled, it can be used as a preset video object detection model. The preset video object detection model prediction service is deployed to predict video files frame by frame, exporting each frame as an image and its corresponding target information. The output of the target information for each frame includes the target type, target ID, and target bounding box coordinates (x, y, w, h). In one example, each video frame is exported as an image, and the output of each frame (frame target information) is written to a text file with the same name. The directory hierarchy is: dir / image path; dir / in / text file path. It is understood that if the preset video object detection model does not exist, the output can be empty.

[0147] In one example, the process of slicing the frame image based on a preset length to obtain a sequence of sub-frame images may include: setting a preset slice time length; and slicing the frame image based on the preset slice time length to divide it into multiple sub-frame image sequences.

[0148] For example, the exported frame image sequence is sliced ​​into multiple sub-frame image sequences according to a certain length unit, and the corresponding target type, coordinate data, etc., are exported to a sub-text file. The process of slicing the frame image into multiple sub-frame image sequences based on the preset slice time length may include: determining the number of slices based on the video duration of the video file and the preset slice time length; determining the number of frames in each slice based on the video frame rate of the video file and the video duration; generating a slice sequence starting from the first frame at intervals of the specified number of frames to obtain several sub-frame image sequences; and storing the sub-frame image sequences and the corresponding frame target information.

[0149] Specifically, such as Figure 3As shown, video slicing divides the preprocessed image sequence into slices according to a certain time unit. Each slice can serve as a basic unit for annotation tasks. The preset slice duration is set to sliceT. If the video duration is T, the number of slices is N = T / (sliceT), and the number of frames in each slice is frameNums ≤ (sliceT) * fps, where fps is the video frame rate. For each video result export directory dir, starting from the first frame, a slice sequence (subframe image sequence) is generated every frameNums frames. The frameNums images are copied to a new subdirectory, and the corresponding text files of the frame output results are also copied to the same subdirectory. The directory hierarchy is: newDir / image path, newDir / in / text file path. Each new subdirectory newDir contains frameNums images and the same number of text files with the same name. The number of images and text files in the last slice (subframe image sequence) may be less than the number of frames frameNums. The number of target categories K in the entire video export result dir is counted, and the maximum value of the target ID in each category is maxID(i), where i = 0 to K-1.

[0150] In one example, setting a preset small target region and using the preset small target region as the ROI region of the subframe image sequence includes: determining a first frame image in the subframe image sequence; obtaining frame target information corresponding to the first frame image, and determining the target rectangle coordinates based on the frame target information; setting at least one preset small target region based on the target rectangle coordinates, and using the preset small target region as the ROI region of the subframe image sequence.

[0151] In one example, the step of annotating the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results includes: generating an ROI configuration file based on the ROI regions of the subframe image sequence and storing the ROI region information in an ROI subfolder; calling the ROI subfolder and determining whether an ROI configuration file exists; when the ROI configuration file is detected, setting the current annotation mode to ROI mode; and in ROI mode, annotating the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

[0152] In one example, the step of generating an ROI configuration file based on the ROI region of the subframe image sequence and storing the ROI region information in an ROI subfolder includes: obtaining the coordinate information and rectangle shape information corresponding to the ROI region of the subframe image sequence; generating an ROI configuration file based on the coordinate information and rectangle shape information; determining the target type and rectangle coordinate information within the ROI region based on the ROI region information, and storing the target type and rectangle coordinate information within the ROI region in an ROI subfolder.

[0153] In one example, the step of annotating the ROI region and ROI region information of the subframe image sequence in the ROI mode to obtain ROI annotation results includes: in the ROI mode, enlarging the ROI region in the screen display area based on the ROI configuration file; and annotating the enlarged ROI region and ROI region information to obtain ROI annotation results.

[0154] In one example, the step of labeling based on the magnified ROI region and ROI region information to obtain ROI labeling results includes: labeling based on the magnified ROI region and ROI region information to obtain magnified image labeling results; obtaining the ROI region display coordinates of a specific target based on the magnified image labeling results; performing coordinate restoration calculation on the ROI region display coordinates to obtain the restored coordinate information of the specific target in the screen display area; and updating the coordinate information in the magnified image labeling results based on the restored coordinate information to obtain ROI labeling results.

[0155] In one example, areas that are difficult for the human eye to distinguish and label, such as small or dense targets, are designated as Regions of Interest (ROIs). Specifically, the process of labeling the ROIs and frame target information of the subframe image sequence to obtain ROI labeling results is exemplified as follows: A preset small target area is set, and the preset small target area is used as the ROI of the subframe image sequence; ROI information corresponding to the ROI is obtained based on the frame target information; and ROIs and ROI information of the subframe image sequence are labeled to obtain ROI labeling results.

[0156] Specifically, when a Region of Interest (ROI) is selected, it is designated as ROI annotation mode. In ROI annotation mode, the segmentation results are annotated. ROI regions are primarily used for annotating small or very dense targets. It should be noted that in ROI annotation mode, the ROI region needs to be enlarged before annotation. The specific ROI region division can be set according to actual needs, based on what is difficult for the human eye to observe and annotate; this embodiment does not impose any limitations on this.

[0157] For example, for a particularly small target area, a suitable ROI region, such as a rectangular ROI region, is selected on the image. The coordinate information, length, and width of this ROI region are exported to the ROI configuration file. Subfiles containing the target type within the ROI region and the processed rectangular bounding box coordinates are exported to the ROI subfolder. Multiple ROI regions can be set. The annotation tool imports the ROI subfolder and determines whether to use ROI annotation mode or normal annotation mode (e.g., main area annotation mode) depending on whether an ROI configuration file exists. For ROI annotation mode, the image is enlarged to a suitable size, and data annotation is performed after the enlargement. The annotation data is then transformed according to the appropriate coordinates and exported to the annotation file.

[0158] In one example, ROI annotation includes: defining a small / dense target area of ​​a certain size as the ROI region, exporting the ROI region information, reloading the exported information, enlarging the image, repeating the target annotation process, and setting the saved annotation information as the ROI annotation result. Specifically, areas that are difficult for the human eye to distinguish and annotate, such as small / dense targets, are designated as ROI regions.

[0159] Specifically, a rectangle is selected from the first frame of the segmented subframe image sequence. This rectangle is sufficient to enclose the small target. Depending on the number and distribution of the target, multiple ROI regions can be selected. These multiple ROI regions are denoted as Region(i)(x1,y1,x2,y2), where i = 0 to n-1, and n is the number of selected ROI regions.

[0160] Specifically, export ROI information for the j-th ROI region. For example, export the coordinates of Region(j): x1, y1, x2, y2 to the image folder / ROI subdirectory, denoted as roi.config (ROI configuration file). Based on Region(j)(x1, y1, x2, y2) and the imported coordinate information from video preprocessing, if a recorded target Object(Cls, id, x, y, w, h) is completely within the Region(j) region: x >= x1, y >= y1, x+w-1 <= x2, y+h-1 <= y2, then continue exporting the Object information to the image folder / ROI(j) / in subfolder.

[0161] Specifically, ROI region loading: When all information of the j-th ROI is exported according to the above process of exporting ROI information of the j-th ROI region, the annotation tool reloads the image folder / ROI(j) subdirectory. When roi.config (ROI configuration file) is detected, the current annotation mode is recorded as ROI mode, which is different from the normal annotation mode when no ROI region is defined.

[0162] Specifically, ROI region magnification and display: After reading roi.config (ROI configuration file) (x1,y1,x2,y2), the corresponding ROI region of the original image is magnified to the image display area. The magnification process includes: based on the ROI region coordinates of the original image: x1, y1, x2, y2, w, h, the original image information: W, H, the screen display area: W, H, and the type and coordinate data of a certain exported target in the / ROI / in subfolder, Object(Cls,ID,xx1,yy1,xx2,yy2), the coordinates of the Object displayed on the screen after the ROI region is loaded can be obtained as follows: X1=x1*W / w, Y1=y1*H / h, X2=x2*W / w, Y2=y2*H / h.

[0163] Specifically, ROI annotation is performed after the ROI area is zoomed in and displayed. It should be noted that the ROI annotation process is similar to the main area annotation process, with the difference being the restoration and saving of the ROI annotation results and the calculation of the size of the target matching area and the search area during target matching.

[0164] For example, for a target annotation result: screen coordinates Q(X1,Y1,X2,Y2), when saving the annotation result, what is actually saved is the target's annotation coordinates q restored from screen coordinates Q to the original image. Therefore, it is necessary to perform a restoration calculation on screen coordinates Q. It can be understood that the specific restoration calculation of screen coordinates Q includes: the corresponding coordinates q in the original image are: src_x1=X1 / W*w+x1, src_y1=Y1 / H*h+y1, Src_x2=X2 / W*w+x1, Src+y2=Y2 / H*h+y2. For the ROI region image matching process, the image to be matched is: the screen-annotated rectangle Q(X1,Y1,X2,Y2); the image to be searched is: the image T of the ROI region in the original image magnified to the screen area (W,H); after matching the result Q1, Q1 needs to be restored, referring to the coordinate restoration process from src_x1 to src_y2 described above. J = 0 to n-1. For each ROI region, repeat the above ROI labeling process, which will not be elaborated here.

[0165] This embodiment uses pedestrian data annotation in a smart city as a specific scenario for illustration. The minimum size of the pedestrian target to be annotated can be 32*64 pixels, and the video duration of the video file to be annotated is half an hour (1800 seconds, 25 frames per second). Pedestrians include ordinary pedestrians (category 0) and cyclists (bicycles, tricycles, motor vehicles, etc., category 1). The annotation results should provide the category C1s, pedestrian ID, and the rectangular coordinates (x1, y1, x2, y2) of all pedestrians. For cyclists, the rectangle should completely enclose both the person and the vehicle.

[0166] Specific scenario: Pedestrian targets at normal density (e.g., 5 to 15 people per frame), including small targets and cyclists. In this scenario, small pedestrian targets are difficult to observe even with the human eye due to their tiny pixel values, resulting in significant errors when drawing bounding boxes. General deep learning models also struggle to detect them, reducing annotation efficiency. Therefore, this embodiment introduces Region of Interest (ROI) annotation. PPHuman's detection of pedestrians cannot include bicycles; therefore, the imported pre-annotated data for pedestrian-cyclist types needs to be discarded and manually annotated again.

[0167] The specific process is as follows: Based on Baidu's open-source PPHuman, a pedestrian detection service is modified and deployed. The input is a video file sample.mp4, the export directory is Dir, and the output is video frame images and the pedestrian target data (id, x1, y1, x2, y2) detected in each frame. The total number of video frames is 1800 * 25 = 45000. The output format is: Dir / 1.jpg, Dir / 2.jpg, Dir / 3.jpg...Dir / 45000.jpg and Dir / in / 1.txt, Dir / in / 2.txt...Dir / in / 45000.txt. The txt format is: Cls, ID, (x1, x2, y1, y2), where x1, x2, y1, y2 are multiple lines. The type value is uniformly set to 0, as PPHuman does not have the ability to finely classify types; this will be modified later using an annotation tool. The maximum value of ID in all txt files is calculated, assumed to be 360. The exported image and text results are segmented into 10-second segments. The total number of segments is 1800 / 10 = 180, and each segment contains 10 * 25 = 250 frames. For example... Figure 3 As shown, the initial value of the new ID for each shard is used to generate a new ID. For example, the initial value of shard 1 is newID = 500, the initial value of shard 2 is newId = 1000, and so on, increasing by 500 every time, to ensure that the newly generated IDs for each shard are not duplicated.

[0168] For example, the process of labeling a person riding a bicycle is as follows: The labeling tool imports the first segment (frames 1-250 and 1-250 txt files), and draws and displays rectangles on the corresponding coordinates of the screen display area based on the pre-stored labeling information in the txt files. Due to the pre-labeling import, most targets do not need to be labeled again; only a small number of targets without rectangles need to be found. For targets without rectangles, there are two cases: one is that the target was not detected at all, in which case a new ID needs to be generated, ID = newID, and newID = newID + 1 (incrementing by 1); the other is that the target was detected in some frames, in which case a new ID does not need to be generated, and the already detected IDs can be reused.

[0169] Specifically, ROI region labeling is performed: Assuming the small target is concentrated in two regions of the image (the division criteria are based on achieving a magnification effect, with an optimal near-square effect), such as... Figure 4 and Figure 5 As shown, the detailed process is as follows: Select the ROI rectangle within the main annotation image area (original image size); export the ROI region image, ROI configuration file, and pre-annotated data within the ROI region to the dir / roi subfolder, in the following format: Dir / roi1 / roi.json

[0170] {

[0171] "xmin": 20,

[0172] “ymin”: 30

[0173] "width": 400,

[0174] "height": 300

[0175] }

[0176] Dir / roi1 / 1.jpg (1.jpg is a small image stored in the roi1 region, and the same applies to subsequent images) ...

[0178] Dir / roi1 / 250.jpg

[0179] Dir / roi1 / in / 1.txt (1.txt is the complete pre-annotated object within the roi1 region, and the same applies to subsequent files) ...

[0181] Dir / roi1 / in / 250.txt.

[0182] Specifically, import ROI region data: The annotation tool imports the roi1 folder and checks if the roi.json configuration file exists to determine if it's currently in ROI mode. If it is in ROI mode, a zoom-in operation is required, and the screen display area size needs to be set to the original image size. For example... Figure 4 As shown, the specific ROI annotation process can be referenced from the main area annotation mode above. The difference is that the coordinates after the screen is enlarged need to be calculated when loading, the enlarged image is also used when matching, and the enlarged screen data needs to be restored to the small image coordinates when saving.

[0183] This embodiment proposes a video dense target detection data annotation device, comprising: a preprocessing module 10 for preprocessing video files to obtain a sequence of sub-frame images and frame target information; a setting module 20 for setting a preset small target region and using the preset small target region as the ROI region of the sub-frame image sequence; an acquisition module 30 for acquiring the ROI region information corresponding to the ROI region based on the frame target information; and an ROI annotation module 40 for annotating the ROI region and ROI region information of the sub-frame image sequence to obtain ROI annotation results. In this embodiment, the ROI region is mainly used for annotating small targets or very dense targets. By selecting and enlarging the ROI region, automatic tracking of small targets, dense targets, and special target types is supported to accelerate the annotation speed and achieve accurate and efficient annotation on small target images.

[0184] In addition, for any technical details not described in detail in the embodiments of the video dense target detection data annotation device, please refer to any embodiment of the video dense target detection data annotation method of the present invention, which will not be repeated here.

[0185] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any limitations on this. It should be noted that the workflow described above is merely illustrative and does not constitute a limitation on the scope of protection of the present invention. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of the solution in this embodiment according to actual needs, and no limitations are imposed here.

[0186] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0187] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0189] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for annotating dense target detection data in video, characterized in that, include: The video file is preprocessed to obtain a sequence of subframe images and frame target information; A preset small target area is defined, and the preset small target area is used as the ROI region of the subframe image sequence; Based on the frame target information, obtain the ROI region information corresponding to the ROI region; The ROI regions and ROI region information of the subframe image sequence are labeled to obtain ROI labeling results; The step of annotating the ROI regions and ROI region information based on the subframe image sequence to obtain ROI annotation results includes: ROI configuration files are generated based on the ROI regions of the subframe image sequence, and the ROI region information is stored in the ROI subfolder; Call the ROI subfolder and determine if an ROI configuration file exists; When the ROI configuration file is detected, the current annotation mode is set to ROI mode; In the ROI mode, ROI regions and ROI region information of the subframe image sequence are labeled to obtain ROI labeling results; In the ROI mode, the ROI region and ROI region information of the subframe image sequence are labeled to obtain ROI labeling results, including: In the ROI mode, the ROI area is enlarged and displayed on the screen display area based on the ROI configuration file; The ROI region and its information, after being enlarged, are labeled to obtain the ROI labeling results. The method further includes: using a preset unsupervised tracking algorithm to annotate the subframe image sequence and frame target information to obtain the main region annotation result; and using a preset merging algorithm to merge the main region annotation result and the ROI annotation result to obtain the target annotation result. The main region annotation result is obtained by annotating the subframe image sequence and frame target information based on a preset unsupervised tracking algorithm. This includes: annotating the target type and target bounding box in the screen display area according to the subframe image sequence and frame target information to obtain a first annotation result; determining whether a target bounding box exists frame by frame in the subframe image sequence; when the target bounding box does not exist in the subframe image sequence, annotating the subframe image sequence based on the preset unsupervised tracking algorithm to obtain a second annotation result; and using the first annotation result and the second annotation result as the main region annotation result. For frames lacking imported information, annotations are added to obtain a second annotation result. The specific annotation process is as follows: Starting from the frame with missing annotation information, denoted as frame i, the target rectangle of the frame is manually drawn, and the annotation area of ​​the target rectangle is denoted as R. The automatic tracking distance D is set, and the search is performed in frame i+1 based on the preset unsupervised tracking algorithm with R as the target object. When the target area is matched in frame i+1, the result is denoted as R1, and R1 is set as the target object. The above process is repeated until frame i+D. The specific fine-tuning process includes: manually fine-tuning the results generated by automatic matching between i+1 and i+D. The fine-tuned result is denoted as M(x1, y1, x2, y2), where i=1 to D, and M is the final annotation rectangle area result of the target. The target category and target ID are the same as the target category and target of frame i.

2. The video dense target detection data annotation method as described in claim 1, characterized in that, The step of setting a preset small target region and using the preset small target region as the ROI region of the subframe image sequence includes: Determine the first frame image in the subframe image sequence; Obtain the frame target information corresponding to the first frame image, and determine the coordinates of the target rectangle based on the frame target information; Based on the coordinates of the target rectangle, at least one preset small target region is set, and the preset small target region is used as the ROI region of the subframe image sequence.

3. The video dense target detection data annotation method as described in claim 1, characterized in that, The step of generating an ROI configuration file based on the ROI regions of the subframe image sequence and storing the ROI region information in the ROI subfolder includes: Obtain the coordinate information and rectangle shape information corresponding to the ROI region of the subframe image sequence; Generate an ROI configuration file based on the coordinate information and the rectangle shape information; Based on the ROI region information, determine the target type and rectangle coordinate information within the ROI region, and store the target type and rectangle coordinate information within the ROI region into the ROI subfolder.

4. The video dense target detection data annotation method as described in claim 1, characterized in that, The step of labeling based on the magnified ROI region and ROI region information to obtain ROI labeling results includes: Based on the magnified ROI region and ROI region information, annotations are performed to obtain the magnified annotation results of the image. The ROI region coordinates of the specific target are obtained based on the annotation results after the image is magnified; The coordinate restoration calculation is performed on the displayed coordinates of the ROI region to obtain the restored coordinate information of the specific target in the screen display area; The coordinate information in the magnified image annotation result is updated based on the restored coordinate information to obtain the ROI annotation result.

5. A video dense target detection data annotation device based on the method described in any one of claims 1-4, characterized in that, include: The preprocessing module is used to preprocess the video file to obtain a sequence of subframe images and frame target information; The setting module is used to set a preset small target area and use the preset small target area as the ROI area of ​​the subframe image sequence; The acquisition module is used to acquire ROI region information corresponding to the ROI region based on the frame target information; The ROI annotation module is used to annotate the ROI regions and ROI region information of the subframe image sequence to obtain ROI annotation results.

6. A video-based dense target detection data annotation device, characterized in that, The video dense target detection data annotation device includes: a memory, a processor, and a video dense target detection data annotation program stored in the memory and executable on the processor, wherein the video dense target detection data annotation program is configured to implement the video dense target detection data annotation method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a video dense target detection data annotation program, which is used to enable the processor to implement the video dense target detection data annotation method as described in any one of claims 1 to 4 when executed.

Images