Annotation examination method and system for re-identification model learning in multi-camera images

Abstract

The present invention relates to a dataset for training an object re-identification artificial intelligence model and an automatic annotation and modification tool therefor. To this end, the present invention provides an annotation verification method for training a re-identification model in multi-camera images, comprising the steps of: classifying and storing images received from an edge device in an image storage unit; recognizing objects by an object recognition model frame by frame in the tracklet generation unit, generating annotations including bounding boxes and object IDs for said objects, generating tracklets by sorting bounding boxes for the same object in chronological order, and assigning unique tracklet IDs; generating trajectories, which are groups sorted chronologically by camera in an trajectory generation unit using the generated tracklets, and assigning unique trajectory IDs; and displaying tracklets to be verified in a tracklet verification unit, receiving selections of bounding boxes included in each tracklet, and separating the selected bounding boxes from the corresponding tracklets. Accordingly, according to the present invention, the quality and accuracy of annotations for training a multi-camera image-based object re-identification model can be improved and the inspection time can be shortened.

Description

Technology Field

[0001] The present invention relates to an annotation verification method and system for training a re-identification model in multi-camera images, and specifically, to a dataset for training an object re-identification artificial intelligence model and an automatic annotation and correction tool therefor. Background Technology

[0002] In a surveillance system, re-identification technology is required to match a specific object observed by a specific camera with the corresponding object observed at a different viewpoint or by a different camera. The surveillance system of the present invention consists of an intelligent CCTV that generates tracklets (a continuous short path of an object) by performing object recognition and re-identification within a single image, and a control system that generates trajectories (a group of multiple tracklets arranged chronologically by camera) by performing re-identification of tracklets within multiple images. To train this object re-identification model, a video annotation tool capable of generating annotations for video data is required.

[0003] Generally, video data consists of tens of frames per second. Manually annotating each image requires a significant amount of labor. Therefore, recent technology is evolving toward providing automated annotation capabilities based on artificial intelligence models. However, since models may not be sufficiently trained in advance, the accuracy of automatically generated annotations may be lower; consequently, human intervention is essential, and this process must also be efficient in terms of time and labor.

[0004] According to published patent No. 10-2020-0123584, prior art describes a device for predicting the error probability of an annotation for input data, wherein an annotation is generated using an algorithm for a first annotation for training input data and a second annotation for evaluation input data, and an annotation evaluation model is trained based on the first annotation and a correction history for the first annotation received from a user to predict the error probability of the second annotation.

[0005] However, conventional technology has the problem that the inspection time takes too long because a model is trained to predict the possibility of annotation errors, and the trained model predicts errors for all images and must be inspected by a human. The problem to be solved

[0006] The present invention aims to solve the aforementioned problems, specifically to provide an efficient annotation verification procedure for training an object re-identification model in multi-camera images and to provide a user interface related thereto. means of solving the problem

[0007] To achieve the above objective, the present invention provides an annotation verification method for training a re-identification model in multi-camera images, comprising the steps of: classifying and storing images received from an edge device in an image storage unit; recognizing objects by an object recognition model for each frame of the images in a tracklet generation unit, generating an annotation including a bounding box and an object ID for said object, generating tracklets by sorting bounding boxes for the same object in chronological order and assigning a unique tracklet ID; generating trajectories, which are groups sorted in chronological order by camera in an trajectory generation unit using the generated tracklets and assigning a unique trajectory ID; and displaying tracklets to be verified in a tracklet verification unit, receiving selections of bounding boxes included in each tracklet, and separating the selected bounding boxes from the corresponding tracklets.

[0008] It is desirable for the above tracklet verification unit to calculate the tracklet reliability by referring to the annotations included in the tracklet, and then mark only the tracklets with a tracklet reliability lower than the set value as targets for inspection.

[0009] It is preferable that the above tracklet verification unit includes a tracklet list unit that displays a stored tracklet ID and a tracklet display unit that displays a tracklet corresponding to the tracklet ID.

[0010] In the above tracklet verification unit, it is desirable to display at least one selected object in the same color if it is entered as the same object.

[0011] According to the present invention, an annotation verification system for learning a re-identification model in multi-camera images is provided, comprising: an image storage unit that classifies and stores images received from an edge device; a tracklet generation unit that recognizes objects by an object recognition model frame by frame on the images, generates an annotation including bounding boxes and object IDs for said objects, generates tracklets by sorting bounding boxes for the same object in chronological order, and assigns unique tracklet IDs; a trajectory generation unit that generates trajectories, which are groups sorted in chronological order by camera, using the generated tracklets, and assigns unique trajectory IDs; and a tracklet verification unit that displays tracklets to be verified, receives selections of bounding boxes included in each tracklet, and separates the selected bounding boxes from the corresponding tracklets. Effects of the invention

[0012] According to the present invention, the quality and accuracy of annotations for training a multi-camera image-based object re-identification model can be improved and the inspection time can be shortened. Brief explanation of the drawing

[0013] FIG. 1 is a configuration diagram showing an annotation verification system according to the present invention; FIG. 2 is a configuration diagram showing the configuration of the annotation generation unit in FIG. 1; FIG. 3 is a configuration diagram showing the screen configuration displayed by the tracklet verification unit; FIG. 4 is an example diagram showing the object selection process in FIG. 3; FIGS. 5 and 6 are example diagrams illustrating the recropping process for an object; FIG. 7 is a configuration diagram showing the screen configuration displayed by the orbit verification unit; FIG. 8 is an example diagram showing the tracklet selection process in FIG. 7; FIG. 9 is an example diagram showing a screen configuration displaying connection candidate tracklets in the track verification unit. Specific details for implementing the invention

[0014] The configuration and operation of a specific embodiment according to the present invention will be described in detail with reference to the drawings.

[0015] Referring to FIG. 1, the annotation verification system (30) according to the present invention may be configured in the form of a server that transmits and receives data with a plurality of edge devices.

[0016] The edge device (10) above is a device capable of collecting and immediately analyzing data, and in this embodiment, it can be configured using a plurality of artificial intelligence cameras.

[0017] The above annotation verification system (30) is configured to include an image storage unit (100), a tracklet generation unit (200), an orbit generation unit (300), a tracklet verification unit (500), and an orbit verification unit (600), and receives various data including image information from a plurality of edge devices.

[0018] The above image loading unit (100) adds image information including a camera identification code, date, time, resolution, shooting location, etc. to the captured image and stores it. The storage can be performed at regular intervals.

[0019] Referring to FIG. 2, the tracklet generation unit (200) is configured to include a spatial information unit (210), an object analysis unit (230), an object information unit (250), and an annotation information unit (270).

[0020] The spatial information unit (210) stores the camera identification code of a specific artificial intelligence camera, the installation location of the artificial intelligence camera, and the location coordinates of the space captured by the artificial intelligence camera. Additionally, the spatial information unit (210) stores the spatial range captured by each artificial intelligence camera, and the location coordinates of an object appearing on the artificial intelligence camera screen may be stored.

[0021] The object analysis unit (230) recognizes a specific object designated per frame, detects the location coordinates of the object on the frame, and creates a bounding box surrounding the object.

[0022] The object may be a moving object such as a car or a person, and in this embodiment, it may refer to a person. Additionally, the object may be recognized using deep learning technology such as CNN or YOLO. To this end, the object analysis unit (230) may be equipped with an object recognition model that has been trained in artificial intelligence.

[0023] The object information section (250) stores object information used for recognizing and identifying an object, such as the object ID of the recognized object, the shooting time, the camera identification number, the feature points of the object, and the recognition accuracy (probability), in chronological order. The feature points may include information such as clothing color, hair color, whether a hat is worn, whether glasses are worn, size, and shape. At this time, if it is determined to be the same object by comparison with an adjacent frame, the same object ID is stored. At this time, the probability of it being the same object may be stored as the recognition accuracy.

[0024] The annotation information unit (270) stores relevant information, including an object ID for the recognized object. That is, the annotation information unit (270) may store annotations such as the camera identification code that captured the image, the time of capture, the frame number, the object ID, the location coordinates of the object, the location and size of the bounding box, and the recognition accuracy.

[0025] The above tracklet generation unit (200) generates tracklets by arranging bounding boxes generated for the same object in chronological order, and assigns a unique tracklet ID to each generated tracklet.

[0026] In other words, the above tracklet refers to a group of bounding boxes with the same object ID among consecutive frames, sorted in chronological order based on the generated object ID.

[0027] When multiple objects are recognized in a single frame, multiple tracklets can be created, with one tracklet created for each object.

[0028] The above trajectory generation unit (300) generates a trajectory, which is a group of tracklets with the same object ID arranged in chronological order by AI camera.

[0029] The above tracklet verification unit (500) includes a tracklet list unit that displays a stored tracklet ID and a tracklet display unit that displays a tracklet corresponding to the tracklet ID, and displays a screen that can verify the tracklet.

[0030] Specifically, referring to FIG. 3, the tracklet verification unit includes a tracklet list unit (510) that displays a stored tracklet ID, a tracklet display unit (530) that displays a tracklet corresponding to a selected tracklet ID, a verification progress display unit (550), and a verification status display unit (570).

[0031] In the above tracklet list section (510), data including the stored tracklet ID, verification progress status, and tracklet reliability is displayed.

[0032] The tracklet display unit (530) displays the tracklet corresponding to the selected tracklet ID. That is, the bounding boxes included in the tracklet are displayed in chronological order, and the shooting date and object reliability are displayed at the bottom.

[0033] The above object reliability represents the probability that the object belongs to the corresponding tracklet, and can be calculated by calculating the average of the similarity with each object belonging to the corresponding tracklet.

[0034] Referring to FIG. 4, the user can select one or more objects belonging to a specific tracklet, and the background of the selected objects is displayed in a specific color, such as blue or red. Additionally, the tracklet verification unit can separate the objects selected by the user's input from the existing tracklet and create a new tracklet. At this time, a new tracklet ID is assigned to the new tracklet.

[0035] Referring to FIGS. 5 and 6, the tracklet verification unit allows the bounding box of a selected object to be changed according to user input. That is, when a recropping signal is received by the user, the tracklet verification unit retrieves and displays the original image corresponding to the bounding box from the storage unit. At this time, the existing bounding box is displayed as a dotted line as in FIG. 6, allowing the user to change the range of the bounding box. Meanwhile, when a deletion signal is received, the tracklet verification unit deletes the selected object.

[0036] When the verification of the corresponding tracklet is completed by the verification complete button, a text or shape indicating that the verification is complete is displayed in the tracklet list section (510) and the verification status display section (570) of the tracklet.

[0037] The verification progress display unit (550) displays the ratio of verified tracks to the total number of tracks when verification is being performed on the selected track.

[0038] Meanwhile, based on user input, the tracklet verification unit can calculate the tracklet reliability for tracklets that have not been inspected, detect tracklets with a tracklet reliability lower than a set value, and display only the detected tracklets on the inspection target screen.

[0039] The above tracklet reliability can be calculated as the average of the object reliabilitys corresponding to each bounding box for multiple bounding boxes included in a single tracklet.

[0040] At this point, tracklets exceeding the set value are changed to the model validation state and then proceed to the tracklet connection step. In the above tracklet connection step, they must be identical tracklets; however, if the tracklet IDs are separated, they can be integrated into a single tracklet ID.

[0041] The above-mentioned track verification unit (600) includes a track list unit that displays a stored track ID and a track display unit that displays a track corresponding to the track ID, and displays a screen that can verify the track.

[0042] Specifically, referring to FIGS. 7 and 8, the orbit verification unit includes an orbit list unit (610) that displays a stored orbit ID, an orbit display unit (630) that displays an orbit corresponding to a selected orbit ID, a verification progress display unit (650), and a verification status display unit (670).

[0043] In the above orbit list section (610), data including the stored orbit ID, verification progress status, and orbit reliability is displayed.

[0044] The above-mentioned orbit display unit (630) displays the orbit corresponding to the selected orbit ID. That is, the bounding boxes included in the orbit are displayed in chronological order, and the shooting date and object reliability are displayed in the lower part.

[0045] The user can select one or more tracklets belonging to a specific orbit, and the background of the selected tracklets is displayed in a specific color, such as blue or red. Additionally, the orbit verification unit can separate the tracklets selected by user input from the existing orbit to create a new orbit. In this case, a new orbit ID is assigned to the new orbit. At this time, non-contiguous tracklets may be combined to form a new orbit.

[0046] When the verification of the corresponding track is completed by the intermediate verification completion button, a phrase (semi-verified) or a shape indicating that the intermediate verification is completed is displayed in the track list section (610) and the track verification status display section (670). The track for which intermediate verification is completed moves on to the track connection stage.

[0047] The verification progress display unit (650) displays the ratio of orbits that have completed intermediate verification for all orbits when verification is being performed on the selected orbit.

[0048] Meanwhile, based on user input, the track verification unit can calculate the track reliability for tracks that have not been inspected, detect tracks with a track reliability lower than a set value, and display only the detected tracks on the inspection target screen.

[0049] The above track reliability can be calculated as the average of the object reliabilitys corresponding to each bounding box for multiple bounding boxes included in a single track.

[0050] At this point, orbits exceeding the set value are changed to the model verification state and then proceed to the orbit connection stage. In the aforementioned orbit connection stage, they must be identical orbits; however, if the orbit IDs are separated, they can be integrated into a single orbit ID.

[0051] Specifically, referring to FIG. 9, when a specific track ID is selected in the track list section, the track verification section displays the track corresponding to the track ID in the track display section, and displays a connection candidate tracklet (680) below the track.

[0052] When a specific orbit ID is selected, the above-mentioned orbit verification unit detects tracklets that are highly likely to be connected to that orbit and marks them as connection candidate tracklets.

[0053] The above-mentioned connection candidate tracklets are detected within a time range determined to be human speed in different areas where the corresponding orbit and observation area do not overlap, taking into account the camera identification code, time of capture, and location where the image was captured. Additionally, in areas where the observation areas overlap, they are detected as tracklets captured at the same time.

[0054] Tracklets detected at the same time in areas where the observation areas do not overlap are excluded, and in areas where the observation areas overlap, the tracklets must be those where the error in the spatial position coordinates of the object is less than the allowable limit.

[0055] When a connection candidate tracklet is selected and an insertion signal is transmitted, the orbit verification unit includes the connection candidate tracklet in the corresponding orbit.

[0056] Although the present invention has been described above with reference to embodiments thereof, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0057] 30 : Annotation verification system 100 : Image storage unit 200 : Tracklet Generation Unit 300 : Orbit Generation Unit 500 : Tracklet Verification Section 510 : Tracklet List Section 530 : Tracklet display unit 600 : Track verification unit 610: Track List Section 630: Track Display Section 680 : Connection Candidate Tracklet

Claims

Claim 1 A step of classifying and storing images received from edge devices in an image storage unit; a step of recognizing objects frame by frame using an object recognition model in a tracklet generation unit, generating an annotation including bounding boxes and object IDs for said objects, generating tracklets by sorting bounding boxes for the same object in chronological order, and assigning unique tracklet IDs; a step of generating trajectories, which are groups sorted chronologically by camera, using the generated tracklets in a trajectory generation unit, and assigning unique trajectory IDs; a step of calculating tracklet reliability, which indicates the accuracy of object recognition for said generated tracklets, in a tracklet verification unit, and selecting and marking tracklets with a tracklet reliability lower than a preset threshold value as targets for inspection. An annotation verification method for learning a re-identification model in multi-camera images, characterized by performing a step of verifying the connection status of a track generated by the track generation unit in the track verification unit, wherein the track verification unit provides a track display unit in which a plurality of bounding boxes included in a tracklet are displayed in chronological order, receives a selection of bounding boxes included in each tracklet from a user, separates the selected bounding box from the corresponding tracklet, assigns a new identification number to the separated bounding box to regenerate it as an independent tracklet, or processes it for deletion by a deletion signal, and wherein when a specific track ID is selected, the track verification unit detects and displays a connection candidate tracklet that is highly likely to be connected to the corresponding tracklet by considering the camera identification code, shooting time, and location where the image was captured, and integrates the connection candidate tracklet into a single track ID by an insertion signal, or separates the tracklet selected by user input from the existing tracklet to create a new track with a new track ID assigned. Claim 2 delete Claim 3 Annotation verification method for learning a re-identification model in multi-camera images, characterized in that, in the first aspect, the tracklet verification unit includes a tracklet list unit that displays a stored tracklet ID and a tracklet display unit that displays a tracklet corresponding to the tracklet ID. Claim 4 Annotation verification method for training a re-identification model in multi-camera images, characterized in that, in paragraph 3, the tracklet verification unit displays at least one selected object in the same color when it is input as the same object. Claim 5 delete Claim 6 delete Claim 7 Annotation verification method for learning a re-identification model in multi-camera images, characterized in that, in the first aspect, the orbit verification unit includes an orbit list unit that displays a stored orbit ID and an orbit display unit that displays an orbit corresponding to the orbit ID. Claim 8 An annotation verification method for learning a re-identification model in multi-camera images, characterized in that, in claim 7, the orbit verification unit displays at least one selected object in the same color when it is input as the same object. Claim 9 An annotation verification system for training a re-identification model in multi-camera images comprises: an image storage unit that classifies and stores images received from an edge device; a tracklet generation unit that recognizes objects by an object recognition model frame by frame on the images, generates an annotation including bounding boxes and object IDs for said objects, generates tracklets by sorting bounding boxes for the same object in chronological order, and assigns unique tracklet IDs; an orbit generation unit that generates orbits, which are groups sorted in chronological order by camera, using the tracklets generated by the tracklet generation unit, and assigns unique orbit IDs; and a tracklet verification unit that performs verification of the tracklets generated by the tracklet generation unit. The annotation verification system for learning a re-identification model in multi-camera images includes a track verification unit that verifies the connection status of a track generated by the track generation unit, wherein the track verification unit provides a track display unit in which a plurality of bounding boxes included in a tracklet are displayed in chronological order, receives a selection of bounding boxes included in each tracklet from a user, separates the selected bounding box from the corresponding tracklet, assigns a new identification number to the separated bounding box to regenerate it as an independent tracklet, or processes it for deletion by a deletion signal, and wherein when a specific track ID is selected, the track verification unit detects and displays a connection candidate tracklet that is highly likely to be connected to the corresponding tracklet by considering the camera identification code, shooting time, and location where the image was captured, and integrates the connection candidate tracklet into the corresponding tracklet into a single track ID by an insertion signal, or separates the tracklet selected by user input from the existing tracklet to create a new tracklet assigned a new track ID. Claim 10 delete

Images