Data labeling, object tracking method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a data annotation and object tracking method, device, equipment and storage medium to improve processing efficiency. The method includes: providing an annotation interface; displaying images of more than one data source in the annotation interface, wherein images of different data sources are associated based on time information; receiving an annotation operation, determining an object on the image corresponding to the annotation operation; annotating the object on the corresponding image, and displaying the annotation information of the object in the annotation interface. Annotation through a visual annotation interface is more intuitive, and after selecting an object in a frame image, the object can be automatically annotated in the frame image and its associated frame images, which is more efficient.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data labeling method and device, an object tracking method and device, an electronic device, and a storage medium. Background Art

[0002] Data annotation can be applied in various scenarios, such as annotating images to monitor the labeled target objects, annotating text for analysis and processing, etc.

[0003] At present, data annotation usually involves labeling each data separately, such as labeling each image, each paragraph of text, etc. However, sometimes the labeled objects in different data may be related to each other. For example, the same labeled object is labeled separately in 5,000 images in order to track the labeled object. If the labeled object is modified in any of the 5,000 images, the other 4,999 images also need to be modified. In this case, the labeled data of each image needs to be modified separately, which is inefficient and will have a greater impact on the system. Summary of the invention

[0004] The embodiment of the present application provides a data labeling method to improve processing efficiency.

[0005] Correspondingly, an embodiment of the present application also provides a data labeling device, an object tracking method and device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above problems, an embodiment of the present application discloses a data annotation method, which includes: providing an annotation interface; displaying images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information; receiving an annotation operation, and determining an object on the image corresponding to the annotation operation; annotating the object on the corresponding image, and displaying the annotation information of the object in the annotation interface.

[0007] An embodiment of the present application also discloses a data annotation method, which includes: receiving images uploaded by different data sources; associating the images from the different data sources based on time information, and providing the associated images to an annotation interface so as to annotate the images through the annotation interface; and obtaining and storing annotation information of objects in the images.

[0008] An embodiment of the present application also discloses an object tracking method, which includes: providing a labeling interface; displaying a surveillance image collected by at least one camera and a point cloud map collected by a radar on the labeling interface, wherein the surveillance image and the point cloud map are associated with corresponding frames through time information; receiving a labeling operation in a specified frame point cloud map, and labeling a corresponding object on the point cloud map by setting a graphic; in at least one frame of surveillance image associated with the specified frame point cloud map, labeling the object by setting a graphic, and displaying the labeling information of the object.

[0009] An embodiment of the present application also discloses an object tracking method, which includes: receiving a surveillance image captured by at least one camera, and receiving a point cloud map captured by a radar; associating corresponding frames of the surveillance image and the point cloud map through time information, and storing the associated surveillance image and point cloud map to provide them to a labeling interface for labeling; obtaining and storing labeling information of objects in the point cloud map and the surveillance image.

[0010] The embodiment of the present application also discloses a data annotation device, which includes: an interface providing module, used to provide an annotation interface; an image display module, used to display images of more than one data source on the annotation interface, wherein images of different data sources are associated based on time information; an annotation module, used to receive an annotation operation, and determine an object on the image corresponding to the annotation operation; and an annotation processing module, used to annotate the object on the corresponding image, and display the annotation information of the object in the annotation interface.

[0011] An embodiment of the present application also discloses a data annotation device, which includes: a receiving module for receiving images uploaded by different data sources; a preprocessing module for associating the images from the different data sources based on time information, and providing the associated images to an annotation interface so as to annotate the images through the annotation interface; and a storage module for acquiring and storing annotation information of objects in the images.

[0012] An embodiment of the present application also discloses an object tracking device, which includes: an interface module for providing a labeling interface; a display module for displaying at least one surveillance image collected by a camera and a point cloud map collected by a radar on the labeling interface, wherein the surveillance image and the point cloud map are associated with corresponding frames through time information; a processing module for receiving a labeling operation in a specified frame point cloud map, and labeling a corresponding object on the point cloud map by setting a graphic; in at least one frame of surveillance image associated with the specified frame point cloud map, labeling the object by setting a graphic, and displaying the labeling information of the object.

[0013] An embodiment of the present application also discloses an object tracking device, which includes: a data receiving module, used to receive a surveillance image captured by at least one camera, and to receive a point cloud map captured by a radar; an association preprocessing module, used to associate the surveillance image and the point cloud map with corresponding frames through time information, and to store the associated surveillance image and point cloud map to provide them to a labeling interface for labeling; and an information storage module, used to obtain and store labeling information of objects in the point cloud map and the surveillance image.

[0014] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0015] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0016] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0017] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0018] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0019] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0020] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0021] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, a processor executes the method described in the embodiments of the present application.

[0022] Compared with the prior art, the embodiments of the present application have the following advantages:

[0023] In an embodiment of the present application, a labeling interface can be provided and images from more than one data source can be displayed in the labeling interface, wherein images from different data sources are associated based on time information, so that labeling can be performed directly on the labeling interface, a corresponding labeling operation is received, an object corresponding to the labeling operation is determined, and then the object can be labeled on the corresponding image, and the labeling information of the object can be displayed in the labeling interface. Labeling through a visual labeling interface is more intuitive, and after an object is selected in a frame image, the object can be automatically labeled in the frame image and its associated frame images, which is more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of processing object selection in an annotation interface example of an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of processing object annotation in the annotation interface example in the embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of processing mapping annotations in the annotation interface example in the embodiment of the present application;

[0027] Figure 4 It is a flowchart of the steps of an embodiment of a data annotation method of the present application;

[0028] Figure 5 is a schematic diagram of processing a rotated image in an annotation interface example in an embodiment of the present application;

[0029] Figure 6 This is a schematic diagram of the process of viewing the annotation object in the annotation interface example in the embodiment of the present application;

[0030] Figure 7 is a schematic diagram of processing for switching image frames in an annotation interface example in an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of an example of a data annotation scenario in an embodiment of the present application;

[0032] Fig. 9 is a flowchart of another data annotation method embodiment of the present application;

[0033] Fig.10 This is a flowchart of an example of determining grouping annotation information in an embodiment of the present application;

[0034] Fig.11 This is a flowchart of an example of adding group annotation information in an embodiment of the present application;

[0035] Fig.12 This is a flowchart of an example of modifying group annotation information in an embodiment of the present application;

[0036] Fig.13 This is a flowchart of an example of querying group annotation information in an embodiment of the present application;

[0037] Fig.14 is a flowchart of steps of another data labeling method embodiment of the present application;

[0038] Fig.15 is a flowchart of steps of an object tracking method embodiment of the present application;

[0039] Fig.16 is a flowchart of another object tracking method embodiment of the present application;

[0040] Fig.17 It is a structural block diagram of an embodiment of a data labeling device of the present application;

[0041] Fig.18 It is a structural block diagram of an optional embodiment of a data annotation device of the present application;

[0042] Fig.19 is a structural block diagram of another data labeling device embodiment of the present application;

[0043] Fig. 20 is a structural block diagram of an embodiment of an object tracking device of the present application;

[0044] Fig.21 is a structural block diagram of another object tracking device embodiment of the present application;

[0045] Fig. 22 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0047] In the embodiment of the present application, an entry can be understood as a unit of data, such as an entry can be a piece of text data, an image data, a piece of audio data, etc. Therefore, an entry may include at least one of the following: text data, image data, audio data. The entry can be annotated, and the annotation method can be set based on the needs, such as recording the position of the annotated object in the entry, adding labels to the annotated object, and other various methods, so that the entry can be analyzed, tracked, and other various processes based on the entry annotation information. Among them, many annotated entries are related to each other, and the entry annotation information such as the annotated objects are associated, for example, the same object is annotated in different entries, and the associated objects are followed in different entries, so that multiple associated entries can be analyzed, tracked, and other processes. In order to facilitate the analysis of the annotated entries, a grouping of entries can be established, and a grouping can be created for entries that meet certain rules. Therefore, for the annotation of entries, the annotation information of the corresponding entries in the group can be modified based on the grouping, thereby improving the processing efficiency.

[0048] Embodiments of the present application can be applied to various annotation scenarios, such as vehicle tracking, face recognition in videos, and the like. Taking image data as an entry, for example, it is applied to the 3D tracking annotation scenario of an unmanned vehicle. Among them, image data from multiple data sources can be annotated, such as radar point cloud images collected by radar, images collected by cameras, and the like. In one example, the radar of an unmanned vehicle can continuously track objects within a certain time range, and collect tracked radar data. Based on the radar data, the corresponding radar point cloud image can be determined, such as a 2D point cloud image or a 3D point cloud image. For images collected by cameras, it can include images collected by cameras of unmanned vehicles, surveillance cameras on roads, etc., or images collected by cameras such as driving recorders.

[0049] Among them, the images of each data source can be grouped according to certain rules, for example, images containing the same object are divided into one group, and images corresponding to data collected within a certain time are divided into one group, so that an image group can be obtained. In some examples, an image group may include images of at least one data source, and images from different data sources can be associated based on time, wherein each frame of the image corresponds to the acquisition time, and images from different data sources can be associated based on the acquisition time, such as one second can correspond to multiple frames of images, so the time can be accurately correlated to milliseconds. Thus, an image group can include images from different data sources that are associated with each other. Among them, the unmanned vehicle continuously tracks objects within a certain time range, and each 3D point cloud image in the image group can be an entry, and marking objects in the point cloud image can describe this section of continuously changing objects.

[0050] Thus, images in an image group can be annotated. By annotating one or more frames of images, images of other associated data sources can be automatically annotated, as well as images before or after the frame of image.

[0051] Reference Figure 1 , Figure 2 , Figure 3 The image annotation processing diagram shown in FIG. 1 can realize group annotation through the following steps: Figure 4 As shown:

[0052] Step 402, providing a marking interface.

[0053] The embodiment of the present application may provide a marking interface, which is used to mark data. In this embodiment, images in an image marking group may be marked on the marking interface. Of course, if the marked items are other types of data such as text books, audio data, etc., the marking interface may also provide marking.

[0054] The annotation interface provides an image display area in which an image is displayed. The image may be an image that has been annotated or an image that needs to be annotated. It also provides an annotation information display area in which information of annotated objects in the image may be displayed.

[0055] like Figure 1 As shown, area 102 is a first image display area. Figure 1-3 The display is a surveillance image captured by the camera; area 104 is a second image display area, such as Figure 1-3 The display is a radar point cloud image collected by the radar on the unmanned vehicle; area 106 is a labeling information display area, each frame of the image can correspond to a labeling information display area, such as the two frames of the image shown in the figure correspond to two labeling information display areas; area 108 is a control area, through which various processing such as labeling can be performed, including mapping controls, frame adjustment controls, such as Figure 1 The mapping control in the middle is displayed as mapping the next frame. The frame adjustment controls include: previous frame, next frame, and frame selection box. Among them, triggering the mapping control of "Map next frame" can map the annotation information in the image to the next frame image. Triggering the "previous frame" control can be adjusted to display the previous frame image. Triggering the "next frame" control can be adjusted to display the next frame image. Entering the frame number in the "frame selection box" control can jump to display the corresponding image of the frame.

[0056] Step 404: display images of more than one data source on the annotation interface.

[0057] The images from different data sources are associated based on time information, which can be understood as the images from different data sources displayed in the annotation interface are associated with each other. The data source includes a camera and / or a radar, and the camera may include a vehicle-mounted camera and / or a surveillance camera. The vehicle-mounted camera includes a camera for collecting data on an unmanned vehicle, and may also include a vehicle-mounted camera of a driving recorder, etc. The surveillance camera includes various cameras such as a road surveillance camera and surveillance cameras of shops on both sides of the road, and the radar includes an unmanned vehicle radar.

[0058] In an optional embodiment, an image captured by a camera may be obtained, such as an image captured by a camera of an unmanned vehicle, an image captured by a vehicle-mounted camera of a driving recorder, an image captured by a surveillance camera of a city road, etc. The image is an image of a video captured by a camera, so the corresponding video can be obtained by combining consecutive frames of images.

[0059] Another optional embodiment is that a point cloud map collected by radar can be obtained. For example, in a scenario of unmanned vehicle tracking, a point cloud map collected by radar on the unmanned vehicle can be obtained, including a two-dimensional point cloud map collected by radar and a three-dimensional point cloud map collected by radar. These point cloud maps include point cloud maps of continuous frames.

[0060] After acquiring images from different data sources, the images from different data sources can be associated. Since multiple frames of images can usually be acquired in 1 second, the images from different data sources can be associated based on millisecond time. For example, each frame of the image corresponds to a timestamp, and the timestamp can be accurate to milliseconds, and the images from different data sources are associated based on the timestamp.

[0061] After the annotation interface is started and more than one corresponding data source is determined, the image corresponding to each data source can be displayed on the annotation interface. Figure 1-3 The images collected by the camera and the radar point cloud map collected by the radar on the unmanned vehicle are respectively displayed; wherein, the monitoring images taken by three cameras are displayed in the first image display area.

[0062] Step 406: Receive a marking operation and determine an object corresponding to the marking operation.

[0063] The annotation operation refers to the operation of annotating an image in the annotation interface, which can be achieved through various operation methods, such as clicking on the image, demarcating an area, etc.

[0064] In an optional embodiment, a marking operation of selecting a target area on the image is received, and an object in the target area is determined. The target area can be selected by a mouse, touch, etc. If it is determined that there is an object in the target area, it indicates that a marking operation of selecting a target area on the image is received, and the object in the target area is determined. Figure 1 In the point cloud image of the radar, the target area is delineated by a white rectangular frame, and the vehicle in the white rectangular frame can be determined as the marked object, and the white rectangular frame is the target area marked by the vehicle.

[0065] In some other optional embodiments, a marking operation of clicking on the image may be received, and the object corresponding to the click position may be determined. A click operation may be received on the image of the marking interface, and it may be determined whether the click position corresponding to the click operation corresponds to an object. If there is an object, it indicates that a marking operation of clicking on the image is received, and the object corresponding to the click position may be determined to be the marked object. Figure 1 If you want to mark the vehicle in the image, you can draw a rectangular frame on the object corresponding to the clicked position to mark the target area of ​​the object.

[0066] In some other optional embodiments of the present application, after determining the annotated object, the annotation type of the object can also be selected, and the annotation type can be added to the annotation information. Among them, the type setting control can be displayed in the annotation interface through a sub-interface, a floating window, etc., and the type of the object can be received through the type setting control, such as setting the annotation type by filling in, selecting, etc. For example, a type setting control with selectable types of "vehicle" and "person" can be displayed, and the selectable type is "vehicle".

[0067] In the embodiment of the present application, after selecting a marked object, the marking information of the object may also be set in the marking interface, including the object name, identifier, etc., as well as attribute information such as the marking type.

[0068] Step 408: marking the object on the corresponding image, and displaying the marking information of the object in the marking interface.

[0069] After determining the marked object, the object can be marked on the image of the marking interface, wherein the marked object can be displayed on the frame image and the corresponding frame images of other data sources associated with the frame image. In one embodiment, for an object marked by selecting a target area, if the object has been marked on the frame image when the target area is selected, the object can be searched in the corresponding frame images of other data sources associated with the frame image, and the object can be marked on the corresponding frame image. Figure 2 In the radar point cloud image, for the vehicle marked by the white rectangular frame, the corresponding vehicle can be marked in the associated frame image taken by the camera. In some other embodiments, for the object determined by clicking, the object can be marked in the frame image and the corresponding frame image of other data sources associated with the frame object.

[0070] The annotation of the object can also be displayed in the annotation information display area corresponding to the frame image, such as displaying the name of the annotated object, etc., so that when multiple objects are annotated on a frame image, they can be divided into areas by name, etc., and the annotation of the object can be displayed on the corresponding frame image by selecting different objects.

[0071] In some optional embodiments, for a three-dimensional (3D) point cloud image, since each object in the image has three-dimensional information, a three-dimensional display area of ​​the object can be set on the point cloud image. After selecting the annotated object, the three-dimensional point cloud image of the object can be displayed in the three-dimensional display area. Figure 1 When the object is initially selected, there is no object in the 3D display area. After the labeled object is determined, Figure 2 The point cloud images of the top, rear and sides of the marked vehicle can be displayed, so that multi-angle information of the object can be viewed.

[0072] In some optional embodiments of the present application, the images, including panoramic surveillance images of cameras, point cloud images, etc., can be viewed by rotating the angle, so as to more accurately identify objects for annotation. Figure 5 In the annotation interface shown, rotate the 3D point cloud image of the radar to see the three-dimensional point cloud of the object in the image. For example, the object annotation box displays a three-dimensional rectangular style during the rotation. By rotating, you can view the image from other angles and select the annotation object.

[0073] After the marking of the object in the image of each data source corresponding to one frame is completed, the same object in the continuous multiple frames of images after the frame of image can be automatically marked.

[0074] In one example, the automatic annotation of the same object in multiple consecutive frames of images can be triggered through the controls in the annotation interface. For example, if a "one-click annotation" control is set on the interface, after triggering the "one-click annotation" control, the annotated object can be automatically identified and annotated from the multiple consecutive frames of images after the frame, so that the corresponding annotated object and annotation information can be displayed when switching to display the corresponding frame image.

[0075] In some other examples, the annotation interface may set a "map next frame" control, and after triggering the control, the annotated object can be identified from the images of the next frame of each data source of the frame, and the object can be annotated in the image and the annotation information of the object can be displayed. Figure 2 If the "Map Next Frame" control is triggered in the control, the object can be marked with a white box in the point cloud image and monitoring image of the next frame after receiving the trigger of the control, and the point cloud images of the top, rear and side of the marked vehicle and the marking information of the object are displayed in the 3D display area. After the mapping is completed, the marking interface is as follows Figure 3 shown.

[0076] In the embodiment of the present application, after completing the annotation of an object, other objects that need to be annotated in the frame image can also be annotated in the same manner as described above. The annotation process is the same as the above steps, so it will not be repeated. After completing the annotation of other objects, different annotated objects can be switched to display by selecting different annotation information. Figure 6 In the example, there are two objects marked in the frame image. If you choose to display the mark of object 2, object 2 can be marked in each frame image, and the radar point cloud image shows the object from different angles. You can also switch to display different image frames to view the objects marked in each image, such as in Figure 6 Trigger the "Next Frame" control to switch to display the 2nd and 3rd frames, such as Figure 7 shown.

[0077] In the embodiment of the present application, after the data is collected, the collected data can also be uploaded to the server for data preprocessing, so as to provide data for the annotation page, and the annotation information of the annotation object can be stored. Of course, in some other examples, data preprocessing and storage can also be performed on the client side, and the embodiment of the present application does not limit this. Take the preprocessing and storage of data on the server as an example. Figure 8 The data annotation scenario diagram shown in FIG. The surveillance camera on the road can collect surveillance images and upload them to the server, and the unmanned vehicle can also collect point cloud images through radar and upload them to the server. The server can then perform data preprocessing, grouping and other processing operations, and provide images for the annotation interface. After the image annotation is completed, it will be returned to the server for storage. The specific steps may include the following (such as Fig. 9 shown):

[0078] Step 902: receiving images uploaded from different data sources.

[0079] Images captured by cameras can be obtained, such as images captured by cameras of unmanned vehicles, images captured by vehicle-mounted cameras of driving recorders, images captured by surveillance cameras on urban roads, etc. Among them, the image is an image of the video captured by the camera, so the corresponding video can be obtained by combining continuous frames of images. Point cloud images captured by radars can also be obtained. For example, in the scene of unmanned vehicle tracking, point cloud images collected by radars on unmanned vehicles can be obtained, including two-dimensional point cloud images collected by radars and three-dimensional point cloud images collected by radars. These point cloud images include point cloud images of continuous frames.

[0080] Step 904 , the images from different data sources are associated according to the time information, and the associated images are provided to the annotation interface so as to annotate the images through the annotation interface.

[0081] Then, the image corresponding to the collected data can be preprocessed, at least one object in the image can be pre-identified, and the object information of the object can be recorded. Among them, various objects included in the image can be identified through preprocessing, such as vehicles, people, buildings, etc. in the point cloud image collected by an unmanned vehicle, and vehicles, people, buildings, etc. in images such as monitoring and vehicle-mounted cameras. Among them, the same object identifier can be used for the same object identified in different data sources, so as to facilitate the determination of the same object contained in the associated frame images of different data sources. After setting the object identifier of the object in the preprocessing, the subsequent annotation process can directly use the annotation.

[0082] The continuous frame images collected by the data source may also be added to an image group, the image group corresponds to at least one data source; and the same object contained in the associated images of different data sources in the image group is determined. Wherein, the image group corresponding to the image can be determined based on preprocessing, and the continuous frame images containing the same object can be added to an image group, wherein, for the case where the same object is contained in different data sources, the continuous frame images of different data sources containing the same object can also be put into an image group. For example, for object A, the continuous 1-100 frame images in the radar point cloud image contain the object A, and the continuous 1-50 frame images in the monitoring image J1 contain the object A, then the continuous 1-100 frame images in the radar point cloud image and the continuous 1-50 frame images in the monitoring image J1 can be put into the same image group.

[0083] In some other examples, one data source may be used as the main data source to determine one or more objects contained in the image of the data source, and then continuous frame images containing at least one of the objects from different data sources may be added to the image group. The objects in the images of different data sources in the image group are partially or completely the same as the objects in the image of the main data source, while the objects contained in the images of other different data sources other than the main data source may be the same or different. Figure 1 The radar point cloud image is associated with the surveillance images of three cameras. In the process of marking object 1 and object 2, surveillance images of different cameras are associated respectively. Figure 3 The annotations and Figure 6 The annotation of object 2 displayed corresponds to the surveillance images of different cameras, that is, the surveillance images of object 1 and object 2 are respectively in the surveillance images of different cameras. These surveillance images of different cameras that contain some objects in the point cloud image may have nothing to do with each other in the surveillance images of different cameras, but can still be put into the same image group based on the association with the point cloud image.

[0084] An image group may include images from more than one data source, so the images from different data sources in the image group are associated according to the timestamps of the images. After acquiring images from different data sources, the images from different data sources may be associated, wherein, since multiple frames of images can usually be acquired in 1 second, the images from different data sources may be associated based on millisecond-level time. For example, each frame of the image corresponds to a timestamp, and the timestamp may be accurate to milliseconds, and the images from different data sources are associated based on the timestamp.

[0085] After the server obtains the collected images and pre-processes them by identifying the objects and grouping them, they can be provided to the annotation interface for annotation. In the annotation interface, annotation can be performed in the above manner. After selecting an annotation object in one of the frames, the object can be determined and annotated in the images of other data sources associated with the frame image. Of course, the associated frame images for each object can be ignored.

[0086] The embodiment of the present application takes the 3D tracking and labeling scene of an unmanned vehicle as an example, and can also be applied to other scenes in actual processing, such as face / object recognition and tracking through road monitoring, etc., and face / object recognition and labeling scenes applied to videos. For surveillance videos and videos of film and television works, the video can be split into image frames for labeling, and each image frame is used as an entry in which related objects are labeled. The image group can describe the associated object labeling of such multiple frames, such as continuously changing gestures, vehicles, expressions, etc., as well as changes in the position of people, etc. Moreover, the embodiment of the present application is not only applicable to object recognition, but also to all continuously labeled scenes, such as faces, gestures, postures, etc.

[0087] Step 906: Obtain and store the annotation information of the object in the image.

[0088] After completing the annotation in the annotation interface, the annotation information can be uploaded to the server for storage. The annotation information of each object includes the description information of the annotated object, such as the object name, identifier, and attribute information such as the annotation location.

[0089] Wherein, obtaining the annotation information of the object in the image includes: obtaining the annotation information of the object annotated in the image in the image group; integrating the annotation information of the object annotated in the image group to determine the group annotation information of the image group. For the objects annotated in each image group, they can also be described by group annotation information, and the group annotation information refers to the description information annotated in the group. The group annotation information can integrate the annotation information of the objects annotated in each frame image, such as merging the same annotation information, merging the annotation information of consecutive frames of the same object, etc. to obtain the group annotation information, and can compress the data volume to reduce the occupation of storage space and transmission resources. Wherein, in order to distinguish the images in the image group, each image can also be provided with an image identifier to distinguish, and the image identifier can be a unique identifier, a number, or a sequence number of the image in the image group. The group annotation information includes an identification field, an image field, and an attribute field. The identification field is used to record the object identification of the annotated object, the image field is used to record the image containing the annotated object, and the attribute field is used to record the attribute information of the annotated object.

[0090] Among them, one grouping element information of the grouping annotation information corresponds to one annotation object, and each grouping element information may include an identification field, an image field and an attribute field to record the information corresponding to one annotation object. Therefore, the images and their annotation information can be sorted according to the annotation object to determine each grouping element information of the grouping annotation information in the image group.

[0091] In an optional embodiment, based on the marked object, the marking information of the marked object in the image group is integrated to determine the grouping marking information of the image group, including: integrating the image containing the marked object into a grouping element information of the grouping marking information, so as to query the grouping element information for each marked object, so as to determine all entries containing the marked object, and also facilitate unified updating of the entries.

[0092] In an optional embodiment, the integration of the annotation information of the annotated objects in the image group to determine the group annotation information of the image group includes: searching according to the object identifier of the annotated object to determine the image identifiers of multiple images containing the object identifier; using the object identifier, multiple image identifiers and the attribute information of the annotated object to determine the corresponding group annotation information. The object identifier of the annotated object in an image can be determined, such as by selecting any target object, or by traversing from the object annotated in the first frame image in the group, etc., and a grouping element information of the group annotation information can be established for each annotated object, and the object identifier of the annotated object is added to the object field. And the attribute information of the target annotated object can also be added to the attribute field of the corresponding grouping element information. Then, the annotation information of the annotated object in each image can be queried in the image group, wherein the image can be queried in sequence according to the frame order of the image in the image group, and the image identifier of the image that queried the object is added to the image field of the corresponding grouping element information, and then the query in other images is continued until all entries in the image group are queried, and the next grouping element information can be processed. The attribute field of the grouping element information can store the common attributes of the annotated object in each image. Therefore, after all images containing the annotated object are queried, the common attributes can be determined and added to the attribute field.

[0093] Based on the above optional embodiments, Fig.10 An example of determining group annotation information can be achieved by the following sub-steps:

[0094] Sub-step 1002, determining the object identifier of the target annotation object from the images in the image group; the target annotation object refers to the annotation object that does not correspond to the grouping element information in the grouping annotation information.

[0095] Sub-step 1004: adding a piece of grouping element information to the grouping annotation information, and adding the object identifier to the object field of the grouping element information.

[0096] Sub-step 1006, searching the image group for the annotation information of the object annotated to the image, and adding the image identifier of the frame image annotated with the object to the entry field of the grouping element information.

[0097] Sub-step 1008 , determining whether all frames of the image in the image group have been queried; if not, returning to sub-step 1006 to continue the query; if so, executing sub-step 1010 .

[0098] Sub-step 1010, determining common attribute information of each image in the image field, and adding the common attribute information to the attribute field of the grouping element information.

[0099] Sub-step 1012, determining whether there are any target annotated objects in the image group; if so, returning to sub-step 1002; if not, ending the process.

[0100] In one example, the grouping element information (metaInfo) may be a record of annotated objects in an image group, and each annotated object in an entry of the image group may correspond to the grouping element information, wherein each object in the example of the schema format of the grouping element information (metaInfo) includes:

[0101] Identification field (id): This field describes the object identifier of the annotated object. The object identifier can be a globally unique id, which is used to identify a certain annotated object. If this annotated object exists in multiple annotated images, the annotated objects in multiple images share the same id.

[0102] Image field (contains): This field describes the images in the image group that contain the annotated object.

[0103] Attribute field (commonLabels): This field describes the attribute information of the annotated object. The attribute information can be a common attribute, such as the common attribute of the annotated object in each image. The attribute field describes the attributes of the object and / or the common attributes of the image to which the object belongs.

[0104] The grouping annotation information may also include other information such as a timestamp.

[0105] Take a JSON string as an example, {"metaInfo":[{"id":"0000-0001","contains":[1,2,3,4,5,6],"commonLabels":{"color":"red","category":"car"}}],"timestamp":"13131231231"}

[0106] In a further optional embodiment, when the object has been marked in consecutive frame images, the image identification of the consecutive frame images is recorded in a compressed manner in the image field of the grouping element information of the object. In the case where multiple consecutive frame images all include the same object, they can also be represented in a compressed manner in the image field to compress the data. For example, the above "contains":[1,2,3,4,5,6] can be adjusted to "contains":[1-6], and for example, "contains":[1,2,3,4,5,6,10,11,12,13,14,56,57,58,59,60] can be adjusted to "contains":[1-6,10-14,56-60].

[0107] Therefore, images with the same annotation object in the image group can be recorded in an image field of the grouping element information, and the annotation objects can be managed uniformly. The grouping annotation information can be recorded in various data records such as data tables and memos.

[0108] The storage of the above-mentioned collected data, image preprocessing, storage of annotation information, etc. can also be processed in the client corresponding to the annotation interface, and the embodiment of the present application does not limit this.

[0109] In some other optional embodiments of the present application, the image may be annotated first, and then the image group of the image annotated with the object may be determined. For example, continuous frame images of different data sources are provided in the annotation interface, and then the selected object in the image is annotated, the annotation position data, object identification and other attribute information of the object are determined, the corresponding annotation information is obtained, and the object is annotated in the associated frame images of other data sources. Each image may correspond to the annotation information of one or more objects, so that after the annotation of each image is completed, an image group may be created based on the objects annotated in the image.

[0110] In the embodiment of the present application, after the related and ordered images are divided into an image group, the edit of the annotations in the image can be updated based on the image group, which can effectively improve the processing efficiency. Among them, the editing operation can generally include adding operation, deleting operation, modifying operation and querying operation, etc. For the update of the annotation of the image in the image group, the group annotation information will be updated accordingly.

[0111] In an optional embodiment, updating the corresponding group annotation information based on the editing operation on the image in the image group includes at least one of the following steps: adding an image identifier of the image in the image field of the group annotation information based on adding the image of the annotated object; deleting the image identifier of the image in the image field of the group annotation information based on deleting the image of the annotated object; updating the attribute field in the group annotation information based on modifying the image of the attribute information of the annotated object; and extracting the group element information corresponding to the object based on the queried object.

[0112] Regarding the adding operation: the images in the image group are related and ordered, so in some cases, after adding an annotated object in one image, the annotation of the object will also be added to the subsequent images. Therefore, rules can be set based on specific environments, needs, experience, etc. After adding an annotated object to an image in the image group, it can be determined based on the rules whether the annotated object needs to be added to the subsequent images, so as to update the corresponding fields of the group annotation information based on the image with the annotated object added.

[0113] In a further optional embodiment, the image identifier of the image is added to the image field of the grouped annotation information based on the image of the annotated object to be added, including: determining the image identifier of the image of the annotated object to be added; determining to add the annotated object to the next frame of the image according to the image identifier and the set rule, then obtaining the image identifier of the next frame of the image; and adding the image identifier of the next frame of the image to the image field of the grouped annotation information. The associated frame images of each frame of the image are similar. When it is necessary to add an annotated object to the image, the image identifier of the image and the object identifier of the annotated object can be determined, and the image identifier can be added to the image field of the grouping element information of the corresponding annotated object. If there is grouping element information for each annotated object in the grouped annotation information, the grouping element information of the annotated object is created. The step of determining the grouping element information is similar to the above embodiment, and the image identifier of the image is added to the image field. Then, according to the set rules, it is determined whether the next frame image of the image also needs to add the annotation object. If it is determined that it needs to be added, the image identifier of the next frame image can be added to the image field of the grouping element information corresponding to the annotation object. Then, the above process can be iterated, and the next frame image is used as the target image to determine whether the next frame image of the target image needs to add the annotation object, until it is determined that the next frame image does not need to add the annotation object or all images in the group have been queried, and the above adding operation process can be ended.

[0114] Based on the above optional embodiments, Fig.11 An example of updating group annotation information according to the adding operation can be implemented by the following sub-steps:

[0115] Sub-step 1102, determining the object identifier of the added annotated object and the image target identifier of the image of the added annotated object; for example, the object identifier is 0000-000X and the image identifier is 2000.

[0116] Sub-step 1104, determining that the marked object corresponds to the grouping element information, adding the image identifier to the image field of the grouping element information; then adding 2000 to the image field of the grouping element information from 0000 to 000X.

[0117] Sub-step 1106, judging whether the next frame image needs to add the annotated object according to the image identifier and the setting rule; if so, executing sub-step 1108, if not, ending the process. For example, the image identifier of the next frame image is 2001.

[0118] Sub-step 1108: Add the image ID of the next frame of image to the image field of the grouping element information. Add 2001 to the image field of the grouping element information from 0000 to 000X.

[0119] Therefore, based on the operation of adding the annotated object in one frame of image, the image to which the annotated object is to be added subsequently can be conveniently added to the group.

[0120] Regarding the deletion operation: each frame image in the image group is associated and ordered. Usually, after deleting an identification object in a frame image, the identification object will also be deleted in the subsequent frame images. Therefore, there is no need to mark the object starting from this frame image. The image identification of this frame image and its subsequent frame images can be included in the grouping element information of the marked object.

[0121] In a further optional embodiment, the image identifier of the image is deleted in the image field of the grouping annotation information according to the image of the deleted annotation object, including: determining the image identifier of the image of the deleted annotation object; deleting the image identifier and the image identifiers of other frame images after the image identifier in the image field of the grouping annotation information. When it is necessary to delete the annotation object in the image, the image identifier of the image and the object identifier of the annotation object can be determined, and then the image identifier and the image identifiers of other frame images after the image identifier can be deleted in the image field of the grouping element information of the annotation object, and the image identifiers of all frame images after the image identifier can be deleted. For example, the object identifier of the deleted annotation object and the image identifier of the image of the deleted annotation object are determined; for example, the object identifier is 0000-000X and the image identifier is 2000. It is determined that the grouping element information corresponding to the annotation object is deleted, and the image identifier of the frame image after the image identifier and the image identifier of the frame image after the image identifier are deleted in the image field of the grouping element information of 0000-000X; then the image identifier after 2000 is deleted in the image field of the grouping element information of 0000-000X.

[0122] Therefore, based on the deletion operation of the annotated object in an image, the annotation of the object in the image and subsequent images in the group can be conveniently deleted.

[0123] Regarding the modification operation: each frame image in the image group is associated and ordered. The attribute field of the grouping element information of a certain annotated object in the image group is usually the common attribute corresponding to each image with the annotated object. Therefore, when modifying the attribute information of a certain image with the annotated object, if the modified attribute information belongs to the common attribute, the attribute field of the grouping element information of the annotated object must also be modified accordingly.

[0124] In a further optional embodiment, the updating of the attribute field in the grouped annotation information based on the image of the modified attribute information of the annotated object includes: determining the object identifier of the modified annotated object in the image, and the modified attribute information; querying the grouped annotation information corresponding to the object identifier; and updating the attribute field of the grouped annotation information based on the modified attribute information. When modifying an annotated object in an image, the image identifier of the modified image, the object identifier of the annotated object, and the attribute information of the modified attribute can be determined, and then it is determined whether the modified attribute information is in the attribute field in the grouped element information corresponding to the annotated object. If it is in the attribute field, the attribute field can be modified based on the modified attribute information. If it is not in the attribute field, there is no need to update the attribute field.

[0125] Based on the above optional embodiments, Fig.12 An example of updating group annotation information according to a modification operation can be implemented by the following sub-steps:

[0126] Sub-step 1202, determining the object identifier of the modified annotated object in the image and the modified attribute information.

[0127] Sub-step 1204, querying the attribute field of the grouping annotation information corresponding to the object identifier.

[0128] Sub-step 1206, determining whether the attribute information belongs to the attribute field. If yes, executing sub-step 1208; if no, ending the process.

[0129] Sub-step 1208: updating the attribute fields of the grouping annotation information according to the modified attribute information.

[0130] Therefore, for the modification of a marked object, the grouping mark information corresponding to the marked object in the image group can be modified.

[0131] Regarding query operations: Sometimes, based on various scenario requirements, it is necessary to query the objects marked in the image, the object's marking information, etc., such as replaying the marked objects of the image, then the attribute fields corresponding to the marked objects of the image can be extracted from the grouped marking information to determine the content marked by the image. It can also be combined with the marking information of the image for comparison, merging and other analysis based on specific scenarios.

[0132] In a further optional embodiment, extracting the group annotation information corresponding to the image according to the queried image includes: determining the image identifier of the queried image; querying the group annotation information whose image field contains the image identifier, and reading the attribute field of the group annotation information. The image identifier of the image to be queried can be determined, and then the component annotation information of the group can be queried, the group element information containing the image identifier in the image field can be determined, and the attribute field and object field of the group element information can be read to obtain each annotated object in the image and its attribute information.

[0133] Taking the playback scene as an example, the image to be played back can be determined, the annotation information and image identifier of the image can be determined, and then the annotation objects and their attribute information of the image can be queried in the group information. The queried annotation objects and their attribute information can be merged and compared with the annotation information of the image, merged, etc. to obtain the data that needs to be played back.

[0134] Based on the above optional embodiments, Fig.13 An example of updating group annotation information according to a query operation can be implemented by the following sub-steps:

[0135] Sub-step 1302, determining the image identifier of the image where the marked object is located.

[0136] Sub-step 1304, query whether the image field of the grouping annotation information has an image identifier. If yes, execute sub-step 1306, if no, execute sub-step 1308.

[0137] Sub-step 1306, reading the attribute field of the grouping annotation information.

[0138] Sub-step 1308, determine whether the group annotation information of the group is searched. If yes, execute sub-step 1310, if not, return to sub-step 1304 to continue the search.

[0139] Sub-step 1310, merging the query results. The queried annotated objects and their attribute information may be merged to obtain the annotated content of the image, or the annotated information may be merged with the queried annotated objects and their attribute information to determine the annotated content of the image, so that the image can be displayed and played back based on the merged content. Among them, the playback object may be determined; the grouping annotation information of the image group corresponding to the playback object may be queried to determine the continuous frame images annotating the playback object; the continuous frame images may be synthesized, and the synthesized data may be played. The synthesized data may be a video, a moving picture, etc.

[0140] In the embodiment of the present application, the annotations of the images in the image group can be edited such as adding, deleting, modifying, and checking, and two data operations can be performed, one is for the annotated objects in the image, and the other is to modify the group annotation information, which greatly improves the performance.

[0141] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide a data annotation method, which can annotate the connection frame images with high efficiency.

[0142] Reference Fig.14 , showing a step flow chart of another data labeling method embodiment of the present application.

[0143] Step 1402, acquiring the captured images and adding the captured continuous frame images into an image group.

[0144] The image captured by the camera can be obtained, wherein the camera includes at least one of the following: a vehicle-mounted camera and a surveillance camera. The camera can also include various cameras such as an infrared camera and a night vision camera. The point cloud image captured by the radar is obtained, wherein the point cloud image includes: a two-dimensional point cloud image and / or a three-dimensional point cloud image. The radar includes a vehicle-mounted radar of an unmanned vehicle.

[0145] Furthermore, the collected continuous frame images are added to an image group, and the image group provides images for the annotation interface. The image group corresponds to at least one data source, and each data source corresponds to continuous frame images; images from different data sources can also be associated according to the timestamps of the images.

[0146] At least one object in the image may be pre-identified and object information of the object may be recorded. The same object contained in the associated images from different data sources in the image group is determined.

[0147] The image group may be provided, and the continuous frame images in the image group may be adjusted according to the adjustment operation. The user may adjust the images in the image group according to the requirements, so that the division of the image group is more flexible.

[0148] Step 1404, providing a marking interface.

[0149] Step 1406 , displaying images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information.

[0150] Step 1408, receiving a marking operation and determining an object corresponding to the marking operation. In particular, a marking operation of selecting a target area on the image may be received and an object within the target area may be determined. Alternatively, a marking operation of clicking on the image may be received and an object corresponding to the clicked position may be determined.

[0151] The object can be marked on the corresponding image, and the marking information of the object can be displayed in the marking interface. The details are as follows:

[0152] Step 1410 , annotating the object in an image of another data source associated with the image, and displaying the three-dimensional point cloud features of the object in a point cloud image, and annotating by setting a graphic.

[0153] Step 1412: determine the annotation information of the object, and display the annotation information of the object in the annotation interface.

[0154] Step 1414: receiving a mapping operation and marking other frame images containing the object.

[0155] In the embodiment of the present application, the marking of the object includes: marking the range of the object in the image by setting a graphic, and the setting graphic is determined according to the object.

[0156] Step 1416: Integrate the annotation information of the annotated objects in the image group to determine the group annotation information of the image group, wherein each object in the group annotation information corresponds to a group element information, and the group element information includes: an attribute field, and the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs.

[0157] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide an object tracking method, which can annotate an object based on an annotation interface, thereby tracking the object.

[0158] Reference Fig.15 , a flowchart of the steps of a client of an object tracking method embodiment of the present application.

[0159] Step 1502, providing a marking interface.

[0160] Step 1504: displaying a monitoring image captured by at least one camera and a point cloud map captured by a radar on the annotation interface, wherein the monitoring image and the point cloud map are associated with corresponding frames through time information.

[0161] Step 1506, receiving the annotation operation in the specified frame point cloud image, and annotating the annotated object by setting the graphics.

[0162] Step 1508: in at least one surveillance image frame associated with the designated frame point cloud image, annotate the object by setting a graphic and display the annotation information of the object, wherein the three-dimensional point cloud features of the object are displayed in the point cloud image and annotated by setting a graphic.

[0163] Step 1510, receiving a mapping operation.

[0164] Step 1512, in the point cloud image and the monitoring image corresponding to the next frame of the specified frame, annotate the object by setting a graphic.

[0165] Step 1514, in the corresponding point cloud images and monitoring images of the consecutive frames after the designated frame, annotate the object by setting a graphic.

[0166] Reference Fig.16 , a flow chart of the steps on the server side of an object tracking method embodiment of the present application.

[0167] Step 1602, receiving a surveillance image captured by at least one camera, and receiving a point cloud image captured by a radar.

[0168] Step 1604 , the surveillance image and the point cloud map are associated with corresponding frames through time information, and the associated surveillance image and point cloud map are stored to provide them to the annotation interface for annotation.

[0169] Step 1608: Obtain and store the annotation information of the objects in the point cloud image and the surveillance image.

[0170] The steps of this embodiment are similar to the corresponding steps in the above embodiments, and thus will not be described in detail.

[0171] In an embodiment of the present application, group annotation information can be stored in texts such as memos, thereby greatly reducing the amount of data transmitted over the network; and can be based on database operations to ensure data consistency; each modification is saved to the database, and the group annotation data will not be lost due to accidents, thereby improving availability; each annotation operation only requires modifying the group element information of the corresponding annotation object in the group, thereby improving scalability and facilitating the updating of group annotation information in the group; and common attributes are extracted and stored through the attribute fields of the group annotation information, without the need for repeated storage, thereby streamlining the amount of data storage.

[0172] The annotation interface provides visual annotation. Users only need to identify the object to be marked on one frame of the image, and the same object can be automatically marked on other images. Users do not need to mark each frame separately, which simplifies the annotation operation and improves the annotation efficiency. In addition, the annotation objects can be automatically identified and marked for images from different data sources, which is more efficient.

[0173] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0174] On the basis of the above embodiments, this embodiment further provides a data labeling device, which is applied to an electronic device of a terminal device.

[0175] Reference Fig.17 , shows a structural block diagram of an embodiment of a data labeling device of the present application, which may specifically include the following modules:

[0176] The interface providing module 1702 is used to provide a marking interface.

[0177] The image display module 1704 is used to display images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information.

[0178] The annotation module 1706 is used to receive an annotation operation and determine an object corresponding to the annotation operation.

[0179] The annotation processing module 1708 is used to annotate the object on the corresponding image and display the annotation information of the object in the annotation interface.

[0180] In summary, a labeling interface can be provided and images from more than one data source can be displayed in the labeling interface, wherein images from different data sources are associated based on time information, so that labeling can be performed directly on the labeling interface, a corresponding labeling operation is received, an object corresponding to the labeling operation is determined, and then the object can be labeled on the corresponding image, and the labeling information of the object can be displayed in the labeling interface. Labeling through a visual labeling interface is more intuitive, and after selecting an object in a frame image, the object can be automatically labeled in the frame image and its associated frame images, which is more efficient.

[0181] Reference Fig.18 , shows a structural block diagram of an optional embodiment of a data annotation device of the present application, which may specifically include the following modules:

[0182] The image preprocessing module 1710 is used to preprocess the collected images.

[0183] The interface providing module 1702 is used to provide a marking interface.

[0184] The image display module 1704 is used to display images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information.

[0185] The annotation module 1706 is used to receive an annotation operation and determine an object corresponding to the annotation operation.

[0186] The annotation processing module 1708 is used to annotate the object on the corresponding image and display the annotation information of the object in the annotation interface.

[0187] The playback module 1712 is used to determine a playback object; query the grouping annotation information of the image group corresponding to the playback object, determine the continuous frame images annotated with the playback object; synthesize the continuous frame images, and play the synthesized data.

[0188] The image preprocessing module 1710 includes: an image acquisition submodule 17102, an image grouping submodule 17104 and an image recognition submodule 17106, wherein:

[0189] In an optional embodiment, the image acquisition submodule 17102 is used to acquire images captured by a camera, wherein the camera includes at least one of the following: a vehicle-mounted camera, a surveillance camera.

[0190] In another optional embodiment, the image acquisition submodule 17102 is used to acquire a point cloud image collected by a radar, wherein the point cloud image includes: a two-dimensional point cloud image and / or a three-dimensional point cloud image. The radar includes a vehicle-mounted radar of an unmanned vehicle.

[0191] The image grouping submodule 17104 is used to add the collected continuous frame images into an image group, and the image group provides images for the annotation interface.

[0192] The image group corresponds to at least one data source, and each data source corresponds to a continuous frame image; the image grouping submodule 17104 is also used to associate images from different data sources according to the timestamps of the images.

[0193] The image grouping submodule 17104 is further used to provide the image group and adjust the continuous frame images in the image group according to the adjustment operation.

[0194] The image recognition submodule 17106 is used to pre-recognize at least one object in the image and record object information of the object.

[0195] In an optional embodiment, the labeling module 1706 is used to receive a labeling operation of selecting a target area on the image, and determine an object in the target area.

[0196] In another optional embodiment, the annotation module 1706 is used to receive an annotation operation of clicking on the image and determine an object corresponding to the clicked position.

[0197] The annotation processing module 1708 includes: an image annotation submodule 17082 and an annotation information determination submodule 17084, wherein:

[0198] The image annotation submodule 17082 is used to annotate the object in the image of other data sources associated with the image.

[0199] The annotation information determination submodule 17084 is used to determine the annotation information of the object and display the annotation information of the object in the annotation interface.

[0200] The image annotation submodule 17082 is also used to display the three-dimensional point cloud features of the object in the point cloud image and to annotate them by setting graphics.

[0201] The image recognition submodule 17106 is further configured to determine the same object contained in the associated images from different data sources in the image group.

[0202] The annotation processing module 1708 further includes: a mapping submodule 17086. The annotation module 1706 is further configured to receive a mapping operation; and the mapping submodule 17086 is configured to annotate other frame images containing the object.

[0203] In the embodiment of the present application, the marking of the object includes: marking the range of the object in the image by setting a graphic, and the setting graphic is determined according to the object.

[0204] The annotation processing module 1708 also includes: a group annotation integration submodule 17088, wherein:

[0205] The group annotation integration submodule 17088 is used to integrate the annotation information of the annotated objects in the image group to determine the group annotation information of the image group. Each object in the group annotation information corresponds to a group element information, and the group element information includes: an attribute field, and the attribute field describes the attributes of the object and / or the common attributes of the image to which the object belongs.

[0206] On the basis of the above-mentioned embodiment, this embodiment further provides a data labeling device, which is applied to an electronic device at a server end.

[0207] Reference Fig.19 , shows a structural block diagram of another data labeling device embodiment of the present application, which may specifically include the following modules:

[0208] The receiving module 1902 is used to receive images uploaded from different data sources.

[0209] The pre-processing module 1904 is used to associate the images from different data sources according to time information, and provide the associated images to the annotation interface so as to annotate the images through the annotation interface.

[0210] The storage module 1906 is used to obtain and store the annotation information of the object in the image.

[0211] In an optional embodiment, the preprocessing module 1904 is further used to add continuous frame images collected by the data source into an image group, and the image group corresponds to at least one data source.

[0212] The pre-processing module 1904 is used to associate the images from different data sources in the image group according to the timestamps of the images.

[0213] In another optional embodiment, the preprocessing module 1904 is further used to pre-identify at least one object in the image and record object information of the object.

[0214] In an optional embodiment, the pre-processing module 1904 is further configured to determine the same object contained in the associated images from different data sources in the image group.

[0215] In an optional embodiment, the storage module 1906 is used to obtain the annotation information of the objects annotated in the images in the image group; integrate the annotation information of the objects annotated in the image group, and determine the group annotation information of the image group.

[0216] The storage module 1906 is used to query according to the object identifier of the marked object to determine the image identifiers of multiple images containing the object identifier; and use the object identifier, multiple image identifiers and attribute information of the marked object to determine corresponding grouping annotation information.

[0217] Each object in the group annotation information corresponds to a group element information, and the group element information includes: an attribute field, and the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs.

[0218] The grouping element information also includes: an image field; the storage module 1906 is also used to record the image identification of the continuous frame images in the image field of the grouping element information of the object in a compressed manner when the object has been marked in the continuous frame images.

[0219] The receiving module 1902 is used to receive an image captured by a camera, wherein the camera includes at least one of the following: a vehicle-mounted camera and a surveillance camera; and / or obtain a point cloud image captured by a radar, wherein the point cloud image includes a two-dimensional point cloud image and / or a three-dimensional point cloud image. The radar includes a vehicle-mounted radar of an unmanned vehicle.

[0220] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide an object tracking device, which can mark objects in images to track the objects and can be applied to client devices.

[0221] Reference Fig. 20, shows a structural block diagram of an object tracking device embodiment of the present application, which may specifically include the following modules:

[0222] The interface module 2002 is used to provide a marking interface.

[0223] The display module 2004 is used to display a monitoring image collected by at least one camera and a point cloud map collected by a radar on the annotation interface, wherein the monitoring image and the point cloud map are associated with corresponding frames through time information.

[0224] Processing module 2006 is used to receive the annotation operation in the specified frame point cloud image, and annotate the annotated object through the set graphic; in at least one frame of monitoring image associated with the specified frame point cloud image, annotate the object through the set graphic, and display the annotation information of the object.

[0225] The processing module 2006 is further used to display the three-dimensional point cloud features of the object in the point cloud image and to mark them by setting graphics.

[0226] The processing module 2006 is further configured to receive a mapping operation; and to annotate the object in the corresponding point cloud image and monitoring image of the next frame of the designated frame by setting a graphic.

[0227] The processing module 2006 is further configured to receive a mapping operation; and to annotate the object in a plurality of consecutive corresponding point cloud images and monitoring images after the designated frame by setting graphics.

[0228] On the basis of the above-mentioned embodiments, the embodiments of the present application further provide an object tracking method, which can mark objects in an image to track the objects, and can be applied to electronic devices on the server side.

[0229] Reference Fig.21 , shows a structural block diagram of another object tracking device embodiment of the present application, which may specifically include the following modules:

[0230] The data receiving module 2102 is used to receive monitoring images collected by at least one camera and receive point cloud images collected by radar.

[0231] The association preprocessing module 2104 is used to associate the monitoring image and the point cloud map with corresponding frames through time information, and store the associated monitoring image and point cloud map to provide them to the annotation interface for annotation.

[0232] The information storage module 2106 is used to obtain and store the annotation information of the objects in the point cloud image and the surveillance image.

[0233] In an embodiment of the present application, group annotation information can be stored in texts such as memos, thereby greatly reducing the amount of data transmitted over the network; and can be based on database operations to ensure data consistency; each modification is saved to the database, and the group annotation data will not be lost due to accidents, thereby improving availability; each annotation operation only requires modifying the group element information of the corresponding annotation object in the group, thereby improving scalability and facilitating the updating of group annotation information in the group; and common attributes are extracted and stored through the attribute fields of the group annotation information, without the need for repeated storage, thereby streamlining the amount of data storage.

[0234] The annotation interface provides visual annotation. Users only need to identify the object to be marked on one frame of the image, and the same object can be automatically marked on other images. Users do not need to mark each frame separately, which simplifies the annotation operation and improves the annotation efficiency. In addition, the annotation objects can be automatically identified and marked for images from different data sources, which is more efficient.

[0235] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0236] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0237] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Fig. 22 An exemplary apparatus 2200 that can be used to implement various embodiments described in this application is schematically shown.

[0238] For one embodiment, Fig. 22 An exemplary apparatus 2200 is shown having one or more processors 2202, a control module (chip set) 2204 coupled to at least one of the (one or more) processors 2202, a memory 2206 coupled to the control module 2204, a non-volatile memory (NVM) / storage device 2208 coupled to the control module 2204, one or more input / output devices 2210 coupled to the control module 2204, and a network interface 2212 coupled to the control module 2204.

[0239] The processor 2202 may include one or more single-core or multi-core processors, and the processor 2202 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 2200 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0240] In some embodiments, the device 2200 may include one or more computer-readable media (e.g., memory 2206 or NVM / storage device 2208) having instructions 2214 and one or more processors 2202 that are combined with the one or more computer-readable media and configured to execute the instructions 2214 to implement the module to perform the actions described in the present disclosure.

[0241] For one embodiment, the control module 2204 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 2202 and / or any suitable device or component in communication with the control module 2204 .

[0242] The control module 2204 may include a memory controller module to provide an interface to the memory 2206. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0243] The memory 2206 may be used, for example, to load and store data and / or instructions 2214 for the device 2200. For one embodiment, the memory 2206 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 2206 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0244] For one embodiment, the control module 2204 may include one or more input / output controllers to provide an interface to the NVM / storage device 2208 and the input / output device(s) 2210 .

[0245] For example, NVM / storage 2208 may be used to store data and / or instructions 2214. NVM / storage 2208 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0246] NVM / storage device 2208 may include storage resources that are physically part of the device on which apparatus 2200 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 2208 may be accessed via input / output device(s) 2210 over a network.

[0247] (One or more) input / output devices 2210 may provide an interface for the apparatus 2200 to communicate with any other appropriate device, and the input / output device 2210 may include a communication component, an audio component, a sensor component, etc. The network interface 2212 may provide an interface for the apparatus 2200 to communicate through one or more networks, and the apparatus 2200 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0248] For one embodiment, at least one of the processor(s) 2202 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 2204. For one embodiment, at least one of the processor(s) 2202 may be packaged together with the logic of one or more controllers of the control module 2204 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 2202 may be integrated on the same die with the logic of one or more controllers of the control module 2204. For one embodiment, at least one of the processor(s) 2202 may be integrated on the same die with the logic of one or more controllers of the control module 2204 to form a system-on-chip (SoC).

[0249] In various embodiments, the device 2200 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 2200 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 2200 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0250] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0251] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0252] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0253] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0254] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0255] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0256] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0257] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0258] The above is a detailed introduction to a data labeling method and device, an object tracking method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data labeling method, characterized in that: The method comprises: Adding the collected continuous frame images to an image group, the image group provides images for the annotation interface, the image group corresponds to more than one data source, and each data source corresponds to the continuous frame images; Provide annotation interface; Displaying images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information; Receiving a labeling operation, and determining an object on the image corresponding to the labeling operation; Annotating the object on the corresponding frame image and corresponding frame images of other data sources associated with the frame image, and automatically annotating the same object in multiple consecutive frame images after the frame image; Displaying the annotation information of the object in the annotation interface; Integrate the image containing the marked object into a grouping element information of the grouping mark information, determine the grouping mark information of the image group, so as to query the grouping element information for each marked object, determine the entry containing the marked object, and facilitate unified update of the entry, each object in the grouping mark information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; When the objects are marked in the continuous frame images, the image identifiers of the continuous frame images are recorded in the image field of the grouping element information of the objects in a compressed manner to compress the data volume.

2. The method according to claim 1, characterized in that Also includes: An image captured by a camera is obtained, wherein the camera includes at least one of the following: a vehicle-mounted camera and a surveillance camera.

3. The method according to claim 1, characterized in that Also includes: A point cloud image collected by a radar is obtained, wherein the point cloud image includes: a two-dimensional point cloud image and / or a three-dimensional point cloud image.

4. The method according to claim 3, characterized in that The radar includes a vehicle-mounted radar of an unmanned vehicle.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: Images from different data sources are associated based on the image timestamps.

6. The method according to claim 5, characterized in that The receiving a marking operation and determining an object corresponding to the marking operation includes: A marking operation of selecting a target area on the image is received, and an object within the target area is determined.

7. The method according to claim 5, characterized in that The receiving a marking operation and determining an object corresponding to the marking operation includes: A marking operation of clicking the image is received, and an object corresponding to the click position is determined.

8. The method according to claim 5, characterized in that The displaying the annotation information of the object in the annotation interface includes: Determine the annotation information of the object, and display the annotation information of the object in the annotation interface.

9. The method according to claim 8, characterized in that Also includes: The three-dimensional point cloud features of the object are displayed in the point cloud image and annotated by setting graphics.

10. The method according to claim 8, characterized in that Also includes: At least one object in the image is pre-identified, and object information of the object is recorded.

11. The method according to claim 10, characterized in that Also includes: The same object contained in the associated images from different data sources in the image group is determined.

12. The method according to claim 8, characterized in that Also includes: A mapping operation is received, and other frame images containing the object are labeled.

13. The method according to claim 1, characterized in that The marking of the object includes: marking the range of the object in the image by setting a graphic, and the setting graphic is determined according to the object.

14. The method according to claim 5, characterized in that Also includes: The image group is provided, and consecutive frame images in the image group are adjusted according to an adjustment operation.

15. The method according to claim 1, characterized in that Also includes: Determine the playback object; Querying the grouping annotation information of the image group corresponding to the playback object, and determining the continuous frame images annotated with the playback object; The continuous frame images are synthesized and the synthesized data is played.

16. A data annotation method, characterized in that: The method comprises: Receiving images uploaded by different data sources, adding the images from the different data sources into an image group, the image group providing images for the annotation interface, each data source corresponding to a continuous frame image; The continuous frame images of the different data sources are associated according to time information, and the associated images are provided to the annotation interface, so as to annotate the images through the annotation interface, wherein objects are annotated on the corresponding frame images and corresponding frame images of other data sources associated with the frame images, and the same object in the continuous multiple frame images after the frame image is automatically annotated; Obtaining annotation information of objects annotated in the images in the image group; Integrate the image containing the marked object into a grouping element information of the grouping mark information, determine and store the grouping mark information of the image group, so as to query the grouping element information for each marked object, determine the entry containing the marked object, and facilitate unified update of the entry, each object in the grouping mark information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; When the objects are marked in the continuous frame images, the image identifiers of the continuous frame images are recorded in the image field of the grouping element information of the objects in a compressed manner to compress the data volume.

17. The method according to claim 16, characterized in that Also includes: The continuous frame images collected by the data source are added into an image group, and the image group corresponds to at least one data source.

18. The method according to claim 17, characterized in that The associating the images from different data sources according to time information includes: The images from different data sources in the image group are associated according to the timestamps of the images.

19. The method according to claim 17, characterized in that Also includes: At least one object in the image is pre-identified, and object information of the object is recorded.

20. The method according to claim 19, characterized in that Also includes: The same object contained in the associated images from different data sources in the image group is determined.

21. The method according to claim 16, characterized in that The step of integrating the image containing the marked object into a piece of grouping element information of the grouping mark information to determine the grouping mark information of the image group includes: Searching according to the object identifier of the marked object to determine the image identifiers of multiple images containing the object identifier; The object identifier, multiple image identifiers and the attribute information of the labeled object are used to determine a corresponding piece of grouping labeling information.

22. The method according to claim 16, characterized in that Receive images uploaded from different data sources, including at least one of the following: Receiving an image captured by a camera, wherein the camera includes at least one of the following: a vehicle-mounted camera and a surveillance camera; A point cloud image collected by a radar is obtained, wherein the point cloud image includes: a two-dimensional point cloud image and / or a three-dimensional point cloud image.

23. The method according to claim 22, characterized in that The radar includes a vehicle-mounted radar of an unmanned vehicle.

24. An object tracking method, characterized in that: The method comprises: Adding the collected continuous frame images to an image group, the image group provides images for the annotation interface, the image group corresponds to more than one data source, each data source corresponds to continuous frame images, and the images include monitoring images and point cloud images; Provide annotation interface; Displaying, on the annotation interface, continuous frames of surveillance images collected by at least one camera and continuous frames of point cloud images collected by a radar, wherein the surveillance images and point cloud images are associated with corresponding frames through time information; Receiving a marking operation in a specified frame point cloud image, marking a corresponding object on the point cloud image by setting a graphic, and automatically marking the same object in multiple consecutive frames of images after the specified frame point cloud image; In at least one surveillance image frame associated with the designated frame point cloud image, the object is annotated by setting a graphic, and the same object is automatically annotated in multiple consecutive frames of images after the at least one surveillance image frame associated with the designated frame, and the annotation information of the object is displayed; Integrate the image containing the marked object into a grouping element information of the grouping mark information, determine the grouping mark information of the image group, so as to query the grouping element information for each marked object, determine the entry containing the marked object, and facilitate unified update of the entry, each object in the grouping mark information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; When the objects are marked in the continuous frame images, the image identifiers of the continuous frame images are recorded in the image field of the grouping element information of the objects in a compressed manner to compress the data volume.

25. The method according to claim 24, characterized in that Also includes: The three-dimensional point cloud features of the object are displayed in the point cloud image and annotated by setting graphics.

26. The method according to claim 24, characterized in that Also includes: Receive mapping operations; In the corresponding point cloud image and monitoring image of the next frame of the designated frame, the object is marked by setting a graphic.

27. The method according to claim 24, characterized in that Also includes: Receive mapping operations; In the corresponding point cloud images and monitoring images of the consecutive multiple frames after the designated frame, the object is marked by setting a graphic.

28. An object tracking method, characterized in that: The method comprises: Receive continuous frame monitoring images collected by at least one camera, and receive continuous frame point cloud images collected by a radar, and add the continuous frame monitoring images and the continuous frame point cloud images into an image group; The surveillance image and the point cloud map are associated with corresponding frames through time information, and the associated surveillance image and point cloud map are stored to provide them to the annotation interface for annotation, wherein, based on the annotation operation in the point cloud map of the specified frame, the corresponding object is annotated by setting a graphic on the point cloud map, and the same object in the continuous multiple frames after the point cloud map of the specified frame is automatically annotated, and in at least one surveillance image frame associated with the point cloud map of the specified frame, the object is annotated by setting a graphic, and the same object in the continuous multiple frames after the at least one surveillance image frame associated with the specified frame is automatically annotated; Integrate a point cloud image containing annotated objects and a surveillance image containing annotated objects into a grouping element information of the grouping annotation information, determine the grouping annotation information of the corresponding image group and store it, so as to query the grouping element information for each annotated object, determine the entry containing the annotated object, and facilitate unified update of the entry, each object in the grouping annotation information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; When the objects are marked in the continuous frame images, the image identifiers of the continuous frame images are recorded in the image field of the grouping element information of the objects in a compressed manner to compress the data volume.

29. A data labeling device, characterized in that: The device comprises: An image preprocessing module, used to add the collected continuous frame images into an image group, the image group provides images for the annotation interface, the image group corresponds to more than one data source, and each data source corresponds to a continuous frame image; The interface providing module is used to provide the annotation interface; An image display module, used for displaying images from more than one data source on the annotation interface, wherein images from different data sources are associated based on time information; A labeling module, used for receiving a labeling operation and determining an object on the image corresponding to the labeling operation; A labeling processing module is used to label the object on the corresponding frame image and the corresponding frame image of other data sources associated with the frame image, and automatically label the same object in multiple consecutive frame images after the frame image; display the labeling information of the object in the labeling interface; integrate the image containing the labeled object into a grouping element information of the grouping labeling information, determine the grouping labeling information of the image group, so as to query the grouping element information for each labeled object and determine the entry containing the labeled object, so as to facilitate the unified update of the entry, each object in the grouping labeling information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; when the object has been labeled in consecutive frame images, the image identifier of the consecutive frame image is recorded in the image field of the grouping element information of the object in a compressed manner to compress the data volume.

30. A data labeling device, characterized in that: The device comprises: A receiving module, used for receiving images uploaded by different data sources, adding the images from different data sources into an image group, the image group providing images for the annotation interface, each data source corresponding to a continuous frame image; A preprocessing module, used to associate the continuous frame images of the different data sources according to time information, and provide the associated images to the annotation interface, so as to annotate the images through the annotation interface, wherein objects are annotated on the corresponding frame images and corresponding frame images of other data sources associated with the frame images, and the same object in the continuous multiple frame images after the frame image is automatically annotated; A storage module is used to obtain the annotation information of the objects annotated by the images in the image group; integrate the images containing the annotated objects into a grouping element information of the grouping annotation information, determine the grouping annotation information of the image group and store it, so as to query the grouping element information for each annotated object and determine the entry containing the annotated object, so as to facilitate the unified update of the entry, each object in the grouping annotation information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attributes of the object and / or the common attributes of the image to which the object belongs; when the object has been annotated in continuous frame images, the image identifier of the continuous frame image is recorded in the image field of the grouping element information of the object in a compressed manner to compress the data volume.

31. An object tracking device, characterized in that: The device comprises: An interface module is used to add the collected continuous frame images to an image group, the image group provides images for the annotation interface, the image group corresponds to more than one data source, each data source corresponds to continuous frame images, the images include monitoring images and point cloud images; provide an annotation interface; A display module, used to display at least one continuous frame monitoring image collected by a camera and a continuous frame point cloud map collected by a radar on the annotation interface, wherein the monitoring image and the point cloud map are associated with corresponding frames through time information; A processing module is used to receive a marking operation in a specified frame point cloud image, and mark the corresponding object by setting a graphic on the point cloud image, and automatically mark the same object in multiple consecutive frames of images after the specified frame point cloud image; in at least one frame of surveillance image associated with the specified frame point cloud image, mark the object by setting a graphic, and automatically mark the same object in multiple consecutive frames of images after at least one frame of surveillance image associated with the specified frame, and display the marking information of the object; integrate the image containing the marked object into a grouping element information of the grouping marking information, determine the grouping marking information of the image group, so as to query the grouping element information for each marked object, determine the entry containing the marked object, and facilitate unified update of the entry, each object in the grouping marking information corresponds to a grouping element information, and the grouping element information includes: an attribute field, the attribute field describes the attribute of the object and / or the common attribute of the image to which the object belongs; when the object has been marked in the continuous frame images, the image identifier of the continuous frame image is recorded in the image field of the grouping element information of the object in a compressed manner to compress the data volume.

32. An object tracking device, characterized in that: The device comprises: A data receiving module, used to receive continuous frame monitoring images collected by at least one camera, and receive continuous frame point cloud images collected by a radar, and add the continuous frame monitoring images and continuous frame point cloud images into an image group; An association preprocessing module, used to associate the surveillance image and the point cloud map with corresponding frames through time information, and store the associated surveillance image and point cloud map to provide them to the annotation interface for annotation, wherein, based on the annotation operation in the point cloud map of a specified frame, the corresponding object is annotated on the point cloud map by setting a graphic, and the same object in the continuous multiple frames after the point cloud map of the specified frame is automatically annotated, and in at least one surveillance image frame associated with the point cloud map of the specified frame, the object is annotated by setting a graphic, and the same object in the continuous multiple frames after the at least one surveillance image frame associated with the specified frame is automatically annotated; An information storage module is used to integrate a point cloud image containing annotated objects and a surveillance image containing the annotated objects into a grouping element information of the grouping annotation information, determine the grouping annotation information of the corresponding image group and store it, so as to query the grouping element information for each annotated object, determine the entry containing the annotated object, and facilitate unified updating of the entry. Each object in the grouping annotation information corresponds to a grouping element information, and the grouping element information includes: an attribute field, which describes the attributes of the object and / or the common attributes of the image to which the object belongs; when the object has been annotated in continuous frame images, the image identifier of the continuous frame image is recorded in the image field of the grouping element information of the object in a compressed manner to compress the data volume.

33. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, cause the processor to perform the method according to one or more of claims 1-15.

34. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method of one or more of claims 1-15.

35. An electronic device, characterized in that: include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as claimed in one or more of claims 16-23.

36. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method of one or more of claims 16-23.

37. An electronic device, characterized in that: include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as claimed in one or more of claims 24-27.

38. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method of one or more of claims 24-27.

39. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to perform the method according to claim 28.

40. One or more machine-readable media having executable code stored thereon, which when executed, causes a processor to perform the method of claim 28.

Images