Image labeling method and device, storage medium and electronic device

Abstract

Disclosed are an image labeling method and device, a storage medium and equipment, wherein the method comprises: obtaining a three-dimensional point cloud of a target scene and at least one to-be-labeled image sequence, labeling static elements in the three-dimensional point cloud of the target scene to obtain labeling information of the static elements in the three-dimensional point cloud, and then projecting the labeling information of each static element in the three-dimensional point cloud to at least one to-be-labeled image set corresponding to the target scene, thereby realizing labeling of static elements in the at least one to-be-labeled image set corresponding to the target scene. Based on the embodiments of the present disclosure, labeling of static elements in all to-be-labeled images in the at least one to-be-labeled image set corresponding to the target scene can be realized at one time, and compared with labeling of to-be-labeled images corresponding to different orientations of the target scene one by one, the image labeling efficiency can be effectively improved.

Description

Technical Field

[0001] This disclosure relates to data annotation technology, and in particular to an image annotation method and apparatus, storage medium and device. Background Technology

[0002] During autonomous vehicle operation, multiple perception and prediction modules are required to provide accurate information about the vehicle's surrounding environment. Perception tasks not only need to detect dynamic objects such as vehicles and pedestrians, but also identify static elements such as road layout and lane markings. In practical applications, a BEV (Birds-Eye View) perception model can be used to convert visual information from multiple cameras or radars on the vehicle into a bird's-eye view for relevant perception tasks. This provides a wider field of view for autonomous driving perception and allows for the simultaneous completion of multiple perception tasks. In related technologies, to ensure the accuracy of the BEV perception model, images acquired by visual sensors facing different orientations are typically labeled individually. Current methods mostly rely on manual labeling, which suffers from low labeling efficiency and poor labeling quality. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides an image annotation method and apparatus, a storage medium, and an electronic device.

[0004] One aspect of this disclosure provides an image annotation method, comprising: acquiring a three-dimensional point cloud corresponding to a target scene and at least one sequence of images to be annotated, wherein each sequence of images to be annotated comprises multiple frames of images to be annotated, the multiple frames of images to be annotated being sorted based on the image acquisition time or the driving trajectory of the mobile device where the image acquisition sensor is located; performing static feature annotation on the three-dimensional point cloud to obtain annotation information of at least one static feature; and projecting the annotation information of the at least one static feature onto the at least one sequence of images to be annotated to obtain at least one labeled image sequence with static feature annotation information.

[0005] In another aspect of this disclosure, an image annotation apparatus is provided, comprising: an acquisition module, configured to acquire a three-dimensional point cloud corresponding to a target scene and at least one sequence of images to be annotated, wherein each sequence of images to be annotated comprises multiple frames of images to be annotated, the multiple frames of images to be annotated being sorted based on the image acquisition time or the driving trajectory of the mobile device where the image acquisition sensor is located; an annotation module, configured to perform static element annotation on the three-dimensional point cloud to obtain annotation information of at least one static element; and a projection module, configured to project the annotation information of the at least one static element onto the at least one sequence of images to be annotated to obtain at least one labeled image sequence with static element annotation information.

[0006] In another aspect, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the image annotation method described in any embodiment of the present disclosure.

[0007] In another aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image annotation method according to any embodiment of this disclosure.

[0008] Based on the embodiments of this disclosure, by acquiring a 3D point cloud of a target scene, static element annotation is performed on the 3D point cloud of the target scene to obtain annotation information of at least one static element. Then, the annotation information of the at least one static element is projected onto at least one unannotated image sequence corresponding to the target scene, thereby realizing the static element annotation of at least one unannotated image sequence corresponding to the target scene, and obtaining at least one labeled image sequence with static element annotation information. Based on the embodiments of this disclosure, static element annotation of all images in at least one unannotated image sequence corresponding to the target scene can be realized at once. Compared with annotating images of different orientations corresponding to the target scene one by one, it can effectively improve the image annotation efficiency. Furthermore, it can avoid the misalignment of elements caused by different annotations of the same static element in images of different orientations corresponding to the target scene, and can effectively improve the image annotation quality. When the labeled image sequence is used to train the BEV perception model, it can effectively improve the performance of the trained BEV perception model. Attached Figure Description

[0009] Figure 1 A schematic diagram of an application scenario of an embodiment of this disclosure is shown.

[0010] Figure 2 This is a flowchart illustrating an exemplary embodiment of the image annotation method provided in this disclosure.

[0011] Figure 3 This is an exemplary flowchart illustrating the process of obtaining at least one sequence of images to be labeled in an embodiment of this disclosure.

[0012] Figure 4 This is another exemplary flowchart illustrating the process of obtaining at least one sequence of images to be labeled in this embodiment of the present disclosure.

[0013] Figure 5 This is a flowchart illustrating an image annotation method provided in another exemplary embodiment of this disclosure.

[0014] Figure 6 This is an exemplary schematic diagram of the first annotation information of the three-dimensional point cloud and static features in the embodiments of this disclosure.

[0015] Figure 7 This is a flowchart illustrating step S221 provided in an exemplary embodiment of this disclosure.

[0016] Figure 8 This is an exemplary flowchart illustrating the process of obtaining the remaining elements of a target static element according to an embodiment of this disclosure.

[0017] Figure 9 This is a flowchart illustrating step S230 provided in an exemplary embodiment of this disclosure.

[0018] Figure 10 This is a flowchart illustrating step S231 provided in an exemplary embodiment of this disclosure.

[0019] Figure 11 This is a flowchart illustrating step S232 provided in an exemplary embodiment of this disclosure.

[0020] Figure 12 This is a flowchart illustrating an image annotation method provided in yet another exemplary embodiment of this disclosure.

[0021] Figure 13 This is a flowchart illustrating an image annotation method provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 14 This is a schematic diagram illustrating an application example of the image annotation method according to an embodiment of this disclosure.

[0023] Figure 15 This is a flowchart of another application example of the image annotation method according to the embodiments of this disclosure.

[0024] Figure 16 This is an exemplary schematic diagram of a three-dimensional point cloud containing an labeled image and second annotation information labeled with static features, as described in an embodiment of this disclosure.

[0025] Figure 17 This is a schematic diagram of the structure of an image annotation device provided in an exemplary embodiment of this disclosure.

[0026] Figure 18 This is a schematic diagram of the structure of an image annotation device provided in another exemplary embodiment of the present disclosure.

[0027] Figure 19 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0028] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0029] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0030] Application Overview

[0031] The large amount of training data required for training the BEV perception model is usually generated by manually annotating images collected by visual sensors from different orientations. This process often results in repeated annotation of the same static feature in images collected by visual sensors from different orientations, leading to low annotation efficiency. Furthermore, annotating images collected by visual sensors from different orientations may result in different annotations of the same feature in images collected by visual sensors from different orientations, causing feature misalignment. This leads to poor annotation quality, which will affect the training effect of the BEV perception model when the annotated images are used.

[0032] Therefore, improving the efficiency and quality of image annotation is an urgent problem to be solved.

[0033] Exemplary System

[0034] This disclosure can be applied to BEV perception model training, specifically to generating training samples required for BEV perception model training. Specifically, when training a BEV perception model for static feature detection, this disclosure can first be used to annotate at least one sequence of images to be labeled using image acquisition sensors (also known as vision sensors) with different orientations. The labeled image sequence is then used as training samples to train the BEV perception model until preset training completion conditions are met, thus obtaining the BEV perception model.

[0035] Figure 1 A schematic diagram illustrating an application scenario of an embodiment of this disclosure is shown. For example... Figure 1As shown, a mobile device 1 (e.g., a vehicle) is equipped with multiple image acquisition sensors 2 (e.g., cameras) facing different directions, and a lidar 3. The mobile device 1 travels through a target scene 4 at a certain speed. During the travel of the mobile device 1, each image acquisition sensor 2 on the mobile device 1 acquires images of the target scene 4 at a certain frequency, obtaining a sequence of images to be labeled. Thus, multiple image acquisition sensors 2 obtain multiple sequences of images to be labeled. Each sequence of images to be labeled includes multiple frames of images to be labeled, and the multiple frames of images to be labeled in each sequence are ordered according to the image acquisition time or the travel trajectory of the mobile device 1. During the travel of the mobile device 1, the lidar 3 can also acquire point cloud data of the target scene 4 at a certain frequency, obtaining a point cloud dataset. Based on the above multiple sequences of images to be labeled and / or point cloud datasets, a three-dimensional scene reconstruction is performed to obtain a three-dimensional (3D) point cloud of the target scene 4. Then, static element annotation is performed on the three-dimensional point cloud according to the embodiments of this disclosure to obtain the annotation information of each static element (e.g., road markings, curbs, traffic signs, traffic signals, buildings, etc.) in the three-dimensional point cloud. Then, the annotation information of each static element in the 3D point cloud is projected onto the above-mentioned multiple sequences of maps to be annotated, resulting in multiple sequences of labeled maps with static element annotation information.

[0036] Based on the embodiments of this disclosure, static feature annotation of all images in at least one image sequence to be annotated corresponding to the target scene can be achieved at one time. Compared with annotating images of different orientations corresponding to the target scene one by one, it can effectively improve the efficiency of image annotation. Furthermore, it can avoid feature misalignment caused by different annotations of the same static feature in images of different orientations corresponding to the target scene, and can effectively improve the quality of image annotation. When the annotated image sequence is used to train the BEV perception model, it can effectively improve the performance of the trained BEV perception model.

[0037] Exemplary methods

[0038] Figure 2 This is a flowchart illustrating an exemplary embodiment of the image annotation method provided in this disclosure. This embodiment can be applied to any electronic device, such as a local terminal device or a cloud server, and can also be distributed across terminal devices and cloud servers. Figure 2 As shown, the image annotation method of this embodiment includes the following steps:

[0039] Step S210: Obtain the 3D point cloud and at least one sequence of maps to be labeled corresponding to the target scene.

[0040] Each of the at least one image sequence to be labeled comprises multiple frames of images to be labeled, and each image sequence is acquired by an image acquisition sensor installed on a mobile device. The multiple frames of images to be labeled in each image sequence are ordered based on their respective image acquisition time or the travel trajectory of the mobile device.

[0041] In one implementation, image acquisition sensors facing different directions can be installed on the mobile device. As the mobile device travels along a certain trajectory within the target scene, each image acquisition sensor on the mobile device acquires a video stream of the target scene, or acquires multiple frames of images of the target scene at a certain frequency. For each frame of the video stream of the target scene acquired by each image acquisition sensor, or multiple frames extracted from the video stream according to a preset method (e.g., frame-by-frame extraction), or multiple frames directly acquired, they are sorted according to the image acquisition time or the travel trajectory of the mobile device to obtain a sequence of images to be labeled. The mobile device may include, but is not limited to, any one or more of the following: vehicles, robots, aircraft, ships, trains, etc. The image acquisition sensors may include, but are not limited to: monocular cameras, binocular cameras, depth cameras, etc.

[0042] In one implementation, a 3D scanning device mounted on a mobile device can be used to scan the target scene at a certain acquisition frequency to obtain multiple point cloud data with a temporal relationship. These multiple point cloud data can then be used to reconstruct the 3D scene, resulting in a 3D point cloud of the target scene. Alternatively, at least one sequence of images to be labeled can be used to reconstruct the 3D scene, obtaining a 3D point cloud of the target scene. The 3D scanning device may include, but is not limited to, LiDAR (Light Laser Detection and Ranging), structured light sensors, and TOF (Time of Flight) cameras.

[0043] Step S220: Perform static feature annotation on the 3D point cloud to obtain annotation information for at least one static feature.

[0044] The static elements in this disclosure can include, but are not limited to, any one or more of the following: road markings, curbs, traffic signs, traffic signals, buildings, greenery, and other static objects. Road markings can include, but are not limited to, any one or more of the following: ground arrows, lane lines, zebra crossings, etc. Correspondingly, vehicles, pedestrians, robots, ships, trains, and other moving objects are considered dynamic elements.

[0045] In some implementations, the annotation information of static features may include one or more of the following: a static feature identifier (ID), vector information, and attribute annotation information. The static feature identifier is used to uniquely identify a static feature. The vector information of the static feature includes data used to determine the position and / or shape of the static feature. The attribute annotation information of the static feature may include, but is not limited to, the type to which the static feature belongs, the category (also called subtype) under that type, the annotation graphic and color corresponding to the static feature, etc. The annotation graphic can be a geometric element such as a point, line, surface, or volume that matches the static feature. For example, the type to which the static feature belongs may be an arrow, and the category under that type may be a left-turn arrow, a straight arrow, a right-turn arrow, etc.

[0046] In one specific implementation, static elements of a 3D point cloud can be labeled using automatic and / or manual annotation methods.

[0047] Step S230: Project the annotation information of the at least one static feature onto the at least one sequence of maps to be annotated to obtain at least one sequence of labeled maps with static feature annotation information.

[0048] In this process, each of the at least one labeled map sequence includes multiple frames of labeled images, and each frame of labeled images contains annotation information of static features.

[0049] In one implementation, for any static feature in a 3D point cloud, the position coordinates of the static feature in the world coordinate system (or vehicle coordinate system, i.e., mobile device coordinate system) can be transformed into position coordinates in the image coordinate system corresponding to each image sequence to be labeled. Then, based on the position coordinates of the static feature in the image coordinate system, the pixel (hereinafter referred to as the target pixel) corresponding to the static feature in the image sequence to be labeled is determined, and the labeling information of the static feature is projected onto the target pixel to achieve the labeling of any static feature in each image sequence to be labeled.

[0050] In this embodiment, by acquiring a 3D point cloud of a target scene, static element annotation is performed on the 3D point cloud of the target scene to obtain annotation information of at least one static element. Then, the annotation information of the at least one static element is projected onto at least one unannotated image sequence corresponding to the target scene, thereby realizing the static element annotation of at least one unannotated image sequence corresponding to the target scene, and obtaining at least one labeled image sequence with static element annotation information. Based on this embodiment, static element annotation of all images in at least one unannotated image sequence corresponding to the target scene can be realized at once. Compared with annotating images of different orientations corresponding to the target scene one by one, it can effectively improve the image annotation efficiency. Furthermore, it can avoid element misalignment caused by different annotations of the same static element in images of different orientations corresponding to the target scene, and can effectively improve the image annotation quality. When the labeled image sequence is used to train the BEV perception model, it can effectively improve the performance of the trained BEV perception model.

[0051] In the embodiments of this disclosure, the three-dimensional point cloud of the target scene can be obtained in a variety of ways.

[0052] For example, in the first embodiment, a three-dimensional scene can be reconstructed based on at least one sequence of maps to be labeled corresponding to the target scene to obtain a three-dimensional point cloud of the target scene, wherein each point in the three-dimensional point cloud has a corresponding position coordinate.

[0053] In the first implementation, as a specific approach, for each frame of the image to be labeled in the sequence of images corresponding to the target scene, the depth information of the pixels in each frame of the image to be labeled and the intrinsic parameter information of the image acquisition sensor that acquired the image to be labeled can be obtained. Then, using the intrinsic parameter information of the image acquisition sensor and the depth information, the position coordinates of each key pixel in the image to be labeled, determined according to a preset method, are transformed from the image coordinate system to the world coordinate system (or vehicle coordinate system) to obtain several three-dimensional points. Then, using the above-mentioned several three-dimensional points, sparse reconstruction is performed through the SFM (Structure From Motion) algorithm, and dense reconstruction is performed through the MVS (Multi-View Stereo) algorithm to obtain the three-dimensional point cloud of the target scene. When the image acquisition sensor for acquiring the image to be labeled is not a depth camera, a pre-trained neural network for determining depth information can be used to determine the depth information of pixels in each frame of the image to be labeled. The neural network can be CNN (Convolutional Neural Networks), DCNN (Deep Convolutional Neural Network), etc., and this disclosure does not limit it.

[0054] In this embodiment, the 3D point cloud obtained by reconstructing a 3D scene using multiple sequences of images to be labeled can be fused and aligned. This not only ensures that the points in the 3D point cloud correspond one-to-one with the pixels in the sequences of images to be labeled, improving the accuracy of the projection of the annotation information of static elements in the 3D point cloud onto the sequences of images to be labeled, but also improves the fusion effect of static elements in multiple sequences of labeled images, thereby improving the annotation quality of the sequences of labeled images.

[0055] In the second implementation, the target scene can be scanned at a certain frequency by a 3D scanning device (such as LiDAR) on the mobile device during its operation. Each set of point cloud data obtained from the scan can carry the time of the scan. The multiple sets of point cloud data obtained from the multiple scans are sorted according to the scan time or the trajectory of the mobile device to obtain the point cloud dataset of the target scene.

[0056] Preprocessing such as denoising and repair can be selectively performed on each point cloud data in the point cloud dataset to obtain the target point cloud dataset. Then, combined with algorithms such as ICP (Iterative Closest Point) and SLAM (Simultaneous Localization and Mapping), a 3D scene is reconstructed based on the target point cloud dataset to obtain a 3D point cloud of the target scene, in which each point has corresponding position coordinates.

[0057] Figure 3 This is an exemplary flowchart illustrating a process for obtaining at least one sequence of images to be labeled, as described in an embodiment of this disclosure. Figure 3 As shown, at least one sequence of maps to be labeled can be obtained in the following way:

[0058] Step S310: Multiple video streams are obtained by acquiring images of the target scene simultaneously using multiple image acquisition sensors with different preset orientations on the mobile device.

[0059] Each of these multiple video streams comprises multiple frames. The preset orientation can be set according to actual needs, and may include, but is not limited to, any one or more of the following orientations: forward view, left forward view, left rear view, right forward view, right rear view, and rear view. The target scene can be captured simultaneously by front view image acquisition sensors, left front view image acquisition sensors, left rear view image acquisition sensors, right front view image acquisition sensors, right rear view image acquisition sensors, and rear view image acquisition sensors, resulting in video streams of the target scene in six orientations.

[0060] Step S320: Based on multiple video streams, at least one sequence of images to be labeled is obtained.

[0061] Each of the at least one image sequence to be labeled includes multiple frames of images to be labeled that have a temporal relationship.

[0062] In one specific implementation, the multiple frames of images included in each video stream can be directly used as a corresponding sequence of images to be labeled.

[0063] Alternatively, in another specific implementation, for each video stream, multiple frames can be selected as images to be labeled according to a preset time interval or frame interval (e.g., one frame is selected every k frames, where k is an integer greater than 0), and these multiple images to be labeled are sorted in chronological order to obtain a sequence of images to be labeled corresponding to the video line.

[0064] Optionally, after obtaining at least one sequence of images to be labeled, interpolation and other methods can be used to perform time alignment on the images to be labeled in each sequence, and to perform denoising, sharpening, cropping, etc. on each image to be labeled, so as to improve the image quality of the images to be labeled.

[0065] In this embodiment of the disclosure, the target scene is captured by multiple image acquisition sensors with different preset orientations on a mobile device to obtain multiple video streams. Then, multiple image sequences to be labeled corresponding to the target scene are obtained based on the multiple video streams, thereby improving the acquisition efficiency of multiple image sequences to be labeled corresponding to the target scene.

[0066] Figure 4 This is another exemplary flowchart illustrating the process of obtaining at least one sequence of images to be labeled, as described in this disclosure. Figure 4 As shown, at least one sequence of maps to be labeled can also be obtained in the following way:

[0067] Step S410: Obtain the point cloud dataset of the target scene using a 3D scanning device on a mobile device.

[0068] In this embodiment, at least one 3D scanning device can be installed at a preset location on the mobile device to collect data on the target scene. The preset location may include, but is not limited to, the top, front, left, and right sides of the mobile device. In one implementation, the 3D scanning device on the mobile device can collect data on the target scene at a preset acquisition frequency to obtain multiple point cloud data sets of the target scene. These multiple point cloud data sets are then arranged in chronological order to obtain a point cloud dataset.

[0069] Step S420: Convert each point cloud data in the target scene into an image to obtain multiple bird's-eye view images.

[0070] Among them, bird's-eye view images are three-dimensional drawings created by looking down at the undulations of the ground from a high point using the perspective principle. They are images seen from the air when looking down at a certain area and are more realistic than two-dimensional images.

[0071] For each point cloud data point in the point cloud dataset, a pre-trained third neural network for converting point clouds into bird's-eye views can be used to transform the point cloud data into a bird's-eye view. Alternatively, the point cloud data can be converted into a bird's-eye view through coordinate system transformation. For example, the X-axis of the coordinate system corresponding to the bird's-eye view can be set to be opposite to the Y-axis of the radar coordinate system in the point cloud data, and the Y-axis of the coordinate system corresponding to the bird's-eye view can be set to be opposite to the X-axis of the radar coordinate system in the point cloud data. Then, the coordinate system is transformed to obtain the corresponding bird's-eye view. The third neural network can be a CNN, DCNN, etc.

[0072] The above transformation is performed on each point cloud data in the point cloud dataset to obtain multiple images corresponding to the target scene.

[0073] Step S430: Based on multiple bird's-eye view images, at least one sequence of images to be labeled is constructed.

[0074] Multiple bird's-eye view images can be arranged in chronological order to obtain at least one sequence of images to be labeled.

[0075] In this embodiment of the disclosure, multiple bird's-eye view images of the target scene are obtained by converting the point cloud data of the target scene, and these multiple bird's-eye view images are used to form a sequence of images to be labeled. This not only broadens the methods for obtaining the sequence of images to be labeled of the target scene, but also saves image labeling costs because the point cloud data of the target scene can also be used to construct the 3D point cloud of the target scene.

[0076] Figure 5 This is a flowchart illustrating an image annotation method provided in another exemplary embodiment of this disclosure. For example... Figure 5 As shown, in an optional embodiment, step S220 includes the following steps:

[0077] Step S221: Using the pre-trained first perception model, perform semantic segmentation and static feature extraction on the 3D point cloud of the target scene to obtain the first annotation information of at least one static feature.

[0078] In this embodiment of the disclosure, the first annotation information of each static element may include: the semantic segmentation result of the static element and its position information in the 3D point cloud. The position information of the static element in the 3D point cloud may include the position information of points belonging to the static element in the 3D point cloud.

[0079] The first perception model can be set up on a local server or in the cloud. In one implementation, the first perception model can first extract features from the 3D point cloud based on preset static elements, and then perform semantic segmentation based on the extracted feature data to obtain the semantic segmentation results of each static element. The semantic segmentation result of each static element can include the points in the 3D point cloud that belong to that static element (with position coordinates) and the category identifier (ID) of that static element. The category identifier of each static element is used to identify the static element belonging to that category and can be preset. For example, the category identifier of a lane line can be set to white. In a specific implementation, the first perception model can be implemented through a feature extraction network and a semantic segmentation network. The feature extraction network can include, but is not limited to, CNN, DCNN, etc., and the semantic segmentation network can include, but is not limited to, FCN (Fully Convolutional Networks), SegNet, etc.

[0080] In this embodiment of the disclosure, the first perception model can be trained using a 3D point cloud with a large number of semantic segmentation results annotated with static features.

[0081] Optionally, after obtaining the semantic segmentation results of each static element, the points belonging to each static element can be vectorized based on the semantic segmentation results of each static element to obtain the vectorized information of each static element. In this case, the position information of the static element in the three-dimensional point cloud can include the vectorized information of the static element in the three-dimensional point cloud in the first annotation information of each static element.

[0082] In this embodiment of the disclosure, the three-dimensional point cloud of the target scene and the first annotation information of each static element form a point cloud map of the target scene.

[0083] Step S222: Display the 3D point cloud of the target scene and the first annotation information of at least one of the above static elements through a visual interactive interface.

[0084] In one implementation, the annotation information of the 3D point cloud of the target scene and at least one static feature can be visualized so that the 3D point cloud of the target scene and the annotation information of at least one static feature can be displayed on a visual interactive interface. For example, the first annotation information of the 3D point cloud of the target scene and at least one static feature can be visualized using the visualization module in PCL (Point Cloud Library) and Cloud Compare visualization software.

[0085] The aforementioned visual interactive interface can be used to display the 3D point cloud of the target scene and the first annotation information of each static element, and can also be used to edit the displayed 3D point cloud and the first annotation information of each static element according to the user's operation instructions. The visual interactive interface can display data through a monitor (e.g., LCD monitor, plasma display) and can receive user operation instructions through a keyboard, mouse, touch screen, etc.

[0086] For example, Figure 6 This is an exemplary schematic diagram of the first annotation information of the 3D point cloud and static features in an embodiment of this disclosure. For example... Figure 6 As stated above, A indicates the first annotation information of the road surface arrow, which belongs to the static element; B indicates the first annotation information of the lane line, which belongs to the static element; and C indicates the first annotation information of the road, which belongs to the static element.

[0087] Step S223: Receive annotation instructions for the target static elements in the 3D point cloud of the target scene through a visual interactive interface.

[0088] The target static element is one of at least one static element, and it is the static element that will be annotated based on the annotation instructions. In the specific implementation, the user can select a static element as the target static element based on the first annotation information of the static elements in the visual interactive interface.

[0089] Step S224: Based on the first annotation information and annotation instructions of the target static element, the target static element is annotated in the 3D point cloud to obtain the second annotation information of the target static element.

[0090] The second annotation information includes attribute annotation information. This attribute annotation information may include, but is not limited to, the type to which the static feature belongs, the category (or subtype) under that type, the annotation graphic and color corresponding to the static feature, etc. The annotation graphic may be a geometric element such as a point, line, surface, or volume that matches the static feature.

[0091] In one implementation, the target static element in the 3D point cloud is labeled with attribute labeling information according to the first labeling information of each static element and the labeling instruction indicating the target static element to be labeled, thereby obtaining the second labeling information of the target static element.

[0092] Accordingly, in step S230 of this embodiment, the second annotation information of the target static feature in the 3D point cloud can be projected onto the at least one image sequence to be annotated, to obtain at least one labeled image sequence. In a specific implementation, the pixel corresponding to the target static feature in the image to be annotated can be determined by coordinate system transformation based on the position information of the target static feature in the 3D point cloud. Then, the second annotation information of the target static feature in the 3D point cloud is projected onto the pixel corresponding to it in the image to be annotated, so as to complete the annotation of the target static feature on the image to be annotated.

[0093] In a specific implementation, one static element can be selected from the above-mentioned at least one static element as the target static element, and the above operation can be performed to achieve the annotation of the image to be annotated.

[0094] In this embodiment, a first perception model is used to perform semantic segmentation and static feature extraction on static features in a 3D point cloud, obtaining first annotation information for the static features. This achieves automatic annotation of static features. Then, attribute annotation information is added to the static features based on the first annotation information, thereby achieving efficient, accurate, and comprehensive annotation of static features. Furthermore, visualizing the 3D point cloud and the first annotation information helps users intuitively annotate the static features based on the first annotation information and determine the accuracy of the annotation results, thus improving annotation efficiency and accuracy.

[0095] Figure 7 This is a flowchart illustrating step S221 provided in an exemplary embodiment of this disclosure. Figure 7 As shown, in an optional embodiment, step S221 includes the following steps:

[0096] Step S2211: Divide the 3D point cloud into multiple point cloud blocks.

[0097] In one implementation, a 3D point cloud can be divided into multiple cloud blocks according to a preset segmentation rule. For example, the preset rule may include preset size information, which may include the width and height of the point cloud blocks to be segmented. For instance, the 3D point cloud can be divided into multiple point cloud blocks with a preset size of 300×300.

[0098] Step S2212: Using the first perception model, semantic segmentation and static feature extraction are performed on each point cloud block in the multiple point cloud blocks to obtain the static feature annotation information of each point cloud block.

[0099] In one implementation, for any point cloud block among all point cloud blocks, a first perception model is used to perform semantic segmentation and static feature extraction on that point cloud block to obtain static feature annotation information for that point cloud block. This static feature annotation information includes: the semantic segmentation result of the static feature and its position information in the 3D point cloud.

[0100] Step S2213: Aggregate the static feature annotation information of multiple point cloud blocks to obtain the first annotation information of at least one static feature.

[0101] Specifically, based on the semantic segmentation results of static elements in each point cloud block and their position information in the 3D point cloud, points belonging to the same static element in each point cloud block can be aggregated to obtain the overall static element, and the first annotation information of at least one static element can be obtained.

[0102] In this embodiment of the disclosure, the three-dimensional point cloud of the target scene is first divided into multiple point cloud blocks, and then the first perception model is used to perform semantic segmentation and static feature extraction on each point cloud block to obtain the static feature annotation information of each point cloud block. This reduces the difficulty and amount of data processed by the first perception model in a single operation, and helps to improve the annotation efficiency and the accuracy of the annotation results.

[0103] In an optional embodiment, step S223 of this disclosure may include: receiving a labeling instruction sent by a user after selecting target static features and setting attribute information in a 3D point cloud via a visual interactive interface. The labeling instruction includes the target static feature selection information and attribute setting information.

[0104] In one implementation, a user can select a target static feature from a 3D point cloud using the first annotation information of the target static feature displayed on a visual interactive interface, thereby sending the selection information for the target static feature. Preset attribute options can be preset, including but not limited to: multiple preset static feature types, and preset categories under each preset static feature type. The user can select the preset category / preset static feature type corresponding to the target static feature from the preset attribute options as the attribute setting information for the target static feature. For example, the aforementioned preset static feature type can be an arrow, and the preset categories under this preset static feature type can be left-turn arrows, straight arrows, right-turn arrows, etc.

[0105] Accordingly, in this embodiment, step S224 may include: determining the target static feature in the 3D point cloud based on the first annotation information of the target static feature and the selection information of the target static feature, and annotating the target static feature with attributes based on the attribute setting information of the target static feature in the annotation instruction.

[0106] In this embodiment, annotation instructions can be received through a visual interactive interface. By selecting the target static element and setting the attribute information in the annotation instructions, the target static element in the 3D point cloud can be selected and the attribute annotation information can be annotated. This realizes the annotation of the attribute annotation information of the target static element in the 3D point cloud. Visual annotation can improve the accuracy and efficiency of annotation of static elements.

[0107] In an optional embodiment, the selection information for the target static element in this disclosure can be sent by the user after selecting the target static element using a geometric element that matches the target static element. The aforementioned geometric elements may include, but are not limited to, any of the following: points, lines, surfaces, volumes, etc.; where surfaces and volumes can be of any shape or a preset shape, for example, surfaces can be circular planes, square planes, curved surfaces, etc., and volumes can be cubes, cylinders, cones, etc. In this case, the selection information for the target static element may include geometric information of the target geometric element used to select the target static element. This geometric information may include the shape information of the target geometric element and the position coordinates of the target geometric element in the 3D point cloud (for example, for a square plane, this could be the coordinates of two diagonal points, etc.). The target geometric element is the geometric element specifically used to select the target static element.

[0108] The matching principle between the target static feature and the geometric element can be set according to actual needs. For example, it can be set that lane lines are matched with lines as the geometric element for static features, and road surface arrows are matched with rectangular planes as the geometric element for static features, etc.

[0109] In one implementation, geometric element options can be preset, from which the user can select the corresponding geometric elements, and then use the selected geometric elements to select the target static element and confirm, thereby sending the selection information of the target static element.

[0110] Accordingly, in this embodiment, in step S224, the target static element corresponding to the target geometric element can be determined in the three-dimensional point cloud based on the first annotation information of the target static element and the geometric information of the target geometric element (i.e., the shape information of the target geometric element and the position coordinates of the target geometric element in the three-dimensional point cloud).

[0111] In this embodiment, users can select target static elements using geometric features through a visual interactive interface, thereby achieving intuitive, efficient, and accurate selection of target static elements, which helps to improve the selection efficiency and accuracy of static elements.

[0112] In an optional embodiment, the user's selection information for the target static element in the 3D point cloud may include the user's drawing information for the target static element in the 3D point cloud. Specifically, the user can directly draw the target static element on the 3D point cloud using geometric elements to obtain the drawing information.

[0113] For example, users can draw lane lines (corresponding to static elements) based on the first annotation information of the lane lines in the visual interactive interface, using lines to draw the lane lines along the extension direction of the lane lines in the 3D point cloud, from the starting position to the ending position of the lane lines, and obtain the drawing information of the lane lines.

[0114] In this embodiment, users can directly draw target static elements in a 3D point cloud through a visual interactive interface. Based on the drawing information input by the user, the selection of target static elements in the 3D point cloud can be realized.

[0115] Alternatively, in another optional embodiment, in step S223, selection information sent by the user after selecting a portion of the target static features on the 3D point cloud can be received.

[0116] In other words, users can use geometric elements on a 3D point cloud, through a visual interactive interface, to select only a portion of the features belonging to a target static feature. For example, for a lane line, they can select or draw only a small segment of the lane line, confirm, and then send the selection information. This portion of the feature can be the feature corresponding to the starting position of the target static feature in the 3D point cloud. For instance, users can use geometric elements to draw on the feature corresponding to the starting position of the target static feature in the 3D point cloud, or select the feature corresponding to the starting position of the target static feature in the 3D point cloud, and then send the selection information.

[0117] Accordingly, in this embodiment, in step S224, the remaining elements in the 3D point cloud that belong to the target static element can be determined based on the preset tracking configuration information, the first annotation information of the target static element, and the selection information.

[0118] The preset tracking configuration information may include: tracking step size. In a specific implementation, based on the position information in the first annotation information of the target static feature, a portion of the features corresponding to the selected information can be determined in the 3D point cloud. This portion of the features is used as the starting point for tracking. The extension direction of the target static feature is determined based on the semantic segmentation result in the first annotation information. Based on the tracking step size in the preset tracking configuration information, the remaining features belonging to the target static feature are tracked and determined in the aforementioned extension direction in the 3D point cloud. The remaining features and the portion of the features together form the target static feature in the 3D point cloud.

[0119] In this embodiment, the user only needs to select some elements of the target static element through the visual interactive interface. Based on the preset tracking configuration information, the first annotation information of the target static element and the selection information, the remaining elements belonging to the target static element can be automatically tracked and determined in the 3D point cloud. The user does not need to draw the complete target static element, thereby reducing manual operation, improving the user experience, improving the efficiency of determining the target static element in the 3D point cloud, and thus improving the annotation efficiency of the target static element.

[0120] Figure 8 This is an exemplary flowchart illustrating the process of obtaining the remaining elements of a target static element according to an embodiment of this disclosure.

[0121] like Figure 8 As shown, it may include the following steps:

[0122] Step S2241: Based on the preset tracking configuration information, the first annotation information of the target static element, and the selection information, determine a point cloud region in the 3D point cloud used to track the target static element, and use it as the current point cloud region.

[0123] The preset tracking configuration information may further include: point cloud region shape and size information. In one embodiment, the shape and area of ​​the current point cloud region can be determined based on the point cloud region shape and area information. Then, based on the position information in the first annotation information of the target static element, the target partial static element corresponding to the selected partial element in the 3D point cloud can be determined. After that, in the current point cloud region, the points belonging to the target static element can be tracked and scanned starting from the target partial static element.

[0124] Step S2242: Determine the partial features in the current point cloud region that belong to the target static features.

[0125] In one implementation, points in the current point cloud region that have the same semantic segmentation results as some elements tracked in the previous point cloud region can be determined based on the semantic segmentation results in the first annotation information of the target static element. These points are then identified as belonging to the target static element, and some elements belonging to the target static element are formed from the points in the current point cloud region that belong to the target static element.

[0126] Step S2243: Based on the preset tracking configuration information, the first annotation information of the target static feature, and some features belonging to the target static feature in the current point cloud region, determine the next point cloud region in the 3D point cloud used to track the target static feature, and use it as the current point cloud region.

[0127] In one implementation, the shape and size of the next point cloud region are determined based on the shape and size information of the point cloud region in the preset tracking configuration information. Specifically, the next point cloud region can be determined in the 3D point cloud along the tracking direction based on the tracking step size and the preset tracking configuration information. For example, the tracking direction can be determined based on the drawing direction and drawing boundary when drawing some features on the 3D point cloud, so that the next point cloud region is tracked along the tracking direction.

[0128] Step S2244 iterates the operation of step 2242 until no part of the current point cloud region belonging to the target static feature is determined, indicating that all the static features belonging to the target have been determined.

[0129] Then proceed to step S2245.

[0130] Step S2245: Fit and vectorize the partial elements belonging to the static elements of the target in the 3D point cloud to obtain the static elements of the target.

[0131] In one implementation, a subset of elements in the 3D point cloud that belong to the target static element—that is, the target static element and the remaining elements belonging to the target static element—are fitted using a preset fitting method (e.g., normal fitting) to obtain a fitting result. This fitting result is then vectorized to obtain the target static element. This target static element carries vector information. This vector information is the positional information of feature points used to determine the shape and position of the target static element. In this embodiment, using preset tracking configuration information, the first annotation information of the target static element, and selection information, iterative tracking of the subset of elements in the 3D point cloud belonging to the target static element is achieved until all elements belonging to the target static element are determined. Then, all the subsets in the 3D point cloud belonging to the target static element are fitted and vectorized to obtain the complete target static element. Thus, accurate tracking and precise positioning of the target static element in the 3D point cloud are achieved.

[0132] Figure 9 This is a flowchart illustrating step S230 provided in an exemplary embodiment of this disclosure. Figure 9 As shown, in an optional embodiment, step S230 includes the following steps:

[0133] Step S231: Determine the images to be labeled that correspond to the target static features in the 3D point cloud from at least one of the above-mentioned image sequences to be labeled.

[0134] In this context, the image to be labeled corresponding to the target static feature in the 3D point cloud is the image to be labeled that includes the target static feature. For example, when the left lane line in the 3D point cloud is the target static feature, the image to be labeled that includes the left lane line in the at least one image sequence to be labeled is the image to be labeled that corresponds to the left lane line in the 3D point cloud.

[0135] In one implementation, a pre-trained neural network for detecting static features can be used to perform static feature detection on each image in each sequence of images to be labeled, and the images to be labeled that include the target static features can be determined based on the detection results. Here, the neural network for detecting static features may include, but is not limited to, CNN, DCNN, etc.

[0136] Step S232: Project the second annotation information of the target static element in the 3D point cloud onto the corresponding image to be annotated.

[0137] In this way, the second annotation information of the target static element in the 3D point cloud can be projected onto the corresponding image to be annotated by means of coordinate system transformation.

[0138] In this embodiment, the second annotation information of the target static element in the three-dimensional point cloud can be projected onto the corresponding image to be annotated, thereby realizing the annotation of the target static element of the above-mentioned at least one image sequence to be annotated.

[0139] Figure 10 This is a flowchart illustrating step S231 provided in an exemplary embodiment of this disclosure. Figure 10 As shown, in an optional embodiment, step S231 may include the following steps:

[0140] Step S2311: Obtain the position of the target static element in the 3D point cloud in the preset coordinate system.

[0141] The preset coordinate system can be set according to actual needs. For example, it can be the radar coordinate system corresponding to the 3D point cloud, or the vehicle coordinate system or the world coordinate system.

[0142] In one implementation, each point in the 3D point cloud has a position in a preset coordinate system. The position of each point in the 3D point cloud can be determined when acquiring point cloud data for reconstructing the 3D point cloud.

[0143] Step S2312: Obtain the position of each image to be labeled in at least one image sequence in the above-mentioned preset coordinate system.

[0144] In this process, when collecting each image to be labeled in at least one image sequence, a positioning device, such as GPS (Global Positioning System), can be used simultaneously to obtain the location information of the collection location corresponding to each image to be labeled in a preset coordinate system, and the location of the corresponding image to be labeled can be determined based on the obtained location information.

[0145] For example, multiple image acquisition sensors and at least one positioning device can be set on a mobile device. For each of the multiple image acquisition sensors, the image acquisition sensor acquires the image to be labeled at a certain frequency, and at the same time, the positioning device also determines the currently detected position information at a certain frequency, and associates each image to be labeled with a position information according to the time sequence or the acquisition sequence, thereby obtaining the position of each image to be labeled in a preset coordinate system.

[0146] Step S2313: Based on the position of the target static element in the 3D point cloud in the preset coordinate system and the position of each image to be labeled in the preset coordinate system, determine the image to be labeled corresponding to the target static element in the 3D point cloud.

[0147] In one implementation, a preset threshold can be set in advance, and the position of the target static feature can be compared with the position in each image to be labeled. The image to be labeled that has a difference between the two that is less than or equal to the preset threshold is determined as the image to be labeled corresponding to the target static feature.

[0148] In this embodiment, the image to be labeled corresponding to the target static element can be quickly located by the position of the target static element in the preset coordinate system and the position of each image to be labeled in the preset coordinate system, so as to improve the overall labeling efficiency.

[0149] Figure 11 This is a flowchart illustrating step S232 provided in an exemplary embodiment of this disclosure. Figure 11 As shown, in an optional embodiment, step S232 may include the following steps:

[0150] Step S2321: Obtain the intrinsic and extrinsic parameter information of each image acquisition sensor corresponding to each image sequence to be labeled.

[0151] For each image sequence to be labeled, the intrinsic and extrinsic information of the image acquisition sensor that acquired the image sequence is obtained. The intrinsic information includes: horizontal focal length, vertical focal length, position of the optical center of the horizontal imaging plane in the pixel coordinate system, and position of the optical center of the vertical imaging plane in the pixel coordinate system. The extrinsic information includes: rotation matrix and translation matrix.

[0152] Step S2322: Based on the intrinsic and extrinsic parameters of each image acquisition sensor and the position of the target static element in the preset coordinate system, the second annotation information of the target static element is projected onto the corresponding image to be annotated.

[0153] Specifically, the position of the target static element in the preset coordinate system can be transformed to the image coordinate system of the corresponding image to be labeled, so as to obtain the position coordinates of the target static element in the image coordinate system. Then, based on the position coordinates of the target static element in the image coordinate system, the corresponding pixel in the corresponding image to be labeled is determined as the target pixel, and the second labeling information of the target static element is projected onto the target pixel, that is, the second labeling information of the target static element is labeled on the target pixel.

[0154] In one implementation, the preset coordinate system can be the world coordinate system. The position of the target static feature in the preset coordinate system can be transformed to the pixel coordinate system to obtain the coordinates of the target static feature in the pixel coordinate system. This method specifically includes:

[0155] The coordinates of the target static element in the pixel coordinate system can be determined according to the formulas (1)-(3) shown below. Specifically, any point in the target static element can be selected as a key point to represent the target static element. For example, the vertex of the target static element can be selected as the key point of the target static element. The following uses the target static element A as an example for explanation. The image acquisition sensor that acquires the image to be labeled corresponding to the target static element A is called the target image acquisition sensor. P0(x0,y0,z0) is the coordinate of the target static element A in the world coordinate system (corresponding to the position of the target static element in the preset coordinate system).

[0156] First, P0(x0,y0,z0) is transformed to the camera coordinate system using equation (1) to obtain the coordinates P1(x1,y1,z1) of the target static element A in the camera coordinate system; then, P1(x1,y1,z1) is transformed to the image coordinate system using equation (2) to obtain the coordinates P2(u) of the target static element A in the image coordinate system. x ,u y ,u z Then, P2(u) is converted using equation (3). x ,u y ,u z Transform to pixel coordinates to obtain the coordinates P3(d) of the target static feature A in the pixel coordinate system. x ,d y ).

[0157]

[0158]

[0159] P3=[d x d y ] = [u x u y u z ] / u z =P0×T (3)

[0160] Where R is the rotation matrix of the target image acquisition sensor with a size of 3×3, and T is the translation matrix of the target image acquisition sensor with a size of 3×1. R and T constitute the extrinsic parameter matrix of the target image acquisition sensor. f x f is the horizontal focal length of the target image acquisition sensor. y c is the vertical focal length of the sensor used to acquire the target image. x c represents the position of the optical center of the horizontal imaging plane of the target image acquisition sensor. y The position of the optical center of the vertical imaging plane of the image acquisition sensor; The intrinsic parameter matrix of the sensor for acquiring the target image.

[0161] In this embodiment, the second annotation information of the target static element is projected onto the corresponding image to be annotated by using the position of the target static element in a preset coordinate system and the intrinsic and extrinsic parameters of the image acquisition sensor, thereby completing the annotation of the image to be annotated. This achieves efficient and accurate annotation of the image to be annotated.

[0162] Figure 12 This is a flowchart illustrating an image annotation method provided in yet another exemplary embodiment of this disclosure. For example... Figure 12 As shown, in an optional embodiment, the image annotation method further includes the following steps:

[0163] Step S510: Display the three-dimensional point cloud, the first annotation information of at least one of the above-mentioned static elements, and the second annotation information of the target static element through a visual interactive interface.

[0164] Specifically, the first annotation information of the 3D point cloud, at least one static element, and the second annotation information of the target static element are visualized to obtain a visualization result, which is then displayed to achieve real-time display of the first annotation information of the 3D point cloud, at least one static element, and the second annotation information of the target static element.

[0165] Step S520: In response to receiving a label modification instruction sent by the user through a visual interactive interface, the label modification instruction is used to modify the second label information of the target static feature in the 3D point cloud.

[0166] In one implementation, the annotation modification command may include modification information for the position of static features, modification information for attribute annotation information, etc. The user displays a 3D point cloud, first annotation information for at least one static feature, and second annotation information for the target static feature through a visual interactive interface. The user can then confirm whether there are any annotation errors in the second annotation information of the target static feature. If an error is found, a annotation modification command can be sent to modify the second annotation information of the target static feature.

[0167] Step S530: Based on the first annotation information and annotation modification instructions of at least one static element, modify the second annotation information of the target static element in the 3D point cloud, and project the modification result of the second annotation information of the target static element onto the at least one image sequence to be annotated.

[0168] Specifically, based on the first annotation information and annotation modification instructions of at least one static element, the second annotation information of the target static element is modified in the 3D point cloud to obtain the modification result of the second annotation information. Then, the modification result of the second annotation information of the target static element can be projected onto the image to be annotated corresponding to the target static element in real time so that the user can confirm whether the modified second annotation information is accurate.

[0169] In this embodiment, a visual interactive interface can be used to display the first annotation information of a 3D point cloud and at least one static feature, as well as the second annotation information of the target static feature. This allows the user to confirm whether there are any annotation errors in the second annotation information of the target static feature, and if errors are found, to modify the second annotation information and immediately project the modified second annotation information into the corresponding sequence of images to be annotated. This achieves real-time visual display and correction of the second annotation information of the target static feature, helping to improve the accuracy of the annotation results.

[0170] Figure 13 This is a flowchart illustrating an image annotation method provided in yet another exemplary embodiment of this disclosure. For example... Figure 13 As shown, after obtaining at least one labeled map sequence with static feature annotation information through any embodiment of this disclosure, the following steps may also be included:

[0171] Step S610: Receive playback instructions sent by the user through a visual interactive interface.

[0172] This playback command is used to play at least one labeled map sequence and the corresponding 3D point cloud (i.e., carrying the first label information, also known as a point cloud map) in a certain playback order (e.g., in forward or reverse chronological order, in forward or reverse chronological order, etc., according to the travel trajectory of the mobile device).

[0173] Step S620: Based on the playback command, play back at least one of the above-mentioned labeled map sequences and 3D point clouds.

[0174] Among them, a visual interactive interface can be used to synchronously play the 3D point cloud and at least one corresponding labeled map sequence.

[0175] Step S630: In response to receiving a label modification instruction sent by the user through a visual interactive interface.

[0176] The annotation modification command is used to modify the second annotation information of a specified static feature in a 3D point cloud.

[0177] In one implementation, while playing the aforementioned 3D point cloud and at least one corresponding labeled image sequence on a visual interactive interface, the user can confirm whether there are errors in the second annotation information of static features in the labeled images. For example, the user can quantitatively determine whether there are errors in the second annotation information of static features in the labeled images using methods such as projection difference or pixel difference. Alternatively, the user can manually check whether there are errors in the second annotation information of static features in the labeled images. When it is determined that there is erroneous second annotation information, the static feature with the erroneous second annotation information is designated as the specified static feature, and a labeling modification instruction is sent to modify the second annotation information of the specified static feature.

[0178] Step S640: Based on the first annotation information and annotation modification instructions of the specified static element, modify the second annotation information of the specified static element in the 3D point cloud, and project the modification result of the second annotation information of the specified static element onto the above-mentioned at least one sequence of maps to be annotated.

[0179] Specifically, erroneous second annotation information can be deleted, and then the second annotation information of the target static element can be re-annotated using the method of annotating the target static element in any of the above embodiments of this disclosure. During the modification process, the annotation process and modification results of the second annotation information of the specified static element can be displayed in real time through a visual interactive interface.

[0180] Step S650: In response to receiving a annotation instruction sent by the user through a visual interactive interface.

[0181] The annotation command is used to annotate the second annotation information of specified static features in a 3D point cloud. When a user determines that annotation is needed for the second annotation information of a static feature, they send the annotation command. Annotations for the second annotation information of a static feature can include explanations, clarifications, or supplementary information. For example, annotations for the second annotation information of a static feature can include the annotation date and modification information.

[0182] Step S660: Annotate the second annotation information of the specified static feature in the 3D point cloud.

[0183] In this process, the second annotation information of a specified static feature in the 3D point cloud is annotated, that is, the second annotation information that is incorrect is annotated, so that the incorrect second annotation information can be uniformly modified in the future.

[0184] It should be noted that in the embodiments of this disclosure, steps S640 and S650 can be executed selectively, or both can be executed. There is no specific order in which steps S640 and S650 are executed.

[0185] In this embodiment, by synchronously replaying the 3D point cloud and at least one corresponding labeled image sequence on the visual interactive interface, it can be confirmed whether there is erroneous second labeling information in the labeled image. If there is erroneous second labeling information, it can be modified or annotated in time for subsequent unified modification.

[0186] In an optional embodiment, after obtaining at least one labeled image sequence based on the above embodiments of this disclosure, a second perception model can be trained using at least one labeled image sequence. After the second perception model is trained, it can be deployed to a mobile device for static feature detection based on images collected by sensors on the mobile device.

[0187] The second perceptual model mentioned above can be a neural network to be trained. For example, the neural network to be trained can include, but is not limited to: CNN, DCNN, LSTM (Long Short-Term Memory), RNN (Recurrent Neural Network), GAN (Generative Adversarial Network), GNN (Graph Neural Network), ResNet (Residual Network), DenseNet (Deep Network), EfficientNet (Time-Sensitive Network), etc.

[0188] In one implementation, each labeled image from at least one labeled image sequence can be sequentially input into a second perception model. The second perception model outputs prediction information for the static features corresponding to each labeled image. Based on the labeling and prediction information of the static features in each labeled image, a preset loss function is used to determine the loss value. The loss value can be reduced by adjusting the parameters of the second perception model so that the second perception model meets a preset training stopping condition, thereby obtaining a trained second perception model. For example, the parameters of the second perception model can be adjusted using a parameter optimizer. For instance, the parameter optimizer can be used to calculate the gradient of each parameter of the second perception model to be trained. The gradient represents the direction in which the loss value decreases the most. The parameters are adjusted along the gradient direction, and the above steps are repeated until the loss value no longer decreases, indicating that the preset training stopping condition has been met, thus obtaining a trained second perception model.

[0189] The parameter optimizer can include, but is not limited to, SGD (Stochastic Gradient Descent), Adagrad (Adaptive Gradient Descent), Adam (Adaptive Moment Estimation), RMSprop (Root Mean Square Prop), and LBFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno). The preset loss function can be the cross-entropy error function, the mean square error function, etc.

[0190] In one application instance Figure 14 This invention illustrates an application example of labeling an image sequence and training a second perceptual model using the resulting labeled image sequence, as described in this embodiment. Figure 15 A flowchart illustrating an application example of the image annotation method in this disclosure is shown.

[0191] In this application example, taking a mobile device or vehicle as an example, the first vehicle is equipped with six cameras (corresponding image acquisition sensors) facing preset directions and one LiDAR. The six directions can be: forward-looking direction, left forward-looking direction, left rear-looking direction, right forward-looking direction, right rear-looking direction, and rear-looking direction.

[0192] Specifically, such as Figure 14 and Figure 15As shown, the first vehicle travels through the target scene within a preset time period. During this journey, six cameras positioned on the first vehicle with preset orientations capture images of the target scene at a certain frequency, resulting in six video streams. Based on these six video streams, six sequences of images to be labeled are generated. Furthermore, during the vehicle's movement, a LiDAR system mounted on the first vehicle can also capture images of the target scene, obtaining a point cloud dataset of the target scene.

[0193] A 3D point cloud of the target scene can be constructed based on six sequences of images to be labeled. The 3D point cloud is then sent to the cloud, where a first perception model is used for semantic segmentation and static feature extraction to obtain the first annotation information for each static feature. The 3D point cloud and its first annotation information are then visualized, and the visualization results are displayed through an interactive interface, such as... Figure 6 As shown.

[0194] Through any of the above embodiments of this disclosure, target static elements are determined in a 3D point cloud, and attribute annotation information is added to the target static elements to obtain second annotation information for the target static elements. The second annotation information of the target static elements is projected onto six sequences of images to be annotated, resulting in six sequences of annotated images. The annotated image sequences and the 3D point cloud are replayed to determine whether there are any static elements in the annotated images that require modification of the annotation information. If so, the second annotation information of the static element requiring modification in the 3D point cloud is modified; otherwise, the annotation is completed.

[0195] A second perception model was trained using six labeled image sequences, and then deployed on a second vehicle. The trained second perception model can perform static feature detection on images captured by sensors on the second vehicle. Additionally, the six labeled image sequences can also be used to train a first perception model.

[0196] in, Figure 16 This is an example of content displayed on a visual interactive interface. The interface shows the first and second annotation information for a 3D point cloud and multiple lane lines (as multiple static elements). For example... Figure 16As shown, part a is a 3D point cloud (also called a point cloud map) with second annotation information of labeled static features. Parts bg are labeled images of this 3D point cloud from different viewpoints. Each labeled image in part bg is obtained by projecting the second annotation information of lane lines in the 3D point cloud in part a onto part bg. Specifically, part b is a labeled image in the labeled image sequence corresponding to the left front view, part c is a labeled image in the labeled image sequence corresponding to the right front view, part d is a labeled image in the labeled image sequence corresponding to the rear view, part e is a labeled image in the labeled image sequence corresponding to the front view, part f is a labeled image in the labeled image sequence corresponding to the left rear view, and part g is a labeled image in the labeled image sequence corresponding to the right rear view.

[0197] Exemplary device

[0198] Figure 17 This is a structural block diagram of an image annotation device according to one embodiment of this disclosure. For example... Figure 17 As shown, the image annotation device includes: an acquisition module 710, an annotation module 720, and a projection module 730.

[0199] The acquisition module 710 is used to acquire a 3D point cloud corresponding to the target scene and at least one sequence of images to be labeled. Each sequence of images to be labeled includes multiple frames of images to be labeled. The multiple frames of images to be labeled are sorted based on the image acquisition time or the driving trajectory of the mobile device where the image acquisition sensor is located.

[0200] The annotation module 720 is used to perform static feature annotation on the three-dimensional point cloud to obtain annotation information of at least one static feature.

[0201] The projection module 730 is used to project the annotation information of the at least one static element onto the at least one sequence of maps to be annotated, so as to obtain at least one sequence of labeled maps with static element annotation information.

[0202] In some optional examples, such as Figure 18 As shown, the annotation module 720 provided in the embodiments of this disclosure includes:

[0203] The segmentation and extraction submodule 721 is used to perform semantic segmentation and static feature extraction on the three-dimensional point cloud using a pre-trained first perception model to obtain first annotation information of at least one static feature.

[0204] Display submodule 722 is used to display the first annotation information of the three-dimensional point cloud and the at least one static element through a visual interactive interface;

[0205] The instruction receiving submodule 723 is used to receive, through the visual interactive interface, annotation instructions for a target static element in the three-dimensional point cloud, wherein the target static element is one of the at least one static elements.

[0206] The annotation submodule 724 is used to annotate the target static feature in the 3D point cloud based on the first annotation information and the annotation instruction, and obtain the second annotation information of the target static feature, wherein the second annotation information includes: attribute annotation information;

[0207] The projection module 730 provided in the embodiments of this disclosure is further configured to project the second annotation information onto the at least one sequence of images to be annotated, thereby obtaining the at least one sequence of annotated images.

[0208] In some optional examples, the segmentation and extraction submodule 721 in the above embodiments of this disclosure is further configured to segment the three-dimensional point cloud into multiple point cloud blocks; use the first perception model to perform semantic segmentation and static feature extraction on each of the multiple point cloud blocks to obtain static feature annotation information of each point cloud block; and aggregate the static feature annotation information of the multiple point cloud blocks to obtain the first annotation information of the at least one static feature.

[0209] In some optional examples, the instruction receiving submodule 723 in the above embodiments of this disclosure is further configured to receive, through the visual interactive interface, a labeling instruction sent by the user after selecting the target static element and setting the attribute information in the three-dimensional point cloud, the labeling instruction including: the target static element selection information and the attribute setting information;

[0210] In some optional examples, the annotation submodule 724 in the above embodiments of this disclosure is further configured to determine the target static element in the three-dimensional point cloud based on the first annotation information and the selection information of the target static element, and to annotate the target static element with attributes based on the attribute setting information of the target static element.

[0211] In some optional examples, the selection information of the target static element in the above embodiments of this disclosure is sent by the user after selecting the target static element using a geometric element that matches the target static element; the geometric element includes any one of the following: point, line, surface, volume; the selection information of the target static element includes geometric information of the target geometric element used to select the target static element;

[0212] The annotation submodule 724 in the above embodiments of this disclosure is further used to determine the target static feature corresponding to the target geometric element in the three-dimensional point cloud based on the first annotation information and the geometric information of the target geometric element.

[0213] In some optional examples, the user's selection information of the target static element in the three-dimensional point cloud in the above embodiments of this disclosure includes the user's drawing information of the target static element in the three-dimensional point cloud.

[0214] In some optional examples, the instruction receiving submodule 723 in the above embodiments of this disclosure is also used to receive selection information sent by the user after selecting a portion of the elements belonging to the target static element on the three-dimensional point cloud;

[0215] The annotation submodule 724 in the above embodiments of this disclosure is further used to determine the remaining elements in the three-dimensional point cloud that belong to the target static element based on preset tracking configuration information, the first annotation information and the selection information.

[0216] In some optional examples, the annotation submodule 724 in the above embodiments of this disclosure, based on the preset tracking configuration information, the first annotation information, and the selection information, determines a point cloud region in the 3D point cloud used for tracking the target static feature, as the current point cloud region; determines some features in the current point cloud region that belong to the target static feature; based on the preset tracking configuration information, the first annotation information, and some features in the current point cloud region that belong to the target static feature, determines the next point cloud region in the 3D point cloud used for tracking the target static feature, as the current point cloud region; iteratively executes the operation of determining some features in the current point cloud region that belong to the target static feature until no features in the current point cloud region that belong to the target static feature are determined; and performs fitting and vectorization processing on the features in the 3D point cloud that belong to the target static feature to obtain the target static feature.

[0217] In some alternative examples, the projection module 730 in the above embodiments of this disclosure includes:

[0218] The determination submodule 731 is used to determine the image to be labeled in the at least one image to be labeled sequence that corresponds to the target static feature;

[0219] The projection submodule 732 is used to project the second annotation information of the target static feature onto the corresponding image to be annotated.

[0220] In some optional examples, the determining submodule 731 in the above embodiments of this disclosure is further configured to obtain the position of the target static feature in a preset coordinate system; obtain the position of each image to be labeled in the at least one image to be labeled sequence in the preset coordinate system; and determine the image to be labeled corresponding to the target static feature in the at least one image to be labeled sequence based on the position of the target static feature in the preset coordinate system and the position of each image to be labeled in the preset coordinate system.

[0221] In some optional examples, the projection submodule 732 in the above embodiments of this disclosure is further used to obtain the intrinsic and extrinsic parameter information of each image acquisition sensor corresponding to each image sequence to be labeled; and to project the second labeling information of the target static element onto the corresponding image to be labeled according to the intrinsic and extrinsic parameter information of each image acquisition sensor and the position of the target static element in the preset coordinate system.

[0222] In some optional examples, the image annotation apparatus in the above embodiments of this disclosure further includes:

[0223] The first display module 740 is used to display the three-dimensional point cloud, the first annotation information of the at least one static element, and the second annotation information of the target static element through the visual interactive interface.

[0224] The first receiving module 750 is configured to respond to receiving a label modification instruction sent by a user through the visual interactive interface, wherein the label modification instruction is used to modify the second label information of the target static element in the three-dimensional point cloud;

[0225] The first modification module 760 is used to modify the second annotation information of the target static element in the three-dimensional point cloud based on the first annotation information and the annotation modification instruction, and project the modification result of the second annotation information of the target static element onto the at least one image sequence to be annotated.

[0226] In some optional examples, the image annotation apparatus in the above embodiments of this disclosure further includes:

[0227] The second receiving module 770 is used to receive playback instructions sent by the user through the visual interactive interface;

[0228] The playback module 780 is used to play back the at least one labeled image sequence and the three-dimensional point cloud based on the playback instruction;

[0229] The third receiving module 790 is used to respond to receiving a label modification instruction sent by a user through the visual interactive interface, wherein the label modification instruction is used to modify the second label information of a specified static element in the three-dimensional point cloud;

[0230] The second modification module 800 is used to modify the second annotation information of the specified static element in the three-dimensional point cloud based on the first annotation information and the annotation modification instruction, and project the modification result of the second annotation information of the specified static element onto the at least one image sequence to be annotated.

[0231] The fourth receiving module 810 is used to respond to receiving a labeling instruction sent by a user through the visual interactive interface, wherein the labeling instruction is used to annotate the second labeling information of a specified static element in the three-dimensional point cloud.

[0232] The annotation module 820 is used to annotate the second annotation information of the specified static feature in the three-dimensional point cloud.

[0233] In some optional examples, the image annotation apparatus in the above embodiments of this disclosure further includes:

[0234] The training module 830 is used to train a second perception model using the at least one labeled image sequence. After the second perception model is trained, it is deployed to a mobile device to perform static feature detection on images acquired by the image acquisition sensor on the mobile device.

[0235] In the image annotation device disclosed herein, the various optional embodiments, optional implementation methods and optional examples disclosed above can be flexibly selected and combined as needed to achieve the corresponding functions and effects. This disclosure does not list them all.

[0236] The image annotation apparatus disclosed herein corresponds to the embodiments of the image annotation methods described above, and the relevant contents can be referred to each other, which will not be repeated here.

[0237] The beneficial technical effects corresponding to the exemplary embodiment of this image annotation apparatus can be found in the corresponding beneficial technical effects in the exemplary method section above, and will not be repeated here.

[0238] Exemplary electronic devices

[0239] Figure 19 A structural diagram of an electronic device provided in an embodiment of this disclosure includes at least one processor 910 and a memory 920.

[0240] The processor 910 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0241] The memory 920 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 910 may execute one or more computer program instructions to implement the image annotation methods and / or other desired functions of the various embodiments of this disclosure described above.

[0242] In one example, the electronic device may also include an input device 930 and an output device 940, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0243] The input device 930 may also include, for example, a keyboard, a mouse, etc.

[0244] The output device 940 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0245] Of course, for the sake of simplicity, Figure 19 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0246] Exemplary computer program products and computer-readable storage media

[0247] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the image annotation methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0248] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0249] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the image annotation methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0250] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0251] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0252] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An image annotation method, comprising: Acquire a 3D point cloud corresponding to the target scene and at least one sequence of images to be labeled. Each sequence of images to be labeled includes multiple frames of images to be labeled. The multiple frames of images to be labeled are sorted based on the image acquisition time or the driving trajectory of the mobile device where the image acquisition sensor is located. Static feature annotation is performed on the three-dimensional point cloud to obtain annotation information for at least one static feature; The annotation information of the at least one static element is projected onto the at least one image sequence to be annotated to obtain at least one labeled image sequence with static element annotation information. Each labeled image sequence in the at least one labeled image sequence includes multiple frames of labeled images, and each frame of labeled images contains annotation information of static elements. Static feature annotation is performed on the 3D point cloud to obtain annotation information for at least one static feature, including: Using a pre-trained first perception model, semantic segmentation and static feature extraction are performed on the three-dimensional point cloud to obtain first annotation information for at least one static feature; The first annotation information of the three-dimensional point cloud and the at least one static element is displayed through a visual interactive interface; Through the visual interactive interface, annotation instructions for target static elements in the three-dimensional point cloud are received, wherein the target static element is one of the at least one static elements. Based on the first annotation information and the annotation instruction, the target static feature is annotated in the 3D point cloud to obtain the second annotation information of the target static feature, the second annotation information including: attribute annotation information; Projecting the annotation information of the at least one static feature onto the at least one sequence of images to be annotated includes: The second annotation information is projected onto the at least one sequence of images to be annotated to obtain the at least one sequence of annotated images.

2. The method of claim 1, wherein, Using a pre-trained first perception model, semantic segmentation and static feature extraction are performed on the 3D point cloud to obtain first annotation information for at least one static feature, including: The three-dimensional point cloud is divided into multiple point cloud blocks; Using the first perception model, semantic segmentation and static feature extraction are performed on each point cloud block in the plurality of point cloud blocks to obtain the static feature annotation information of each point cloud block. The static feature annotation information of the multiple point cloud blocks is aggregated to obtain the first annotation information of the at least one static feature.

3. The method of claim 1, wherein, Through the visual interactive interface, annotation instructions for target static features in the 3D point cloud are received, including: Through the visual interactive interface, the annotation command sent by the user after selecting the target static element and setting the attribute information in the three-dimensional point cloud is received. The annotation command includes: the selection information and attribute setting information of the target static element. Based on the first annotation information and the annotation instructions, the target static features are annotated in the 3D point cloud, including: Based on the first annotation information and the selection information of the target static element, the target static element is determined in the 3D point cloud, and the target static element is annotated with attributes based on the attribute setting information of the target static element.

4. The method of claim 3, wherein, The selection information of the target static element is sent by the user after selecting the target static element using a geometric element that matches the target static element; the geometric element includes any one of the following: point, line, surface, volume; the selection information of the target static element includes the geometric information of the target geometric element used to select the target static element. Based on the first annotation information and the selection information of the target static feature, the target static feature is determined in the 3D point cloud, including: Based on the first annotation information and the geometric information of the target geometric element, the target static feature corresponding to the target geometric element is determined in the three-dimensional point cloud.

5. The method according to claim 3, wherein, The user's selection information for the target static element in the 3D point cloud includes the user's drawing information for the target static element in the 3D point cloud.

6. The method according to claim 3, wherein, Receiving user selection information for the target static feature in the 3D point cloud, including: Receive selection information sent by the user after selecting a portion of the target static features on the 3D point cloud; Determining the target static element based on the first annotation information and the selection information of the target static element includes: Based on the preset tracking configuration information, the first annotation information, and the selection information, the remaining elements in the 3D point cloud that belong to the target static element are determined.

7. The method according to claim 6, wherein, Based on preset tracking configuration information, the first annotation information, and the selection information, the remaining elements in the 3D point cloud that belong to the target static element are determined, including: Based on the preset tracking configuration information, the first annotation information, and the selection information, a point cloud region in the 3D point cloud is determined for tracking the target static element, and is taken as the current point cloud region. Identify the subset of elements in the current point cloud region that belong to the target static element; Based on the preset tracking configuration information, the first annotation information, and some elements in the current point cloud region that belong to the target static element, the next point cloud region in the 3D point cloud used to track the target static element is determined as the current point cloud region. Iteratively execute the operation of determining the partial elements belonging to the target static element in the current point cloud region until no partial elements belonging to the target static element in the current point cloud region are determined. The target static elements are obtained by fitting and vectorizing a portion of the elements in the three-dimensional point cloud that belong to the target static elements.

8. The method according to any one of claims 1-7, wherein, Projecting the second annotation information onto the at least one sequence of images to be annotated includes: Identify the image to be labeled in the at least one image sequence that corresponds to the target static feature; The second annotation information of the target static feature is projected onto the corresponding image to be annotated.

9. The method according to claim 8, wherein, Determining the image to be labeled corresponding to the target static feature in the at least one image sequence to be labeled includes: Obtain the position of the target static element in the preset coordinate system; Obtain the position of each image to be labeled in the preset coordinate system in the at least one image to be labeled sequence; Based on the position of the target static feature in the preset coordinate system and the position of each image to be labeled in the preset coordinate system, determine the image to be labeled corresponding to the target static feature in the at least one image to be labeled sequence.

10. The method according to claim 9, wherein, Projecting the second annotation information of the target static feature onto the corresponding image to be annotated includes: Obtain the intrinsic and extrinsic parameter information of each image acquisition sensor corresponding to each image sequence to be labeled; Based on the intrinsic and extrinsic parameters of each image acquisition sensor and the position of the target static element in the preset coordinate system, the second annotation information of the target static element is projected onto the corresponding image to be annotated.

11. The method according to any one of claims 1-7, after projecting the second annotation information onto the at least one sequence of maps to be annotated to obtain the at least one sequence of labeled maps, further comprising: The visualization interface displays the 3D point cloud, the first annotation information of the at least one static element, and the second annotation information of the target static element. In response to receiving a label modification instruction sent by a user through the visual interactive interface, the label modification instruction is used to modify the second label information of the target static element in the 3D point cloud; Based on the first annotation information and the annotation modification instruction, the second annotation information of the target static feature is modified in the 3D point cloud, and the modification result of the second annotation information of the target static feature is projected onto the at least one image sequence to be annotated.

12. The method according to any one of claims 1-7, after obtaining at least one labeled map sequence with static feature annotation information, further comprising: The user can receive playback commands through the visual interactive interface. Based on the playback command, the at least one labeled map sequence and the 3D point cloud are played back; In response to receiving a label modification instruction sent by a user through the visual interactive interface, the label modification instruction is used to modify the second label information of a specified static element in the 3D point cloud; Based on the first annotation information and the annotation modification instruction, the second annotation information of the specified static element is modified in the 3D point cloud, and the modification result of the second annotation information of the specified static element is projected onto the at least one image sequence to be annotated; In response to receiving a annotation instruction sent by a user through the visual interactive interface, the annotation instruction is used to annotate the second annotation information of a specified static element in the 3D point cloud; The second annotation information of the specified static element is annotated in the three-dimensional point cloud.

13. The method according to any one of claims 1-7, further comprising: A second perception model is trained using the at least one labeled image sequence. After the second perception model is trained, it is deployed to a mobile device to perform static feature detection on images acquired by the image acquisition sensor on the mobile device.

14. An image annotation apparatus, comprising: The acquisition module is used to acquire a 3D point cloud corresponding to the target scene and at least one sequence of images to be labeled. Each sequence of images to be labeled includes multiple frames of images to be labeled, and the multiple frames of images to be labeled are sorted based on the image acquisition time or the driving trajectory of the mobile device where the image acquisition sensor is located. The annotation module is used to annotate the three-dimensional point cloud with static features to obtain annotation information for at least one static feature. The projection module is used to project the annotation information of the at least one static feature onto the at least one image sequence to be annotated, so as to obtain at least one labeled image sequence with static feature annotation information. Each labeled image sequence in the at least one labeled image sequence includes multiple frames of labeled images, and each frame of labeled images has annotation information of static features. The annotation module includes: The segmentation and extraction submodule is used to perform semantic segmentation and static feature extraction on the three-dimensional point cloud using a pre-trained first perception model, and to obtain the first annotation information of at least one static feature. The display submodule is used to display the first annotation information of the three-dimensional point cloud and the at least one static feature through a visual interactive interface; The instruction receiving submodule is used to receive, through the visual interactive interface, annotation instructions for a target static element in the three-dimensional point cloud, wherein the target static element is one of the at least one static elements. The annotation submodule is used to annotate the target static feature in the 3D point cloud based on the first annotation information and the annotation instruction, and obtain the second annotation information of the target static feature, wherein the second annotation information includes: attribute annotation information; The projection module is further configured to project the second annotation information onto the at least one sequence of images to be annotated, thereby obtaining the at least one sequence of annotated images.

15. A computer-readable storage medium storing a computer program for performing the image annotation method according to any one of claims 1-13.

16. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image annotation method according to any one of claims 1-13.

Images