Image annotation method and image annotation device

Abstract

An image annotation method and apparatus, the method comprising: acquiring an image to be annotated, the image to be annotated including a reference lane already annotated by a user and an auxiliary lane to be annotated; acquiring boundary line information of the reference lane and attribute information of the auxiliary lane, the attribute information including at least width information; determining the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane; annotating the auxiliary lane based on the boundary line information of the auxiliary lane, and displaying the annotated image. The image annotation method and apparatus according to embodiments of the present application can automatically annotate other lanes in the image to be annotated based on the reference lane already annotated by the user, thereby reducing the workload of manual annotation by the user and improving lane annotation efficiency.

Description

Technical Field

[0001] The present application relates to the field of image annotation technology, and more specifically to an image annotation method and an image annotation device. Background Art

[0002] In recent years, my country has continuously guided the development of the autonomous driving industry to promote the digitalization of transportation infrastructure. With the rise of computer vision technology, autonomous driving offers new solutions for improving traffic safety and efficiency. Lane recognition is crucial for autonomous driving system perception and decision-making, and accurate lane recognition algorithm training relies on a large amount of lane marking data. Therefore, efficient lane marking tools based on 3D point cloud data have become an indispensable requirement for the development of autonomous driving. Due to the sparseness, discontinuity, and irregularity of 3D point clouds, the road positions formed by individual points are difficult to discern. Furthermore, potential road damage and regional variations in traffic conditions complicate the process of quickly assessing road conditions and identifying lane positions. These and other factors contribute to the overall low efficiency of lane marking and the significant labor costs involved.

[0003] To improve labeling efficiency, existing lane marking tools use methods such as multi-channel sensor fusion labeling and intelligent adjustment labeling to reduce the burden of manual operations. Multi-channel sensor fusion labeling refers to the fusion of different information collected by multiple types of sensors. For example, the 2D image captured by a camera and the 3D image obtained by a 3D scanner are linked and mapped, allowing road conditions to be assessed from multiple dimensions during labeling. Automatic adjustment labeling refers to the automatic adjustment of manually labeled points, lines, boxes, etc. through artificial intelligence methods, reducing the time required for manual fine-tuning. These methods can improve lane marking efficiency to a certain extent, but the entire process still relies heavily on manual identification and manual labeling. When there are a large number of lanes to be labeled, there are multiple lanes in parallel, or the same lane appears in consecutive frames, manual labeling is required, frame by frame and lane by lane. This process involves a large number of tedious and repetitive operations, and there is still much room for reduction in time consumption.

[0004] Therefore, the overall marking efficiency that can be achieved by existing autonomous driving lane marking tools still needs to be improved. Summary of the Invention

[0005] According to one aspect of the present application, a method for image annotation is provided, the method comprising: acquiring an image to be annotated, the image to be annotated including a reference lane that has been annotated by a user and an auxiliary lane to be annotated; acquiring boundary line information of the reference lane and attribute information of the auxiliary lane, the attribute information including at least width information; determining the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane; performing lane line annotation on the auxiliary lane based on the boundary line information of the auxiliary lane, and displaying the annotated image.

[0006] In an embodiment of the present application, the boundary line information includes left boundary line information and / or right boundary line information; the determining of the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane includes: obtaining the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane; and / or obtaining the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane.

[0007] In an embodiment of the present application, the obtaining of the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane includes: using the left boundary line information of the reference lane as the right boundary line information of the first auxiliary lane on the left side of the reference lane, and combining it with the width information of the first auxiliary lane on the left side to calculate the left boundary line information of the first auxiliary lane on the left side; using the left boundary line information of the first auxiliary lane on the left side as the right boundary line information of the second auxiliary lane on the left side of the reference lane, and combining it with the width information of the second auxiliary lane on the left side to calculate the left boundary line information of the second auxiliary lane on the left; and so on, to obtain the boundary line information of all auxiliary lanes on the left side of the reference lane. Boundary information; the obtaining of the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane includes: using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane, and combining it with the width information of the first auxiliary lane on the right side to calculate the right boundary line information of the first auxiliary lane on the right side; using the right boundary line information of the first auxiliary lane on the right side as the left boundary line information of the second auxiliary lane on the right side of the reference lane, and combining it with the width information of the second auxiliary lane on the right side to calculate the right boundary line information of the second auxiliary lane on the right side; and so on, to obtain the boundary line information of all auxiliary lanes on the right side of the reference lane.

[0008] In an embodiment of the present application, the left boundary line information includes a left line segment set, and the right boundary line information includes a right line segment set; the left boundary line information of the first auxiliary lane on the left side of the reference lane is used as the right boundary line information of the first auxiliary lane on the left side, combined with the width information of the first auxiliary lane on the left side, to calculate the left boundary line information of the first auxiliary lane on the left side, including: using the left line segment set of the reference lane as the right line segment set of the first auxiliary lane on the left side, using the width of the first auxiliary lane on the left side as the first left displacement, and calculating the line segment position of each line segment in the right line segment set of the first auxiliary lane on the left side after being translated vertically to the left by the first left displacement. , obtaining the left line segment set of the first auxiliary lane on the left; using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane, combined with the width information of the first auxiliary lane on the right side, to calculate the right boundary line information of the first auxiliary lane on the right side, including: using the right line segment set of the reference lane as the left line segment set of the first auxiliary lane on the right side, using the width of the first auxiliary lane on the right side as a first right side displacement, calculating the line segment position of each line segment in the left line segment set of the first auxiliary lane on the right side after being translated vertically to the right by the first right side displacement, to obtain the right line segment set of the first auxiliary lane on the right side.

[0009] In an embodiment of the present application, the method further includes: fine-tuning the starting point position and the ending point position of each line segment in the calculated set of left line segments of the auxiliary lane to the left of the reference lane; and / or fine-tuning the starting point position and the ending point position of each line segment in the calculated set of right line segments of the auxiliary lane to the right of the reference lane.

[0010] In an embodiment of the present application, the calculated starting point position and ending point position of each line segment in the left line segment set of the auxiliary lane to the left of the reference lane are fine-tuned, including: sequentially obtaining the line segments in the left line segment set of the auxiliary lane to the left of the reference lane, extending the current line segment, taking the intersection point of the current line segment or its extension line with the next line segment or its extension line as the new ending point position of the current line segment, and taking this point as the starting point position of the next line segment, until the ending point position of the last line segment in the left line segment set is reached. Fine-tuning is completed; the starting point position and the ending point position of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane are fine-tuned, including: sequentially obtaining the line segments in the right line segment set of the auxiliary lane to the right of the reference lane, extending the current line segment, taking the intersection point of the current line segment or its extended line with the next line segment or its extended line as the new ending point position of the current line segment, and taking this point as the starting point position of the next line segment, until the fine-tuning of the ending point position of the last line segment in the right line segment set is completed.

[0011] In an embodiment of the present application, the left and right sides of the reference lane include auxiliary lanes of the same or similar number.

[0012] In an embodiment of the present application, the attribute information further includes direction information and / or speed information, and the method further includes: displaying the boundary line information of the auxiliary lane and the attribute information as annotation information of the auxiliary lane in the image to be annotated.

[0013] In an embodiment of the present application, the method further includes: after the image to be labeled is labeled, obtaining the next frame image of the image to be labeled; when the next frame image is a delayed frame image, obtaining lane line labeling information of the image to be labeled as at least part of the labeling information of the delayed frame image.

[0014] In an embodiment of the present application, whether the next frame image is a deferred frame image is determined by a frame cutting rule. When the frame cutting rule is a deferred frame cutting rule, the next frame image is a deferred frame image, and the lane line annotation information of the image to be annotated is saved in a cache for being obtained as at least part of the annotation information of the deferred frame image.

[0015] According to another aspect of the present application, an image annotation device is also provided, comprising: an image acquisition module for acquiring an image to be annotated, wherein the image to be annotated includes a reference lane that has been annotated by a user and an auxiliary lane to be annotated; an information acquisition module for acquiring boundary line information of the reference lane and attribute information of the auxiliary lane, wherein the attribute information includes at least width information; a boundary line determination module for determining the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane; and an annotation display module for annotating the lane lines of the auxiliary lane based on the boundary line information of the auxiliary lane and displaying the annotated image.

[0016] According to another aspect of the present application, an image annotation device is provided, comprising a memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the processor executes the above-mentioned image annotation method.

[0017] According to another aspect of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is run, the image annotation method described above is executed.

[0018] The image annotation method and image annotation device according to the embodiments of the present application can automatically annotate other lanes in the image to be annotated based on the reference lanes that have been annotated by the user in the image to be annotated, thereby reducing the workload of manual annotation by the user and improving the efficiency of lane annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 A schematic block diagram of an example electronic device for implementing the image annotation method and image annotation apparatus according to an embodiment of the present invention is shown.

[0021] Figure 2 A schematic flowchart of an image annotation method according to an embodiment of the present application is shown.

[0022] Figure 3 A schematic block diagram of an image annotation apparatus according to an embodiment of the present application is shown.

[0023] Figure 4 A schematic block diagram of an image annotation apparatus according to another embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application more apparent, the following is a detailed description of example embodiments according to the present application with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0025] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0026] The workflow of the lane marking tool is usually to first receive the images, point cloud images and other information obtained by each sensor, parse and fuse them, and then render them on the page. Then, after saving the lane lines, traffic signs and other information marked by the user in the current frame, switch to the next frame, continue to obtain and save the next frame of marking data until the marking is completed. However, due to factors such as the sparsity and irregularity of the 3D point cloud map and the complexity and diversity of traffic conditions, the overall lane marking efficiency is not high. Especially when there are a large number of lanes, multiple similar lanes are parallel, or the same lane appears in multiple frames, users need to manually identify and mark them frame by frame and road by road. The complicated and repetitive process adds a burden to the operation.

[0027] Based on this, this application provides a new image annotation solution. Figure 1 An example electronic device 100 for implementing the image tagging method and apparatus according to the embodiments of the present invention will be described.

[0028] like Figure 1As shown, the electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, and an output device 108, which are interconnected via a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as needed.

[0029] The processor 102 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0030] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions and / or other desired functions in the embodiments of the present invention (implemented by the processor) described below. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.

[0031] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. In addition, the input device 106 may also be any interface for receiving information.

[0032] The output device 108 can output various information (such as images or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, etc. In addition, the output device 108 can also be any other device with an output function.

[0033] For example, an exemplary electronic device for implementing the image annotation method and the image annotation apparatus according to the embodiments of the present invention may be implemented as a terminal such as a smart phone, a tablet computer, a camera, or the like.

[0034] Below, we will refer to Figure 2 The image annotation method 200 according to an embodiment of the present application is described. Figure 2 As shown, the image annotation method 200 may include the following steps:

[0035] In step S210 , an image to be annotated is acquired, where the image to be annotated includes a reference lane that has been annotated by a user and an auxiliary lane to be annotated.

[0036] In step S220, the boundary line information of the reference lane and the attribute information of the auxiliary lane are obtained, where the attribute information at least includes width information.

[0037] In step S230 , the boundary line information of the auxiliary lane is determined based on the boundary line information of the reference lane and the width information of the auxiliary lane.

[0038] In step S240 , lane lines of the auxiliary lane are marked based on the boundary line information of the auxiliary lane, and the marked image is displayed.

[0039] In an embodiment of the present application, a user first selects at least one lane in the image to be annotated as a reference lane and annotates the reference lane, for example, by at least annotating the reference lane's boundary line information. Next, the image to be annotated is acquired, and based on the annotated reference lane in the image to be annotated, lane annotation is performed on other lanes in the image (referred to as auxiliary lanes, so named to distinguish them from the reference lanes). Specifically, the boundary line information of the auxiliary lane can be determined based on the boundary line (lane line) information of the reference lane and the width information of the auxiliary lane (this width information can also be annotated by the user), thereby automatically annotating the lane lines of the auxiliary lane. This can reduce the workload of manual annotation by the user, especially in situations where there are many lanes or multiple similar lanes in parallel, as the user does not need to annotate each lane one by one, greatly improving annotation efficiency. Finally, the lane line annotation information of the auxiliary lane can be displayed in the image to be annotated for the user to review, allowing the user to confirm whether the annotation results are accurate, thereby further ensuring the accuracy of the lane annotation. Moreover, the image to be annotated including such annotated information is displayed to the user, which has the effect of one-click rendering of the auxiliary lane. It can also provide a reference for the user to assist in judging road conditions, identifying lanes and other matters.

[0040] Therefore, the image annotation method according to the embodiment of the present application can automatically annotate other lanes in the image to be annotated based on the reference lanes that have been annotated by the user in the image to be annotated, thereby reducing the workload of manual annotation by the user and improving the efficiency of lane annotation.

[0041] In an embodiment of the present application, the boundary line information may include left boundary line information and / or right boundary line information; determining the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane in step S230 may include: obtaining the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane; and / or obtaining the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane.

[0042] The expression "and / or" is used here because the reference lane selected by the user may be any lane. For example, in the image to be annotated, the user-selected reference lane may be the middle lane. In this case, the reference lane includes auxiliary lanes on both the left and right sides. The boundary line information of the left auxiliary lane can be obtained based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane, and the boundary line information of the right auxiliary lane can be obtained based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane. For another example, in the image to be annotated, the user-selected reference lane may be the leftmost lane. In this case, the left side of the reference lane does not include auxiliary lanes, but the right side does. The boundary line information of the right auxiliary lane can be obtained based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane. For another example, in the image to be annotated, the user-selected reference lane may be the rightmost lane. In this case, the right side of the reference lane does not include auxiliary lanes, but the left side does. The boundary line information of the left auxiliary lane can be obtained based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane. Typically, users select the center lane in the image to be annotated as the reference lane. This means that the reference lane has an equal or similar number of auxiliary lanes on its left and right sides. This way, the auxiliary lanes to the left of the reference lane are determined based on the left boundary of the reference lane, and the auxiliary lanes to the right of the reference lane are determined based on the right boundary of the reference lane. This results in more accurate annotation than if all auxiliary lanes were determined based solely on the left or right boundary. Generally, when a reference lane has an equal number of auxiliary lanes on its left and right sides, the reference lane is also referred to as the center lane.

[0043] In an embodiment of the present application, obtaining the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane can include: using the left boundary line information of the reference lane as the right boundary line information of the first auxiliary lane on the left side of the reference lane, combined with the width information of the first auxiliary lane on the left, to calculate the left boundary line information of the first auxiliary lane on the left; using the left boundary line information of the first auxiliary lane on the left as the right boundary line information of the second auxiliary lane on the left side of the reference lane, combined with the width information of the second auxiliary lane on the left, to calculate the left boundary line information of the second auxiliary lane on the left; and so on, to obtain the boundary line information of all auxiliary lanes on the left side of the reference lane. Similarly, obtaining the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane can include: using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane, combined with the width information of the first auxiliary lane on the right side, to calculate the right boundary line information of the first auxiliary lane on the right side; using the right boundary line information of the first auxiliary lane on the right side as the left boundary line information of the second auxiliary lane on the right side of the reference lane, combined with the width information of the second auxiliary lane on the right side, to calculate the right boundary line information of the second auxiliary lane on the right; and so on, to obtain the boundary line information of all auxiliary lanes on the right side of the reference lane.

[0044] In this embodiment, from the reference lane to each auxiliary lane to the left of the reference lane, the left boundary line information of the previous lane is used as the right boundary line information of the current lane, combined with the width information of the current lane, and the left boundary line information of the current lane is calculated until the boundary line information of all lanes on the left is calculated. From the reference lane to each auxiliary lane to the right of the reference lane, the right boundary line information of the previous lane is used as the left boundary line information of the current lane, combined with the width information of the current lane, and the left boundary line information of the current lane is calculated until the boundary line information of all lanes on the right is calculated. This solution can quickly obtain the boundary line information of the auxiliary lanes and realize efficient automatic lane marking of the auxiliary lanes.

[0045] In an embodiment of the present application, the aforementioned left boundary line information includes a left line segment set consisting of one or more line segments, and the aforementioned right boundary line information includes a right line segment set consisting of one or more line segments. Based on this, the left boundary line information of the reference lane is used as the right boundary line information of the first auxiliary lane on the left side of the reference lane, combined with the width information of the first auxiliary lane on the left side, to calculate the left boundary line information of the first auxiliary lane on the left side, which may include: using the left line segment set of the reference lane as the right line segment set of the first auxiliary lane on the left side, using the width of the first auxiliary lane on the left side as the first left displacement, calculating the line segment position of each line segment in the right line segment set of the first auxiliary lane on the left side after being translated to the left vertical direction by the first left displacement, and obtaining the left line segment set of the first auxiliary lane on the left side. Then, the left segment set of the first auxiliary lane on the left can be used as the right segment set of the second auxiliary lane on the left, and the width of the second auxiliary lane on the left can be used as the second left displacement. The position of each segment in the right segment set of the second auxiliary lane on the left can be calculated after being vertically shifted to the left by the second left displacement, thus obtaining the left segment set of the second auxiliary lane on the left. Similarly, the left and right segment sets of all auxiliary lanes on the left side of the reference lane can be obtained.

[0046] Similarly, using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane to the right of the reference lane, combined with the width information of the first auxiliary lane to the right, to calculate the right boundary line information of the first auxiliary lane to the right, can include: using the right line segment set of the reference lane as the left line segment set of the first auxiliary lane to the right, using the width of the first auxiliary lane to the right as a first right displacement, calculating the line segment position of each line segment in the left line segment set of the first auxiliary lane to the right after being translated vertically to the right by the first right displacement, to obtain the right line segment set of the first auxiliary lane to the right. Then, using the right line segment set of the first auxiliary lane to the right as the left line segment set of the second auxiliary lane to the right, using the width of the second auxiliary lane to the right as a second right displacement, calculating the line segment position of each line segment in the left line segment set of the second auxiliary lane to the right after being translated vertically to the right by the second right displacement, to obtain the right line segment set of the second auxiliary lane to the right. Similarly, the left and right line segment sets of all auxiliary lanes to the right of the reference lane can be obtained.

[0047] In this embodiment, the boundary line of each lane is represented by a set of line segments, which can further improve the accuracy of lane line marking.

[0048] In an embodiment of the present application, method 200 may further include (not shown): fine-tuning the starting point position and the ending point position of each line segment in the calculated left line segment set of the auxiliary lane to the left of the reference lane; and / or fine-tuning the starting point position and the ending point position of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane. In this embodiment, the starting point position and the ending point position of each line segment in the calculated left line segment set and / or right line segment set of the auxiliary lane may be adjusted to make them more consistent with the actual lane line. This fine-tuning step may be performed after step S240, or after step S230 and before step S240. The adjusted line segment set may also be used as the final result of step S230.

[0049] In an embodiment of the present application, fine-tuning the starting point and ending point positions of each line segment in the calculated left line segment set of the auxiliary lane to the left of the reference lane may include: sequentially obtaining line segments in the left line segment set of the auxiliary lane to the left of the reference lane, extending the current line segment, taking the intersection point where the current line segment or its extension intersects with the next line segment or its extension as the new ending point position of the current line segment, and using this point as the starting point position of the next line segment, until the ending point position of the last line segment in the left line segment set is fine-tuned. Similarly, fine-tuning the starting point and ending point positions of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane may include: sequentially obtaining line segments in the right line segment set of the auxiliary lane to the right of the reference lane, extending the current line segment, taking the intersection point where the current line segment or its extension intersects with the next line segment or its extension as the new ending point position of the current line segment, and using this point as the starting point position of the next line segment, until the ending point position of the last line segment in the right line segment set is fine-tuned. The above-mentioned fine-tuning process is merely exemplary. In other examples, fine-tuning may be performed in any other suitable manner.

[0050] In an embodiment of the present application, in addition to the width information of the auxiliary lane, the attribute information obtained in step S220 may also include other attribute information such as direction information (i.e., whether the auxiliary lane is an execution lane, a left-turn lane, a right-turn lane, a U-turn lane, etc.), speed information (i.e., whether the auxiliary lane is a fast lane, a slow lane, a motor vehicle lane, a non-motor vehicle lane, etc.). In this embodiment, method 200 may further include (not shown): displaying the boundary line information and attribute information of the auxiliary lane as annotation information of the auxiliary lane in the image to be annotated. In this embodiment, the relevant information of all lanes in the image to be annotated (including lane line information and other information) can be displayed in the image to be annotated, so that the user can view the complete road annotation information.

[0051] In an embodiment of the present application, method 200 may further include (not shown): after the image to be annotated is completed, obtaining the next frame of the image to be annotated; when the next frame is a deferred frame, obtaining the lane line annotation information of the image to be annotated as at least part of the annotation information of the deferred frame (the deferred frame may also include other lanes to be annotated or other objects to be annotated, and new annotation information can be generated for them). In this embodiment, the annotation information in the current frame of the image to be annotated can be deferred to the next frame of the image to be annotated as an effective reference for the annotation of the next frame of the image to be annotated, which can further improve the annotation efficiency. This can greatly improve the annotation efficiency for scenarios where multiple frames of the same lane appear.

[0052] In an embodiment of the present application, whether the next image frame is a delayed frame image is determined by a frame cutting rule. When the frame cutting rule is a delayed frame cutting rule, the next image frame is a delayed frame image, and the lane line annotation information of the image to be annotated is saved in a cache to be used to obtain at least part of the annotation information for the delayed frame image. In this embodiment, whether the next image frame is a delayed frame image can be determined based on the frame cutting rule. In other embodiments, whether the next image frame is a delayed frame image can also be determined through other means, such as user interaction.

[0053] The above exemplifies the image annotation method 200 according to an embodiment of the present application. The following describes a more specific process that may be included in the above image annotation method 200 with reference to a specific example. Step 1: Receive 2D and 3D data acquired by each sensor, render the data to be annotated, and start annotation. Step 2: Detect whether there is new annotation information submitted for the current frame. If so, execute step 3; if not, execute step 9. Step 3: Detect whether the annotation type is a lane line. If so, execute step 4; otherwise, execute step 8 after saving other types of annotation information. Step 4: Obtain the boundary line position information of the center lane (i.e., the reference lane) and the auxiliary lane information on the left and right sides, and convert the left boundary line and right boundary line of the center lane into sets consisting of one or more line segments, i.e., the left segment set and the right segment set. Step 5: Taking the center lane as the reference lane, determine the left segment set and the right segment set of the first auxiliary lane adjacent to the left of the center lane; taking the first left auxiliary lane as the reference lane, obtain the left and right segment sets of the second left auxiliary lane; and so on, until the boundary line information of all auxiliary lanes on the left is determined. Step 6: Using the center lane as the reference lane, determine the left and right line segment sets of the first auxiliary lane adjacent to the right of the center lane; using the first right auxiliary lane as the reference lane, obtain the left and right line segment sets of the second right auxiliary lane; and so on, until the boundary line information of all auxiliary lanes on the right is determined. Step 7: Based on the left and right line segment sets of all auxiliary lanes on both sides and the direction and attribute marking information of each auxiliary road, generate complete road labeling information for each auxiliary road. Step 8: Single labeling is completed, jump to step 2. Step 9: Check whether there is the next frame of data. If so, continue to step 10. Otherwise, labeling is completed and all labeling information is generated. Step 10: Receive the frame switching command and determine whether the frame switching rule is a delayed frame switching. If so, save the user-submitted data as the current frame annotation data, save the lane line delayed data to the cache, complete the frame switching, load the new frame file import data, and obtain the data saved in the previous frame from the cache, and jump to step 2; otherwise, save the user-submitted data as the current frame annotation data, complete the frame switching, load the new frame file import data, and jump to step 2.

[0054] Based on the above description, the image annotation method according to the embodiment of the present application can automatically annotate other lanes in the image to be annotated based on the reference lanes that have been annotated by the user in the image to be annotated, thereby reducing the workload of manual annotation by the user and improving the efficiency of lane annotation.

[0055] The following combination Figures 3 to 4According to another aspect of the present application, an image annotation device is provided, which can be used to perform the image annotation method according to the embodiment of the present application described above. Those skilled in the art can understand the structure and specific operation of the image annotation device according to the embodiment of the present application in combination with the above content. For the sake of brevity, the specific details are not repeated here, and only some key operations are described.

[0056] Figure 3 FIG. 3 shows a schematic block diagram of an image annotation apparatus 300 according to an embodiment of the present application. Figure 3 As shown, the image annotation device 300 includes an image acquisition module 310, an information acquisition module 320, a boundary line determination module 330, and an annotation display module 340. The image acquisition module 310 is used to acquire an image to be annotated, which includes a reference lane that has been annotated by the user and an auxiliary lane to be annotated. The information acquisition module 320 is used to obtain boundary line information of the reference lane and attribute information of the auxiliary lane, the attribute information including at least width information. The boundary line determination module 330 is used to determine the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane. The annotation display module 340 is used to annotate the auxiliary lane based on the boundary line information of the auxiliary lane and display the annotated image.

[0057] In an embodiment of the present application, the boundary line information includes left boundary line information and / or right boundary line information; the boundary line determination module 330 determines the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane, including: obtaining the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane; and / or obtaining the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane.

[0058] In an embodiment of the present application, the boundary line determination module 330 obtains the boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane, including: using the left boundary line information of the reference lane as the right boundary line information of the first auxiliary lane on the left side of the reference lane, combined with the width information of the first auxiliary lane on the left, to calculate the left boundary line information of the first auxiliary lane on the left; using the left boundary line information of the first auxiliary lane on the left as the right boundary line information of the second auxiliary lane on the left side of the reference lane, combined with the width information of the second auxiliary lane on the left, to calculate the left boundary line information of the second auxiliary lane on the left; and so on, to obtain the boundary line information of all auxiliary lanes on the left side of the reference lane. The boundary line determination module 330 obtains the boundary line information of the auxiliary lane on the right side based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane, including: using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane, combined with the width information of the first auxiliary lane on the right side, to calculate the right boundary line information of the first auxiliary lane on the right side; using the right boundary line information of the first auxiliary lane on the right side as the left boundary line information of the second auxiliary lane on the right side of the reference lane, combined with the width information of the second auxiliary lane on the right side, to calculate the right boundary line information of the second auxiliary lane on the right side; and so on, to obtain the boundary line information of all auxiliary lanes on the right side of the reference lane.

[0059] In an embodiment of the present application, the left boundary line information includes a left line segment set, and the right boundary line information includes a right line segment set. The boundary line determination module 330 uses the left boundary line information of the reference lane as the right boundary line information of the first auxiliary lane on the left side of the reference lane, and combines it with the width information of the first auxiliary lane on the left side to calculate the left boundary line information of the first auxiliary lane on the left side. This includes: using the left line segment set of the reference lane as the right line segment set of the first auxiliary lane on the left side, using the width of the first auxiliary lane on the left side as a first left displacement, calculating the line segment position of each line segment in the right line segment set of the first auxiliary lane on the left side after being translated vertically to the left by the first left displacement, and obtaining the left line segment set of the first auxiliary lane on the left side. The boundary line determination module 330 uses the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane, and combines it with the width information of the first auxiliary lane on the right side to calculate the right boundary line information of the first auxiliary lane on the right side, including: using the right line segment set of the reference lane as the left line segment set of the first auxiliary lane on the right side, using the width of the first auxiliary lane on the right side as the first right side displacement, calculating the line segment position of each line segment in the left line segment set of the first auxiliary lane on the right side after being translated in the vertical direction to the right by the first right side displacement, and obtaining the right line segment set of the first auxiliary lane on the right side.

[0060] In an embodiment of the present application, the device also includes a fine-tuning module (not shown), which is used to: fine-tune the starting point position and the ending point position of each line segment in the calculated set of left line segments of the auxiliary lane to the left of the reference lane; and / or fine-tune the starting point position and the ending point position of each line segment in the calculated set of right line segments of the auxiliary lane to the right of the reference lane.

[0061] In an embodiment of the present application, the fine-tuning module fine-tunes the starting point position and the ending point position of each line segment in the calculated left line segment set of the auxiliary lane to the left of the reference lane, including: sequentially obtaining the line segments in the left line segment set of the auxiliary lane to the left of the reference lane, extending the current line segment, taking the intersection point of the current line segment or its extended line and the next line segment or its extended line as the new ending point position of the current line segment, and using this point as the starting point position of the next line segment, until the fine-tuning of the ending point position of the last line segment in the left line segment set is completed. The fine-tuning module fine-tunes the starting point position and the ending point position of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane, including: sequentially obtaining the line segments in the right line segment set of the auxiliary lane to the right of the reference lane, extending the current line segment, taking the intersection point of the current line segment or its extended line with the next line segment or its extended line as the new ending point position of the current line segment, and taking this point as the starting point position of the next line segment, until the fine-tuning of the ending point position of the last line segment in the right line segment set is completed.

[0062] In an embodiment of the present application, the left and right sides of the reference lane include the same or similar number of auxiliary lanes.

[0063] In an embodiment of the present application, the attribute information also includes direction information and / or speed information. The identification display module 340 is further configured to display the boundary line information and attribute information of the auxiliary lane as annotation information of the auxiliary lane in the image to be annotated.

[0064] In an embodiment of the present application, the image acquisition module 310 is also used to: acquire the next frame image of the image to be annotated after the image to be annotated is completed; when the next frame image is a delayed frame image, the annotation display module 340 acquires the lane line annotation information of the image to be annotated as at least part of the annotation information of the delayed frame image.

[0065] In an embodiment of the present application, whether the next frame image is a deferred frame image is determined by a frame cutting rule. When the frame cutting rule is a deferred frame cutting, the next frame image is a deferred frame image, and the lane line annotation information of the image to be annotated is saved in the cache by the annotation display module 340 for use in obtaining at least part of the annotation information of the deferred frame image.

[0066] Based on the above description, the image labeling device according to the embodiment of the present application can automatically label other lanes in the image to be labeled based on the reference lanes that have been labeled by the user in the image to be labeled, thereby reducing the workload of manual labeling by the user and improving the efficiency of lane labeling.

[0067] Figure 4 FIG. 4 shows a schematic block diagram of an image annotation apparatus 400 according to another embodiment of the present application. Figure 4 As shown, the image annotation apparatus 400 according to an embodiment of the present application may include a memory 410 and a processor 420. The memory 410 stores a computer program executed by the processor 420. When the computer program is executed by the processor 420, the processor 420 executes the image annotation method according to the embodiment of the present application described above. Those skilled in the art can understand the specific operation of the image annotation apparatus 400 according to the embodiment of the present application in combination with the above content. For the sake of brevity, the specific details are not repeated here.

[0068] In addition, according to an embodiment of the present application, a storage medium is further provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the corresponding steps of the image annotation method of the embodiment of the present application are executed. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0069] Based on the above description, the image annotation method and image annotation device according to the embodiments of the present application can automatically annotate other lanes in the image to be annotated based on the reference lanes that have been annotated by the user in the image to be annotated, thereby reducing the workload of manual annotation by the user and improving the efficiency of lane annotation.

[0070] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present application. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0071] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units described is merely a logical function division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not performing some features.

[0073] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0074] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach of the present application should not be interpreted as reflecting the intention that the application claimed for protection requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present application.

[0075] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0076] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of this application and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0077] The various component embodiments of the present application can be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules according to the embodiments of the present application. The application can also be implemented as a part or all of a device program (e.g., a computer program and a computer program product) for performing the method described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0078] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbols placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0079] The above description is merely a specific embodiment or illustration of a specific embodiment of the present application, and the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. The scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image annotation method, characterized in that: The method comprises: Acquire an image to be annotated, where the image to be annotated includes a reference lane that has been annotated by a user and an auxiliary lane to be annotated; Acquire boundary line information of the reference lane and attribute information of the auxiliary lane, wherein the attribute information includes at least width information; determining the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane; Marking lane lines of the auxiliary lane based on boundary line information of the auxiliary lane and displaying the marked image; The boundary line information of the reference lane and the boundary line information of the auxiliary lane are both represented by a set of line segments.

2. The method according to claim 1, characterized in that The boundary line information includes left boundary line information and / or right boundary line information; The determining of the boundary line information of the auxiliary lane based on the boundary line information of the reference lane and the width information of the auxiliary lane includes: Acquiring boundary line information of the auxiliary lane on the left side based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane; and / or The boundary line information of the right auxiliary lane is acquired based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane.

3. The method according to claim 2, characterized in that The acquiring the boundary line information of the left auxiliary lane based on the left boundary line information of the reference lane and the width information of the auxiliary lane on the left side of the reference lane includes: The left boundary line information of the reference lane is used as the right boundary line information of the first auxiliary lane on the left side of the reference lane, and combined with the width information of the first auxiliary lane on the left side, the left boundary line information of the first auxiliary lane on the left side is calculated; the left boundary line information of the first auxiliary lane on the left side is used as the right boundary line information of the second auxiliary lane on the left side of the reference lane, and combined with the width information of the second auxiliary lane on the left side, the left boundary line information of the second auxiliary lane on the left side is calculated; and so on, the boundary line information of all auxiliary lanes on the left side of the reference lane is obtained; The acquiring the boundary line information of the right auxiliary lane based on the right boundary line information of the reference lane and the width information of the auxiliary lane on the right side of the reference lane includes: The right boundary line information of the reference lane is used as the left boundary line information of the first auxiliary lane on the right side of the reference lane, and combined with the width information of the first auxiliary lane on the right side, the right boundary line information of the first auxiliary lane on the right side is calculated; the right boundary line information of the first auxiliary lane on the right side is used as the left boundary line information of the second auxiliary lane on the right side of the reference lane, and combined with the width information of the second auxiliary lane on the right side, the right boundary line information of the second auxiliary lane on the right side is calculated; and so on, the boundary line information of all auxiliary lanes on the right side of the reference lane is obtained.

4. The method according to claim 3, characterized in that The left boundary line information includes a left line segment set, and the right boundary line information includes a right line segment set; The method of using the left boundary line information of the reference lane as the right boundary line information of the first auxiliary lane on the left side of the reference lane and calculating the left boundary line information of the first auxiliary lane on the left side in combination with the width information of the first auxiliary lane on the left side includes: Using the left line segment set of the reference lane as the right line segment set of the first left auxiliary lane, using the width of the first left auxiliary lane as a first left displacement, calculating the position of each line segment in the right line segment set of the first left auxiliary lane after being translated leftward in a vertical direction by the first left displacement, to obtain the left line segment set of the first left auxiliary lane; The method of using the right boundary line information of the reference lane as the left boundary line information of the first auxiliary lane on the right side of the reference lane and calculating the right boundary line information of the first auxiliary lane on the right side in combination with the width information of the first auxiliary lane on the right side includes: Taking the right line segment set of the reference lane as the left line segment set of the first auxiliary lane on the right, and taking the width of the first auxiliary lane on the right as the first right displacement, calculate the line segment position of each line segment in the left line segment set of the first auxiliary lane on the right after being translated vertically to the right by the first right displacement, to obtain the right line segment set of the first auxiliary lane on the right.

5. The method according to claim 4, characterized in that The method further comprises: Fine-tuning the calculated starting point position and ending point position of each line segment in the set of left line segments of the auxiliary lane to the left of the reference lane; and / or Fine-tune the starting point position and the ending point position of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane.

6. The method according to claim 5, characterized in that Fine-tuning the starting point position and the ending point position of each line segment in the calculated left line segment set of the auxiliary lane to the left of the reference lane includes: Sequentially obtain line segments from the left line segment set of the auxiliary lane to the left of the reference lane, extend the current line segment, take the intersection point of the current line segment or its extension with the next line segment or its extension as the new end point position of the current line segment, and use this point as the starting point position of the next line segment, until the end point position of the last line segment in the left line segment set is fine-tuned; Fine-tuning the starting point position and the ending point position of each line segment in the calculated right line segment set of the auxiliary lane to the right of the reference lane includes: Sequentially obtain the line segments in the right line segment set of the auxiliary lane to the right of the reference lane, extend the current line segment, take the intersection point of the current line segment or its extended line with the next line segment or its extended line as the new end point position of the current line segment, and use this point as the starting point position of the next line segment, until the end point position of the last line segment in the right line segment set is fine-tuned.

7. The method according to any one of claims 1 to 6, characterized in that The left and right sides of the reference lane include auxiliary lanes of the same or similar number.

8. The method according to any one of claims 1 to 6, characterized in that The attribute information further includes direction information and / or speed information, and the method further includes: The boundary line information and the attribute information of the auxiliary lane are displayed in the image to be labeled as the labeling information of the auxiliary lane.

9. The method according to any one of claims 1 to 6, characterized in that The method further comprises: After the image to be labeled is labeled, obtaining the next frame of the image to be labeled; When the next frame image is a delayed frame image, obtaining lane line annotation information of the image to be annotated as at least part of the annotation information of the delayed frame image; Among them, whether the next frame image is a deferred frame image is determined by a frame cutting rule. When the frame cutting rule is a deferred frame cutting, the next frame image is a deferred frame image, and the lane line annotation information of the image to be annotated is saved in the cache for being obtained as at least part of the annotation information of the deferred frame image.

10. An image annotation device, characterized in that: The device includes a memory and a processor, wherein a computer program executed by the processor is stored in the memory, and when the computer program is executed by the processor, the processor executes the image annotation method according to any one of claims 1 to 9.

Images