A method and system for annotating 3D frames of images based on vanishing lines

Abstract

The present invention relates to a method and system for annotating a 3D frame of an image based on vanishing lines. The 3D frame annotation method comprises: obtaining an initial 3D frame based on a target outline, determining a vanishing point of the initial 3D frame, and generating vanishing lines in a horizontal direction passing through the vanishing point; adjusting the position of the vanishing line, and calculating and adjusting the positions of the vertices of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective to obtain a final 3D frame that accurately fits the target; first, the vanishing line of the 3D frame is obtained using the principle of perspective; then, a user can adjust multiple vertices of the 3D frame by adjusting the vanishing line, and conversely, when the user adjusts certain vertices of the 3D frame, the vanishing line will also be adjusted accordingly; finally, the 3D frame can accurately fit the outline of the object; the user changes the outline of the 3D frame by adjusting the vanishing line. In most cases, the position of the vanishing line can be determined based on the position of the distant horizon, eliminating the need to manually imagine the positions of vertices blocked by the vehicle body, thereby making it easier to draw and adjust the 3D frame.

Description

Technical Field

[0001] The present invention relates to the field of data annotation, and in particular to a method and system for annotating 3D frames of images based on vanishing lines. Background Art

[0002] In recent years, with the development of fields like AI (artificial intelligence), autonomous driving, 5G technology, intelligent transportation, and smart manufacturing, data volumes have grown exponentially. The data services sector is booming, with the data annotation industry experiencing explosive growth. A data annotation industry chain has also emerged upstream of the AI ​​industry. Data annotation is an essential component of AI development. The quantity and quality of data determine the success of AI model training. As we all know, more data and better data quality lead to higher recognition rates for trained AI models.

[0003] In fields like autonomous driving and intelligent transportation, it's often necessary to identify vehicles on the road, necessitating labeled vehicle datasets for model training. During the annotation process, vehicles can be annotated using 2D or 3D boxes. However, 2D boxes only capture the vehicle's maximum outline, preventing AI algorithms from obtaining depth information. 3D boxes contain both the vehicle's outline and its depth.

[0004] Currently, in the process of annotating vehicles with 3D frames, the eight vertices of the 3D frame cannot be accurately drawn due to the occlusion problem of most vehicles themselves. The innermost rectangular part of the 3D frame needs to be manually filled in. As a result, it is not easy for users to draw the 3D frame to fit the outline of the vehicle. It is necessary to manually adjust the 3D frame vertices multiple times, which is time-consuming and labor-intensive. Summary of the Invention

[0005] In response to the technical problems existing in the prior art, the present invention provides a method and system for annotating 3D frames of images based on vanishing lines, which solves the problem that ordinary 3D frame image annotations are difficult to fit the contour and require multiple vertex adjustments.

[0006] According to a first aspect of the present invention, a method for annotating a 3D frame of an image based on vanishing lines is provided, comprising: step 1, obtaining an initial 3D frame based on a target outline, determining a vanishing point of the initial 3D frame, and generating a horizontal vanishing line passing through the vanishing point;

[0007] Step 2: Adjust the position of the vanishing line, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time based on the principle of one-point perspective, and obtain a final 3D frame that accurately fits the target.

[0008] On the basis of the above technical solution, the present invention can also make the following improvements.

[0009] Optionally, step 1 includes:

[0010] Step 101, drawing an initial 2D rectangular frame based on the target outline;

[0011] Step 102 : generating the initial 3D frame from the initial 2D rectangular frame based on a preset algorithm of the target type.

[0012] Optionally, when the volume size of the generated initial 3D frame is 0, step 1 to step 2 are executed again.

[0013] Optionally, the method for calculating and adjusting the vertex positions of the 3D box corresponding to the vanishing line in real time based on the principle of one-point perspective in step 2 is:

[0014] A constraint relationship of the coordinates of eight vertices is set based on the principle of single-point perspective, and based on the constraint relationship and the equation of the vanishing line, the vertex positions of the 3D frame corresponding to the vanishing line that are adjusted in real time are calculated.

[0015] Optionally, the constraint relationship is:

[0016] = ;

[0017] Among them, A1 (x1, y1), A2 (x2, y1), A3 (x2, y2), A4 (x1, y2), B1 (x4, y4), B2 (x3, y4), B3 (x3, y3) and B4 (x4, y3) are the coordinates of the eight vertices of the 3D box. A1, A2, A3 and A4 are the upper left, upper right, lower right and lower left vertices of the rectangle in front of the 3D box, respectively. B1, B2, B3 and B4 are the upper left, upper right, lower right and lower left vertices of the rectangle behind the 3D box, respectively.

[0018] Optionally, when calculating and adjusting the vertex positions of the 3D frame corresponding to the vanishing line in step 2, the positions of the four vertices of the rectangle in front of the 3D frame remain unchanged, and the positions of the four vertices of the rectangle behind the 3D frame are changed.

[0019] According to a second aspect of the present invention, there is provided a vanishing line-based image 3D frame annotation system, comprising: a vanishing line generation module and a 3D frame generation module;

[0020] The vanishing line generation module is configured to obtain an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point;

[0021] The 3D frame generation module is used to adjust the position of the vanishing line, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective, and obtain a final 3D frame that accurately fits the target.

[0022] According to a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the processor is configured to implement the steps of a method for annotating a 3D image frame based on vanishing lines when executing a computer management program stored in the memory.

[0023] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored. When the computer management program is executed by a processor, the steps of the image 3D box annotation method based on vanishing lines are implemented.

[0024] The present invention provides a method, system, electronic device, and storage medium for annotating 3D frames based on vanishing lines. The method first utilizes the principle of perspective to obtain the vanishing lines of the 3D frame. The user then adjusts the vanishing lines to adjust multiple vertices of the 3D frame. Conversely, when the user adjusts certain vertices of the 3D frame, the vanishing lines are also adjusted accordingly. Finally, the 3D frame can precisely fit the outline of the object. By adjusting the vanishing lines, the user can alter the 3D frame's outline. In most cases, the position of the vanishing line can be determined based on the position of the distant horizon, eliminating the need to manually determine the positions of vertices obscured by vehicle bodies. This makes it easier to draw and adjust the 3D frame. By observing the horizon, the user can quickly adjust the vanishing lines, and thus the position of the 3D frame. This is more efficient, significantly improving annotation efficiency and saving time. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for annotating 3D frames of images based on vanishing lines provided by the present invention;

[0026] Figure 2 A schematic diagram of an embodiment of generating vanishing lines provided by the present invention;

[0027] Figure 3 This is a schematic diagram of an embodiment of generating a 3D frame when 3Dside is 0 provided by the present invention;

[0028] Figure 4 This is a rendering of an embodiment of the vanishing line-based 3D frame annotation of an image provided by the present invention;

[0029] Figure 5 This is a structural diagram of a 3D frame annotation of an image based on vanishing lines provided by the present invention;

[0030] Figure 6 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0031] Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0032] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0033] Figure 1 The present invention provides a flowchart of a method for annotating 3D frames of images based on vanishing lines, as shown in FIG. Figure 1 As shown, the 3D box annotation method includes:

[0034] Step 1: Draw an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point.

[0035] According to the principle of one-point perspective, two parallel lines in the natural world intersect at a single point in the coordinate system of a visual 2D image. This point is known as the vanishing point. Given the eight vertex coordinates of a 3D box, the location of the vanishing point can be calculated, thereby determining the location of the vanishing line.

[0036] Step 2: Adjust the position of the vanishing line. Based on the principle of single-point perspective, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time to obtain the final 3D frame that accurately fits the target.

[0037] Since the vanishing line L is determined by the eight vertices of the 3D frame, when the user adjusts the vanishing line L, the position of the vertex coordinates of the 3D frame can be adjusted according to certain conditional constraints. When the 3D frame is adjusted in real time until it accurately fits the target, the 3D frame at this time is the final 3D frame.

[0038] To address the problems that ordinary 3D frame image annotations are difficult to fit the contour and require multiple vertex adjustments, the present invention provides a 3D frame annotation method for images based on vanishing lines. First, the vanishing line of the 3D frame is obtained by using the principle of perspective. Then, the user can adjust multiple vertices of the 3D frame by adjusting the vanishing line. Conversely, when the user adjusts certain vertices of the 3D frame, the vanishing line will also be adjusted accordingly. Finally, the 3D frame can accurately fit the contour of the object. The user changes the contour of the 3D frame by adjusting the vanishing line. In most cases, the position of the vanishing line can be determined based on the position of the distant horizon, without having to manually imagine the position of vertices blocked by the vehicle body, making it easier to draw and adjust the 3D frame.

[0039] Example 1

[0040] The embodiment 1 provided by the present invention is an embodiment of a method for marking a 3D frame of an image based on vanishing lines provided by the present invention, Figure 1 It can be seen that the embodiment of the 3D box annotation method includes:

[0041] Step 1: Draw an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point.

[0042] like Figure 2 Schematic diagram of an embodiment of generating vanishing lines provided by the present invention is shown. Straight lines A1B1, A2B2, A3B3, and A4B4 intersect at point P. Point P is the vanishing point, and the horizontal straight line passing through point P is the vanishing line L.

[0043] In one possible embodiment, step 1 includes:

[0044] Step 101: draw an initial 2D rectangular frame based on the target outline.

[0045] Step 102 : Based on a preset algorithm of the target type, an initial 3D frame is generated from the initial 2D rectangular frame.

[0046] For example, a vehicle is the target. First, the user draws an initial 2D rectangle based on the vehicle's outline using the mouse. The coordinates of this initial 2D rectangle are determined based on the front or rear of the vehicle. A pre-set algorithm calculates the vehicle's initial 3D bounding box based on its general structure.

[0047] In a specific implementation, after obtaining the initial 3D frame, a property dialog box pops up on the page, and the user can fill in relevant property information.

[0048] In a possible embodiment, when the volume size of the generated initial 3D box is 0, step 1 to step 2 are executed again.

[0049] like Figure 3 The figure shows an embodiment of generating a 3D frame when 3Dside is 0 provided by the present invention. In the process of 3D frame annotation, it may be possible to encounter a situation where the front of an object is almost invisible and only the side can be seen, such as a vehicle on the road. Figure 2 The 3D box annotation in does not meet the annotation requirements, so it is necessary to use Figure 3 In the diagram, for a 3D box with side 0, A2 overlaps with A1, B2 overlaps with B1, B3 overlaps with B4, and A3 overlaps with A4. For this special case of side 0, repeat steps 1 and 2.

[0050] Step 2: Adjust the position of the vanishing line. Based on the principle of single-point perspective, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time to obtain the final 3D frame that accurately fits the target.

[0051] In a possible embodiment, the method for calculating and adjusting the vertex positions of the 3D box corresponding to the vanishing line in real time based on the principle of one-point perspective in step 2 is:

[0052] Based on the principle of single-point perspective, the coordinate constraints of the eight vertices are set. Based on the constraints and the equation of the vanishing line, the vertex positions of the 3D box corresponding to the vanishing line are calculated in real time.

[0053] In a possible embodiment, when the user adjusts the position of the vanishing line (which may refer to the position of the horizon), the program adjusts the positions of the 3D box vertices in real time.

[0054] It is known that the side edges of the 3D box are perpendicular to the ground, so we can assume that the coordinates of the eight vertices are A1 (x1, y1), A2 (x2, y1), A3 (x2, y2), A4 (x1, y2), B1 (x4, y4), B2 (x3, y4), B3 (x3, y3), and B4 (x4, y3). Based on the definition of the vanishing point, we can derive the following constraint relationship:

[0055] =

[0056] Assuming that the equation of the vanishing line L is y=y0, combined with the constraint relationship, the coordinates of x4 and y4 can be calculated. The coordinates of other points remain unchanged. That is, when the vanishing line is adjusted, the coordinates of the rectangle A1A2A3A4 in the front remain unchanged, while the coordinates of the rectangle B1B2B3B4 in the back will change.

[0057] In a possible embodiment, when calculating and adjusting the vertex positions of the 3D box corresponding to the vanishing line in step 2, the positions of the four vertices of the rectangle in front of the 3D box remain unchanged, and the positions of the four vertices of the rectangle behind the 3D box are changed.

[0058] like Figure 4 Shown is an effect diagram of an embodiment of an image 3D frame annotation based on vanishing lines provided by the present invention.

[0059] Example 2

[0060] Embodiment 2 of the present invention is an embodiment of a 3D frame annotation system for images based on vanishing lines provided by the present invention. Figure 5 This is a structural diagram of a 3D frame annotation system for images based on vanishing lines provided by an embodiment of the present invention, combined with Figure 5 It can be seen that the embodiment of the 3D box annotation system includes: a vanishing line generation module and a 3D box generation module.

[0061] The vanishing line generation module is used to obtain an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point.

[0062] The 3D frame generation module is used to adjust the position of the vanishing line. Based on the principle of single-point perspective, it calculates and adjusts the vertex positions of the 3D frame corresponding to the vanishing line in real time to obtain the final 3D frame that accurately fits the target.

[0063] It can be understood that the image 3D frame annotation system based on vanishing lines provided by the present invention corresponds to the image 3D frame annotation method based on vanishing lines provided in the aforementioned embodiments. The relevant technical features of the image 3D frame annotation system based on vanishing lines can refer to the relevant technical features of the image 3D frame annotation method based on vanishing lines, and will not be repeated here.

[0064] See also Figure 6 , Figure 6 Schematic diagram of an embodiment of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored in the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, the following steps are implemented: Step 1, obtaining an initial 3D frame based on the target outline, determining the vanishing point of the initial 3D frame, and generating a vanishing line in the horizontal direction passing through the vanishing point; Step 2, adjusting the position of the vanishing line, calculating and adjusting the vertex position of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective, and obtaining a final 3D frame that accurately fits the target.

[0065] See also Figure 7 , Figure 7 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 7 As shown, this embodiment provides a computer-readable storage medium 1400, on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, the following steps are implemented: Step 1, based on the target outline, an initial 3D frame is obtained, the vanishing point of the initial 3D frame is determined, and a vanishing line in the horizontal direction and passing through the vanishing point is generated; Step 2, the position of the vanishing line is adjusted, and the vertex position of the 3D frame corresponding to the vanishing line is calculated and adjusted in real time based on the principle of single-point perspective to obtain a final 3D frame that accurately fits the target.

[0066] The embodiments of the present invention provide a method, system, electronic device and storage medium for annotating an image 3D frame based on vanishing lines. To address the problems that ordinary 3D frame image annotations are difficult to fit the contour and require multiple vertex adjustments, the method first uses the perspective principle to obtain the vanishing line of the 3D frame; then the user can adjust multiple vertices of the 3D frame by adjusting the vanishing line. Conversely, when the user adjusts certain vertices of the 3D frame, the vanishing line will also be adjusted accordingly; finally, the 3D frame can accurately fit the contour of the object; the user changes the contour of the 3D frame by adjusting the vanishing line. In most cases, the position of the vanishing line can be determined based on the position of the distant horizon, without having to manually imagine the position of vertices blocked by the vehicle body, making it easier to draw and adjust the 3D frame; the user can quickly adjust the vanishing line by observing the horizon position, thereby adjusting the 3D frame position, which is more efficient, can greatly improve the annotation efficiency, and save work time.

[0067] It should be noted that, in the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0068] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1The function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0072] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0073] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for annotating 3D frames of images based on vanishing lines, characterized in that: The 3D frame annotation method includes: Step 1: Obtain an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point; Step 2: Adjust the position of the vanishing line, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective, and obtain a final 3D frame that accurately fits the target; The step 1 comprises: Step 101, drawing an initial 2D rectangular frame based on the target outline; Step 102 , generating the initial 3D frame from the initial 2D rectangular frame using a preset algorithm based on the target type; When calculating and adjusting the vertex positions of the 3D frame corresponding to the vanishing line in step 2, the positions of the four vertices of the rectangle in front of the 3D frame remain unchanged, and the positions of the four vertices of the rectangle behind the 3D frame are changed.

2. The 3D box annotation method according to claim 1, characterized in that: When the volume size of the generated initial 3D frame is 0, step 1 to step 2 are executed again.

3. The 3D box annotation method according to claim 1, characterized in that: The method for calculating and adjusting the vertex positions of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective in step 2 is: A constraint relationship of the coordinates of eight vertices is set based on the principle of single-point perspective, and based on the constraint relationship and the equation of the vanishing line, the vertex positions of the 3D frame corresponding to the vanishing line that are adjusted in real time are calculated.

4. The 3D box annotation method according to claim 3, characterized in that: The constraint relationship is: = ； Among them, A1 (x1, y1), A2 (x2, y1), A3 (x2, y2), A4 (x1, y2), B1 (x4, y4), B2 (x3, y4), B3 (x3, y3) and B4 (x4, y3) are the coordinates of the eight vertices of the 3D box. A1, A2, A3 and A4 are the upper left, upper right, lower right and lower left vertices of the rectangle in front of the 3D box, and B1, B2, B3 and B4 are the upper left, upper right, lower right and lower left vertices of the rectangle behind the 3D box.

5. A 3D frame annotation system for images based on vanishing lines, characterized in that: The 3D frame annotation system includes: a vanishing line generation module and a 3D frame generation module; The vanishing line generation module is configured to obtain an initial 3D frame based on the target outline, determine the vanishing point of the initial 3D frame, and generate a horizontal vanishing line passing through the vanishing point; The 3D frame generation module is used to adjust the position of the vanishing line, calculate and adjust the vertex positions of the 3D frame corresponding to the vanishing line in real time based on the principle of single-point perspective, and obtain a final 3D frame that accurately fits the target; The vanishing line generation module includes: Step 101, drawing an initial 2D rectangular frame based on the target outline; Step 102 , generating the initial 3D frame from the initial 2D rectangular frame using a preset algorithm based on the target type; When the 3D frame generation module calculates and adjusts the vertex positions of the 3D frame corresponding to the vanishing line, the positions of the four vertices of the rectangle in front of the 3D frame remain unchanged, and the positions of the four vertices of the rectangle behind the 3D frame are changed.

6. An electronic device, characterized in that: The system comprises a memory and a processor, wherein the processor is configured to implement the steps of the method for annotating a 3D frame of an image based on vanishing lines according to any one of claims 1 to 4 when executing a computer management program stored in the memory.

7. A computer-readable storage medium, characterized in that A computer management program is stored thereon, and when the computer management program is executed by the processor, the steps of the image 3D frame annotation method based on vanishing lines according to any one of claims 1 to 4 are implemented.

Images