A fusion annotation method, system, electronic device and storage medium

Abstract

The present invention provides a fusion annotation method, system, electronic device, and storage medium. The method includes: calculating the corresponding coordinates of the left and right bottom edges of each image in a point cloud coordinate system based on the coordinates of the bottom edges of the left and right edges of a pre-annotated image, the internal and external parameters of a camera, and the external parameters of a laser radar; obtaining a point cloud annotation frame and a pre-annotated image corresponding to the point cloud annotation frame, and constructing vectors from the origin of the point cloud coordinate system to the coordinates of the bottom edges of the left and right edges of the pre-annotated image in the point cloud coordinate system; if the center point of the point cloud annotation frame is within the inferior angle formed by the two vectors, drawing a picture annotation frame in the corresponding pre-annotated image based on the point cloud annotation frame. This method can achieve fusion annotation of a single-frame point cloud and multiple-frame images, improve annotation efficiency, and effectively ensure the correspondence between point cloud data and image data.

Description

Technical Field

[0001] The present invention belongs to the field of data fusion processing, and in particular relates to a fusion annotation method, system, electronic equipment and storage medium. Background Art

[0002] As autonomous driving research continues to deepen, the requirements are becoming increasingly stringent. To achieve better autonomous driving results, manufacturers are installing more sensors on vehicles, resulting in an increasing number of lidars and cameras. These vehicles also collect richer data while driving, including large amounts of single-frame point cloud data and multi-view images.

[0003] When fusing point clouds and images, generally only one frame of point cloud data and one matching image can be annotated simultaneously. For data sources with multiple lidars and multiple cameras, since images are not easily stitched and fused together like point cloud data, annotation will require annotating one frame of point cloud data and multiple matching images simultaneously. Conventional point cloud and image fusion annotation methods can only annotate the point cloud and images separately and then merge them. This not only reduces annotation efficiency, but also fails to guarantee that the point cloud annotation data and the image annotation data are completely consistent. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a fusion annotation method, system, electronic device, and storage medium for solving the problem of low efficiency in the existing fusion annotation of point cloud and image data.

[0005] In a first aspect of an embodiment of the present invention, a fusion annotation method is provided, comprising:

[0006] Calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the pre-annotated image's left and right bottom edge coordinates, the camera's internal and external parameters, and the lidar's external parameters.

[0007] Get the point cloud annotation box and the pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system;

[0008] If the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors, the image annotation box is drawn in the corresponding pre-annotated image according to the point cloud annotation box.

[0009] In a second aspect of an embodiment of the present invention, a fusion annotation system is provided, including:

[0010] The coordinate conversion module is used to calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the coordinates of the left and right bottom edges of the pre-annotated images, the internal and external parameters of the camera, and the external parameters of the lidar;

[0011] A vector construction module is used to obtain the point cloud annotation box and the pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the coordinates of the lower edges of the left and right boundaries of the pre-annotated image in the point cloud coordinate system;

[0012] A calculation and judgment module is used to calculate and judge whether the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors;

[0013] The image annotation module is used to draw an image annotation box in the corresponding pre-annotated image according to the point cloud annotation box when the center point of the point cloud annotation box is located in the inferior angle formed by two vectors.

[0014] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.

[0015] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.

[0016] In this embodiment of the present invention, the bottom edge of the pre-annotated image boundary is converted to the point cloud coordinate system. Whether to perform image annotation is determined based on whether the center of the point cloud annotation box is within the inferior angle of the two vectors formed by the center point and the bottom edges of the left and right boundaries. This enables the fusion annotation of a single-frame point cloud and multiple images, and automated annotation of point clouds and images. This effectively improves annotation efficiency, saves manpower and material costs, and ensures that the point cloud and image annotation data are completely consistent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a fusion annotation method provided in an embodiment of the present invention;

[0019] Figure 2Another schematic diagram of a flow chart of a fusion annotation method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the structure of a fusion annotation system provided by an embodiment of the present invention;

[0021] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] It should be understood that the terms "including" and similar expressions in the specification, claims, and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, or apparatus comprising a series of steps or units is not limited to the listed steps or units. Furthermore, the terms "first" and "second" are used to distinguish between different objects and are not intended to describe a specific order.

[0024] See also Figure 1 , a flowchart of a fusion annotation method provided by an embodiment of the present invention includes:

[0025] S101, calculating the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the pre-annotated left and right bottom edge coordinates of the image, the camera's internal and external parameters, and the lidar's external parameters;

[0026] The pre-labeled image is a picture that needs to be labeled, which is generally acquired by multiple different cameras. The left and right bottom edges of the pre-labeled image are the bottom edges of the left and right sides of the image, that is, the lower left and right corners of the image.

[0027] The camera's internal parameters include the camera's focal length, pixels, etc.; the camera's external parameters, that is, the camera's posture, generally include the camera's position, rotation direction, etc.; the lidar's external parameters include the radar's position, rotation angle, etc.

[0028] Convert the coordinates of the lower left and right edges of the pre-annotated image to the point cloud coordinate system to determine whether the point cloud annotation box should appear in the image.

[0029] Specifically, the coordinates corresponding to the lower edges of the left and right boundaries of the pre-annotated image in the point cloud coordinate system are calculated according to formula (1):

[0030]

[0031] Among them, P lidar Indicates the coordinates corresponding to the lower edge of the left or right boundary of the image in the point cloud coordinate system. (x, y, z) indicates the three-dimensional coordinates in the point cloud coordinate system. P camera Indicates the coordinates of the bottom left or right edge of the image, K -1 Represents the inverse matrix of the K matrix, the K matrix is ​​the camera internal parameter matrix, T lidar represents the laser radar offset, T camera Represents the camera offset, is the lidar rotation matrix R lidar The inverse matrix, R camera is the camera's rotation matrix.

[0032] S102: Obtain a point cloud annotation frame and a pre-annotated image corresponding to the point cloud annotation frame, and construct vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system;

[0033] Multiply the vectors starting from the origin of the point cloud coordinate system and pointing to the coordinates of the lower edge of the left and right boundaries of the pre-annotated image in the point cloud coordinate system by the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the center point of the point cloud annotation box to obtain two vectors. Based on the vectors, determine whether the coordinates of the center point are within the inferior angle (180°) formed by the vectors.

[0034] Among them, the z-axis coordinate values ​​of the center point of the point cloud annotation box and the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system are set to zero respectively.

[0035] Specifically, the judgment vector is calculated according to formula (2) by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the left boundary of the pre-annotated image in the point cloud coordinate system:

[0036]

[0037] According to formula (3), the judgment vector is calculated by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the right boundary of the pre-annotated image in the point cloud coordinate system:

[0038]

[0039] In the formula, (x L ,y L ,z L ) represents the vector R L The coordinates of (x R ,y R ,z R) represents the vector R R The coordinates of P left0 Represents the vector with the origin of the point cloud coordinate system as the starting point and pointing to the coordinates of the lower left edge of the pre-annotated image in the point cloud coordinate system when the z value is 0, P right0 Indicates the vector with the origin of the point cloud coordinate system as the starting point and the coordinate of the lower right edge of the pre-annotated image in the point cloud coordinate system when the z value is 0, (x boxCenter ,y boxCenter , 0) represents the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the center point of the point cloud annotation box when the z value is 0.

[0040] S103: If the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors, a picture annotation box is drawn in the corresponding pre-annotated picture according to the point cloud annotation box.

[0041] Optionally, if the center point of the point cloud annotation box is not within the inferior angle formed by the two vectors, the corresponding pre-annotated image is marked, and the annotation box is not drawn for the marked pre-annotated image.

[0042] Determine whether the coordinates of the center point of the annotation box fall within the inferior angle formed by the two vectors. If so, the point cloud annotation box should appear in the image corresponding to this set of corresponding vectors; otherwise, it should not appear. Draw the image annotation box in the corresponding image and assign the corresponding attributes to the annotation box.

[0043] In this embodiment, the fusion annotation of one frame of point cloud and multiple pictures is realized, which improves the efficiency of annotation and ensures that the point cloud annotation data is completely corresponding to the picture annotation data.

[0044] It should be noted that, for formula (1), P camera Indicates the coordinates of the bottom left or right edge of the image. Its meaning is as shown in Definition 1. lidar and R camera The meaning of is as shown in Definition 2, where r x1 , r y1 , r z1 , r x2 , r y2 , r z2 Both are 3x1 matrices, and their meanings are as shown in Definition 3. In Definition 3, q ​​and p are both 4x1 matrices, q represents the laser radar rotation, and its value can be obtained from the laser radar external parameters; p represents the camera rotation, and its value can be obtained from the camera external parameters; q -1 and p -1 The meaning of is shown in Definition 4. The rightmost side of the equal sign of each equation in Definition 3 is calculated from left to right, and the calculation between each two 4x1 matrices is performed according to the rules shown in Definition 5.

[0045] Definition 1: If the image is x pixels wide and y pixels high, the coordinates of the bottom left edge of the image are The coordinates of the lower edge of the right boundary are Definition 2: R lidar =(r x1 r y1 r z1 ), R camera =(r x2 r y2 r z2 ).

[0046] Definition 3:

[0047] Definition 4: If but like but

[0048] Definition 5:

[0049] For formulas (2) and (3), if the corresponding coordinates of the lower left edge of the image are but If the corresponding coordinates of the lower right edge of the image are but

[0050] After calculating the corresponding coordinates of each group, when annotating the point cloud, take the coordinates of the center point of the point cloud annotation frame Then take the corresponding coordinates of each group in turn, substitute them into formula (2) and formula (3) respectively to calculate and obtain multiple groups of R L and R R For each group of R L and R R The value is judged, when z L >0 and z R <0 (z L , z R is the deviation value or height value relative to the vector plane), indicating that the point cloud annotation box should appear in this set of R L and R R According to the conclusion, the conventional point cloud and image fusion annotation method is used to calculate and draw the corresponding image annotation box in each image and assign the corresponding attributes.

[0051] In another embodiment, Figure 2 As shown in the figure, the fusion annotation process of single-frame point cloud and multi-frame images is as follows:

[0052] Obtain the lidar offset T1 and rotation Q1 from the lidar extrinsic parameters, obtain the intrinsic and extrinsic parameters of multiple cameras, and save them to array C. Obtain the intrinsic and extrinsic parameters C1 of a camera from C. Obtain the matrix K, offset T2, and rotation Q2 from C1. Define the coordinates PL and PR of the lower left and right edges of the image. Substitute PL, PR, T1, Q1, K, T2, and Q2 into formula (1) to calculate the corresponding coordinates PPL and PPR of PL and PR, and store them in array PP. Repeat the above steps until all the parameters in C are calculated.

[0053] Obtain a point cloud annotation data set S, obtain a point cloud annotation box data B from S, and obtain the image set PS corresponding to B according to the order of camera parameters in C. Obtain the center point coordinates BC of the point cloud annotation box from B and define the result array RS. Obtain a set of PPL and PPR from PP, and substitute PPL and BC into formula (2) to calculate the result RL. Substitute PPR and BC into formula (3) to calculate the result RR. Determine whether the z value of RL is greater than 0 and the z value of RR is less than 0. If so, add a record true to RS, otherwise add a record false to RS. Repeat the above steps until all values ​​in PP are calculated.

[0054] Take out one picture PSQ from PS one by one, and take out one record RSQ from RS in the same order. If RSQ is true, draw the picture BP corresponding to B in PSQ. Repeat this process until all data in PS and RS are judged and drawn. Clear PS and RS, and draw all annotation boxes on the corresponding pictures.

[0055] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0056] Figure 3 A schematic diagram of the structure of a fusion annotation system provided in an embodiment of the present invention, the system includes:

[0057] The coordinate conversion module 310 is used to calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the pre-annotated left and right bottom edge coordinates of the image, the camera's internal and external parameters, and the lidar's external parameters;

[0058] Specifically, the coordinate conversion module 310 includes:

[0059] The calculation unit calculates the coordinates corresponding to the lower edge coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system according to formula (1):

[0060]

[0061] Among them, P lidar Indicates the coordinates corresponding to the lower edge of the left or right boundary of the image in the point cloud coordinate system. (x, y, z) indicates the three-dimensional coordinates in the point cloud coordinate system. P camera Indicates the coordinates of the bottom left or right edge of the image, K -1 Represents the inverse matrix of the K matrix, the K matrix is ​​the camera internal parameter matrix, T lidar represents the laser radar offset, T camera Represents the camera offset, is the lidar rotation matrix R lidar The inverse matrix, R camera is the camera's rotation matrix.

[0062] A vector construction module 320 is used to obtain a point cloud annotation box and a pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system;

[0063] Among them, the z-axis coordinate values ​​of the center point of the point cloud annotation box and the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system are set to zero respectively.

[0064] Specifically, according to formula (2), the judgment vector is calculated by multiplying the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the center point of the point cloud annotation box with the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the lower edge of the left boundary of the pre-annotated image in the point cloud coordinate system:

[0065]

[0066] According to formula (3), the judgment vector is calculated by multiplying the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the center point of the point cloud annotation box with the vector starting from the origin of the point cloud coordinate system and pointing to the coordinates of the lower edge of the right boundary of the pre-annotated image in the point cloud coordinate system:

[0067]

[0068] In the formula, (x L ,y L ,z L ) represents the vector R L The coordinates of (x R ,y R ,z R ) represents the vector R R The coordinates of P left0 Represents the vector pointing from the origin of the point cloud coordinate system to the coordinates of the lower left edge of the pre-annotated image in the point cloud coordinate system when the z value is 0, P right0 Indicates the vector pointing from the origin of the point cloud coordinate system to the coordinates of the lower right edge of the pre-annotated image in the point cloud coordinate system when the z value is 0, (xboxCenter ,y boxCenter , 0) represents the vector pointing from the origin of the point cloud coordinate system to the center point of the point cloud annotation box when the z value is 0.

[0069] A calculation and judgment module 330 is used to calculate and judge whether the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors;

[0070] The image annotation module 340 is configured to draw an image annotation box in the corresponding pre-annotated image according to the point cloud annotation box when the center point of the point cloud annotation box is located within the inferior angle formed by two vectors.

[0071] Among them, if the center point of the point cloud annotation box is not within the inferior angle formed by the two vectors, the corresponding pre-annotated image is marked, and the annotation box is not drawn for the marked pre-annotated image.

[0072] Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0073] Figure 4 This is a structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device is used for fusion annotation of a single-frame point cloud and multiple-frame images. Figure 4 As shown, the electronic device 4 of this embodiment includes: a memory 410, a processor 420 and a system bus 430, wherein the memory 410 includes an executable program 4101 stored thereon. It can be understood by those skilled in the art that Figure 4 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0074] The following combination Figure 4 A detailed introduction to the various components of electronic equipment:

[0075] The memory 410 can be used to store software programs and modules. The processor 420 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 410. The memory 410 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cached data), etc. In addition, the memory 410 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0076] The memory 410 includes an executable program 4101 for the network request method. The executable program 4101 can be divided into one or more modules / units. The one or more modules / units are stored in the memory 410 and executed by the processor 420 to implement data fusion annotation, etc. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions. The instruction segments are used to describe the execution process of the computer program 4101 in the electronic device 4. For example, the computer program 4101 can be divided into a coordinate conversion module, a vector construction module, a calculation and judgment module, and an image annotation module.

[0077] Processor 420 is the control center of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 410 and accessing data stored in memory 410, it performs various functions of the electronic device and processes data, thereby monitoring the overall status of the electronic device. Optionally, processor 420 may include one or more processing units; preferably, processor 420 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, application programs, etc., and the modem processor primarily handles wireless communications. It is understood that the modem processor described above may not be integrated into processor 420.

[0078] The system bus 430 connects the various functional components within the computer and can transmit data, address information, and control information. It can be a PCI bus, an ISA bus, a VESA bus, or other bus types. Instructions from the processor 420 are transmitted to the memory 410 via the bus, and the memory 410 feeds data back to the processor 420. The system bus 430 is responsible for the exchange of data and instructions between the processor 420 and the memory 410. Of course, the system bus 430 can also connect to other devices, such as network interfaces and display devices.

[0079] In an embodiment of the present invention, the executable program executed by the processing 420 included in the electronic device includes:

[0080] Calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the pre-annotated image's left and right bottom edge coordinates, the camera's internal and external parameters, and the lidar's external parameters.

[0081] Get the point cloud annotation box and the pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system;

[0082] If the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors, the image annotation box is drawn in the corresponding pre-annotated image according to the point cloud annotation box.

[0083] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0084] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0085] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion annotation method, characterized in that: include: Calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the pre-annotated image's left and right bottom edge coordinates, the camera's internal and external parameters, and the lidar's external parameters. Among them, the coordinates corresponding to the lower edge coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system are calculated according to formula (1): ；（1） in, Indicates the coordinates corresponding to the lower edge of the left or right boundary of the image in the point cloud coordinate system. (x, y, z) indicates the three-dimensional coordinates in the point cloud coordinate system. Indicates the coordinates of the bottom left or right edge of the image. Represents the inverse matrix of the K matrix, the K matrix is ​​the camera intrinsic parameter matrix, represents the lidar offset, Represents the camera offset, is the lidar rotation matrix The inverse matrix of is the camera's rotation matrix; Get the point cloud annotation box and the pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system; Among them, the judgment vector is calculated according to formula (2) by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the left boundary of the pre-annotated image in the point cloud coordinate system: ；（2） According to formula (3), the judgment vector is calculated by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the right boundary of the pre-annotated image in the point cloud coordinate system: ；（3） Where, ( x L ,y L ,z L ) represents a vector R L The coordinates of x R ,y R ,z R ) represents a vector R R The coordinates of P left0 Indicates the vector of the coordinates of the lower left edge of the pre-annotated image in the point cloud coordinate system when the z value is 0. P right0 Indicates that the origin of the point cloud coordinate system when the z value is 0 points to the vector of the coordinates of the lower right edge of the pre-annotated image in the point cloud coordinate system, ( x boxCenter ， y boxCenter ，0 ) represents the vector from the origin of the point cloud coordinate system to the center of the point cloud annotation box when the z value is 0; If the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors, the image annotation box is drawn in the corresponding pre-annotated image according to the point cloud annotation box.

2. The method according to claim 1, characterized in that Before constructing vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system, the method includes: Set the z-axis coordinate values ​​of the center point of the point cloud annotation box and the lower edge coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system to zero respectively.

3. The method according to claim 1, characterized in that If the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors, drawing the image annotation box in the corresponding pre-annotated image according to the point cloud annotation box further includes: If the center point of the point cloud annotation box is not within the inferior angle formed by the two vectors, the corresponding pre-annotated image is marked, and the annotation box is not drawn for the marked pre-annotated image.

4. A fusion annotation system, characterized in that: include: The coordinate conversion module is used to calculate the corresponding coordinates of the left and right bottom edges of each image in the point cloud coordinate system based on the coordinates of the left and right bottom edges of the pre-annotated images, the internal and external parameters of the camera, and the external parameters of the lidar; Among them, the coordinates corresponding to the lower edge coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system are calculated according to formula (1): ；（1） in, Indicates the coordinates corresponding to the lower edge of the left or right boundary of the image in the point cloud coordinate system. (x, y, z) indicates the three-dimensional coordinates in the point cloud coordinate system. Indicates the coordinates of the bottom left or right edge of the image. Represents the inverse matrix of the K matrix, the K matrix is ​​the camera intrinsic parameter matrix, represents the lidar offset, Represents the camera offset, is the lidar rotation matrix The inverse matrix of is the camera's rotation matrix; A vector construction module is used to obtain the point cloud annotation box and the pre-annotated image corresponding to the point cloud annotation box, and construct vectors from the origin of the point cloud coordinate system to the coordinates of the lower edges of the left and right boundaries of the pre-annotated image in the point cloud coordinate system; Among them, the judgment vector is calculated according to formula (2) by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the left boundary of the pre-annotated image in the point cloud coordinate system: ；（2） According to formula (3), the judgment vector is calculated by multiplying the vector from the origin of the point cloud coordinate system to the coordinates of the center point of the point cloud annotation box and the vector from the origin of the point cloud coordinate system to the coordinates of the lower edge of the right boundary of the pre-annotated image in the point cloud coordinate system: ；（3） Where, ( x L ,y L ,z L ) represents a vector R L The coordinates of x R ,y R ,z R ) represents a vector R R The coordinates of P left0 Indicates the vector of the coordinates of the lower left edge of the pre-annotated image in the point cloud coordinate system when the z value is 0. P right0 Indicates that the origin of the point cloud coordinate system when the z value is 0 points to the vector of the coordinates of the lower right edge of the pre-annotated image in the point cloud coordinate system, ( x boxCenter ， y boxCenter ，0 ) represents the vector from the origin of the point cloud coordinate system to the center of the point cloud annotation box when the z value is 0; A calculation and judgment module is used to calculate and judge whether the center point of the point cloud annotation box is located within the inferior angle formed by the two vectors; The image annotation module is used to draw an image annotation box in the corresponding pre-annotated image according to the point cloud annotation box when the center point of the point cloud annotation box is located in the inferior angle formed by two vectors.

5. The system according to claim 4, characterized in that Before constructing vectors from the origin of the point cloud coordinate system to the lower coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system, the method includes: Set the z-axis coordinate values ​​of the center point of the point cloud annotation box and the lower edge coordinates of the left and right boundaries of the pre-annotated image in the point cloud coordinate system to zero respectively.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the fusion annotation method according to any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of the fusion annotation method according to any one of claims 1 to 3 are implemented.

Images