Data annotation method, data annotation device, electronic equipment and program product

Abstract

The invention is suitable for the technical field of computer vision, and provides a data annotation method, a data annotation device, electronic equipment and a program product. The data labeling method comprises the following steps: determining a visible surface in six geometric surfaces of a cuboid object in a color image; determining four first vertex coordinates of the visible surface in a camera coordinate system; determining first pose information of the visible surface based on the four first vertex coordinates; the first pose information refers to pose information of the visible surface under the camera coordinate system; the four first vertex coordinates and the first pose information are labeling data of the visible surface, the labeling data of the visible surface is used for labeling the visible surface in a target image, and the target image at least comprises the color image. According to the invention, the problems of low manual labeling precision, poor consistency, low efficiency and the like can be solved.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and in particular relates to a data annotation method, a data annotation device, an electronic device, and a program product. Background Technology

[0002] In industrial settings such as warehousing, sorting, and loading / unloading, open plastic boxes, cardboard boxes, tabletops, and pallets—rectangular objects—are the most frequently encountered and numerous key objects for robots during movement and handling operations. For robots to perform actions such as grasping, placing, stacking, and obstacle avoidance in unstructured environments, they must achieve centimeter-level or even millimeter-level precision in locating these objects. Precise location of these objects requires large-scale, high-precision, and highly consistent annotation data. Currently, manual annotation is the primary method, but this suffers from low accuracy, poor consistency, and low efficiency. Summary of the Invention

[0003] This application provides a data annotation method, data annotation device, electronic device, and program product, which can solve the problems of low accuracy, poor consistency, and low efficiency of manual annotation.

[0004] In a first aspect, embodiments of this application provide a data annotation method, including:

[0005] Identify the visible faces among the six geometric faces of a cuboid object in a color image;

[0006] Determine the coordinates of the four first vertices of the visible surface in the camera coordinate system;

[0007] Based on the coordinates of the four first vertices, the first pose information of the visible face is determined; the first pose information refers to the pose information of the visible face in the camera coordinate system.

[0008] The coordinates of the four first vertices and the first pose information constitute the annotation data of the visible faces. The annotation data of the visible faces is used to annotate the visible faces in the target image, and the target image includes at least the color image.

[0009] In this embodiment, by determining the visible surfaces of the color image on the six geometric faces of the cuboid object, the coordinates of the four first vertices of the visible surfaces in the camera coordinate system can be determined. Based on these four first vertex coordinates, the first pose information of the visible surfaces can be determined, thereby automatically generating annotation data including the coordinates of the four first vertices and the first pose information. The entire process does not require manual interaction, solving the problems of low accuracy, poor consistency and low efficiency of manual annotation, and can achieve millimeter-level accuracy annotation at the scale of millions of images.

[0010] Secondly, embodiments of this application provide a data annotation apparatus, comprising:

[0011] The first determining module is used to determine the visible faces among the six geometric faces of the cuboid object in the color image;

[0012] The second determining module is used to determine the coordinates of the four first vertices of the visible surface in the camera coordinate system;

[0013] The third determining module is used to determine the first pose information of the visible face based on the coordinates of the four first vertices; the first pose information refers to the pose information of the visible face in the camera coordinate system.

[0014] The coordinates of the four first vertices and the first pose information constitute the annotation data of the visible faces. The annotation data of the visible faces is used to annotate the visible faces in the target image, and the target image includes at least the color image.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the data annotation method as described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computer, implements the data annotation method as described in the first aspect above.

[0017] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the data annotation method as described in the first aspect above to be executed.

[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the data annotation method provided in an embodiment of this application;

[0021] Figure 2This is an example diagram of a color image rendered and output in a simulation scene provided in the embodiments of this application;

[0022] Figure 3 This is an example image of the depth image rendered in the simulation scene provided in the embodiments of this application;

[0023] Figure 4 This is an example diagram of the overall mask of a cuboid object rendered in a simulation scene provided in this application embodiment;

[0024] Figure 5 These are the pose and size information of the cuboid object rendered in the simulation scene provided in this application embodiment, in the camera coordinate system.

[0025] Figure 6 This is an example diagram of the pixel-level mask for each visible face in the simulation scenario provided in the embodiments of this application;

[0026] Figure 7 This is an example diagram of the first pose information of each visible face in the simulation scenario provided in the embodiments of this application;

[0027] Figure 8 This is an example diagram of ten consecutive color images provided in an embodiment of this application;

[0028] Figure 9 This is an example diagram of the fourth pose information of the camera provided in the embodiments of this application;

[0029] Figure 10 This is an example diagram showing the pose information of each AprilTag in a color image provided in the embodiments of this application;

[0030] Figure 11 This is an example diagram of point clouds from different perspectives fused together to form a point cloud from the same perspective, as provided in the embodiments of this application.

[0031] Figure 12 This is a schematic diagram of the data annotation device provided in the embodiments of this application;

[0032] Figure 13 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0038] In industrial settings such as warehousing, sorting, and loading / unloading, open plastic boxes, cardboard boxes, tabletops, and pallets—rectangular objects—are the most frequently encountered and numerous key objects for robots to handle during movement and transport. For robots to perform actions such as grasping, placing, stacking, and obstacle avoidance in unstructured environments, they must achieve centimeter-level or even millimeter-level precision in locating these objects.

[0039] Since cuboids have known three-dimensional geometric constraints, their positioning strategies are generally divided into two categories:

[0040] In the first category, when only one side is visible, the four vertices of the surface are first fitted using a two-dimensional image, and then the occluded three-dimensional information is completed using pre-set length, width, height and other size information, thereby restoring the complete pose.

[0041] The second type, when two or three sides are visible, directly uses the three-dimensional coordinates of the visible vertices in the image-depth space to estimate the object's size and pose in one go through rigid body constraints.

[0042] With the rise of learning-based visual perception algorithms, the geometric reasoning process in the aforementioned localization strategies is gradually being replaced by end-to-end artificial intelligence (AI) models. However, whether it's instance segmentation, keypoint detection, or pose regression networks, all require large-scale, high-precision, and highly consistent labeled data. Currently, manual annotation is the primary method, which has three main drawbacks:

[0043] Low precision: Depth edges and occlusion boundaries are difficult to accurately define with the naked eye, and vertex position errors usually exceed 5px.

[0044] Poor consistency: Different annotators have inconsistent understandings of the correspondence between "face" and "whole", resulting in the pose standard deviation of the same dataset being greater than 2cm;

[0045] Low efficiency: A single RGB-D image with a resolution of 1920×1080 takes an average of 8 to 12 minutes to process, which is insufficient to support the construction of datasets with hundreds of thousands of images.

[0046] Furthermore, manual annotation is typically limited to interaction on a two-dimensional display screen. Annotators cannot intuitively manipulate vertices or planes in three-dimensional space, which presents the following difficulties for the crucial annotation of "the three-dimensional pose of each face of a cuboid-like object":

[0047] Lacking depth perception, the error rate in judging occlusion relationships is high;

[0048] Mouse dragging can only move within the image plane; in the depth direction, it is necessary to repeatedly switch the viewpoint and input values, which results in large cumulative errors.

[0049] When multiple faces are visible simultaneously, it is difficult to guarantee that the vertices of each face are collinear, coplanar, or orthogonal, resulting in a serious deviation between the final 3D pose and the real geometry.

[0050] Therefore, there is an urgent need for a fully automated annotation method for cuboid objects that is suitable for industrial scenarios, requires no manual intervention, and can be directly applied to RGB-D images or binocular stereo images, so as to generate high-quality training and evaluation data in batches at low cost.

[0051] Based on this, embodiments of this application provide a data annotation method, a data annotation device, an electronic device, and a program product. In embodiments of this application, by determining the visible surfaces of a color image on the six geometric faces of a cuboid object, the coordinates of the four first vertices of the visible surfaces in the camera coordinate system can be determined. Based on these four first vertex coordinates, the first pose information of the visible surfaces can be determined, thereby automatically generating annotation data including the coordinates of the four first vertices and the first pose information. The entire process requires no manual interaction, solving the problems of low accuracy, poor consistency, and low efficiency of manual annotation, and enabling millimeter-level accuracy annotation at the scale of millions of images.

[0052] The data annotation method provided in this application can be applied to robots, robot control chips, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, desktop computers, servers, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and other electronic devices. This application does not limit the specific type of electronic device; it can be selected according to actual needs. It should be understood that this application does not limit the form of the robot; for example, it can be a humanoid robot.

[0053] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0054] Please see Figure 1 , Figure 1 The diagram shown is a schematic flowchart of the data annotation method provided in an embodiment of this application. It is provided as an example and not as a limitation. Taking the application of this data annotation method to a robot as an example, it illustrates the following. Figure 1 The process shown is explained in detail. This data annotation method includes the following steps:

[0055] Step 101: Determine the visible faces among the six geometric faces of the cuboid object in the color image.

[0056] This application does not limit the image format of the color image. For example, if the image format of the color image is RGB, then the color image is an RGB image; if the image format of the color image is YUV, then the color image is a YUV image; if the image format of the color image is HSV, then the color image is an HSV image.

[0057] The visible surfaces mentioned above may refer to the geometric surfaces that are actually visible from the current perspective among the six geometric surfaces.

[0058] It should be noted that the data standards in this application can be divided into fully automatic offline annotation and real data annotation. Fully automatic offline annotation refers to generating annotation data in a simulated scene. Real data annotation refers to generating annotation data in a real camera shooting scene.

[0059] Step 102: Determine the coordinates of the four first vertices of the visible surface in the camera coordinate system.

[0060] In the simulation scenario, the size information of the cuboid object is known. Based on this, the second pose information of the cuboid object in the color image can be rendered. The second pose information and the size information of the cuboid object can be converted into vertex coordinates in the cuboid object coordinate system to obtain the coordinates of the eight second vertices of the cuboid object. The eight second vertex coordinates are then converted to the camera coordinate system to obtain the coordinates of the eight first vertices of the cuboid object in the camera coordinate system. After determining the visible surfaces in the cuboid object, the coordinates of the four first vertices of the visible surfaces can be determined from these eight first vertex coordinates.

[0061] In a real-world camera shooting scenario, after determining the pose information (i.e., the third pose information) of the cuboid object in the world coordinate system in the color image through subsequent multi-view correlation and bundle adjustment (BA) optimization, the third pose information can be transformed into the camera coordinate system based on the camera's fourth pose information. After projecting it onto the image plane of the color image, the vertex coordinates of the two-dimensional quadrilaterals of the six geometric faces of the cuboid object can be obtained, that is, the coordinates of the eight second vertices of the cuboid object. Transforming the eight second vertex coordinates into the camera coordinate system yields the coordinates of the eight first vertices of the cuboid object in the camera coordinate system. After determining the visible faces in the cuboid object, the coordinates of the four first vertices of the visible faces can be determined from these eight first vertex coordinates.

[0062] Step 103: Determine the first pose information of the visible face based on the coordinates of the four first vertices; the first pose information refers to the pose information of the visible face in the camera coordinate system; the coordinates of the four first vertices and the first pose information are the annotation data of the visible face, which is used to annotate the visible faces in the target image, and the target image includes at least a color image.

[0063] In some embodiments, the target image may include not only a color image but also a depth image acquired simultaneously.

[0064] Optionally, the pose information in this application may refer to a six-degrees-of-freedom (6-DOF) pose.

[0065] In this embodiment, the entire process can be completed without human interaction in both simulated and real camera shooting scenarios, solving the problems of low accuracy, poor consistency, and low efficiency of manual annotation, and achieving millimeter-level accuracy annotation at the scale of millions of images.

[0066] In this embodiment, by determining the visible surfaces of the color image on the six geometric faces of the cuboid object, the coordinates of the four first vertices of the visible surfaces in the camera coordinate system can be determined. Based on these four first vertex coordinates, the first pose information of the visible surfaces can be determined, thereby automatically generating annotation data including the coordinates of the four first vertices and the first pose information. The entire process does not require manual interaction, solving the problems of low accuracy, poor consistency and low efficiency of manual annotation, and can achieve millimeter-level accuracy annotation at the scale of millions of images.

[0067] In some embodiments of this application, the annotation data of the visible surface also includes a pixel-level mask of the visible surface; the pixel-level mask of the visible surface is determined as follows:

[0068] Pixels that exist simultaneously within the visible surface and the overall mask of the cuboid object are defined as the actual visible pixels of the visible surface.

[0069] Generate a pixel-level mask for the visible surface based on the actual visible pixels of the visible surface;

[0070] The overall mask is the annotation data of the cuboid object, and the annotation data of the cuboid object is used to annotate the cuboid objects in the target image.

[0071] In this embodiment, the pixels that exist simultaneously in both the visible surface and the overall mask can be determined by finding the intersection of the visible surface and the overall mask, thereby determining the actual visible pixels of the visible surface.

[0072] In both simulated and real camera shooting scenarios, electronic devices can determine all the actual visible pixels (i.e., the set of actual visible pixels) of the visible surface by finding the intersection between the visible surface and the overall mask, thereby generating a pixel-level mask for the visible surface.

[0073] In simulated or real camera shooting scenarios, in some other embodiments of this application, when the visible surface is an open surface, the electronic device can also determine the pixel-level mask of the open surface in the following manner:

[0074] Calculate the second absolute difference between the actual depth and the corresponding theoretical depth of each pixel in the open surface;

[0075] If the second absolute difference is less than the second preset difference, then the pixel is determined to be an actual visible pixel of the open surface;

[0076] A pixel-level mask for the open surface is generated based on the actual visible pixels of the open surface.

[0077] Optionally, a second absolute difference can be set based on actual needs or empirical values. For example, the second absolute difference could be 1 cm.

[0078] In a color image, the depth map can be sampled pixel by pixel inside the two-dimensional quadrilateral of the open surface to obtain the actual depth of each pixel in the open surface. The theoretical depth of each pixel in the open surface can be calculated using the plane equation of the open surface.

[0079] The actual visible pixels of the aforementioned open surface can refer to the unobstructed pixels on the four thin edges of the open surface.

[0080] If the second absolute difference is less than the second preset difference, then the pixel is determined to be an actual visible pixel of the open surface (i.e., the unobstructed pixels on the four thin edges of the open surface); if the second absolute difference is greater than or equal to the second preset difference, then the pixel may be inside the cuboid object (e.g., inside the box opening) or a material area, and the pixel needs to be discarded.

[0081] The open surface is usually located on the top surface of a cuboid object, hence it can be called the open top surface. Of course, it should be understood that the open surface can also be located on the bottom or side surface of a cuboid object, and this application does not limit it in this regard.

[0082] In this embodiment, addressing the unique characteristics of the open top surface annotation (i.e., the depth discontinuity problem of "no top cover"), a dual determination using a plane equation and a second preset difference achieves pixel-level precision extraction of the truly visible area of ​​the open surface. This effectively distinguishes the top surface from the internal space of the box, resulting in a misclassification rate of less than 0.5% for the open top surface. This significantly improves the safety of cuboid object grasping and stacking scenarios and resolves the annotation ambiguity problem caused by depth discontinuity. The resulting mask possesses extremely high geometric realism and physical reliability, providing a crucial perceptual foundation for the robot's safe and reliable operation of the open box.

[0083] In some embodiments of this application, before determining the pixels that simultaneously exist within the visible surface and in the overall mask of the cuboid object as the actual visible pixels of the visible surface in both simulated and real camera shooting scenarios, the method further includes:

[0084] Calculate the first absolute difference between the actual depth and the corresponding theoretical depth of each pixel within the visible surface;

[0085] If there is a pixel in the visible surface whose first absolute difference is less than or equal to the first difference threshold, then that pixel is determined as an unoccluded pixel in the visible surface.

[0086] Pixels that exist simultaneously within the visible surface and the overall mask of the cuboid object are defined as the actual visible pixels of the visible surface, including:

[0087] Pixels that exist simultaneously in all unmasked pixels and in the overall mask are identified as the actual visible pixels of the face.

[0088] The actual depth of each pixel within the visible surface can refer to the depth of that pixel in the depth image corresponding to the color image. The theoretical depth of each pixel within the visible surface can refer to the depth calculated based on the plane equation of the visible surface in the camera coordinate system, the image coordinates of the pixel, and camera intrinsic parameters. The plane equation of the visible surface in the camera coordinate system can be expressed as Ax + By + Cz + D = 0, where (A, B, C) is the second normal vector of the visible surface in the camera coordinate system, and D is the distance from the visible surface to the second origin, which is the origin of the camera coordinate system.

[0089] The color image and the corresponding depth image are a set of spatiotemporally aligned paired images, in which the color information of each pixel position in the color image corresponds one-to-one with the depth information in the depth image, together representing the scene from the same camera viewpoint.

[0090] Optionally, a first difference threshold can be set based on actual needs or experience.

[0091] In this embodiment, by calculating the absolute difference between the actual depth and the theoretical depth of each pixel within the visible surface, and filtering out unoccluded pixels based on a first difference threshold, the truly visible area and the occluded area of ​​the visible surface can be accurately separated. This measure significantly improves the geometric accuracy and physical reliability of the pixel-level mask of the visible surface, effectively overcoming annotation interference caused by depth noise and complex occlusion, thereby providing high-quality and reliable ground truth data for subsequent visual perception model training.

[0092] In the simulation scenario, target images such as color images and depth images are obtained through simulation. In some embodiments of this application, the electronic device also performs the following steps:

[0093] In a simulation environment, a color image, a depth image, an overall mask, and the second pose information of a cuboid object are obtained through a single multi-channel rendering. The second pose information refers to the pose information of the cuboid object in the camera coordinate system.

[0094] Before determining the visible faces of the six geometric faces of the cuboid object in the color image, the process also includes:

[0095] The second pose information and the size information of the cuboid object are converted into vertex coordinates in the cuboid object coordinate system to obtain the coordinates of the eight second vertices of the cuboid object.

[0096] The coordinates of the eight second vertices are transformed to the camera coordinate system and projected onto the image plane of the color image to obtain six geometric surfaces;

[0097] The size information and second pose information of the cuboid object constitute the annotation data of the cuboid object. The annotation data of the cuboid object is used to annotate the cuboid object in the target image.

[0098] In this embodiment, the electronic device can, in a simulated scene, provide the size information and pose information of the cuboid object in the world coordinate system according to its category (e.g., open plastic box, cardboard box, desktop, tray, etc.), apply the corresponding material, and perform a multi-channel rendering using software such as Blender, NVISII, or Unreal Engine. Simultaneously, it outputs a color image, a depth image, an overall mask, the second pose information of the cuboid object, and the size information of the cuboid object. The size information of the cuboid object can include its length, width, and height.

[0099] For any geometric surface, the corresponding two-dimensional quadrilateral can be used to represent the geometric surface, and the coordinates of the four vertices of the two-dimensional quadrilateral on the image plane are saved in a counterclockwise direction to facilitate subsequent mask calculation and geometric analysis.

[0100] In a simulation scenario, after determining the six geometric faces of a cuboid object, based on the object's category, at least one solid face among these six faces can be identified. Furthermore, for each solid face, the method for determining the visible face within that solid face includes:

[0101] For each solid face, calculate the angle between the first normal vector of the solid face in the camera coordinate system and the direction of the camera's line of sight;

[0102] If the angle between the first normal vector and the camera's line of sight is less than or equal to a preset angle, then the solid surface is determined to be a visible surface.

[0103] In the simulation scenario, the first normal vector of the solid face can be determined based on the second pose information of the cuboid object. Specifically, the rotation part of the second pose information of the cuboid object defines the orientation of its own coordinate system in camera space, and the first normal vector of each solid face is along these axes or their opposite directions.

[0104] The camera's line of sight to a solid surface can refer to the direction from the center of the camera to the center of the solid surface.

[0105] like Figure 2 The image shown is an example of a color image rendered in a simulated scene. Figure 3 The image shown is an example of a depth image rendered in a simulated scene. Figure 4 The image shown is an example of the overall mask of a cuboid object rendered in a simulation scene. Figure 4 The numbers 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, etc., displayed on a rectangular prism object represent the object's number. For example... Figure 5The image shows the pose and size information of a cuboid object rendered in the camera coordinate system in a simulated scene. Figure 6 The image shown is an example of a pixel-level mask for each visible surface in a simulated scene. Figure 7 The image shown is an example of the first pose information for each visible face in a simulation scenario. Figure 7 Each of the values ​​0.86, 0.88, 0.89, 0.90, 0.92, 0.93, and 1.00 represents the occupancy rate of the pixel-level mask of the visible surface in the two-dimensional projection of the visible surface. If the visible surface has cutouts or is occluded, the occupancy rate is less than 1.00.

[0106] In some embodiments of this application, for at least one solid face among six geometric faces, the method for determining the visible face among the at least one solid face includes:

[0107] For each solid face, calculate the angle between the first normal vector of the solid face in the camera coordinate system and the direction of the camera's line of sight;

[0108] If the angle between the first normal vector and the line of sight is less than or equal to the first angle threshold, then the solid surface is determined to be a visible surface.

[0109] Optionally, a first angle threshold can be set according to actual needs or empirical values. For example, the first angle threshold mentioned above is 85°.

[0110] In this embodiment, by calculating the angle between the first normal vector of the solid surface and the camera's line of sight and filtering according to a first angle threshold, faces that are not facing the camera can be efficiently and accurately eliminated early in the process. This measure significantly reduces unnecessary subsequent calculations, improves overall annotation efficiency, and avoids label noise caused by processing illegal geometric projections, thereby ensuring that the final annotation data has higher physical rationality and computational accuracy.

[0111] In some embodiments of this application, the target image further includes a depth image corresponding to the color image. When the target image is captured by a camera, the electronic device also performs the following steps:

[0112] Instance segmentation is performed on the color image to obtain the initial mask of the cuboid object;

[0113] Based on the depth image, detect whether there are outliers in the initial mask;

[0114] If there are outliers in the initial mask, remove the outliers from the initial mask to obtain the overall mask;

[0115] If there are no outliers in the initial mask, then the initial mask is determined as the global mask.

[0116] The aforementioned outliers can refer to pixels in a point cloud generated based on the initial mask and depth image that have an Euclidean distance greater than a preset distance threshold from the main point cloud (i.e., the largest connected component in the point cloud) and are spatially disconnected. These outliers can also refer to background outliers.

[0117] In some embodiments, a pre-trained instance segmentation network can be used to segment a color image to obtain an initial mask for a cuboid object, and then depth filtering can be used to remove background outliers.

[0118] In this embodiment, outlier detection and removal based on depth consistency are performed on the initial mask generated by instance segmentation to achieve automatic mask refinement. This step combines two-dimensional semantic information with three-dimensional geometric information, significantly improving the purity and physical consistency of the point cloud data of the cuboid object. This not only directly improves the accuracy of single-frame plane detection but also provides high-quality input for subsequent multi-view association and joint optimization. It is a crucial foundation for the stable and reliable operation of the entire fully automated annotation process in real noisy scenes, effectively enhancing the practicality and robustness of the method.

[0119] In real-world camera shooting scenarios, after determining the overall mask of the cuboid object, in some embodiments of this application, the visible faces among the six geometric faces of the cuboid object in the color image are determined, including:

[0120] The image coordinates of the pixels within the overall mask and the corresponding depths in the depth image are converted into an initial point cloud;

[0121] The initial point cloud is selected as the candidate point cloud;

[0122] Based on the candidate point cloud, a maximum plane is fitted, and the point clouds in the candidate point cloud that belong to the maximum plane are removed to obtain the remaining point cloud;

[0123] Determine whether the number of remaining point clouds is less than the first number, or whether the number of fitted planes reaches the second number;

[0124] If the number of remaining point clouds is greater than or equal to the first number, and the number of fitted planes does not reach the second number, then the remaining point clouds are determined as candidate point clouds, and the process returns to the step of fitting a maximum plane based on the candidate point clouds and subsequent steps, until the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number.

[0125] If the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number, then all fitted planes are determined as visible planes.

[0126] Optionally, the first and second quantities can be set according to actual needs or empirical values. For example, the first quantity is 5% of the total number of points in the entire point cloud converted from all pixels within the overall mask. The second quantity is 3.

[0127] In this embodiment, an iterative plane fitting algorithm enables the automatic and robust detection of visible faces of cuboid objects from a single-frame point cloud. This transforms unstructured point clouds into a set of planar observations with clear geometric meaning, not only strictly adhering to 3D geometric constraints to improve the accuracy of face-level segmentation but also adapting to changes in the number of visible faces from different viewpoints. This step provides high-quality, structured input data for subsequent multi-view association and joint optimization, serving as the core guarantee for achieving high-precision and robust processing at the single-frame level in the entire fully automated annotation process. It strongly supports the fully automated generation from raw data to final labels.

[0128] In this embodiment, after fitting all visible surfaces, the index of each point within each visible surface and the second plane parameters of each visible surface can be recorded. The second plane parameters include the third normal vector in the camera coordinate system and the distance to the second origin, which is the origin in the camera coordinate system.

[0129] The aforementioned color images can be multiple frames, which are captured by a camera from different perspectives of the cuboid object. The visible faces of the cuboid object in each frame of the color image can be determined using the aforementioned iterative plane fitting algorithm. Based on this, in some embodiments of this application, after determining the visible faces of the cuboid object in multiple frames of color images, the electronic device can perform the following steps:

[0130] Cluster the visible surfaces of multiple color images to obtain visible surfaces belonging to the same geometric surface;

[0131] Using the dimensions and third pose information of a cuboid object as variables to be optimized, and the observation data of all visible faces as constraints, a problem of minimizing reprojection error is established. The problem of minimizing reprojection error includes in-plane point geometric constraints and vertex consistency constraints. The in-plane point geometric constraints are used to minimize the distance error between points in each visible face belonging to the same geometric face and the same geometric face after being projected onto the coordinate system of the cuboid object. The vertex consistency constraints are used to ensure that the three-dimensional coordinates of the same vertex in the world coordinate system are consistent under different viewpoints. The observation data of a visible face includes the first plane parameters of the visible face and the three-dimensional coordinates of the points in the visible face in the world coordinate system. The first plane parameters include the second normal vector in the world coordinate system and the distance to the first origin. The first origin is the origin in the world coordinate system. The third pose information refers to the pose information of the cuboid object in the world coordinate system.

[0132] Solve the problem of minimizing the reprojection error to obtain the size information and third pose information of the cuboid object;

[0133] Based on the fourth pose information of the camera from different perspectives, the third pose information is transformed into the camera coordinate system to obtain the second pose information of the cuboid object in the multi-frame color images; the fourth pose information is the pose information of the camera in the world coordinate system, and the second pose information refers to the pose information of the cuboid object in the camera coordinate system.

[0134] The size information and second pose information of the cuboid object constitute the annotation data of the cuboid object. The annotation data of the cuboid object is used to annotate the cuboid object in the target image at the corresponding viewpoint.

[0135] The aforementioned in-plane point geometric constraints ensure that when a point in the plane is reprojected onto the coordinate system of the cuboid object, it should strictly fall on the corresponding geometric plane (e.g., the ±X / ±Y / ±Z plane). The aforementioned vertex consistency constraints ensure that the coordinates of the same vertex in the world coordinate system should be consistent under different viewpoints.

[0136] It should be noted that in real-world camera shooting scenarios, the geometric faces of a cuboid object can be referred to as physical faces, representing the actual surface areas of the cuboid object captured and extracted by a camera in a real-world scene. When there are multiple frames of color images, the number of depth images is also multiple frames.

[0137] In some embodiments, a handheld RGB-D camera can be used to take pictures around a cuboid object, obtaining temporally continuous RGB and depth images. Simultaneously, the six-degree-of-freedom pose (i.e., fourth pose information) of the camera for each frame is recorded using AprilTag or OptiTrack. Alternatively, multiple RGB-D cameras can be set up around the cuboid object to obtain RGB and depth images with the same temporal frame, while simultaneously recording the fourth pose information of the camera for each frame using AprilTag or OptiTrack. This embodiment uses RGB-D cameras to acquire images and uses AprilTag or OptiTrack to record the fourth pose information of the camera for each frame. Based on this, subsequent multi-view joint optimization using the rigid body constraints of the cuboid can achieve a pose error of less than 2mm and a vertex coordinate error of less than 1px. This solves the problem of "pose differences exceeding 2cm caused by different people annotating the same image" in manual annotation, ensuring consistent scale between the training and test sets, and improving model convergence speed and final accuracy.

[0138] In some embodiments, Ceres or g2o can be used to solve the problem of minimizing the reprojection error to obtain the size information and third pose information of the cuboid object.

[0139] like Figure 8The image shown is an example of ten consecutive color images provided in an embodiment of this application. Figure 9 The image shown is an example of the camera's fourth pose information. Figure 9 The image only shows the camera's pose information from nine different viewpoints. Figure 9 The units for the three coordinate axes are in meters (m). For example... Figure 10 The image shown is an example of the pose information for each AprilTag in a color image. Figure 11 The image shown is an example of point clouds from different viewpoints fused into a single point cloud from the same viewpoint. By utilizing color and depth images from different viewpoints, as well as the camera's fourth pose information, all point clouds from different viewpoints can be fused into a single point cloud from the same viewpoint. Figure 9 and Figure 10 The three line segments at the same origin represent the camera's X-axis, Y-axis, and Z-axis, respectively.

[0140] In this embodiment, through joint optimization of multi-view visible surface clustering and bundle adjustment, globally consistent and high-precision estimation of the size and pose information of cuboid objects is achieved, and complete annotations for each frame of the image are automatically generated. This method completely eliminates manual annotation, achieving an order-of-magnitude improvement in efficiency and a significant reduction in cost. The generated annotation data has millimeter-level accuracy and absolute cross-view consistency, fundamentally solving the core pain point of high noise in training data. At the same time, the plug-and-play feature and complete label output significantly lower the threshold for the implementation of robot vision systems and provide a unified, high-quality source of truth for downstream tasks such as instance segmentation, pose estimation, and grasping planning, powerfully promoting the intelligent upgrading of industrial scenarios.

[0141] In some embodiments of this application, before clustering the visible surfaces of multiple frames of color images to obtain visible surfaces belonging to the same geometric surface, the method further includes:

[0142] Transform all visible second plane parameters to the world coordinate system to obtain all visible first plane parameters; the second plane parameters include the third normal vector in the camera coordinate system and the distance to the second origin, which is the origin in the camera coordinate system.

[0143] Calculate the angle between the second normal vectors of the visible surfaces from any two different viewpoints and the third absolute difference in distance to the first origin;

[0144] Clustering the visible surfaces of multiple color images yields visible surfaces belonging to the same geometric surface, including:

[0145] Based on the rule that the angle between the second normal vectors is less than the second angle threshold and the third absolute difference in distance to the first origin is less than the third difference threshold, the visible surfaces of multiple frames of color images are clustered to obtain visible surfaces belonging to the same geometric surface.

[0146] Optionally, a second angle threshold and a third difference threshold can be set according to actual needs or experience.

[0147] In this embodiment, by unifying multi-frame observations to the world coordinate system and performing geometric clustering, automatic and accurate association of cross-view visible surfaces is achieved. This establishes the correct multi-view constraint relationship for subsequent joint optimization and is an indispensable key step in transforming single-frame local observations into a global accurate model of the object. It fundamentally ensures the feasibility of the fully automatic annotation process and the accuracy of the final result.

[0148] In a real-world camera shooting scenario, after obtaining the second pose information of the cuboid object in multiple frames of color images, the electronic device can also perform the following steps:

[0149] For any cuboid object in a color image, the second pose information is projected onto the image plane of the color image to obtain the six geometric faces of the cuboid object in the color image.

[0150] In this embodiment, after obtaining the optimized object pose (i.e., the second pose information), it is projected back onto the image plane of each frame, realizing the instantiation mapping from the global 3D model to the accurate geometric truth of each frame's 2D image. This step provides an accurate geometric boundary reference for generating pixel-level masks and is a key operation for producing the final labeled data. Simultaneously, it ensures complete uniformity in output format and quality standards between the real data labeling process and the simulation process, forming a fully automated closed loop from multi-view optimization to single-frame label generation. This allows the high-precision model obtained through optimization to be transformed into a supervision signal that can be directly used by downstream tasks.

[0151] In both simulated and real-world camera shooting scenarios, in some embodiments, the first pose information of the visible face is determined based on the coordinates of four first vertices, including:

[0152] Calculate the coordinates of the center point of the visible surface based on the coordinates of the four first vertices;

[0153] Calculate the two adjacent edge vectors of the visible face based on the coordinates of three consecutive adjacent vertices among the four first vertex coordinates;

[0154] The cross product of two adjacent edge vectors yields the third normal vector of the visible surface;

[0155] After normalizing the third normal vector, construct the rotation matrix of the visible surface;

[0156] The coordinates of the center point of the visible face and the rotation matrix constitute the first pose information.

[0157] The rotation matrix mentioned above has a dimension of 3×3.

[0158] In some embodiments, after normalizing the third normal vector, it can be filled into a 3×3 matrix to construct the rotation matrix of the visible face.

[0159] It should be noted that, for any visible face, after the visible face is determined, the coordinates of the four first vertices of the visible face can be stored in a counterclockwise direction, and the two adjacent edge vectors of the visible face can be calculated by selecting three consecutive adjacent first vertex coordinates according to the storage order.

[0160] In this embodiment, the center point and rotation matrix of the visible face are automatically calculated by using the coordinates of the first vertex of the visible face. This can efficiently and accurately generate the first pose information of the face, ensuring strict consistency between the face-level pose information and its underlying geometric vertices, and providing fine-grained, high-fidelity supervisory ground truth for downstream tasks.

[0161] In some embodiments of this application, for each frame of the target image, the overall mask, second pose information, size information of the cuboid object, pixel-level mask of each visible face, and first pose information can be output simultaneously to meet the training requirements of multi-task tasks such as instance segmentation, keypoint detection, pose estimation, and grasping planning. For each frame, the image file name, camera intrinsics, identification information and category of each cuboid object, and name of each visible face can also be output simultaneously.

[0162] Optionally, the annotation data of the cuboid object and the annotation data of each visible face can be saved as a structured file (e.g., JSON format). The overall mask and the pixel-level masks of each visible face can be compressed using run-length encoding (RLE), and the compressed encoding results can be stored in the corresponding fields of the structured file.

[0163] It should be noted that all label data needs to be perfectly aligned with each frame of the target image (e.g., color and depth images) to facilitate subsequent training and evaluation.

[0164] For simulation scenarios, this application embodiment can obtain labeled data with zero manual intervention through a single rendering. For real camera shooting scenarios, this application embodiment requires no manual interaction during the determination of labeled data. It can automatically generate pixel-level masks and first pose information with the same format as simulation labels on sequences of real color and depth images, which can be directly used for model training and evaluation. Moreover, the labeling time for a single frame image is reduced from the traditional 8 to 12 minutes to the millisecond level, saving more than 95% of labor costs.

[0165] This application embodiment can simultaneously use two commonly used robot vision solutions, RGB-D and binoculars, without the need for additional robotic arms or laser trackers. AprilTag / OptiTrack is only used to calibrate the fourth pose information of the camera, without relying on high-precision external fixtures, and can be quickly deployed in existing warehouses and production lines.

[0166] The embodiments of this application can achieve fully automated acquisition and processing, and can generate more than 100,000 labeled images in 24-hour continuous operation; if a new cuboid object category is added, only the instance segmentation network label needs to be adjusted, without redesigning the labeling rules, which can meet the needs of large-scale deep learning for "tens of thousands of instances and millions of images".

[0167] The embodiments of this application can output label data including the overall mask, second pose information, pixel-level mask of each visible face and first pose information, etc., which can cover the ground values ​​required by algorithms such as instance segmentation, key point detection, pose estimation and grasping planning at one time, avoiding repeated labeling.

[0168] The tag data generated through the embodiments of this application enables enterprises to complete the production of a dataset of 100,000 entries within a week using existing cameras and labeling boards, without having to purchase labeled data at high prices or build a professional labeled team. This shortens the iteration cycle of humanoid robot vision algorithms by more than 50% and accelerates industrial replication and deployment.

[0169] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0170] Corresponding to the data annotation method described in the above embodiments, Figure 12 A schematic diagram of the data annotation device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0171] Reference Figure 12 The data annotation device includes:

[0172] The first determining module 1201 is used to determine the visible faces among the six geometric faces of the cuboid object in the color image;

[0173] The second determining module 1202 is used to determine the coordinates of the four first vertices of the visible surface in the camera coordinate system;

[0174] The third determining module 1203 is used to determine the first pose information of the visible face based on the coordinates of the four first vertices; the first pose information refers to the pose information of the visible face in the camera coordinate system.

[0175] The coordinates of the four first vertices and the first pose information constitute the annotation data of the visible faces. The annotation data of the visible faces is used to annotate the visible faces in the target image, and the target image includes at least the color image.

[0176] In some embodiments, the labeled data of the visible surface further includes a pixel-level mask of the visible surface; the data labeling device further includes:

[0177] A pixel determination module is used to determine pixels that exist simultaneously within the visible surface and in the overall mask of the cuboid object as actual visible pixels of the visible surface.

[0178] A mask generation module is used to generate a pixel-level mask for the visible surface based on the actual visible pixels of the visible surface.

[0179] The overall mask is the annotation data of the cuboid object, and the annotation data of the cuboid object is used to annotate the cuboid object in the target image.

[0180] In some embodiments, the data annotation apparatus further includes:

[0181] The difference calculation module is used to calculate the first absolute difference between the actual depth and the corresponding theoretical depth of each pixel in the visible surface;

[0182] An unoccluded determination module is used to determine a pixel as an unoccluded pixel in the visible surface if there is a pixel in the visible surface whose first absolute difference is less than or equal to a first difference threshold.

[0183] The pixel determination module described above is specifically used for:

[0184] The pixels that exist simultaneously in all unmasked pixels and in the overall mask are determined as the actual visible pixels of the visible face.

[0185] In some embodiments, the target image further includes a depth image corresponding to the color image; when the target image is obtained from simulation, the data annotation device further includes:

[0186] The data acquisition module is used to acquire the color image, the depth image, the overall mask, and the second pose information of the cuboid object through a single multi-channel rendering in a simulation environment; the second pose information refers to the pose information of the cuboid object in the camera coordinate system.

[0187] The information conversion module is used to convert the second pose information and the size information of the cuboid object into vertex coordinates in the coordinate system of the cuboid object, so as to obtain the coordinates of the eight second vertices of the cuboid object.

[0188] The coordinate projection module is used to transform the coordinates of the eight second vertices to the camera coordinate system and project them onto the image plane of the color image to obtain the six geometric surfaces;

[0189] The size information of the cuboid and the second pose information constitute the annotation data of the cuboid, which is used to annotate the cuboid in the target image.

[0190] In some embodiments, for at least one solid face among the six geometric faces, the method for determining the visible face among the at least one solid face includes:

[0191] For each of the solid surfaces, calculate the angle between the first normal vector of the solid surface in the camera coordinate system and the camera's line of sight.

[0192] If the angle between the first normal vector and the camera's line of sight is less than or equal to a first angle threshold, then the solid surface is determined to be the visible surface.

[0193] In some embodiments, when the visible surface is an open surface, the method for determining the pixel-level mask of the open surface further includes:

[0194] Calculate the second absolute difference between the actual depth and the corresponding theoretical depth of each pixel in the open surface;

[0195] If the second absolute difference is less than the second preset difference, then the pixel is determined to be the actual visible pixel of the open surface;

[0196] A pixel-level mask for the open surface is generated based on the actual visible pixels of the open surface.

[0197] In some embodiments, the target image further includes a depth image corresponding to the color image, and when the target image is captured by a camera, the data annotation device further includes:

[0198] An instance segmentation module is used to perform instance segmentation on the color image to obtain the initial mask of the cuboid object;

[0199] An outlier detection module is used to detect whether there are outliers in the initial mask based on the depth image;

[0200] An outlier removal module is used to remove the outlier from the initial mask if the outlier exists in the initial mask, thereby obtaining the overall mask.

[0201] A mask determination module is used to determine the initial mask as the overall mask if the outlier does not exist in the initial mask.

[0202] In some embodiments, the target image further includes a depth image corresponding to the color image. When the target image is captured by a camera, the first determining module 1201 is specifically used for:

[0203] The image coordinates of the pixels within the overall mask and the corresponding depths in the depth image are converted into an initial point cloud;

[0204] The initial point cloud is determined as a candidate point cloud;

[0205] Based on the candidate point cloud, a maximum plane is fitted, and the point clouds in the candidate point cloud that belong to the maximum plane are removed to obtain the remaining point cloud;

[0206] Determine whether the number of remaining point clouds is less than a first number, or whether the number of fitted planes reaches a second number;

[0207] If the number of remaining point clouds is greater than or equal to the first number, and the number of fitted planes does not reach the second number, then the remaining point clouds are determined as candidate point clouds, and the process returns to the step of fitting a maximum plane based on the candidate point clouds and subsequent steps, until the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number.

[0208] If the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number, then all fitted planes are determined as the visible planes.

[0209] In some embodiments, the number of color images is multiple frames, and the multiple frames of color images are obtained by a camera capturing the cuboid object from different perspectives; the data annotation device further includes:

[0210] The clustering module is used to cluster the visible surfaces of multiple frames of the color images to obtain visible surfaces belonging to the same geometric surface;

[0211] The problem reconstruction module is used to establish a problem of minimizing reprojection error, using the size information and third pose information of the cuboid object as variables to be optimized, and the observation data of all visible surfaces as constraints. The problem of minimizing reprojection error includes in-plane point geometric constraints and vertex consistency constraints. The in-plane point geometric constraints are used to minimize the distance error between points in each visible surface belonging to the same geometric surface and the same geometric surface after being projected into the coordinate system of the cuboid object. The vertex consistency constraints are used to make the three-dimensional coordinates of the same vertex in the world coordinate system consistent under different viewpoints. The observation data of a visible surface includes the first plane parameter of the visible surface and the three-dimensional coordinates of the points in the visible surface in the world coordinate system. The first plane parameter includes the second normal vector in the world coordinate system and the distance to the first origin. The first origin is the origin in the world coordinate system. The third pose information refers to the pose information of the cuboid object in the world coordinate system.

[0212] The problem-solving module is used to solve the problem of minimizing the reprojection error, and obtain the size information of the cuboid object and the third pose information.

[0213] The information conversion module is used to convert the third pose information to the camera coordinate system based on the fourth pose information of the camera under different viewpoints, so as to obtain the second pose information of the cuboid object in the multiple frames of the color images; the fourth pose information is the pose information of the camera in the world coordinate system, and the second pose information refers to the pose information of the cuboid object in the camera coordinate system.

[0214] The size information of the cuboid and the second pose information are the annotation data of the cuboid, and the annotation data of the cuboid is used to annotate the cuboid in the target image from the corresponding viewpoint.

[0215] In some embodiments, the data annotation apparatus further includes:

[0216] The parameter transformation module is used to transform the second plane parameters of all visible surfaces to the world coordinate system to obtain the first plane parameters of all visible surfaces; the second plane parameters include the third normal vector in the camera coordinate system and the distance to the second origin, where the second origin is the origin of the camera coordinate system;

[0217] The data calculation module is used to calculate the angle between the second normal vectors of visible surfaces from any two different viewpoints and the third absolute difference in distance to the first origin.

[0218] The clustering module is specifically used for:

[0219] Based on the rule that the angle between the second normal vectors is less than the second angle threshold and the third absolute difference in distance to the first origin is less than the third difference threshold, the visible surfaces of multiple frames of the color images are clustered to obtain visible surfaces belonging to the same geometric surface.

[0220] In some embodiments, the data annotation apparatus further includes:

[0221] The geometric surface determination module is used to project the second pose information of the cuboid object in any of the color images onto the image plane of the color image to obtain the six geometric surfaces of the cuboid object in the color image.

[0222] In some embodiments, the third determining module 1203 is specifically used for:

[0223] Based on the coordinates of the four first vertices, calculate the coordinates of the center point of the visible surface;

[0224] Based on the coordinates of three consecutive adjacent vertices among the four first vertex coordinates, calculate the two adjacent edge vectors of the visible face;

[0225] The cross product of the two adjacent edge vectors is used to obtain the third normal vector of the visible face.

[0226] After normalizing the third normal vector, the rotation matrix of the visible surface is constructed;

[0227] The center point coordinates of the visible surface and the rotation matrix constitute the first pose information.

[0228] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0229] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 As shown, the electronic device 13 of this embodiment includes: at least one processor 1300 ( Figure 13 (Only one is shown in the diagram), memory 1301, and computer program 1302 stored in said memory 1301 and executable on said at least one processor 1300, which, when executing said computer program 1302, implements the steps of any of the above method embodiments.

[0230] The electronic device may include, but is not limited to, a processor 1300 and a memory 1301. Those skilled in the art will understand that... Figure 13This is merely an example of electronic device 13 and does not constitute a limitation on electronic device 13. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0231] The processor 1300 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0232] In some embodiments, the memory 1301 may be an internal storage unit of the electronic device 13, such as a hard disk or memory of the electronic device 13. In other embodiments, the memory 1301 may be an external storage device of the electronic device 13, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 13. Furthermore, the memory 1301 may include both internal and external storage units of the electronic device 13. The memory 1301 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 1301 can also be used to temporarily store data that has been output or will be output.

[0233] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0235] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0236] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0237] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0238] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data annotation method, characterized in that, include: Identify the visible faces among the six geometric faces of a cuboid object in a color image; Determine the coordinates of the four first vertices of the visible surface in the camera coordinate system; Based on the coordinates of the four first vertices, the first pose information of the visible face is determined; the first pose information refers to the pose information of the visible face in the camera coordinate system. The coordinates of the four first vertices and the first pose information constitute the annotation data of the visible faces. The annotation data of the visible faces is used to annotate the visible faces in the target image, and the target image includes at least the color image.

2. The data annotation method according to claim 1, characterized in that, The labeled data of the visible surface also includes the pixel-level mask of the visible surface; the data labeling method further includes: The pixels that exist simultaneously within the visible surface and in the overall mask of the cuboid object are determined as the actual visible pixels of the visible surface. Based on the actual visible pixels of the visible surface, a pixel-level mask for the visible surface is generated; The overall mask is the annotation data of the cuboid object, and the annotation data of the cuboid object is used to annotate the cuboid object in the target image.

3. The data annotation method according to claim 2, characterized in that, Before determining the pixels that simultaneously exist within the visible surface and in the overall mask of the cuboid object as the actual visible pixels of the visible surface, the method further includes: Calculate the first absolute difference between the actual depth and the corresponding theoretical depth of each pixel within the visible surface; If there is a pixel in the visible surface whose first absolute difference is less than or equal to the first difference threshold, then the pixel is determined as an unoccluded pixel in the visible surface. The step of determining the pixels that simultaneously exist within the visible surface and in the overall mask of the cuboid object as the actual visible pixels of the visible surface includes: The pixels that exist simultaneously in all unmasked pixels and in the overall mask are determined as the actual visible pixels of the visible face.

4. The data annotation method according to claim 2, characterized in that, The target image also includes a depth image corresponding to the color image. If the target image is obtained through simulation, the data annotation method further includes: In a simulation environment, the color image, the depth image, the overall mask, and the second pose information of the cuboid object are obtained through a single multi-channel rendering; the second pose information refers to the pose information of the cuboid object in the camera coordinate system. Before determining the visible faces of the six geometric faces of the cuboid object in the color image, the process also includes: The second pose information and the size information of the cuboid object are converted into vertex coordinates in the cuboid object coordinate system to obtain the coordinates of the eight second vertices of the cuboid object. The coordinates of the eight second vertices are transformed to the camera coordinate system and projected onto the image plane of the color image to obtain the six geometric surfaces; The size information of the cuboid and the second pose information constitute the annotation data of the cuboid, which is used to annotate the cuboid in the target image.

5. The data annotation method according to claim 4, characterized in that, For at least one solid face among the six geometric faces, the method for determining the visible face among the at least one solid face includes: For each of the solid surfaces, calculate the angle between the first normal vector of the solid surface in the camera coordinate system and the camera's line of sight. If the angle between the first normal vector and the camera's line of sight is less than or equal to a first angle threshold, then the solid surface is determined to be the visible surface.

6. The data annotation method according to claim 2, characterized in that, When the visible surface is an open surface, the method for determining the pixel-level mask of the open surface further includes: Calculate the second absolute difference between the actual depth and the corresponding theoretical depth of each pixel in the open surface; If the second absolute difference is less than the second preset difference, then the pixel is determined to be the actual visible pixel of the open surface; A pixel-level mask for the open surface is generated based on the actual visible pixels of the open surface.

7. The data annotation method according to claim 2, characterized in that, The target image also includes a depth image corresponding to the color image. If the target image is captured by a camera, the data annotation method further includes: The color image is segmented to obtain the initial mask of the cuboid object; Based on the depth image, detect whether there are outliers in the initial mask; If the outlier exists in the initial mask, then the outlier is removed from the initial mask to obtain the overall mask; If the outlier does not exist in the initial mask, then the initial mask is determined as the overall mask.

8. The data annotation method according to claim 2, characterized in that, The target image also includes a depth image corresponding to the color image. When the target image is captured by a camera, determining the visible faces among the six geometric faces of the cuboid object in the color image includes: The image coordinates of the pixels within the overall mask and the corresponding depths in the depth image are converted into an initial point cloud; The initial point cloud is determined as a candidate point cloud; Based on the candidate point cloud, a maximum plane is fitted, and the point clouds in the candidate point cloud that belong to the maximum plane are removed to obtain the remaining point cloud; Determine whether the number of remaining point clouds is less than a first number, or whether the number of fitted planes reaches a second number; If the number of remaining point clouds is greater than or equal to the first number, and the number of fitted planes does not reach the second number, then the remaining point clouds are determined as candidate point clouds, and the process returns to the step of fitting a maximum plane based on the candidate point clouds and subsequent steps, until the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number. If the number of remaining point clouds is less than the first number, or the number of fitted planes reaches the second number, then all fitted planes are determined as the visible planes.

9. The data annotation method according to any one of claims 1 to 8, characterized in that, The number of color images is multiple frames, which are captured by a camera from different perspectives of the cuboid object; after determining the visible faces of the six geometric faces of the cuboid object in the color images, the method further includes: Cluster the visible surfaces of multiple frames of the color images to obtain visible surfaces belonging to the same geometric surface; Using the size information and third pose information of the cuboid object as variables to be optimized, and the observation data of all visible surfaces as constraints, a problem of minimizing reprojection error is established. The problem of minimizing reprojection error includes in-plane point geometric constraints and vertex consistency constraints. The in-plane point geometric constraints are used to minimize the distance error between points in each visible surface belonging to the same geometric surface and the same geometric surface after being projected onto the coordinate system of the cuboid object. The vertex consistency constraints are used to ensure that the three-dimensional coordinates of the same vertex in the world coordinate system are consistent under different viewpoints. The observation data of a visible surface includes the first plane parameters of the visible surface and the three-dimensional coordinates of the points in the visible surface in the world coordinate system. The first plane parameters include the second normal vector in the world coordinate system and the distance to the first origin. The first origin is the origin in the world coordinate system. The third pose information refers to the pose information of the cuboid object in the world coordinate system. Solving the problem of minimizing the reprojection error yields the size information of the cuboid object and the third pose information. Based on the fourth pose information of the camera under different viewpoints, the third pose information is transformed into the camera coordinate system to obtain the second pose information of the cuboid object in multiple frames of the color images; the fourth pose information is the pose information of the camera in the world coordinate system, and the second pose information refers to the pose information of the cuboid object in the camera coordinate system. The size information of the cuboid and the second pose information are the annotation data of the cuboid, and the annotation data of the cuboid is used to annotate the cuboid in the target image from the corresponding viewpoint.

10. The data annotation method according to claim 9, characterized in that, Before clustering the visible surfaces of multiple frames of the color images to obtain visible surfaces belonging to the same geometric surface, the method further includes: The second plane parameters of all visible surfaces are transformed into the world coordinate system to obtain the first plane parameters of all visible surfaces; the second plane parameters include the third normal vector in the camera coordinate system and the distance to the second origin, where the second origin is the origin of the camera coordinate system; Calculate the angle between the second normal vectors of the visible surfaces from any two different viewpoints and the third absolute difference in distance to the first origin; The step of clustering visible surfaces of multiple frames of the color images to obtain visible surfaces belonging to the same geometric surface includes: Based on the rule that the angle between the second normal vectors is less than the second angle threshold and the third absolute difference in distance to the first origin is less than the third difference threshold, the visible surfaces of multiple frames of the color images are clustered to obtain visible surfaces belonging to the same geometric surface.

11. The data annotation method according to claim 9, characterized in that, After obtaining the second pose information of the cuboid object in multiple frames of the color images, the method further includes: For the second pose information of the cuboid object in any of the color images, the second pose information is projected onto the image plane of the color image to obtain the six geometric faces of the cuboid object in the color image.

12. The data annotation method according to any one of claims 1 to 8, characterized in that, Determining the first pose information of the visible face based on the coordinates of the four first vertices includes: Based on the coordinates of the four first vertices, calculate the coordinates of the center point of the visible surface; Based on the coordinates of three consecutive adjacent vertices among the four first vertex coordinates, calculate the two adjacent edge vectors of the visible face; The cross product of the two adjacent edge vectors is used to obtain the third normal vector of the visible face. After normalizing the third normal vector, the rotation matrix of the visible surface is constructed; The center point coordinates of the visible surface and the rotation matrix constitute the first pose information.

13. A data annotation device, characterized in that, include: The first determining module is used to determine the visible faces among the six geometric faces of the cuboid object in the color image; The second determining module is used to determine the coordinates of the four first vertices of the visible surface in the camera coordinate system; The third determining module is used to determine the first pose information of the visible face based on the coordinates of the four first vertices; the first pose information refers to the pose information of the visible face in the camera coordinate system. The coordinates of the four first vertices and the first pose information constitute the annotation data of the visible faces. The annotation data of the visible faces is used to annotate the visible faces in the target image, and the target image includes at least the color image.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the data annotation method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, Includes a computer program, which, when run, causes the data annotation method as described in any one of claims 1 to 12 to be performed.

Images