A retail scene image labeling data quality evaluation method based on multi-modal alignment

Abstract

The application discloses a retail scene image labeling data quality evaluation method based on multi-modal alignment and belongs to the field of computer vision data processing. A two-dimensional color image of a target retail commodity, an initial labeled two-dimensional instance segmentation mask and a three-dimensional depth map are acquired, the depth values of effective labeled pixel points in the mask are extracted, corresponding pixel points are converted to a global three-dimensional world coordinate system, a three-dimensional point cloud subset is formed, a standard physical fitting surface is obtained by fitting, and a point-to-surface iterative closest point algorithm is used to establish the corresponding relationship between the point cloud and the standard physical fitting surface; the depth distance, the normal direction, the edge continuity and the surface coverage residual are calculated according to the point-to-surface corresponding relationship, the three-dimensional space wrapping degree score is obtained, and the labeling quality is judged in combination with a dynamic qualified threshold. Residual space distribution coding features are extracted, a defect mode template library is matched, abnormal points and uncovered areas are back-projected to a two-dimensional image, and pixel-level defect position marks and suggestions are generated.

Description

Technical Field

[0001] This invention belongs to the field of computer vision data processing technology, specifically relating to a method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment. Background Technology

[0002] In computer vision tasks such as object detection and image segmentation, model performance is highly dependent on the boundary accuracy of the labeled masks in the training data.

[0003] For product images in retail scenarios, existing labeling quality audits mainly rely on manual sampling or automated verification based on two-dimensional pixel intersection-over-union ratio.

[0004] However, the two-dimensional plane verification method can only reflect the coverage of the mask in the image coordinate system, and cannot characterize the over-boundary or floating state of the labeled area in the depth direction, resulting in the evaluation results lacking the ability to measure the true three-dimensional volume distribution of the target.

[0005] Furthermore, the existing assessment process lacks objective physical morphological truth values ​​as a comparison benchmark, and it is difficult to unify the standards for manual review. Directly using the original depth map as a reference is also prone to introducing sensor acquisition noise, making it difficult to obtain a pure and ideal physical boundary.

[0006] Existing automated evaluation tools typically only output a single global pass rate indicator, which cannot refine and quantify deviations from local dimensions such as surface fit and edge continuity, nor can they adapt to noise interference in complex acquisition environments to provide pixel-level defect location and correction guidance, resulting in high costs for subsequent data rework. Summary of the Invention

[0007] The purpose of this invention is to provide a method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment, in order to solve the problem of how to overcome the deficiency of traditional two-dimensional evaluation in lacking three-dimensional spatial measurement capabilities, construct an objective physical geometric evaluation benchmark under the premise of suppressing sensor noise interference, and realize the multi-dimensional quantification of annotation deviation and the automated tracing of pixel-level defect root causes.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for quality assessment of image annotation data in retail scenarios based on multimodal alignment, comprising: Step 1: For the target retail product, obtain its 2D color image, initially labeled 2D instance segmentation mask, 3D depth map, and camera intrinsic and extrinsic parameter matrix; extract the standard physical depth values ​​of the effectively labeled pixels in the mask; based on the camera intrinsic and extrinsic parameter matrix, map the 2D pixel coordinates and standard physical depth values ​​to the global 3D world coordinate system to generate an initial 3D point cloud subset; Step 2: Based on the category label of the target retail product, call the physical geometry primitive prior model to fit the initial 3D point cloud subset to generate a standard physical fitting surface; use the point-to-surface iterative nearest point algorithm for fine registration to update the point cloud position and cumulative transformation matrix, and output the registered 3D point cloud subset, the final point-to-surface correspondence set, and the final cumulative fine-tuning spatial transformation matrix. Step 3: Construct a multidimensional residual feature vector based on the final point-to-plane correspondence set and calculate the three-dimensional spatial wrapping score. Combine the dynamic qualification threshold to determine whether the initial labeled data is qualified. For unqualified data, supplement the residual spatial distribution coding features to form a complete attribution feature vector. Determine the upper-level defect pattern through feature matching and generate pixel-level defect location markers and correction suggestions.

[0009] Preferably, the specific process for obtaining a two-dimensional color image, an initially labeled two-dimensional instance segmentation mask, a three-dimensional depth map, and a camera intrinsic and extrinsic parameter matrix for a target retail product is as follows: Read a two-dimensional color image containing the target retail product, an initially labeled two-dimensional instance segmentation mask, and a three-dimensional depth map from the same time and perspective from the retail product data acquisition system; A checkerboard calibration board of known physical size was placed in front of the image acquisition device, and images from multiple angles were acquired. The coordinates of the corner points were extracted using the Zhang Zhengyou calibration method, and the camera intrinsic and extrinsic parameter matrices were obtained by optimization using the least squares method.

[0010] Preferably, the specific process for extracting the standard physical depth values ​​of valid labeled pixels in the mask is as follows: Iterate through each pixel in the initially labeled 2D instance segmentation mask and determine whether the pixel value of the current pixel is equal to the preset foreground identifier value; If equal to, then the pixel is determined to belong to the target retail product and marked as a valid labeled pixel; Obtain the 2D pixel coordinates of valid labeled pixels in the 2D color image, extract the original depth value from the same position in the 3D depth map, and convert it into a standard physical depth value that matches the physical units of the camera's intrinsic parameters.

[0011] Preferably, the specific process of mapping two-dimensional pixel coordinates and standard physical depth values ​​to the global three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameter matrix to generate an initial three-dimensional point cloud subset is as follows: Based on the acquired camera intrinsic parameter matrix, the two-dimensional pixel coordinates and standard physical depth values ​​of the effectively labeled pixels are transformed into coordinates in the three-dimensional camera coordinate system; the acquired camera extrinsic parameter matrix is ​​analyzed to extract the rotation matrix and translation vector. Based on the rotation matrix and translation vector, the coordinates in the 3D camera coordinate system are transformed to the coordinates in the global 3D world coordinate system; Extract the first three elements of the homogeneous coordinate form of the coordinates in the transformed global 3D world coordinate system to obtain the final 3D coordinates in the global 3D world coordinate system. The 3D coordinates of all valid labeled pixels are collected to generate an initial 3D point cloud subset of the target retail product.

[0012] Preferably, the specific process of calling the physical geometric primitive prior model based on the category label of the target retail product to fit the initial 3D point cloud subset to generate a standard physical fitting surface is as follows: Read the category label of the target retail product from the metadata of the initially labeled two-dimensional instance segmentation mask, and call the corresponding physical geometry primitive prior model; Initialize the random sampling consensus algorithm. In each iteration, randomly select the smallest sample point set and substitute it into the standard mathematical parametric equation to construct a geometric equation system. Solve the candidate model parameter vector, calculate the distance residual from all points in the initial 3D point cloud subset to the estimated geometric surface, count the number of inliers that are less than the preset inlier distance threshold, and select the inlier with the largest number of inliers as the optimal model parameter vector. Substituting the optimal model parameter vector into the mathematical equation yields the target geometric analytical equation. Surface sampling is carried out within the three-dimensional bounding box of the initial three-dimensional point cloud subset. Redundant points are eliminated and valid sampling points are retained. The unit normal vector of the surface is calculated analytically, and the standard physical fitting surface is constructed by combining the results.

[0013] Preferably, the point-to-surface iterative nearest-point algorithm is used for fine-grained registration to update the point cloud position and cumulative transformation matrix. The specific process is as follows: Assign the initial 3D point cloud subset to the current iteration point cloud subset, and initialize the cumulative fine-tuning spatial transformation matrix; In each fine-tuning iteration, the shortest normal projection point from the point in the current iteration point cloud subset to the standard physical fitting surface and the surface unit normal vector are calculated point by point. The point cloud point coordinates, the shortest normal projection point coordinates and the surface unit normal vector are combined into matching point pairs, and summarized to form a set of point-to-surface correspondences. Construct a point-to-surface target error function, solve for the optimal single-step fine-tuning rotation matrix and the optimal single-step fine-tuning translation vector using the linear least squares method, and construct the single-step fine-tuning spatial transformation matrix by splicing the matrix according to the matrix splicing rules of the homogeneous coordinate system. Apply it to the current iteration subset of the point cloud to update the point cloud position, and left-multiply it to the cumulative fine-tuning spatial transformation matrix to update the cumulative fine-tuning spatial transformation matrix.

[0014] Preferably, the specific process of outputting the registered 3D point cloud subset, the final set of point-to-plane correspondences, and the final cumulative fine-tuning spatial transformation matrix is ​​as follows: Repeat the process of establishing a set of point-to-plane correspondences, constructing a point-to-plane target error function, solving a single fine-tuning spatial transformation matrix, and updating the point cloud position until the value of the target error function is less than the preset convergence threshold or the maximum number of fine-tuning iterations is reached. After the fine-tuning iteration converges, based on the point-to-plane correspondence set established in the last fine-tuning iteration, the output includes the standard physical fitting surface as the registration reference, the registered 3D point cloud subset precisely aligned with the standard physical fitting surface, the final point-to-plane correspondence set recording the point-to-plane mapping relationship, and the final cumulative fine-tuning spatial transformation matrix recording the global mapping relationship from the initial point cloud to the final registration pose.

[0015] Preferably, the specific process of constructing a multidimensional residual feature vector based on the final point-to-face correspondence set and calculating the three-dimensional spatial wrapping score is as follows: Traverse the final point-to-plane correspondence set, calculate the directed normal distance between the point cloud coordinates and the shortest normal projection point coordinates, and statistically analyze the mean and variance of the absolute directed normal distance and the proportion of inner and outer points to form a depth distance residual feature group. Mark invalid interference point cloud points and calculate their proportion. Calculate the spatial angle between the local surface normal vector and the surface unit normal vector, and statistically combine the mean, variance and the proportion of exceeding the threshold to form the feature group of normal vector direction residuals. Calculate the vertical distance from the edge point to the theoretical boundary and the abrupt change in depth and normal angle between adjacent edge points. After removing natural geometric abrupt change points, the statistical variance combination is used to form the edge continuity residual characteristic group. The total surface uncovering rate and the proportion of the largest continuous uncovered area are combined to form the surface coverage residual feature group by dividing the uniform sampling grid and statistically analyzing the surface coverage residual rate. All statistics are normalized to unify the polarity direction and concatenated into a multidimensional residual feature vector. The three-dimensional spatial wrapping score is calculated by weighting and summing the preset feature dimension weight coefficients.

[0016] Preferably, the specific process for determining whether the initial labeled data is qualified, based on the dynamic qualification threshold, is as follows: During the fitting phase of the random sampling consensus algorithm, the distance residuals of all inliers are statistically analyzed, and the mean and standard deviation of the inlier distance residuals are calculated. The interval formed by adding the mean to the standard deviation by a preset multiple is determined as the confidence interval of the inlier residuals. Using the upper bound of the confidence interval of the in-point residual as the baseline, the calculated proportion of invalid interference point cloud points is input into the preset threshold relaxation function to calculate the threshold relaxation coefficient. Multiply the baseline pass line by the threshold relaxation factor to obtain the dynamic pass threshold for the current sample; If the 3D spatial envelopment score is greater than or equal to the dynamic pass threshold, the initial annotation data is deemed qualified and output. If the 3D spatial envelopment score is less than the dynamic acceptable threshold, the initial annotation data is deemed unacceptable, and the subsequent attribution process is triggered.

[0017] Preferably, for non-compliant data, the residual spatial distribution coding features are supplemented to form a complete attribution feature vector. The higher-level defect pattern is determined through feature matching, and the specific process of generating pixel-level defect location markers and correction suggestions is as follows: Points in the point cloud with an absolute directional normal distance greater than a preset residual threshold are marked as high residual points. The density clustering algorithm is used to obtain the three-dimensional coordinates of the centroid of the largest cluster as the spatial cluster location. The difference between the proportion of outer points and the proportion of inner points is used to obtain the positive and negative residual proportion difference. Connectivity analysis is performed on the uncovered grid to obtain the three-dimensional coordinates of the centroid of the largest connected region as the location of the largest continuous uncovered area. These are combined into residual spatial distribution coding features and spliced ​​into a complete attribution feature vector. Based on localized highlighted areas and areas with missing depth values, interference areas in the acquisition were identified to eliminate unlabeled defects; Calculate the weighted Euclidean distance between the complete attribution feature vector and each standard defect template vector in the defect pattern template library, and select the template with the smallest distance that is less than the preset matching threshold to determine the superior defect pattern. The high residual points and the points in the uncovered surface area are back-projected to generate pixel-level defect location markers. Standardized correction suggestions are extracted, and a quality assessment report is compiled and sent to the annotation terminal.

[0018] Compared with the prior art, the beneficial effects of the present invention are: (1) The effective pixels within the 2D mask are combined with the depth map and mapped to the 3D world coordinate system using camera parameters to generate the initial point cloud. This stretches the originally flat annotation outline into a 3D spatial distribution with a real physical scale. This solves the limitation of traditional 2D evaluation, which can only calculate errors on a plane. Because the old method cannot see whether the annotation is out of bounds or suspended in the depth direction, nor can it measure volume, this solution makes up for the shortcomings of lacking depth perception and volume measurement capabilities.

[0019] (2) Based on the product category, a prior model is used to fit the initial point cloud, and then the iterative nearest point algorithm is used for fine registration. This approach cleverly utilizes the regular geometric shape of the product itself to filter out noise and mislabeled areas in the initial point cloud, thereby extracting the ideal standard physical surface. This is equivalent to providing an objective and stable three-dimensional physical "ruler" for quality assessment, changing the previous situation of high dependence on manual sampling and inconsistent subjective evaluation standards, making the assessment benchmark more reliable.

[0020] (3) Based on the point-to-face correspondence, multidimensional residual features are calculated to obtain the wrapping score, and dynamic thresholds are used to determine whether the data is qualified. For unqualified data, spatial distribution features are supplemented to match the defect library, and pixel-level error marks and modification suggestions are directly output. This transforms the previous "black box" scoring that only gave a single total score into a "white box" attribution that can see the cause. This scheme not only quantifies the fit of the annotation in multiple dimensions, but also directly guides the annotators to make targeted modifications, linking evaluation, positioning and correction into a complete data cleaning closed loop, which significantly reduces the rework cost of large-scale data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1; Please see Figure 1 This invention provides a method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment, comprising: Step 1: For the target retail product, acquire its 2D color image, initially labeled 2D instance segmentation mask, 3D depth map, and camera intrinsic and extrinsic parameter matrix; extract the standard physical depth values ​​of the effectively labeled pixels in the mask; based on the camera intrinsic and extrinsic parameter matrix, map the 2D pixel coordinates and standard physical depth values ​​to the global 3D world coordinate system to generate an initial 3D point cloud subset. The specific process is as follows: For each retail product labeled sample to be evaluated, read the two-dimensional color image containing the retail product, the initial labeled two-dimensional instance segmentation mask, and the three-dimensional depth map from the retail product data acquisition system at the same time and from the same perspective; Among them, the initial labeled two-dimensional instance segmentation mask refers to the pixel-level binary region identifier generated on a two-dimensional color image for a specific retail product, which is used to define the outline boundary of the target product in the two-dimensional image. Its sources include manual annotation, artificial intelligence pre-annotation, or automatic generation by large models. A 3D depth map is an image that is strictly registered with a 2D color image at the pixel level. Each pixel value represents the physical depth distance of the corresponding spatial point from the optical center of the image acquisition device.

[0024] Obtain the camera intrinsic parameter matrix and camera extrinsic parameter matrix of the image acquisition device; Furthermore, the process of obtaining the intrinsic parameter matrix and the extrinsic parameter matrix of the camera is as follows: A checkerboard calibration board of known physical size is placed in front of the image acquisition device, and multiple images of the calibration board from different angles are acquired. Using Zhang Zhengyou's calibration method, the two-dimensional pixel coordinates of the corner points of the calibration board and the known three-dimensional world coordinates are extracted. The least squares method is used to optimize the solution, thereby obtaining the camera intrinsic parameter matrix and the camera extrinsic parameter matrix.

[0025] Since Zhang Zhengyou's calibration method is a conventional camera calibration method in this field, its specific calculation process will not be elaborated further here.

[0026] Iterate through each pixel in the initially labeled 2D instance segmentation mask, and determine whether the pixel value of the current pixel is equal to the preset foreground identifier value (e.g., 1 or 255 in the binarized mask). If it is equal to the preset foreground identifier value, then the pixel is determined to belong to the target retail product and is marked as a valid labeled pixel. For any valid labeled pixel, obtain its two-dimensional pixel coordinates in the two-dimensional color image. And extract the corresponding pixel value from the same coordinate position in the 3D depth map, and define the pixel value as the original depth value; The raw depth value is converted to a depth value that matches the physical units of the camera intrinsics and denoted as the standard physical depth value d (for example, if the camera intrinsics are based on meters, the raw depth value in millimeters is divided by 1000 to convert it to meters to achieve uniformity of physical units). Based on the acquired camera intrinsic parameter matrix, the two-dimensional pixel coordinates of the effectively labeled pixels will be... Transform the standard physical depth value d into coordinates in the 3D camera coordinate system The specific conversion formula is as follows: ; in, These are the focal lengths along the horizontal and vertical axes of the camera intrinsic parameter matrix, respectively. These are the x and y coordinates of the optical center in the camera intrinsic parameter matrix, respectively.

[0027] The specific meaning of this formula is as follows: Based on the pinhole camera imaging model, using the geometric principle of similar triangles in pinhole imaging, the pixel coordinates on the two-dimensional image plane are combined with depth information to reverse derive and restore the physical space coordinates in the three-dimensional camera coordinate system.

[0028] The acquired camera extrinsic parameter matrix is ​​analyzed to extract the rotation matrix R and translation vector T; based on the rotation matrix R and translation vector T, the coordinates in the 3D camera coordinate system are... Coordinates transformed to the global 3D world coordinate system The specific analysis and calculation formulas are as follows: ; Where R is the rotation matrix and T is the translation vector. Let R be the transpose of the rotation matrix R; The column vector on the right side of the formula represents the homogeneous coordinate form of the coordinates in the 3D camera coordinate system, and the column vector on the left side of the formula represents the homogeneous coordinate form of the calculated global 3D world coordinate system. After performing the matrix multiplication operation described above, extracting the first three elements of the column vector on the left side of the formula yields the final 3D coordinates in the global 3D world coordinate system. ; Furthermore, the rotation matrix R and translation vector T are directly extracted by analyzing the aforementioned camera extrinsic matrix (usually a 3×4 or 4×4 homogeneous transformation matrix), representing the rotation attitude and translation position of the camera coordinate system relative to the world coordinate system, respectively.

[0029] The specific meaning of this formula is as follows: Based on the rigid transformation principle of the three-dimensional rectangular coordinate system, through inverse rotation and inverse translation operations, the three-dimensional points in the local camera coordinate system are mapped to the globally unified world coordinate system.

[0030] Collect all the coordinates of the valid labeled pixels in the global 3D world coordinate system after conversion to generate an initial 3D point cloud subset of the target retail product; This subset represents the physical distribution in three-dimensional space of the initial labeled two-dimensional instance segmentation mask of the target retail product.

[0031] It should be noted that conventional image annotation quality assessment is usually limited to calculating the intersection-union ratio within a two-dimensional pixel coordinate system. This method cannot characterize the out-of-bounds or floating states of the annotated region in the depth direction.

[0032] This step involves parsing the camera intrinsic and extrinsic parameter matrices, and then inversely mapping the valid labeled pixels within the initially labeled 2D instance segmentation mask to the global 3D world coordinate system by combining the standard physical depth value d provided by the 3D depth map.

[0033] This transformation process converts two-dimensional pixel coordinates Restored to 3D coordinates in the global 3D world coordinate system .

[0034] Through the coordinate transformation described above, the generated initial 3D point cloud subset objectively records the 3D spatial distribution of the target retail goods in the actual data collection scenario.

[0035] This provides data support with a real physical metric for subsequent steps to calculate the three-dimensional spatial residuals, overcoming the deficiency of conventional two-dimensional evaluation methods in lacking depth perception capabilities.

[0036] Step 2: Based on the category label of the target retail product, call the prior model of physical geometric primitives to fit the initial 3D point cloud subset to generate a standard physical fitting surface; use the point-to-surface iterative nearest point algorithm for fine-tuning registration to update the point cloud position and cumulative transformation matrix; output the registered 3D point cloud subset, the final point-to-surface correspondence set, and the final cumulative fine-tuning spatial transformation matrix. The specific process is as follows: Read the category label (e.g., "beverage bottle" or "snack box") of the target retail product being evaluated from the metadata of the initially labeled 2D instance segmentation mask. Based on the category label, the corresponding physical geometry primitive prior model is matched and invoked from the pre-built retail product prior knowledge base; Among them, the physical geometric primitive prior model refers to a standardized geometric reference template that combines the general shape characteristics of retail goods to make regular geometric abstractions, binds the actual physical size of the goods, and can be expressed through standard mathematical parametric equations. Including but not limited to: parameterized primitive types that match the forms of different product categories; For example: cylindrical primitives to match upright cylindrical rotating products such as bottled beverages, cans, and canned food; cuboid primitives to match square-shaped products such as boxed milk, square packaging boxes, cigarette boxes, boxed cosmetics, and paper express boxes; spherical primitives to match spherical fresh citrus fruits, spherical candies, and spherical packaged products; frustum primitives to match products with constricted / flared openings such as bucket instant noodles, conical paper cups, and conical snack buckets; and multi-basic primitive splicing and combination models to adapt to products with special shapes. Furthermore, the process of constructing the prior knowledge base for retail goods is as follows: 3D scan data or physical size specifications of various standard retail goods are collected in advance; the core regular geometric features of the goods' shape are extracted through geometric abstraction; standard mathematical parametric equations of corresponding geometric primitives are established for each type of goods (such as establishing a cylinder equation containing axis vector, cross-sectional radius, and cylinder height parameters for beverage bottles); the parametric equations are bound to the corresponding product category labels, thus forming the prior physical geometric primitive model of that type of goods; and finally, the prior models of all categories are uniformly stored in the knowledge base to complete the construction.

[0037] Using the Random Sample Consensus (RANSAC) algorithm combined with a prior model of physical geometric primitives, a local physical surface is fitted to an initial subset of 3D point clouds to generate a standard physical fitting surface representing the ideal physical boundary of the target retail product. The specific analysis and iteration process is as follows: Initialize the iteration counter for the random sample consensus algorithm and set the maximum number of iterations and the preset interior point distance threshold; enter the iteration loop and perform the following operations in each iteration: First, randomly select the minimum set of sample points from the initial 3D point cloud subset to satisfy the parameters required to solve the prior model of the current physical geometric primitives; Based on the extracted minimum sample point set, the three-dimensional coordinates of each sample point are substituted into the standard mathematical parametric equations corresponding to the currently invoked physical geometric primitive prior model to construct a set of geometric equations that match the number of parameters to be solved. The parameter vector of the current physical geometric primitive prior model is calculated by solving this system of equations, and the model parameter vector obtained by this iteration is marked as the candidate model parameter vector Θ; (the specific dimensions and physical meaning of the model parameter vector Θ are determined by the current prior model; for example, if it is a cylindrical model, then Θ includes the axial direction, a point on the axial direction, and the radius; if it is a planar model, then Θ includes the normal vector and the intercept). Secondly, calculate the distance residuals from all points in the initial 3D point cloud subset to the estimated geometric surface defined by the candidate model parameter vector Θ. The specific analysis and calculation formulas are as follows: ; in, Let be the 3D coordinate vector of the i-th point in the initial 3D point cloud subset. This is the function for calculating the shortest Euclidean distance from a point to a geometric surface. Its specific analytical expression is determined by the mathematical equations of the prior model of the currently invoked physical geometric primitive (e.g., if it is a cylindrical model, then...). This is the absolute value of the radius minus the perpendicular distance from the point to the axis of the cylinder; if it is a planar model, then... (To calculate the absolute value of the perpendicular distance from a point to a plane). The meaning of this formula is: to quantify the degree to which each point in the initial 3D point cloud subset deviates from the ideal physical boundary defined by the candidate model parameter vector; Statistical distance residuals The number of points whose distance to an inlier is less than the preset inlier distance threshold is recorded as the number of inliers corresponding to the current candidate model parameter vector. Store the current candidate model parameter vector and the corresponding number of interior points into the candidate result record set, and then increment the iteration counter by 1; If the iteration counter has not reached the preset maximum number of iterations, return to the "randomly select the minimum sample point set" step to start the next round of iteration; When the iteration counter reaches the preset maximum number of iterations, the iteration loop exits. From all the recorded candidate model parameter vectors, the candidate model parameter vector with the most interior points is selected as the optimal model parameter vector. The more interior points there are, the higher the consistency between the geometric surface defined by the candidate parameter and the real commodity point cloud, the less interference from outlier noise points, and the stronger the robustness of the coarse fitting result.

[0038] Based on the optimal model parameter vector, it is substituted into the mathematical equation of the currently invoked physical geometric primitive prior model to obtain the target geometric analytical equation with definite parameters. The calculation of the 3D spatial bounding box of the initial 3D point cloud subset is as follows: extract the maximum and minimum coordinate values ​​of all points in the initial 3D point cloud subset in each coordinate axis direction of the global 3D world coordinate system, and construct an axis-aligned 3D spatial bounding box based on the maximum and minimum coordinate values ​​in each coordinate axis direction.

[0039] Based on the target geometric analytical equation, surface sampling is carried out within the three-dimensional spatial bounding box of the initial three-dimensional point cloud subset; Among them, the sampling interval is combined with the spatial resolution setting of the depth camera at the current shooting distance, and the sampling density is matched with the original point cloud density; Substitute the surface parameter coordinates generated by sampling into the target geometric analytical equation to solve the corresponding three-dimensional spatial coordinates, eliminate redundant points that exceed the bounding box of the initial three-dimensional point cloud, and retain valid sampling points that are strictly located on the surface defined by the target geometric analytical equation and aligned with the product space covered by the point cloud. Based on the partial derivatives of the target geometric analytical equation, the surface unit normal vector at the location of each effective sampling point is calculated analytically. By combining all valid sampling points and their corresponding surface unit normal vectors, a standard physical fitting surface representing the ideal physical boundary of the target retail product is constructed.

[0040] After generating the standard physical fit surface, in order to eliminate the slight spatial deviation caused by the coarse fitting of the random sampling consensus algorithm, the Point-to-Plane ICP algorithm is used for fine spatial registration. The purpose is to construct an accurate spatial mapping relationship between the initial 3D point cloud subset and the standard physical fit surface. The specific process is as follows: Define a point cloud set for dynamic updating during iteration, assign its initial state to a subset of the initial 3D point cloud, and name this point cloud set the current iteration point cloud subset; Meanwhile, a matrix is ​​defined to record the cumulative results of all fine-tuning transformations. Its initial state is set to the identity matrix of the same dimension, and this matrix is ​​named the cumulative fine-tuning space transformation matrix. Enter the fine-tuning iteration loop and perform the following operations in each fine-tuning iteration: First, for any point in the current iteration's point cloud subset, calculate the shortest normal projection point to the standard physical fitting surface point by point, and simultaneously obtain the surface unit normal vector at the position of each projection point. Combine the point cloud point coordinates, the corresponding shortest normal projection point coordinates, and the surface unit normal vector into a matching point pair, and summarize all matching point pairs into the point-to-surface correspondence set for this iteration. Based on the set of point-to-plane correspondences, the point-to-plane iterative nearest point algorithm is used to calculate the single-step fine-tuning spatial transformation matrix between the current iterative point cloud subset and the standard physical fitting surface, and to construct the point-to-plane target error function. Specifically: ; Where N is the total number of matching point pairs in the point-to-face correspondence set, and k is the index number of the matching point pair. The coordinates of the points in the current iteration point cloud subset contained in the k-th matching point pair. The coordinates of the shortest normal projection point contained in the matching point pair. The surface unit normal at the point where the shortest normal is projected. To represent the unit normal vector of the surface transpose, Let be the single-step fine-tuning rotation matrix to be solved. Let be the single fine-tuning translation vector to be solved.

[0041] The optimal single-step fine-tuning rotation matrix that minimizes the objective error function is obtained by solving the linear least squares method. and the optimal single fine-tuning translation vector ; The optimal single-step fine-tuning rotation matrix is ​​used as the rotation submatrix, and the optimal single-step fine-tuning translation vector is used as the translation subvector. According to the matrix splicing rules of the homogeneous coordinate system, they are spliced ​​together to construct a homogeneous transformation matrix, and this homogeneous transformation matrix is ​​defined as the single-step fine-tuning space transformation matrix. The single-step fine-tuning spatial transformation matrix is ​​applied to the current iteration subset of the point cloud to update the point cloud positions, resulting in the updated current iteration subset of the point cloud. Simultaneously, the single-step fine-tuning spatial transformation matrix is ​​left-multiplied by the cumulative fine-tuning spatial transformation matrix to update the cumulative fine-tuning spatial transformation matrix. Repeat the process of setting up the point-to-plane correspondence set, constructing the point-to-plane target error function, solving the single fine-tuning spatial transformation matrix, and updating the point cloud position until the value of the target error function is less than the preset convergence threshold or the maximum number of fine-tuning iterations is reached. After the fine-tuning iteration converges, based on the normal projection correspondence set established by the last fine-tuning iteration, the output includes the standard physical fitting surface as the registration reference, the registered 3D point cloud subset precisely aligned with the standard physical fitting surface (i.e. the current iteration point cloud subset updated at the end of the iteration), the final point-to-plane correspondence set recording the point-to-plane mapping relationship, and the final cumulative fine-tuning spatial transformation matrix recording the global mapping relationship from the initial point cloud to the final registration pose.

[0042] It should be noted that the initial 3D point cloud subset inevitably contains annotation bias and depth acquisition noise. Directly calculating errors based on this noisy point cloud can easily lead to distorted evaluation results.

[0043] This step introduces a prior model of physical geometric primitives and uses a random sampling consensus algorithm for noise-resistant fitting to generate a standard physical fitting surface. Subsequently, a point-to-surface iterative nearest-point algorithm is employed for fine-tuning registration. During the registration iteration process, by continuously updating and accumulating the fine-tuning spatial transformation matrix, a final set of point-to-surface correspondences between the initial 3D point cloud subset and the standard physical fitting surface is established.

[0044] This final set of point-to-plane correspondences precisely records the coordinates of each point in the point cloud. The coordinates of its corresponding shortest normal projection point The spatial mapping relationship between them is established. Using a mathematical geometric model as an objective truth reference eliminates the subjective judgment error of manual sampling. Simultaneously, the precisely aligned point-to-surface mapping relationship provides a reliable data foundation for subsequent calculations of multidimensional residual features, ensuring the stability of the evaluation results.

[0045] Step 3: Construct a multi-dimensional residual feature vector based on the final point-to-plane correspondence set and calculate the 3D spatial wrapping score; determine whether the initial labeled data is qualified by combining the dynamic qualification threshold; for unqualified data, supplement the residual spatial distribution encoding features to form a complete attribution feature vector, determine the upper-level defect pattern through feature matching, and generate pixel-level defect location markers and correction suggestions. The specific process is as follows: Iterate through each matching point pair in the final point-to-plane correspondence set. For the k-th matching point pair, obtain the coordinates of the point cloud points in the registered 3D point cloud subset contained in that matching point pair. The coordinates of its corresponding shortest normal projection point And the surface unit normal vector pointing to the outside of the standard physical fit surface at the shortest normal projection point. Calculate the directed normal distance between the coordinates of a point in the point cloud and the coordinates of the point projected by the shortest normal. The specific calculation formula is as follows: ; in, A positive value indicates that the point cloud points are located outside the standard physical fit surface. A negative value indicates that the point cloud points are located inside the standard physical fitting surface; The mean and variance of the absolute directed normal distances of all matching point pairs are calculated, and the proportions of outer and inner points are also calculated. These are combined into a depth distance residual feature set. Points in the point cloud whose absolute directed normal distance is greater than the preset outlier threshold are marked as invalid interference points. The proportion of invalid interference points in the total number of points in the registered 3D point cloud subset is recorded as the invalid interference point ratio.

[0046] For the k-th point in the registered 3D point cloud subset, the average spacing of the point cloud at the current shooting distance of the depth camera is obtained. The average spacing of the point cloud is multiplied by a preset scaling factor to obtain the preset neighborhood search radius. Construct a local spherical neighborhood with the point cloud point as the center and a preset neighborhood search radius as the radius, and obtain all neighborhood point cloud points located within the local spherical neighborhood; Principal component analysis is used to calculate the local surface normal vector of a point in the point cloud by fitting it to all the obtained neighboring point cloud points. .

[0047] Obtain the surface unit normal vector pointing to the outer side of the standard physical fit surface at the shortest normal projection point corresponding to the point cloud point. The local surface normal vector of the point cloud. With the corresponding surface unit normal vector The spatial angle between The calculation formula is: ; Calculate the mean and variance of the spatial angles between all point cloud points, and simultaneously calculate the local surface normal vectors. With surface unit normal vector The proportion of point cloud points whose spatial angle between them is greater than a preset angle threshold is combined to form the normal vector direction residual feature group.

[0048] From the initial 3D point cloud subset, extract the 3D points corresponding to the edge pixels of the initially labeled 2D instance segmentation mask to construct an edge point cloud subset; combine the standard physical fitting surface theoretical boundary to calculate the edge alignment residual and edge smoothing residual, and construct the edge continuity residual feature set, specifically: Edge alignment residual: For each edge point in the edge point cloud subset, calculate the vertical distance from the edge point to the nearest standard physical fit surface theoretical boundary, and calculate the mean and variance of the offset distances of all edge points; Edge smoothing residual: Based on the topological connection order of edge pixels, obtain the 3D depth value of the j-th edge point in the edge point cloud subset. and edge surface normal vector And obtain the three-dimensional depth value of its successor adjacent edge points in the topological connection order of the edge pixels. and edge surface normal vector ; The formula for calculating the jump magnitude of the current edge point and its successor adjacent edge points in the 3D depth value is as follows: ; Calculate the abrupt change in angle between the current edge point and its successor neighboring edge points along the direction of the edge surface normal vector. The calculation formula is: ; Based on the target geometric analytical equation of the standard physical fitting surface, the surface curvature is calculated analytically. Regions with surface curvature greater than a preset curvature threshold are identified as theoretical edge positions. Edge points located at theoretical edge positions with depth jump amplitude greater than a preset jump threshold are identified as natural geometric jump points. Natural geometric jump points are then removed from the statistical sample. Finally, the variance of the jump amplitude and the variance of the angle change of all edge points that were not removed were calculated. The mean and variance of edge offsets obtained from edge alignment residuals, and the variances of depth jumps and normal jumps obtained from edge smoothing residuals are combined into an edge continuity residual feature set.

[0049] Obtain the average spacing of the initial 3D point cloud subset, and divide the standard physical fitting surface into a uniform sampling grid with an interval consistent with the average spacing of the initial 3D point cloud subset. For each uniform sampling grid, calculate the shortest distance from each point in the registered 3D point cloud subset to the surface of the uniform sampling grid. If there are registered point cloud points whose shortest distance is within the inlier distance threshold determined by the random sampling consensus algorithm, then mark the uniform sampling grid as a covered grid; otherwise, mark the uniform sampling grid without corresponding matching points as an uncovered grid. The proportion of all uncovered grids to the total number of uniformly sampled grids is recorded as the total surface uncovering rate. Spatial connectivity is partitioned into uniformly sampled grids marked as uncovered grids. The connected region containing the most uncovered grids is obtained, and the proportion of uncovered grids in this connected region to the total number of uncovered grids is calculated and denoted as the proportion of the largest continuous uncovered region. These are combined to form the surface cover residual feature group.

[0050] For the current retail product labeling samples to be evaluated, all statistics contained in the depth distance residual feature group, normal vector direction residual feature group, edge continuity residual feature group, and surface coverage residual feature group are normalized. The purpose is to eliminate the differences in physical dimensions and orders of magnitude between different statistics, and at the same time unify the polarity direction of all features, so that all normalized feature values ​​are transformed into positive indicators that the larger the value, the better the labeling quality of that dimension. Among them, for negative features where smaller values ​​represent better quality, reverse minimax normalization is used; For percentage-based features, directly scale linearly to the preset range; All normalized features are concatenated in a fixed order to construct a multidimensional residual feature vector for the current labeled retail product sample to be evaluated.

[0051] For the current retail product labeling sample to be evaluated, the constructed multidimensional residual feature vector is input into the quality evaluation function to calculate the three-dimensional spatial wrapping score S of the current retail product labeling sample to be evaluated. The quality evaluation function formula is: ; in, The preset feature dimension weight coefficients are the values ​​corresponding to the i-th normalized feature value. Let be the i-th normalized eigenvalue in the multidimensional residual eigenvector, where i is the index number of the feature dimension, and H is the total number of dimensions of the multidimensional residual eigenvector.

[0052] During the fitting phase of the random sampling consensus algorithm, the distance residuals of all inliers are statistically analyzed, and the mean and standard deviation of the inlier distance residuals are calculated. The interval formed by adding a preset multiple of the standard deviation to the mean is determined as the confidence interval of the inlier residuals. The upper bound of the confidence interval of the inlier residuals is used as the baseline pass line. The calculated proportion of invalid interference point cloud points is input into a preset threshold relaxation function to calculate the threshold relaxation coefficient. The baseline pass line is multiplied by the threshold relaxation coefficient to obtain the dynamic pass threshold of the current sample. If the 3D spatial envelopment score is greater than or equal to the dynamic pass threshold, the initial annotation data is deemed qualified and output. If the 3D spatial envelopment score is less than the dynamic acceptable threshold, the initial annotation data is deemed unacceptable, and the subsequent attribution process is triggered.

[0053] The specific process of the attribution process is as follows: For the initial labeled data that is deemed unqualified, a residual spatial distribution coding feature is added to the multidimensional residual feature vector (this residual spatial distribution coding feature is used to characterize the geometric distribution shape and clustering characteristics of the residual in three-dimensional space). Point cloud points whose absolute directional normal distance is greater than the preset residual threshold are marked as high residual points. Density clustering algorithm is used to cluster the high residual points, and the centroid three-dimensional coordinates of the largest cluster (i.e. the cluster containing the most high residual points) are obtained as the spatial clustering location of the high residual points. Furthermore, the specific process of using density clustering algorithm to cluster high residual points is as follows: taking each high residual point as the center, search for high residual points within a preset neighborhood radius. If the number of high residual points in the neighborhood is greater than the preset density threshold, they are assigned to the same cluster, and this process is continuously expanded outward until it can no longer be expanded, thus obtaining multiple clusters. Since density clustering algorithm is a mature clustering method in this field, its specific iterative details will not be elaborated further here.

[0054] The difference between the proportion of outer points and the proportion of inner points is used to obtain the difference between the proportions of positive and negative residuals. Connectivity analysis is performed on the uniformly sampled grid marked as uncovered grids, and the three-dimensional coordinates of the centroid of the connected region with the largest area (i.e. the connected region containing the most uncovered grids) are obtained as the location of the largest continuous uncovered region. Furthermore, the specific process of performing connected component analysis on uniformly sampled grids marked as uncovered grids is as follows: In three-dimensional space, uniformly sampled grids that are spatially adjacent and are all marked as uncovered grids are aggregated into a connected component, thereby obtaining multiple connected components; since connected component analysis is a mature grid processing method in this field, its specific search details will not be elaborated further here.

[0055] The spatial clustering location of high residual points, the difference in the proportion of positive and negative residuals, and the location of the largest continuous uncovered area are combined into residual spatial distribution coding features. The residual spatial distribution coding features are then concatenated with the multidimensional residual feature vector to form a complete attribution feature vector V.

[0056] Based on the local highlight areas of the two-dimensional color image and the depth value missing areas of the three-dimensional depth map, the acquisition interference area is determined. The specific determination process is as follows: calculate the local pixel gradient of the two-dimensional color image, and determine the area where the pixel gradient is greater than the preset gradient threshold as the local highlight area. In the 3D depth map, regions with empty depth values ​​or depth value jumps greater than a preset jump threshold are defined as depth value missing regions; the union of local bright regions and depth value missing regions on the image plane is defined as acquisition interference regions.

[0057] The percentage of high residual points that fall within the acquisition interference area is counted. If the percentage of such points is greater than the preset interference threshold, it is directly determined to be an acquisition quality problem, and is marked separately and output. Obtain a pre-built defect pattern template library, which contains P standard defect template vectors. And its corresponding defect type label, where m is the template index number; Furthermore, the process of constructing the defect pattern template library is as follows: collect various unqualified annotation samples that have been confirmed in historical annotation data, extract their complete attribution feature vectors, and manually assign corresponding defect type labels; Clustering algorithms are used to cluster complete attribution feature vectors with the same defect type label. The coordinate vector of the cluster center is used as the standard defect template vector. The corresponding defect type label is bound to the standard defect template vector. Finally, the standard defect template vectors of all categories are summarized to form a defect pattern template library.

[0058] Calculate the complete attribution feature vector V of the current retail product labeled sample to be evaluated and the corresponding standard defect templates in the defect pattern template library. Weighted Euclidean distance between vectors The calculation formula is: ; Where W is the feature weight diagonal matrix, and T represents the matrix transpose.

[0059] From all the calculated weighted Euclidean distances, the standard defect template vector with the smallest weighted Euclidean distance that is less than the preset matching threshold is selected as the target matching template, and the defect type label corresponding to the target matching template is determined as the superior defect pattern of the current initial annotation data.

[0060] Based on the camera intrinsic and extrinsic parameters, points with high residuals and uncovered surface areas in 3D space are back-projected back to the initially labeled 2D instance segmentation mask plane to generate pixel-level defect location markers and match the standardized correction suggestions corresponding to the target matching template. Furthermore, the process of obtaining standardized correction suggestions is as follows: When pre-establishing a defect pattern template library, corresponding text correction guidelines and pixel-level operation prompt templates are configured for each defect type, and the defect type label, residual feature range, correction guidelines, and operation prompt templates are stored together. When the complete attribution feature vector matches a standard defect template vector in the defect pattern template library, the correction guidelines corresponding to that standard defect template vector are read, and combined with the pixel-level defect location markers, standardized correction suggestions for the current target retail product label sample are generated.

[0061] The quality assessment report of the target retail product annotation sample is generated by compiling the 3D spatial envelopment score, dynamic pass threshold, pass or fail judgment result, higher-level defect mode, pixel-level defect location mark and standardized correction suggestions. The quality assessment report is then sent to the annotation terminal so that the annotation personnel can correct the data of the initial 2D instance segmentation mask or manually review it.

[0062] It should be noted that existing automated evaluation tools typically only output a single global pass rate metric. This approach cannot refine and quantify deviations at a local level, nor can it adapt to noise interference in complex data acquisition environments.

[0063] This step calculates the coordinates of point cloud points and the coordinates of the shortest normal projection point based on the final set of point-to-plane correspondences. Coordinates of the shortest normal projection point Directed normal distance between A multidimensional residual feature vector is constructed by combining the normal vector direction, edge continuity, and surface coverage integrity. Simultaneously, a qualified threshold is dynamically calculated by combining the confidence interval of the interior point residuals and the proportion of invalid interference point cloud points to suppress sensor noise interference under different environments.

[0064] For initially labeled data deemed unqualified, residual spatial distribution encoding features are supplemented to form a complete attribution feature vector V. This is achieved by calculating the weighted Euclidean distance. The defect pattern template library is matched to backproject high residual regions in three-dimensional space onto a two-dimensional image plane.

[0065] This process directly generates pixel-level defect location markers and standardized correction suggestions, constructing a complete iterative process from quality assessment to data correction, reducing the cost of rework for large-scale retail labeled data.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment, characterized in that, include: For the target retail product, obtain its two-dimensional color image, initially labeled two-dimensional instance segmentation mask, three-dimensional depth map, and camera intrinsic and extrinsic parameter matrix; Extract the standard physical depth values ​​of the valid labeled pixels in the mask; map the two-dimensional pixel coordinates and standard physical depth values ​​to the global three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameter matrix to generate an initial three-dimensional point cloud subset; Based on the category label of the target retail product, the physical geometry primitive prior model is invoked to fit the initial 3D point cloud subset to generate a standard physical fitting surface; A point-to-surface iterative nearest point algorithm is used for fine-grained registration to update the point cloud position and cumulative transformation matrix, and outputs the registered 3D point cloud subset, the final point-to-surface correspondence set, and the final cumulative fine-tuning spatial transformation matrix. Based on the final point-to-face correspondence set, a multidimensional residual feature vector is constructed and a three-dimensional spatial wrapping score is calculated. Combined with a dynamic qualification threshold, the initial labeled data is judged to be qualified. For unqualified data, residual spatial distribution coding features are added to form a complete attribution feature vector. The upper-level defect pattern is determined by feature matching, and pixel-level defect location markers and correction suggestions are generated. The specific process for determining whether the initial labeled data is qualified based on the dynamic qualification threshold is as follows: During the fitting phase of the random sampling consensus algorithm, the distance residuals of all inliers are statistically analyzed, and the mean and standard deviation of the inlier distance residuals are calculated. The interval formed by adding the mean to the standard deviation by a preset multiple is determined as the confidence interval of the inlier residuals. Using the upper bound of the confidence interval of the in-point residual as the baseline, the calculated proportion of invalid interference point cloud points is input into the preset threshold relaxation function to calculate the threshold relaxation coefficient. Multiply the baseline pass line by the threshold relaxation factor to obtain the dynamic pass threshold for the current sample; If the 3D spatial envelopment score is greater than or equal to the dynamic pass threshold, the initial annotation data is deemed qualified and output. If the 3D spatial envelopment score is less than the dynamic acceptable threshold, the initial annotation data is deemed unacceptable, and the subsequent attribution process is triggered.

2. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 1, characterized in that: The specific process for obtaining a 2D color image, an initially labeled 2D instance segmentation mask, a 3D depth map, and the camera intrinsic and extrinsic parameter matrix for a target retail product is as follows: Read a two-dimensional color image containing the target retail product, an initially labeled two-dimensional instance segmentation mask, and a three-dimensional depth map from the same time and perspective from the retail product data acquisition system; A checkerboard calibration board of known physical size was placed in front of the image acquisition device, and images from multiple angles were acquired. The coordinates of the corner points were extracted using the Zhang Zhengyou calibration method, and the camera intrinsic and extrinsic parameter matrices were obtained by optimization using the least squares method.

3. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 2, characterized in that: The specific process for extracting the standard physical depth values ​​of valid labeled pixels in the mask is as follows: Iterate through each pixel in the initially labeled 2D instance segmentation mask and determine whether the pixel value of the current pixel is equal to the preset foreground identifier value; If equal to, then the pixel is determined to belong to the target retail product and marked as a valid labeled pixel; Obtain the 2D pixel coordinates of valid labeled pixels in the 2D color image, extract the original depth value from the same position in the 3D depth map, and convert it into a standard physical depth value that matches the physical units of the camera's intrinsic parameters.

4. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 3, characterized in that: The specific process of mapping two-dimensional pixel coordinates and standard physical depth values ​​to the global three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameter matrices to generate an initial subset of three-dimensional point clouds is as follows: Based on the acquired camera intrinsic parameter matrix, the two-dimensional pixel coordinates and standard physical depth values ​​of the effectively labeled pixels are transformed into coordinates in the three-dimensional camera coordinate system; the acquired camera extrinsic parameter matrix is ​​analyzed to extract the rotation matrix and translation vector. Based on the rotation matrix and translation vector, the coordinates in the 3D camera coordinate system are transformed to the coordinates in the global 3D world coordinate system; Extract the first three elements of the homogeneous coordinate form of the coordinates in the transformed global 3D world coordinate system to obtain the final 3D coordinates in the global 3D world coordinate system. The 3D coordinates of all valid labeled pixels are collected to generate an initial 3D point cloud subset of the target retail product.

5. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 4, characterized in that: The specific process of using a prior physical geometric primitive model based on the category label of the target retail product to fit an initial 3D point cloud subset to generate a standard physically fitted surface is as follows: Read the category label of the target retail product from the metadata of the initially labeled two-dimensional instance segmentation mask, and call the corresponding physical geometry primitive prior model; Initialize the random sampling consensus algorithm. In each iteration, randomly select the smallest sample point set and substitute it into the standard mathematical parametric equation to construct a geometric equation system. Solve the candidate model parameter vector, calculate the distance residual from all points in the initial 3D point cloud subset to the estimated geometric surface, count the number of inliers that are less than the preset inlier distance threshold, and select the inlier with the largest number of inliers as the optimal model parameter vector. Substituting the optimal model parameter vector into the mathematical equation yields the target geometric analytical equation. Surface sampling is carried out within the three-dimensional bounding box of the initial three-dimensional point cloud subset. Redundant points are eliminated and valid sampling points are retained. The unit normal vector of the surface is calculated analytically, and the standard physical fitting surface is constructed by combining the results.

6. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 5, characterized in that: The specific process of using the point-to-surface iterative nearest-point algorithm for fine-grained registration to update the point cloud positions and cumulative transformation matrix is ​​as follows: Assign the initial 3D point cloud subset to the current iteration point cloud subset, and initialize the cumulative fine-tuning spatial transformation matrix; In each fine-tuning iteration, the shortest normal projection point from the point in the current iteration point cloud subset to the standard physical fitting surface and the surface unit normal vector are calculated point by point. The point cloud point coordinates, the shortest normal projection point coordinates and the surface unit normal vector are combined into matching point pairs, and summarized to form a set of point-to-surface correspondences. Construct a point-to-surface target error function, solve for the optimal single-step fine-tuning rotation matrix and the optimal single-step fine-tuning translation vector using the linear least squares method, and construct the single-step fine-tuning spatial transformation matrix by splicing the matrix according to the matrix splicing rules of the homogeneous coordinate system. Apply it to the current iteration subset of the point cloud to update the point cloud position, and left-multiply it to the cumulative fine-tuning spatial transformation matrix to update the cumulative fine-tuning spatial transformation matrix.

7. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 6, characterized in that: The specific process of outputting the registered 3D point cloud subset, the final set of point-to-plane correspondences, and the final cumulative fine-tuning spatial transformation matrix is ​​as follows: Repeat the process of establishing a set of point-to-plane correspondences, constructing a point-to-plane target error function, solving a single fine-tuning spatial transformation matrix, and updating the point cloud position until the value of the target error function is less than the preset convergence threshold or the maximum number of fine-tuning iterations is reached. After the fine-tuning iteration converges, based on the point-to-plane correspondence set established in the last fine-tuning iteration, the output includes the standard physical fitting surface as the registration reference, the registered 3D point cloud subset precisely aligned with the standard physical fitting surface, the final point-to-plane correspondence set recording the point-to-plane mapping relationship, and the final cumulative fine-tuning spatial transformation matrix recording the global mapping relationship from the initial point cloud to the final registration pose.

8. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 7, characterized in that: The specific process of constructing a multidimensional residual feature vector based on the final point-to-plane correspondence set and calculating the three-dimensional spatial wrapping score is as follows: Traverse the final point-to-plane correspondence set, calculate the directed normal distance between the point cloud coordinates and the shortest normal projection point coordinates, and statistically analyze the mean and variance of the absolute directed normal distance and the proportion of inner and outer points to form a depth distance residual feature group. Mark invalid interference point cloud points and calculate their proportion. Calculate the spatial angle between the local surface normal vector and the surface unit normal vector, and statistically combine the mean, variance and the proportion of exceeding the threshold to form the feature group of normal vector direction residuals. Calculate the vertical distance from the edge point to the theoretical boundary and the abrupt change in depth and normal angle between adjacent edge points. After removing natural geometric abrupt change points, the statistical variance combination is used to form the edge continuity residual characteristic group. The total surface uncovering rate and the proportion of the largest continuous uncovered area are combined to form the surface coverage residual feature group by dividing the uniform sampling grid and statistically analyzing the surface coverage residual rate. All statistics are normalized to unify the polarity direction and concatenated into a multidimensional residual feature vector. The three-dimensional spatial wrapping score is calculated by weighting and summing the preset feature dimension weight coefficients.

9. The method for evaluating the quality of image annotation data in retail scenarios based on multimodal alignment according to claim 8, characterized in that: For non-compliant data, the residual spatial distribution coding features are supplemented to form a complete attribution feature vector. The higher-level defect pattern is determined through feature matching, and the specific process of generating pixel-level defect location markers and correction suggestions is as follows: Points in the point cloud with an absolute directional normal distance greater than a preset residual threshold are marked as high residual points. The density clustering algorithm is used to obtain the three-dimensional coordinates of the centroid of the largest cluster as the spatial cluster location. The difference between the proportion of outer points and the proportion of inner points is used to obtain the positive and negative residual proportion difference. Connectivity analysis is performed on the uncovered grid to obtain the three-dimensional coordinates of the centroid of the largest connected region as the location of the largest continuous uncovered area. These are combined into residual spatial distribution coding features and spliced ​​into a complete attribution feature vector. Based on localized highlighted areas and areas with missing depth values, interference areas in the acquisition were identified to eliminate unlabeled defects; Calculate the weighted Euclidean distance between the complete attribution feature vector and each standard defect template vector in the defect pattern template library, and select the template with the smallest distance that is less than the preset matching threshold to determine the superior defect pattern. The high residual points and the points in the uncovered surface area are back-projected to generate pixel-level defect location markers. Standardized correction suggestions are extracted, and a quality assessment report is compiled and sent to the annotation terminal.

Images